Crawl Parameters

You can customize or set the following parameters in each web crawl.

ParameterDescription
Crawl Name The name of your web crawl
URL List The starting list of URLs from which your crawl will start. This can be one or many URLs.
Depth Level How far down the crawl will go from your URL list before stopping. A depth level of 0 means only your URL list will be crawled.
Crawl Size How many total URLs your web crawl will crawl before stopping.
80apps The web scraper that will run with your crawl. The 80app determines what data is scraped from each URL you crawl, along with which URLs are crawled from each URL.