You can customize or set the following parameters in each web crawl.
Parameter | Description |
---|---|
Crawl Name | The name of your web crawl |
URL List | The starting list of URLs from which your crawl will start. This can be one or many URLs. |
Depth Level | How far down the crawl will go from your URL list before stopping. A depth level of 0 means only your URL list will be crawled. |
Crawl Size | How many total URLs your web crawl will crawl before stopping. |
80apps | The web scraper that will run with your crawl. The 80app determines what data is scraped from each URL you crawl, along with which URLs are crawled from each URL. |