In this walkthrough, we'll explain how you can use 80legs to count the # of URLs a given website has.
The steps we'll take are:
Using the Web Portal
1. Upload a URL list containing the website
Click "Create URL list" when done.
2. Create a crawl using the URL list and the CrawlInternalLinks 80app
Go back to the "My Crawls" page and click "Create a new crawl". Give your crawl a name. Select the URL list you just uploaded, set your depth to 20, and your maximum URLs as high as possible for your account. Select the CrawlInternalLinks 80app. This 80app will crawl only links on the domain in your URL list.
Click "Create Crawl" when you're done.
3. Use the # of URLs crawled to determine # of URLs on the website
Once the crawl completes, check the # of URLs crawled. That's how many URLs are on the website!
Using the API
Here are those same steps using the API: