Counting How Many URLs a Website Has

In this walkthrough, we'll explain how you can use 80legs to count the # of URLs a given website has.

The steps we'll take are:

  1. Upload a URL list containing the website
  2. Create a crawl using the URL list and the CrawlInternalLinks 80app
  3. Using the # of URLs crawled to determine # of URLs on the website

Using the Web Portal

1. Upload a URL list containing the website

Login to the web portal and go to the "My URL Lists" page.  Click on "Create a URL list" and then "Type In a List".  Enter only the website you want to crawl, like so:

Click "Create URL list" when done.

2. Create a crawl using the URL list and the CrawlInternalLinks 80app

Go back to the "My Crawls" page and click "Create a new crawl".  Give your crawl a name.  Select the URL list you just uploaded, set your depth to 20, and your maximum URLs as high as possible for your account.  Select the CrawlInternalLinks 80app.  This 80app will crawl only links on the domain in your URL list.

Click "Create Crawl" when you're done.

3. Use the # of URLs crawled to determine # of URLs on the website

Once the crawl completes, check the # of URLs crawled.  That's how many URLs are on the website!

Using the API

Here are those same steps using the API: