Creating and Uploading a New 80app with the Web Portal

Once you've started to run crawls, you may want to start customizing how your crawler behaves.  By creating and using your own custom 80app, you can:

  • Tell the crawler which links to crawl next from the URL it's currently crawling
  • Tell the crawler what data to return from the URL it's currently crawling

The steps we'll take to create a new 80app are:

  1. Copy the source code for an existing 80app into a local text file
  2. Edit the file and save the code as a new 80app
  3. Upload the 80app to your account

1. Copy the source code for an existing 80app into a local text file

We host a collection of public 80apps on our Github repo.  Go to it and copy the source code for one of the 80apps to a text file on your local  computer.

2. Edit the file and save the code as a new 80app

Let's say you want to add on some functionality to the DocumentData 80app.  Just edit the file on your computer and then save it as a new .js file (e.g., "new80app.js").

Note: The 80legs web crawler uses an extended version of Cheerio, a lighter weight version of jQuery. Though it implements many of the core jQuery functions, it does not have functionality for all of them. Please go here to learn more about how this affects how you build 80apps.

3. Test your 80app

Use our testing site to test your 80app.

4. Upload the 80app to your account

Go to the web portal and click on the "My 80apps" tab.  Then click on "Upload an 80app".  You can select your file here and upload it.

You can also click on "Type In an 80app", and copy-paste your source code into the form and submit.  This is also an easy way to copy over code from our public repo.

Once the 80app's been uploaded, you can use when creating new crawls.