The Data Collector product offers a JSON API to control and inspect your data collectors programmatically.
Getting started:
- Signup (or login) to our control panel
- Insert target URL (website to collect data from) or choose a ready-made template
- Set up the delivery preference of collector: Batch or Real-Time
- Initiate by API
NOT A DEVELOPER? Contact our support for easy onboarding.
The data collector is divided into two crawling systems:
- Batch: this is an efficient way of handling large amounts of data. Data is collected in batches over a period of time. After the data is collected, entered, and processed, the batch results are generated. You can deliver the results all at once or stream them to your system as they’re collected.
- Real-Time: this is an ideal way to get a fast response for one request. This is useful in environments where you have a limited amount of time to crawl a page and get the data.
The Data Collector API has four general workflows:
Categories