Getting started with Web Scraper IDE

What is the Data Collector?

Data Collectors are automated tools that enable businesses to automatically collect all types of public online data on a mass scale, while heavily reducing in-house expenses on proxy maintenance and development. The Data Collector ...

What is Data Collector IDE?

Data Collector's IDE is its integrated development environment. The IDE is a Public web data on any scale at your fingertips, you can: Build your collector in minutes Debug and diagnose with ease Bring to production quickly Brows...

What is an “input” when using a Data Collector?

When collecting data, your “input” are the parameters you'll enter to run your collection with. This can include keywords, URL, search items, product ID, ASIN, profile name, check in and check out dates, etc.

What is an “output” when using a Data Collector?

The output is the data that you've collected from a platform based on your input parameters. You'll receive your data as JSON/NDJSON/CSV/XLSX.

How many free records are included with my free trial?

Each free trial includes 100 records (note: 100 records does not mean 100 page loads).

Why did I receive more statistic records than inputs?

You'll always receive a higher number of records than the inputs you've requested.

What are the most frequent data points collected from social media?

Number of followers, average number of likes for posts, level of engagement, account theme, social and demographic portrait of the audience, social listening: keywords/ brand mentions, sentiments, viral trends.

Is there a way to know if my publicly-available data was collected by Bright Data's collection platform?

Yes, you can check if your public data was collected here:https://brightdata.com/check_your_data.

Do you only collect public data or do you also collect private data?

We never collect private data. We only collect publicly-available data.

Can I collect data from multiple platforms?

Yes, we can collect data from large numbers of websites at the same time.

Can I add additional information to my Data Collector?

Yes you can, you can ask your account manager for help, or you can open a ticket related to the specific Data Collector by selecting 'Report an issue.' Then request that fields be added or removed from your Data Collector.

What is a search collector?

In cases where you don't know a specific URL, you can search for a term and get data based on that term.

What is a discovery collector?

With a discovery collector, you enter a URL(s) and collect all data from that page(s). You'll receive data without having to specify a specific product or keyword.

Can I change the code in the IDE by myself?

Yes, the code is in JS, and you can change it according to your requirements.

What are the options to initiate requests?

We have 3 option to initiate requests: Initiate by API - regular request, queue request and replace request. Initiate manually. Schedule mode

How to start using the Data Collector?

Click here to check our guide on how to start using the Data Collector

What is a queue request?

When you are sending more than one API request, a “queue request” means that you'd like your next request to start automatically after your first request is completed, and so on with all other requests.

What is a CPM?

CPM = 1000 page loads