Quick Tour of Web scraper IDE

How to start using the data collection tool

There are two ways to use the data collection tool : - Develop a self-managed collector on your own - Request a managed collector  

Develop a self-managed collector

To develop a custom collector using our Integrated development environment (IDE), you will need to insert an URL and start interacting with the development environment using Javascript language. Step 1 - Start from scratch You can ...

Request a managed collector

To request a custom collector, fill out a form so we can understand what you want to collect and in what structure you’d like to receive the data. Step 1 - Create and define the project A : Name the projectB : Select a Project Own...

I requested that you build a new Data Collector. How can I confirm that someone is working on it?

You'll receive an email that the developer is working on your new Data Collector, and you will be notified when your collector is ready. Status of the request can also be found on your dashboard : 

Data Collector Dashboard

Any collector you create using a template or a custom collector will appear on your Data Collector dashboard. Dashboard overview : - Free trial : As part of the 7-day free trial, you’re entitled to 1,000-page loads- Update availab...

Dashboard - Collector action menu

The collector action menu allows performing different actions with the collector. Initiate by API - start a data collection without having to enter the control panel Initiate manually - Bright Data's control panel makes it easy t...

Dashboard - Properties

Maintainer of the collector :  Self-serve: collector is maintained by you Full-service: collector is maintained by Bright Data Developers Type of the collector : Search : The collector input is a keyword (i.e., iPhone) PDP : ...

Dashboard - Delivery preferences

Choose when to get the data : Batch : an efficient way of managing large amounts of data Split batch : deliver the data in smaller batches as soon as it's ready Note: when using this setting, filenames will append numbers and ...

Dashboard - Output configuration

Schema defines the data point structure and how the data will be organized. You can change the schema structure and modify the data points to suit your needs.   Managed Collector :  You can define the input and output configuration...

Initiate collector and get collection results

Initiate Collector To start collecting the data, you have three options: A. Initiate by APIB. Initiate manuallyC. Schedule a collector   Get collection results Once the data collection is completed, click the "three dots" icon an...

Initiate collector - Initiate by API

You can start a data collection through API without accessing the Bright Data control panel : Getting started with API documentation Before initiating an API request, please create an API token. To create an API token, go to:Dashbo...

Initiate collector - Initiate Manually

Bright Data's control panel makes it easy to get started collecting data. 1. Trigger behavior - you can add several requests in parallel that are activated according to the order they're defined. You can add another job run to the ...

Initiate collector - Schedule Configuration

Choose when to initiate the collector. Step One :1. Choose a date and time for the collector to start.2. Select the frequency it will run (hourly, daily, weekly, etc.)3. Set a deadline for when a collector is complete.4. Review you...

Statistics

The statistics page presents essential information about the success of the data collection. Below is a list of all the terms included in the statistics table: Job ID - The unique id of the collection Trigger - The person who ini...