Web scraper IDE - quick tour

  • How to start using the data collection tool

    There are two ways to use the data collection tool : - Develop a self-managed collector on your own - Request a managed collector  

  • Develop a self-managed collector

    To develop a custom collector using our Integrated development environment (IDE), you will need to insert an URL and start interacting with the development environment using Javascript language. Step 1 - Start from scratch You can start from scratch or ...

  • Request a managed collector

    To request a custom collector, fill out a form so we can understand what you want to collect and in what structure you’d like to receive the data. Step 1 - Create and define the project A : Name the projectB : Select a Project Owner userC : Select a Da...

  • I requested that you build a new Data Collector. How can I confirm that someone is working on it?

    You'll receive an email that the developer is working on your new Data Collector, and you will be notified when your collector is ready. Status of the request can also be found on your dashboard : 

  • Data Collector Dashboard

    Any collector you create using a template or a custom collector will appear on your Data Collector dashboard. Dashboard overview : - Free trial : As part of the 7-day free trial, you’re entitled to 1,000-page loads- Update available : A new version of ...

  • Dashboard - Collector action menu

    The collector action menu allows performing different actions with the collector. Initiate by API - start a data collection without having to enter the control panel Initiate manually - Bright Data's control panel makes it easy to get started collecti...

  • Dashboard - Properties

    Maintainer of the collector :  Self-serve: collector is maintained by you Full-service: collector is maintained by Bright Data Developers Type of the collector : Search : The collector input is a keyword (i.e., iPhone) PDP : The collector input is...

  • Dashboard - Delivery preferences

    Choose when to get the data : Batch : an efficient way of managing large amounts of data Split batch : deliver the data in smaller batches as soon as it's ready Note: when using this setting, filenames will append numbers and possibly a period to d...

  • Dashboard - Output configuration

    Schema defines the data point structure and how the data will be organized. You can change the schema structure and modify the data points to suit your needs.   Managed Collector :  You can define the input and output configuration for how you would lik...

  • Initiate collector and get collection results

    Initiate Collector To start collecting the data, you have three options: A. Initiate by APIB. Initiate manuallyC. Schedule a collector   Get collection results Once the data collection is completed, click the "three dots" icon and select "Statistics" ...

  • Initiate collector - Initiate by API

    You can start a data collection through API without accessing the Bright Data control panel : Getting started with API documentation Before initiating an API request, please create an API token. To create an API token, go to:Dashboard side menu settings...

  • Initiate collector - Initiate Manually

    Bright Data's control panel makes it easy to get started collecting data. 1. Trigger behavior - you can add several requests in parallel that are activated according to the order they're defined. You can add another job run to the queue and run more tha...

  • Initiate collector - Schedule Configuration

    Choose when to initiate the collector. Step One :1. Choose a date and time for the collector to start.2. Select the frequency it will run (hourly, daily, weekly, etc.)3. Set a deadline for when a collector is complete.4. Review your setup. Step Two :1....

  • Statistics

    The statistics page presents essential information about the success of the data collection. Below is a list of all the terms included in the statistics table: Job ID - The unique id of the collection Trigger - The person who initiated the data collec...

  • Report an issue

    You can use this form to communicate any issues you have with the platform, the collector, or the dataset results.   1. Select a type of the issue: Collector maintainer * Note : This option is only available for managed collectors. Select a type under ...