Instead of creating duplicate collectors for each delivery type, you can choose a delivery type per job using API.
Step 1. Ensure that your collector is updated to the latest version to improve success rate.
Step 2. Set a delivery type to Batch.
Following error message will be returned from Batch API when delivery type is set to Realtime.
"error": "Cannot trigger a batch job with a real-time collector. Use /trigger_immediate endpoint instead"
Step 3. Trigger the collector using a preferred API.
- Initiate a Batch job (dca/trigger) : Trigger a collector for batch collection method
curl"https://api.brightdata.com/dca/trigger?collector=ID_COLLECTOR&queue_next=1"-H "Content-Type: application/json" -H "Authorization: Bearer API_TOKEN" -d "[{\"url\":\"https://targetwebsite.com/product_id/\"}]"
Batch response sample :
{"collection_id":"j_l3daejgw1wnpjxxxxx","start_eta":"2022-05-19T17:28:48.056Z"}
- Initiate a Realtime job (dca/trigger_immediate) : Trigger a collector for real-time collection
curl"https://api.brightdata.com/dca/trigger_immediate?collector=ID_COLLECTOR"-H "Content-Type: application/json" -H "Authorization: Bearer API_TOKEN" -d "{\"url\":\"https://targetwebsite.com/product_id/\"}"
Realtime response sample :
{"response_id":"z2805t1652973963340rg6252xxxxxx"}
* Note : Batch responses begin with "j_****" and real-time responses begin with "z****"
Step 4. Receive data
- Receive Batch data : Receive batch data
- Receive Realtime data : Receive data from real-time work collector
Categories