触发批量采集

Ori Goren-Bar 阳锐
Ori Goren-Bar 阳锐
  • 更新于
API endpoint: POST /dca/trigger?collector={ID_COLLECTOR}
Parameters :
  • collector={ID_COLLECTOR} - A unique identification of collector
  • version=dev - (Optional) Trigger the development version of the collector
  • name={human_name} - (Optional) A human readable name for the batch
Choose trigger_behavior parameters:
  • queue_next=1 - (Default) If there’s already something in the crawl queue, push this job into the queue
  • queue=‘string’ - (Optional) Send another batch of requests that will start after the last one is finished
  • confirm_cancel=1 - (Optional) Cancel running job and run instead, submit the request and cancel running one
Payload :
  • Header: Authorization Bearer Key -H "Authorization: Bearer API_TOKEN"
  • Header: Content-Type JSON/CSV/Multipart -H "Content-Type: application/json"
  • Body: raw JSON -d '[{"url":"https://targetwebsite.com/product_id/"}]'
Sample Response :
{"collection_id":"ID_DATASET","start_eta":"2021-11-07T13:26:22.702Z"}
 
Shell
curl"https://api.brightdata.com/dca/trigger?collector=ID_COLLECTOR&queue_next=1"-H "Content-Type: application/json" -H "Authorization: Bearer API_TOKEN" -d "[{\"url\":\"https://targetwebsite.com/product_id/\"}]"

Node.js

#!/usr/bin/env noderequire('request-promise')({
	method: 'POST',
	url: 'https://api.brightdata.com/dca/trigger?collector=ID_COLLECTOR&queue_next=1',
	json: [{'url': 'https://targetwebsite.com/product_id/'}],
	headers: {'Authorization': 'Bearer API_TOKEN'},
}).then(function(data){ console.log(data); },
	function(err){ console.error(err); });

Java

packageexample;

importorg.apache.http.HttpHost;
importorg.apache.http.client.fluent.*;

public classExample {
  public static voidmain(String[] args) throwsException {
String body = "[{\"url\":\"https://targetwebsite.com/product_id/\"}]"; String res = Executor.newInstance() .addHeader("Authorization", "Bearer API_TOKEN")

.execute(Request.Post("https://api.brightdata.com/dca/trigger?collector=ID_COLLECTOR&queue_next=1"))
.bodyString(body,ContentType.APPLICATION_JSON))

.returnContent().asString();
System.out.println(res) }

}

C#

usingSystem;
usingSystem.Net;
usingSystem.Net.Http;
usingSystem.Net.Http.Headers;

public classProgram {
  public static asyncTaskMain() {
    var client = newHttpClient();
    var requestMessage = newHttpRequestMessage {
      Method = HttpMethod.Post,
RequestUri = newUri("https://api.brightdata.com/dca/trigger?collector=ID_COLLECTOR&queue_next=1"), Headers = { {"Authorization", "Bearer API_TOKEN"} },
Content =newStringContent(JsonConvert.SerializeObject([
new
{
url ="https://targetwebsite.com/product_id/"
}
]), Encoding.UTF8,"application/json")) }; var response = await client.SendAsync(requestMessage); var responseString = await response.Content.ReadAsStringAsync(); Console.WriteLine(responseString); } }

Python

#!/usr/bin/env python
print('If you get error "ImportError: No module named requests", please install it:\n$ sudo pip install requests');
import requests

data = [{'url': 'https://targetwebsite.com/product_id/'}] headers = {'Content-Type': 'application/json','Authorization': 'Bearer API_TOKEN'} r = requests.post('https://api.brightdata.com/dca/trigger?collector=ID_COLLECTOR&queue_next=1', data=json.dumps(data), headers=headers) print(r.content)