1. Bright Data
  2. Proxies & Scraping Infra
  3. SERP API

Asynchronous Requests

In this article, we'll explain the difference between sync and async requests, highlight the benefits of async API, and describe key parameters with examples.

How to send Asynchronous requests on SERP API

There are 2 ways for us to handle your SERP API requests:

  • Synchronous requests (default)  - you send a request and get the response on demand within a few seconds
  • Asynchronous requests - you send a request without waiting for the full response and later collect your responses at a more convenient time via a designated endpoint API. We store responses for up to 24 hours from the time the request was sent.
    Recommended for: those with high-volume requests who don't need to serve an immediate response on the single level and can wait a few minutes to retrieve all their responses at once.

Why to use Async

  • 99.99% success rate
  • Stability
  • Flexibility - the ability to retrieve  your requests at a later time of your choosing (and not have to wait for the response immediately after sending the request)
  • Most results come back within a few minutes

How it works

Sending requests and receiving responses with Async mode is a two-step flow:

    • Sending the request - This request includes the search parameters, returns a response_ID, and is a direct request (i.e. you will be billed for this request).
    • Collecting the response - This request includes the response_ID and is free of charge (i.e. you will not be billed for this request)
      Note: We store responses for up to 24 hours from the time the request was sent.

Getting started

  1. Enabling "Asynchronous requests"
    Within your specific SERP API zone, enable the "Asynchronous requests" toggle.

  2. [Optional] Setting up a webhook (POST or GET)
    This is where you will get notified on the status of your future requests
    Note: A webhook can also be set per request (see "Initial request parameters" below)

  3. Sending a SERP API async request 
    The request syntax is slightly different from that of synchronous requests and requires an API token for authentication. See a basic example below (for more request parameters see below):
     
    curl -i --silent --compressed "https://api.brightdata.com/serp/req?customer=[ACCOUNT_ID]&zone=[ZONE_NAME]" -H "Content-Type: application/json" -H "Authorization: Bearer [API_TOKEN]" -d '{"country":"us","query":{"q":"pizza","num":"100","hl":"en","gl":"au"}}'

    You'll receive a response to the above that contains an `x-response-id` header with the ID of your request. This is the RESPONSE_ID for this request which you will use when collecting your results in the next step.
    Note: If you want to receive a parsed response in JSON, you'll need to configure this within your initial request using the optional brd_json parameter (see "    Initial request parameters" below). 

  4. Webhook notification
    If you've set up a webhook,  you'll receive a notification immediately when the requests are ready with the following parameters: status, response_id, request_url and hook_data (optional - if you've used the webhook_data parameter in your request). 
  5. Collecting your results
    Using the RESPONSE_ID received in step 3, send the following:
    curl -i --silent --compressed "https://api.brightdata.com/serp/get_result?customer=[ACCOUNT_ID]&zone=[ZONE_NAME]&response_id=${[RESPONSE_ID]}" -H "Authorization: Bearer [API_TOKEN]"
    Note: In step 3 above (sending the initial request), you can easily save the RESPONSE_ID in your script by initializing it along with your request like this: 
    RESPONSE_ID=$(curl -i --silent --compressed "https://api.brightdata.com/serp/req?customer=[ACCOUNT_ID]&zone=[ZONE_NAME]" -H "Content-Type: application/json" -H "Authorization: Bearer API_TOKEN" -d '{"country":"us","query":{"q":"pizza","num":"100","hl":"en","gl":"au"}}' | sed -En 's/^x-response-id: (.*)/\1/p' | tr -d '\r')

Initial request parameters (all optional)

  • webhook_url: Defines the URL to which the job status notification will be sent. If you don't want to setup a default webhook (above) or prefer the URL to be different per request, use this. 
  • webhook_method: POST or GET (Default). Defines the method with which job status notification will be delivered. 
  • webhook_data: Defines the data that will be sent with job status notification
  • query: Defines the query object for the request and supports various Serp API parameters (i.e.country)
  • brd_json: To enable and configure parsing. By default, a SERP API request returns an unparsed structured HTML of the targeted SERP. If you would like to receive a parsed JSON response, add one of the following parameter values:
    • brd_json=1 - Returns a single parsed JSON (instead of a raw HTML)
       
      • curl -i --silent --compressed "https://api.brightdata.com/serp/req?customer=[ACCOUNT_ID]&zone=[ZONE_NAME]" -H "Content-Type: application/json" -H "Authorization: Bearer [API_TOKEN]" -d '{"country":"us","query":{"q":"pizza","num":"100","hl":"en","gl":"au"},"brd_json":"1"}'
    • brd_json=html - Returns a single parsed JSON containing an additional "html" field (with the raw HTML) along with the other parsed fields
    • Note: "query.brd_json" could be used instead of this parameter
  • multi: To run multiple queries within the same request (see below)

Response/collection parameters (mandatory)

  • response_id: Defines the job id. Received in the response to your initial async request. 

How to send multiple queries in an identical API request

SERP API supports sending 2 parallel query requests with one API request using the multi parameter. 

These parallel requests use the same peer IP and session and can be used for collecting additional data, comparison tests, etc. – e.g., making a pair of requests with different parameters/values. They use the same IP and session.

Conditions:

  • Supported only for a zone with asynchronous requests enabled
  • Supported only for Google Search
  • Limited to 2 requests
  • Billed as 2 requests

multi parameter uses:

  • Multiple requests with the same keyword but a different num parameter:
     
    multi:[{"keyword":"pizza","num":20},{"keyword":"pizza","num":100}]
  • Multiple requests with different keywords:
     
    multi:[{"keyword":"pizza"},{"keyword":"burger"}] 

multirequest example:

Send a request:

curl -v --compressed "https://brightdata.com/api/serp/req" -H "Content-Type: application/json" -H "Authorization: Bearer {API_TOKEN}" -d "{\"country\":\"us\",\"multi\":[{\"keyword\":\"pizza\",\"num\":20},{\"keyword\":\"pizza\",\"num\":100}]}"

Collect the result (use the response ID from the 'x-response-id' returned above to collect the result):

curl -k -v --compressed "https://brightdata.com/api/serp/get_result?customer={ACCOUNT_ID&zone={ASYNC_ZONE}&response_id={response_id}" -H "Authorization: Bearer {API_Token}" -o {Your_result_ouput_file}

Was this article helpful?