What products support targeting search engines?
Use one of the following products to collect data from search engines:
* SERP API: this product operates with a proxy interface, performs automatic website unblocking, and provides results in HTML or JSON. It is customized for collecting data from search engines and their different sections, such as hotels, maps, trends, etc.
* Web Scraper IDE: A fully hosted API that allows building and operating a data collection in our IDE. This product includes fully automated unblocking, various output formats, scheduling, batch mode or single query, and more.
What happens if I target search engines through a proxy network?
Datacenter & ISP networks
When attempting to target search engines through a proxy network, you will be blocked, meaning your request will not exit our system and you will receive the following error message in the response headers:
HTTP/1.1 403 Forbidden domain
X-Luminati-Error: Forbidden: requests to this domain are blocked using the Datacenter and ISP proxy networks; please get access via a SERP API zone or contact your account manager to assist
Residential & Mobile networks
When targeting search engines through the Residential or Mobile networks, the request will not go through the Residential or Mobile peer but instead will be sent directly through one of our super proxies. The reason for this behavior is to support the integration of the Residential and Mobile networks with a browser, which might need to load search engine resource endpoints from the target website.
In case you target a search engine domain using Residential or Mobile networks, the request will be passed directly through the super proxy, and the following response header will serve as an indication:
x-luminati-ip: superproxy bypass
Categories