- The customer sends a Search Engine Crawler API request to Super Proxy
- Super Proxy routes the request to a suitable peer
- The request is sent to the search engine from the peer using Bright Data's Web Unlocker methods
- A response returns from the search engine back to the peer, then back to Super Proxy, and then back to the customer
Build and practice Search Engine Crawler's API requests on our Search Engine Crawler Playground , which generates API commands according to the search parameters. Find more about it here
Notes
- Search Engine Crawler is only available via API (supported code languages are: shell, node.js, java, c#, python, VB, PHP, Ruby, Perl)
- Session continuity is not supported as each request is sent from a different IP
- HTTP and HTTPS URLs can be targeted. See How to send Search Engine Crawler HTTPS requests