The main things to consider for IPs are:
- Geolocation: it's generally best to match the peer country to the country of the target.
Using more specific targeting can also help:
City is more specific than State, and State more specific than country
- Pool type: using Shared IPs can be a savings if the target isn't sensitive to multiple requests from the same IP. If the target is sensitive to that, using Dedicated IPs can help.
- Quality: Bright Data offers 4 'normal' proxy networks, in addition to our 'managed' networks.
These are listed below in order of increasing difficulty to detect:
Data Center - ISP - Residential - Mobile
If one of these networks is consistently blocked, trying the next one down in the list may help get past the blocking.
- Consistency: for single-URL scrapes, you should generally be rotating the IP frequently to avoid rate limits. If you're trying to emulate a real user, the tips in the section below can help keep the IP consistent.
- If you're trying to mimic a real user, you should use the session ID username parameter:
Session IP Persistence
This tells us that for a request with the same ID, you want to keep the same IP for the request.
Doing this allows you to appear more like a normal user when using rotating networks like Residential and Mobile.
- If your use-case is single scrapes, you might still want to use this parameter, but in a different way:
When the ID changes, that tells us that you want to use a different IP than the last one.
Iterating over multiple IDs within 60 seconds will allow you to rotate through your allocated IPs.
Request Method / Fingerprint
- Request Method: "Normal users" will typically use a browser to access websites. You can integrate the proxies into a regular browser, either directly or via our extension (available here).
If you're using a different integration, like browser automation, or direct API requests, it's important to keep this in mind and make your requests look as much like a real browser as possible to avoid detection.
- Fingerprint: Part of looking like a real user is the headers and cookies you send with the request, but another part is having the same attributes that a real browser would have. You can find many resources online about this, and we also offer the Web Unlocker and Scraping Browsers, which manage all of this on your behalf.