General Tips to Avoid Blocking

IPs

The main things to consider for IPs are:

  • Geolocation: it's generally best to match the peer country to the country of the target.
    Using more specific targeting can also help:
    City is more specific than State, and State more specific than country
  • Pool type: using Shared IPs can be a savings if the target isn't sensitive to multiple requests from the same IP. If the target is sensitive to that, using Dedicated IPs can help.
  • Quality: Bright Data offers 4 'normal' proxy networks, in addition to our 'managed' networks.
    These are listed below in order of increasing difficulty to detect:
    Data Center - ISP - Residential - Mobile
    If one of these networks is consistently blocked, trying the next one down in the list may help get past the blocking.
  • Consistency: for single-URL scrapes, you should generally be rotating the IP frequently to avoid rate limits. If you're trying to emulate a real user, the tips in the section below can help keep the IP consistent.

Sessions

  • If you're trying to mimic a real user, you should use the session ID username parameter:
    Session IP Persistence
    This tells us that for a request with the same ID, you want to keep the same IP for the request.
    Doing this allows you to appear more like a normal user when using rotating networks like Residential and Mobile.
  • If your use-case is single scrapes, you might still want to use this parameter, but in a different way:
    When the ID changes, that tells us that you want to use a different IP than the last one.
    Iterating over multiple IDs within 60 seconds will allow you to rotate through your allocated IPs.

Request Method / Fingerprint

  • Request Method: "Normal users" will typically use a browser to access websites. You can integrate the proxies into a regular browser, either directly or via our extension (available here).
    If you're using a different integration, like browser automation, or direct API requests, it's important to keep this in mind and make your requests look as much like a real browser as possible to avoid detection.
  • Fingerprint: Part of looking like a real user is the headers and cookies you send with the request, but another part is having the same attributes that a real browser would have. You can find many resources online about this, and we also offer the Web Unlocker and Scraping Browsers, which manage all of this on your behalf.

Was this article helpful?