Tips

Check out our tips below that will turn you into a "power" Proxy Manager user, bringing your scraping efforts up a notch, while keeping them as efficient as possible (thereby saving you money!)

How to reduce the amount of data in a response?

Go to "General" tab
Select "On" from "SSL analyzing"
Go to "Rules" tab
Select "URL" from "Rule type" drop-down list
Choose any file type to apply regex expression
Select "Null response" from "Action type" drop-down list

In the following example below the Proxy Manager will return a null response (0kb and still status code 200) for all requests that end with any of the file type endings .jpg|.png|.gif etc.

Use this option when you want to reduce the amount of data you wish to get in your response.

How to accelerate your web scraping

Motivation (why should I care?)

If the number of requests you rotate through a single IP is higher than what target websites allow, the website you target will identify your IP and block or mislead you with false information. It means that your information collecting can be much slower than what you're used to.

How do I improve the speed of my data harvesting?

Assuming you're running 10 million requests, 1 request per second per IP with 1000 data center IPs, your routine can take about 3 hours. With 10,000,000 residential IPs, your routine can potentially take 1 second.

Guidelines to rotate multiple parallel sessions through Bright Data's residential network:

Open Proxy Manager
Go to the 'Overview' menu
Click the port of your residential zone
Edit in the port settings 'preset' to 'Rotating (IPs)'
Route your requests to 127.0.0.1:{portnum} where the {portnum} is the port of the residential zone

How do I know if I'm getting cloaked?

Motivation

Getting cloaked means that you're getting misleading information from the website you are scraping.

Example: If you are collecting comparative competitive information to feed your automatic pricing algorithms, the target website can return artificially lower prices to your requests, to skew your pricing and profits.

How to know when you're getting cloaked

When using traditional proxy networks (data center based IPs), your target websites may identify your activity quite easily and may cloak your requests. Therefore, the only way to ensure you're not getting cloaked is to rotate your requests through residential IPs.

Guidelines for rotating requests through millions of residential IPs:

Open Proxy Manager
Go to the 'Overview' tab
Click the port of your residential zone
Edit in the port settings 'preset' to 'Rotating (IPs)'
Route your requests to 127.0.0.1:{portnum} where the {portnum} is the port of the residential zone

How to use a specific residential IP for a long period of time in a single session?

First, please mind that a residential/mobile IP are real users' devices' IPs, and therefore can be used only when the user's device is idle (i.e. the device is connected to the internet, has enough battery power, and the user is not currently using it).

If the IP becomes unavailable, our system will automatically assign you with another available residential IP of the same type and geo-location as you used.

Setting Long Single Session when using Proxy Manager

Choose "Long Single Session" on the port settings drop-down, and it will automatically ping the IP to keep the session alive:

Using BrightData in iPhone settings

Open your device Settings app and go to Wi-Fi
Tap the name of the Wi-Fi network you are connected to
Scroll to the bottom and you will find a section for 'HTTP Proxy'. This is set to 'Off' by default. Set it to 'Manual'
- In the server slot enter: zproxy.lum-superproxy.io
- In the port slot enter: 22225
Toggle the Authentication to 'On'
Enter username and password as can be found at https://brightdata.com/cp/zones, click on the Zone name, then click on 'Settings' tab.
Note: if you need to change the proxy credentials you may encounter a bug in iOS proxy settings. Proxy username and passwork are cached by host name and can't easily be changed. To overcome this bug either use Proxy Manager to control proxy credentials or use a different host every time you need to change credentials. You can use session-RANDOM_ID.zproxy.lum-superproxy.io to create a new host domain every time.

Remote DNS

Using Remote DNS gives you a DNS resolve server similar to that of the Proxy peer (the origin of the Proxy's IP). This makes your request look more credible as the resolve server is usually located geographically close to the IP, whereas without using Remote DNS the resolve server is one of Bright Data's Super-Proxies which may be located in another country. To use Remote DNS:

Via Proxy Manager, go to Edit Port > IP control > set "DNS Lookup" to "Remote - resolved by peer"

Via API simply add it to your username string. For example:

lum-customer-{YOUR_CUSTOMER_ID}-zone-{YOUR_ZONE}-dns-remote

Refresh session (IP)

Using the same IP for a long period of time makes it easier for the target website to mark the IP as proxy and can make your request get detectable by the target website. Refreshing your IPs will result in allocation of new IPs from Bright Data instead of your existing ones in your pool allowing you to gain control over your pool and reach higher success rates.

How to use parallel Proxy Manager instances for high performance

When you don't want to overload one Proxy Manager with too many requests you can create several instances of the Proxy manager on different servers.

The best way to do so is by following the next steps:

Install the Proxy Manager locally and set all of the ports as needed
Save the config file
Turn on Sync configuration so all the changes will be kept in sync on all the machines automatically
Install the Proxy Manager on the servers you want to use
Import the config file to the proxy_manager directory in the servers

Split your traffic between the servers
request 1 -> {FIRST SERVER IP}:24000

request 2 -> {SECOND SERVER IP}:24000

Using the Proxy Manager with this method will make sure you are not overloading one Proxy Manager instance and it will help you keep the same configuration across all instances.

This means you can send requests from same port with the same country at the same time on each instance.

If you're having troubles setting up this load-balancing yourself - you could use our Clouds that have load-balancing feature implemented. You'll only need to use 1 UI and control all instances with shared configuration, and then target same URL, we will take the whole load-balancing process ourselves.

Running multiple browser profiles with different proxy settings

You can run a chromium browser which is automatically preconfigured with the Proxy Manager. In the Overview tab click on the Browser button (see image below).

The browser will pop up on the screen and will be automatically configured according to the proxy settings of this port.

Open multiple browsers for each port to have a unique and clean profile for each proxy port.

Browse Chinese sites by using Chinese residential IPs

If you are outside China, and you are using residential Chinese peers to target Chinese sites, no special settings are required.
If you are inside China, and you are using residential Chinese peers to target Chinese sites, you must use our Chinese Super Proxy:
- When using Proxy Manager: on Port settings > IP control: set Super Proxy location to "China".
- When using API commands, use servercountry-cn.zproxy.lum-superproxy.io
  Example:
```
curl --proxy servercountry-cn.zproxy.lum-superproxy.io: 22225 --proxy-user lum-customer-CUSTOMER-zone-ZONE-country-cn:PASSWORD "TARGET CHINESE WEBSITE"
```