Overview
Bright Data's site health monitor is a system designed to ensure that usage of our products does not cause or contribute to performance issues or the overloading of our customers' target sites.
Maintaining a healthy ecosystem.
In the data collection field, it is technically possible to overload a website by consuming its resources and causing performance issues. Many companies take an active approach to avoid overloading their target sites during the data collection process, but this can be complex and require sophisticated systems.
Our solution - the site health monitor
The site health monitor uses several techniques to prevent overloading of target sites.
How it works
- Monitoring: The site health monitor tracks the health of all sites targeted by Bright Data's customers by monitoring their round-trip time (RTT) value from locations worldwide. It also monitors the requests coming from Bright Data and looks for correlations.
- Throttling: If the site health monitor detects that the RTT of a site is worsening compared to its normal state, it will throttle requests coming from Bright Data products. This throttling can happen globally or for specific locations.
- Self-learning: Over time, the site health monitor will learn the usage limits and appropriate throttling patterns and lengths for each site.
By implementing these measures, the site health monitor enforces healthy and ethical data collection and helps maintain a healthier ecosystem in the data collection field.
How to identify if your requests have been blocked by the site health monitor
If your request has been blocked by the site health monitor's throttling system, you will receive an error message saying, "Rate limit for domain '<SITE>' has been reached. Bright Data's health monitor is throttling down these requests to prevent overloading of the target website."
Error code: 429
Categories