feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth. - User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used. - Normal requests use what you pass without issues, of course. - All errors are retried now, and proxies are rotated on retries automatically.
This commit is contained in:
@@ -1 +1,3 @@
|
||||
from .proxy_rotation import ProxyRotator, is_proxy_error, round_robin
|
||||
|
||||
__all__ = ["ProxyRotator", "is_proxy_error", "round_robin"]
|
||||
|
||||
Reference in New Issue
Block a user