feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth. - User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used. - Normal requests use what you pass without issues, of course. - All errors are retried now, and proxies are rotated on retries automatically.
This commit is contained in:
@@ -33,6 +33,8 @@ from typing import (
|
||||
SupportsIndex,
|
||||
)
|
||||
|
||||
# Proxy can be a string URL or a dict (Playwright format: {"server": "...", "username": "...", "password": "..."})
|
||||
ProxyType = Union[str, Dict[str, str]]
|
||||
SUPPORTED_HTTP_METHODS = Literal["GET", "POST", "PUT", "DELETE"]
|
||||
SelectorWaitStates = Literal["attached", "detached", "hidden", "visible"]
|
||||
PageLoadStates = Literal["commit", "domcontentloaded", "load", "networkidle"]
|
||||
|
||||
Reference in New Issue
Block a user