66476d42be
feat(browsers): Add option to block requests to specific domains
2026-02-07 01:26:21 +02:00
f88502718f
feat(proxy control): Force a proxy at request level at any given point
...
And merge request's meta with response's meta
2026-02-02 14:26:52 +02:00
ef8c5bc7d6
feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
...
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.
- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.
- Normal requests use what you pass without issues, of course.
- All errors are retried now, and proxies are rotated on retries automatically.
2026-02-02 00:16:22 +02:00
5aa178a76e
style: update post requests type hints to be more accurate
2026-01-20 18:27:52 +02:00
bf5aa021d4
fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails
...
This has been causing issues with tests for a long time, and now finally found the reason.
2026-01-18 16:46:48 +02:00
a17bb1d976
feat(browsers): Add option to retry tabs that gives errors
2026-01-11 23:40:26 +02:00
d5b9ed11b2
fix(browsers): Close pages that give error
...
This was causing the browser and the whole code to stand still.
2026-01-11 18:34:13 +02:00
c5bc82deb8
fix(browsers): solve a bug with setting referer on the request level
2026-01-11 03:21:03 +02:00
08f3e6efad
fix: add missing line
2026-01-08 01:01:00 +02:00
6a73f9dcd7
refactor: internal API changes to be easily used as indicators for spiders
2026-01-08 00:52:06 +02:00
4947152be2
docs: change docstring style for some methods
2026-01-01 21:06:35 +02:00
f6f0ed4b37
fix(async stealth): wait the coroutine of the bounding box
2025-12-31 00:19:53 +02:00
ee13af9516
build: update deps
...
and docstrings
2025-12-29 16:08:51 +02:00
5e3845f060
docs: shorten docstrings for some functions
2025-12-28 21:56:28 +02:00
9bfe744840
fix: remove forgotten unused code
2025-12-28 00:14:47 +02:00
1b838eabd0
fix: update the cookies type hint for the mcp server
2025-12-28 00:08:44 +02:00
09bbd416b8
fix: remove old proxy logic
...
When I was caching it with the rest of the data
2025-12-28 00:02:50 +02:00
842718ba94
fix: update shell signatures for the autocompletion
...
Added missing fields and reordered classes as well for easier comparisons
2025-12-27 14:30:48 +02:00
cfc667f7dc
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
...
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
2025-12-26 03:21:13 +02:00
f9f34e6ee9
fix: improve the handling of useragent in chrome fetcher
2025-12-24 23:08:30 +02:00
ecb97d3fd6
docs: add docs for timezone_id argument
2025-12-18 01:16:22 +02:00
39f544eae4
feat(dynamicSession): add the option to set timezone of the browser
2025-12-18 01:13:31 +02:00
b507e4d4a0
refactor(fetchers): rename internal api
...
To make it easier to use to use sessions outside `with` context
2025-12-17 01:27:40 +02:00
aed34e43b5
fix(fetchers): better handling for infinite network_idle
2025-12-03 03:11:30 +02:00
f0b01fc253
refactor(browser fetchers): Make all the type hints dynamic + Faster validation
...
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
2025-11-25 20:18:36 +02:00
9e85311ff6
fix(validators): Fix overriding session values with request ones
2025-11-25 19:48:09 +02:00
c487faf4fd
pref(requests): Optimizing memory by specifiying slots
2025-11-23 21:50:54 +02:00
357b170e1c
refactor(requests): Make all the type hints dynamic
...
Made the code shorter by about ~500 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
2025-11-23 20:48:51 +02:00
df073c4680
fix: added raw_response to separate collected response from rendered response
2025-11-08 20:09:40 +02:00
fc4a809985
fix(cloudflare): Limit the waiting period for websites that captcha doesn't disappear after solving
...
Also solves #100
2025-10-27 05:13:12 +03:00
d4a2507666
refactor(validation): All browser based fetchers are ~8-15% faster now in most cases
2025-10-27 04:48:45 +03:00
13b9a4668c
feat(DynamicSession): New option to add extra browser flags
2025-10-26 16:53:27 +03:00
59ec6be201
refactor(fetchers): Less duplicated code and better handling for unstable websites
...
- Websites that never finish loading their requests won't crash the code now if you used `network_idle` with them
- Fixed a typo in `load_dom` in DynamicSession's async_fetch
- Removed dead code
2025-10-26 05:04:19 +03:00
74fa45daea
fix: Addressing the issue of collecting response after page_action in #100
2025-10-26 00:41:45 +03:00
9c7114219b
fix(solver): Improve the logic a bit
2025-10-12 06:41:04 +03:00
a38b92cc87
feat(Dynamic fetcher): Add additional_args option to customize browser as in StealthyFetcher
2025-10-12 05:07:22 +03:00
c34e10faed
feat(fetchers): Add argument to control the user data directory
2025-10-12 04:48:59 +03:00
8c995c936d
fix(CF solver): further improve the detection
2025-10-12 04:15:16 +03:00
073c4910d4
fix(fetchers): More accurate collection of the final response
...
This will make the `response.body` return the raw response for text/json responses as well
2025-10-12 04:12:18 +03:00
6bfa266c90
fix(camoufox): Fix a typo in the async solver
2025-10-12 03:07:18 +03:00
e53ce3f749
refactor: Making the requests logic better and cleaner
...
Also, solve some bugs along the way and a cleaner fix for #97
2025-10-06 05:00:41 +03:00
debe03256b
refactor: Making all the codebase acceptable by PyRight
...
Also fixes #97
2025-10-05 04:03:39 +03:00
e149c715dd
feat(CF solver): better implementation to help with all custom implementations of turnstile
2025-10-02 20:05:39 +03:00
eedfa855ab
fix: Fixes for the type checking in the main init file
...
and a bit of cleaning
2025-10-01 03:50:39 +03:00
f6c122b87f
style: Removing dead code/docstrings
2025-09-29 05:09:23 +03:00
8ad7cd6343
docs: Update all pages/docstrings to reflect recent changes
2025-09-29 03:57:40 +03:00
716a5c3572
feat(cf solver): Make solver able to handle Turnstile and interstitial
2025-09-28 02:53:51 +03:00
82b09620b2
perf: Improving all browser-based fetches by ~15%
2025-09-20 06:20:23 +03:00
970ad40404
perf: Speed up sync fetches by ignoring number of pages checking
2025-09-20 04:31:57 +03:00
67db865aee
feat(DynamicFetcher): Replace rebrowser with patchright
2025-09-20 00:23:29 +03:00
Karim shoair