Commit Graph

20 Commits

Author SHA1 Message Date
Karim shoair aed34e43b5 fix(fetchers): better handling for infinite network_idle 2025-12-03 03:11:30 +02:00
Karim shoair f0b01fc253 refactor(browser fetchers): Make all the type hints dynamic + Faster validation
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
2025-11-25 20:18:36 +02:00
Karim shoair 13b9a4668c feat(DynamicSession): New option to add extra browser flags 2025-10-26 16:53:27 +03:00
Karim shoair 59ec6be201 refactor(fetchers): Less duplicated code and better handling for unstable websites
- Websites that never finish loading their requests won't crash the code now if you used `network_idle` with them
- Fixed a typo in `load_dom` in DynamicSession's async_fetch
- Removed dead code
2025-10-26 05:04:19 +03:00
Karim shoair 74fa45daea fix: Addressing the issue of collecting response after page_action in #100 2025-10-26 00:41:45 +03:00
Karim shoair a38b92cc87 feat(Dynamic fetcher): Add additional_args option to customize browser as in StealthyFetcher 2025-10-12 05:07:22 +03:00
Karim shoair c34e10faed feat(fetchers): Add argument to control the user data directory 2025-10-12 04:48:59 +03:00
Karim shoair debe03256b refactor: Making all the codebase acceptable by PyRight
Also fixes #97
2025-10-05 04:03:39 +03:00
Karim shoair 716a5c3572 feat(cf solver): Make solver able to handle Turnstile and interstitial 2025-09-28 02:53:51 +03:00
Karim shoair 82b09620b2 perf: Improving all browser-based fetches by ~15% 2025-09-20 06:20:23 +03:00
Karim shoair 970ad40404 perf: Speed up sync fetches by ignoring number of pages checking 2025-09-20 04:31:57 +03:00
Abdullah 7ba333c3c9 perf(session): improve page management and speed up execution
Close pages immediately after tasks finish to reduce resource usage and
improve overall performance. Update corresponding test files to reflect
the new page lifecycle management.
2025-09-17 18:49:45 +03:00
Karim shoair 5683965367 fix(sessions): Fixing a bug with max pages counter 2025-09-16 01:47:31 +03:00
Karim shoair 13d7e70cb7 refactor: Make all fetchers as an optional dependency group
+ Removing some dead code
2025-09-13 16:07:48 +03:00
Karim shoair 8c4d6158d6 feat(browser fetchers): A new option to control page JS 2025-09-13 03:59:12 +03:00
Karim shoair 330d03559c style: applying the new ruff rules to all files 2025-09-13 03:22:53 +03:00
Karim shoair 0c7db3c9d9 fix: validation keywords 2025-09-13 02:07:04 +03:00
Karim shoair adb82008e5 style: Using keywords for validation
Less lines of code and more stylish
2025-09-12 04:33:00 +03:00
Karim shoair 9838d741d0 refactor/feat(browser fetchers): make it possible to have a configuration per page in sessions
- Also, no need for the `page_action` argument function to return the page again
2025-09-11 03:48:55 +03:00
Karim shoair ecaec65049 refactor/fix(browser fetchers): Solving a logic bug in page rotation
- Pages that get reused in rotation are possibly contaminated from previous settings used on them, and some are stubborn to remove, so the new approach replaces finished pages with new ones.
- Removed the `max_pages` from sync `StealthySession` to match `DynamicSession` (Doesn't mean anything in sync code)
2025-09-08 16:08:57 +03:00