a17bb1d976
feat(browsers): Add option to retry tabs that gives errors
2026-01-11 23:40:26 +02:00
d5b9ed11b2
fix(browsers): Close pages that give error
...
This was causing the browser and the whole code to stand still.
2026-01-11 18:34:13 +02:00
c5bc82deb8
fix(browsers): solve a bug with setting referer on the request level
2026-01-11 03:21:03 +02:00
6a73f9dcd7
refactor: internal API changes to be easily used as indicators for spiders
2026-01-08 00:52:06 +02:00
4947152be2
docs: change docstring style for some methods
2026-01-01 21:06:35 +02:00
ee13af9516
build: update deps
...
and docstrings
2025-12-29 16:08:51 +02:00
5e3845f060
docs: shorten docstrings for some functions
2025-12-28 21:56:28 +02:00
cfc667f7dc
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
...
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
2025-12-26 03:21:13 +02:00
ecb97d3fd6
docs: add docs for timezone_id argument
2025-12-18 01:16:22 +02:00
b507e4d4a0
refactor(fetchers): rename internal api
...
To make it easier to use to use sessions outside `with` context
2025-12-17 01:27:40 +02:00
f0b01fc253
refactor(browser fetchers): Make all the type hints dynamic + Faster validation
...
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
2025-11-25 20:18:36 +02:00
df073c4680
fix: added raw_response to separate collected response from rendered response
2025-11-08 20:09:40 +02:00
d4a2507666
refactor(validation): All browser based fetchers are ~8-15% faster now in most cases
2025-10-27 04:48:45 +03:00
13b9a4668c
feat(DynamicSession): New option to add extra browser flags
2025-10-26 16:53:27 +03:00
59ec6be201
refactor(fetchers): Less duplicated code and better handling for unstable websites
...
- Websites that never finish loading their requests won't crash the code now if you used `network_idle` with them
- Fixed a typo in `load_dom` in DynamicSession's async_fetch
- Removed dead code
2025-10-26 05:04:19 +03:00
74fa45daea
fix: Addressing the issue of collecting response after page_action in #100
2025-10-26 00:41:45 +03:00
a38b92cc87
feat(Dynamic fetcher): Add additional_args option to customize browser as in StealthyFetcher
2025-10-12 05:07:22 +03:00
c34e10faed
feat(fetchers): Add argument to control the user data directory
2025-10-12 04:48:59 +03:00
073c4910d4
fix(fetchers): More accurate collection of the final response
...
This will make the `response.body` return the raw response for text/json responses as well
2025-10-12 04:12:18 +03:00
debe03256b
refactor: Making all the codebase acceptable by PyRight
...
Also fixes #97
2025-10-05 04:03:39 +03:00
f6c122b87f
style: Removing dead code/docstrings
2025-09-29 05:09:23 +03:00
82b09620b2
perf: Improving all browser-based fetches by ~15%
2025-09-20 06:20:23 +03:00
67db865aee
feat(DynamicFetcher): Replace rebrowser with patchright
2025-09-20 00:23:29 +03:00
7ba333c3c9
perf(session): improve page management and speed up execution
...
Close pages immediately after tasks finish to reduce resource usage and
improve overall performance. Update corresponding test files to reflect
the new page lifecycle management.
2025-09-17 18:49:45 +03:00
7a7097b775
fix: Stop removing browser default page
...
Looks like it's problematic with some systems so it doesn't worth the trouble
2025-09-15 13:43:47 +03:00
13d7e70cb7
refactor: Make all fetchers as an optional dependency group
...
+ Removing some dead code
2025-09-13 16:07:48 +03:00
8c4d6158d6
feat(browser fetchers): A new option to control page JS
2025-09-13 03:59:12 +03:00
60be9dc816
feat: Validate all fetch-level parameters
...
So it's validated like the session-level ones. Needs improvement, but that's for later
2025-09-13 03:22:08 +03:00
adb82008e5
style: Using keywords for validation
...
Less lines of code and more stylish
2025-09-12 04:33:00 +03:00
05a714980d
docs: Update all docstring according to the new changes
2025-09-11 04:13:36 +03:00
9838d741d0
refactor/feat(browser fetchers): make it possible to have a configuration per page in sessions
...
- Also, no need for the `page_action` argument function to return the page again
2025-09-11 03:48:55 +03:00
ecaec65049
refactor/fix(browser fetchers): Solving a logic bug in page rotation
...
- Pages that get reused in rotation are possibly contaminated from previous settings used on them, and some are stubborn to remove, so the new approach replaces finished pages with new ones.
- Removed the `max_pages` from sync `StealthySession` to match `DynamicSession` (Doesn't mean anything in sync code)
2025-09-08 16:08:57 +03:00
710f994339
perf(DynamicSession): Close persistent context default page
2025-09-07 18:48:40 +03:00
6ad1bab2e4
feat(DynamicFetcher): add the ability to add init_script to context ( #56 )
2025-09-02 16:25:13 +03:00
b5be2547d7
fix(dynamicSession): issue with headless
2025-08-31 01:29:47 +03:00
2f402f4835
style: add flags for tests coverage
...
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
2025-08-17 01:02:14 +03:00
009bbb20c0
refactor(dynamic Fetcher): small optimization
2025-08-16 23:20:54 +03:00
6c9a960a44
fix(dynamicFetcher): Fix a bug with real_chrome argument
2025-08-16 16:01:32 +03:00
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
63a9db21a1
refactor(DynamicSession): Optimization + Removing max_pages from sync version
2025-07-06 04:59:22 +03:00
dbf7aa7761
feat(DynamicFetcher): Use persistent context by default for better stealth
...
And might solve the sandbox issue with GitHub
2025-06-27 03:02:46 +03:00
8395db9121
fix(controllers): Fix issue with async adding stealth scripts
2025-06-27 01:39:32 +03:00
37e842b796
fix(controllers): Fix issue with adding stealth scripts
2025-06-27 01:04:58 +03:00
63ff12bcd6
docs: correcting a small mistake
2025-06-23 03:38:34 +03:00
78b81c537f
fix(fetchers): Fix the bug of referer and google_search argument conflict
2025-06-23 03:36:11 +03:00
460b4443c2
refactor(controllers): Improve type validation
2025-06-22 19:59:40 +03:00
fc42487a5f
refactor(controllers): Optimize imports
2025-06-22 18:53:44 +03:00
0a570a7ca7
docs: Updating doc strings
2025-06-22 01:59:38 +03:00
145b710960
feat(fetchers): Adding the foundation of the new browser-based fetchers logic
2025-06-19 04:06:52 +03:00
Karim shoair