ef8c5bc7d6
feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
...
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.
- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.
- Normal requests use what you pass without issues, of course.
- All errors are retried now, and proxies are rotated on retries automatically.
2026-02-02 00:16:22 +02:00
90c52c45c7
feat(parser): replacing tldextract with tld library
...
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).
In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
2026-01-23 00:00:48 +02:00
95955af3be
build: Add w3lib to deps
2026-01-20 16:36:34 +02:00
aa4817f349
feat: Change logger to be flexible enough for the spiders classes
2026-01-11 03:22:20 +02:00
c5bc82deb8
fix(browsers): solve a bug with setting referer on the request level
2026-01-11 03:21:03 +02:00
0634f5796c
feat(spiders): Add follow function to the response
2026-01-10 20:39:43 +02:00
5e3845f060
docs: shorten docstrings for some functions
2025-12-28 21:56:28 +02:00
9bfe744840
fix: remove forgotten unused code
2025-12-28 00:14:47 +02:00
1b838eabd0
fix: update the cookies type hint for the mcp server
2025-12-28 00:08:44 +02:00
842718ba94
fix: update shell signatures for the autocompletion
...
Added missing fields and reordered classes as well for easier comparisons
2025-12-27 14:30:48 +02:00
cfc667f7dc
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
...
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
2025-12-26 03:21:13 +02:00
ac2d4d3833
fix: use the correct signature for the stealthy_fetch shortcut
2025-12-02 02:15:42 +02:00
9b3309f825
fix(shell): use correct argument name in signatures
2025-11-26 02:22:29 +02:00
8940bdeb55
fix(shell): dynamically build the signature of shortcuts after last changes
2025-11-25 21:23:06 +02:00
5376c97996
fix(fetcher): Remove non-keywords arguments
2025-11-25 20:25:09 +02:00
f0b01fc253
refactor(browser fetchers): Make all the type hints dynamic + Faster validation
...
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
2025-11-25 20:18:36 +02:00
357b170e1c
refactor(requests): Make all the type hints dynamic
...
Made the code shorter by about ~500 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
2025-11-23 20:48:51 +02:00
fb80beba04
feat(TextHandler): Add argument to clean method to remove html entities
2025-11-17 01:36:06 +02:00
3565f9d500
feat(fetcher): Make impersonate able to randomize fingerprint
2025-11-16 16:33:58 +02:00
debe03256b
refactor: Making all the codebase acceptable by PyRight
...
Also fixes #97
2025-10-05 04:03:39 +03:00
eedfa855ab
fix: Fixes for the type checking in the main init file
...
and a bit of cleaning
2025-10-01 03:50:39 +03:00
f6c122b87f
style: Removing dead code/docstrings
2025-09-29 05:09:23 +03:00
8ad7cd6343
docs: Update all pages/docstrings to reflect recent changes
2025-09-29 03:57:40 +03:00
74f20d2a0e
refactor: better implementation for the mcp mode
2025-09-28 20:14:04 +03:00
4a661b4875
feat: Make mcp able to use http transport
2025-09-28 04:54:35 +03:00
3da806210b
perf: General code restructure to not use more than needed memory
2025-09-23 18:34:54 +03:00
2d704b2a8b
fix(shell): Fixing a bug with content converting
2025-09-19 04:50:15 +03:00
a9d05cc0ef
style: Removing dead code/docstrings and correcting type hints
2025-09-19 04:49:30 +03:00
bf5fe6d451
fix(shell): Fix view command edge case
2025-09-19 04:21:15 +03:00
504ecdf7dc
fix(shell): Fix parsing curl --data-raw
2025-09-19 04:20:16 +03:00
05c2d9aa00
feat(CLI): Make the session classes available in the shell
2025-09-15 23:34:36 +03:00
1547366c85
fix: fixing body issue with cli view command
2025-09-15 21:32:57 +03:00
f848e6b7f2
fix: Fixes to handle the new extra deps group
2025-09-15 00:19:19 +03:00
67ca139ff9
fix: Fixes for multiple encoding issues ( #80 & #81 )
2025-09-14 20:20:29 +03:00
0b3509a295
fix(shell): Fix a small typo
2025-09-14 05:05:55 +03:00
13d7e70cb7
refactor: Make all fetchers as an optional dependency group
...
+ Removing some dead code
2025-09-13 16:07:48 +03:00
330d03559c
style: applying the new ruff rules to all files
2025-09-13 03:22:53 +03:00
af9a62a539
fix: fix for shell imported by mistake issue ( #76 )
2025-09-02 15:23:15 +03:00
2f402f4835
style: add flags for tests coverage
...
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
2025-08-17 01:02:14 +03:00
13700e2692
fix: improve error handling
2025-08-15 04:52:11 +03:00
83a19f3b17
perf(parser): A lot of optimizations to speed things up
2025-08-01 06:28:52 +03:00
f93348b915
style: remove unused code
2025-08-01 05:43:31 +03:00
df48662c00
perf(parser): A lot of optimizations to speed things up
2025-08-01 05:42:32 +03:00
af5f3688c1
fix: moving types to use Union again
2025-07-30 05:39:25 +03:00
27658b33ff
fix: fix invalid return type
2025-07-30 03:39:33 +03:00
32cb76604c
style: Adjustments to the translator
2025-07-30 03:28:16 +03:00
5bb1266fa5
style: using isinstance function as the main way for type checking
2025-07-30 03:14:23 +03:00
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
ca12a11b7e
style(utils): optimize imports
2025-07-30 02:08:01 +03:00
db781dcc32
style: Removing dead code
2025-07-30 02:03:39 +03:00
Karim shoair