5f557a7f9f
test: add tests for the new feature
2026-02-07 01:28:58 +02:00
66476d42be
feat(browsers): Add option to block requests to specific domains
2026-02-07 01:26:21 +02:00
d00c34ee7f
style(parser): Improve the type hint for find_by_text and find_by_regex
2026-02-06 03:19:00 +02:00
05f62fba81
fix(response): Force the body of the response to always be bytes
2026-02-06 03:14:23 +02:00
9acc8194c5
test: update the tests accordingly
2026-02-06 02:45:13 +02:00
aa7a95fb70
feat(parser)!: Make all selection return selector objects by default
...
- The strings/Texthandlers are only returned by `get`/`getall`/`extract`/`extract_first`. This makes the type checking/autocompletion experience consistent.
- Removed `css_first` and `xpath_first` since it doesn't make sense to leave them now.
- Made the type hints more accurate in multiple places.
2026-02-06 02:43:51 +02:00
d8fe909c8b
docs: make translation urls absolute
...
For PyPI, Docker, and other external websites featuring this
2026-02-05 02:44:12 +02:00
151e4e7a7c
docs: adding new sponsor
2026-02-05 02:39:51 +02:00
f88502718f
feat(proxy control): Force a proxy at request level at any given point
...
And merge request's meta with response's meta
2026-02-02 14:26:52 +02:00
6e0277f23b
test: add new tests for the proxy rotation logic
2026-02-02 00:16:40 +02:00
ef8c5bc7d6
feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
...
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.
- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.
- Normal requests use what you pass without issues, of course.
- All errors are retried now, and proxies are rotated on retries automatically.
2026-02-02 00:16:22 +02:00
65f1a23358
fix(browsers): Improve stealth and speed by adjusting flags
2026-01-29 00:58:04 +02:00
0c3d359b46
fix(browsers): disable autoplay for all browsers
2026-01-29 00:57:30 +02:00
90c52c45c7
feat(parser): replacing tldextract with tld library
...
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).
In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
2026-01-23 00:00:48 +02:00
5dbbd84b3d
fix(requests): return the cached fingerprint on second call
2026-01-20 20:28:13 +02:00
d90189e675
test: update tests accordingly
2026-01-20 20:22:47 +02:00
122923c80e
feat(spiders/requests): Use a reliable method for fingerprint generation
2026-01-20 20:21:42 +02:00
5aa178a76e
style: update post requests type hints to be more accurate
2026-01-20 18:27:52 +02:00
d0f52c3855
test: remove the tests for the removed html_utils module
2026-01-20 16:37:17 +02:00
95955af3be
build: Add w3lib to deps
2026-01-20 16:36:34 +02:00
0ec78e2335
test: update tests accordingly
2026-01-19 16:40:18 +02:00
6947066761
feat(spiders): Make graceful shutdown always enabled
2026-01-19 16:40:02 +02:00
94bc759c6a
test: updates tests to include new logic
2026-01-19 16:13:29 +02:00
25850db92c
fix(spiders): handle errors with manual pause during stream mode
2026-01-19 16:13:03 +02:00
fbc8be5bcc
fix(spiders): fix a bug with session stats counter
2026-01-19 16:00:06 +02:00
4ecf8a64d3
fix(spiders): Fix for when the pausing system has large queue
2026-01-19 15:04:40 +02:00
cf0de7ea6d
Merge branch 'main' into v4
2026-01-19 14:47:59 +02:00
9229cef895
docs: adding new sponsor
2026-01-19 14:12:32 +02:00
d247b5b691
test: add tests for the spiders system
...
It's generated by Opus on Claude Code. It's very good as per my review and instructions, but I will have another look later.
2026-01-19 03:46:00 +02:00
0d003f62e3
ops: update bandit checklist
2026-01-19 03:45:39 +02:00
1f0f86e3a2
feat(spiders): Add pause/resume system for crawls
2026-01-19 03:43:10 +02:00
acdb39988c
ops: update bandit checklist
2026-01-19 03:40:57 +02:00
bf5aa021d4
fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails
...
This has been causing issues with tests for a long time, and now finally found the reason.
2026-01-18 16:46:48 +02:00
02c9dffc77
style(spiders): make parse function abstract
2026-01-18 00:01:04 +02:00
afe6b2e97b
feat(spiders): Change items hook to make it for processing items + add a stat for this
2026-01-17 20:52:49 +02:00
41c7a7e793
fix(spiders engine): Improve logging
2026-01-17 19:57:04 +02:00
dac854a129
fix(parser): handle responses with empty body
...
It makes root's value None
2026-01-13 02:35:26 +02:00
35d570120a
feat(spiders): Add option to logging format and date format
2026-01-12 17:11:52 +02:00
3466bcab2c
feat: add hook to run after collecting an item
2026-01-12 15:58:33 +02:00
e0f94cf7d0
feat(spiders): Add a stream mode to collect items in realtime
2026-01-12 15:53:29 +02:00
927cfd85ff
feat(spiders): The option to start sessions lazily
2026-01-12 02:09:50 +02:00
f2c6216968
style: remove unwanted import from spiders main init file
2026-01-12 01:21:26 +02:00
72c3c2792a
feat(spiders): add simple export system for the results
2026-01-12 01:21:05 +02:00
a17bb1d976
feat(browsers): Add option to retry tabs that gives errors
2026-01-11 23:40:26 +02:00
5a11c2466b
fix: make retry_blocked_request take the response as well
2026-01-11 22:54:17 +02:00
736873a7c6
feat(spiders): Add logic to detect blocked requests and retry them
2026-01-11 21:56:57 +02:00
774364b6d3
style(spiders): make the usage of uvloop optional
2026-01-11 19:57:03 +02:00
0bfdab7158
feat: spiders use uvloop/winloop when available
2026-01-11 18:39:42 +02:00
d5b9ed11b2
fix(browsers): Close pages that give error
...
This was causing the browser and the whole code to stand still.
2026-01-11 18:34:13 +02:00
32f4f2adf4
build: Add AnyIO to the dependencies
2026-01-11 17:01:33 +02:00
Karim shoair