90c52c45c7
feat(parser): replacing tldextract with tld library
...
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).
In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
2026-01-23 00:00:48 +02:00
5dbbd84b3d
fix(requests): return the cached fingerprint on second call
2026-01-20 20:28:13 +02:00
d90189e675
test: update tests accordingly
2026-01-20 20:22:47 +02:00
122923c80e
feat(spiders/requests): Use a reliable method for fingerprint generation
2026-01-20 20:21:42 +02:00
5aa178a76e
style: update post requests type hints to be more accurate
2026-01-20 18:27:52 +02:00
d0f52c3855
test: remove the tests for the removed html_utils module
2026-01-20 16:37:17 +02:00
95955af3be
build: Add w3lib to deps
2026-01-20 16:36:34 +02:00
0ec78e2335
test: update tests accordingly
2026-01-19 16:40:18 +02:00
6947066761
feat(spiders): Make graceful shutdown always enabled
2026-01-19 16:40:02 +02:00
94bc759c6a
test: updates tests to include new logic
2026-01-19 16:13:29 +02:00
25850db92c
fix(spiders): handle errors with manual pause during stream mode
2026-01-19 16:13:03 +02:00
fbc8be5bcc
fix(spiders): fix a bug with session stats counter
2026-01-19 16:00:06 +02:00
4ecf8a64d3
fix(spiders): Fix for when the pausing system has large queue
2026-01-19 15:04:40 +02:00
cf0de7ea6d
Merge branch 'main' into v4
2026-01-19 14:47:59 +02:00
9229cef895
docs: adding new sponsor
2026-01-19 14:12:32 +02:00
d247b5b691
test: add tests for the spiders system
...
It's generated by Opus on Claude Code. It's very good as per my review and instructions, but I will have another look later.
2026-01-19 03:46:00 +02:00
0d003f62e3
ops: update bandit checklist
2026-01-19 03:45:39 +02:00
1f0f86e3a2
feat(spiders): Add pause/resume system for crawls
2026-01-19 03:43:10 +02:00
acdb39988c
ops: update bandit checklist
2026-01-19 03:40:57 +02:00
bf5aa021d4
fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails
...
This has been causing issues with tests for a long time, and now finally found the reason.
2026-01-18 16:46:48 +02:00
02c9dffc77
style(spiders): make parse function abstract
2026-01-18 00:01:04 +02:00
afe6b2e97b
feat(spiders): Change items hook to make it for processing items + add a stat for this
2026-01-17 20:52:49 +02:00
41c7a7e793
fix(spiders engine): Improve logging
2026-01-17 19:57:04 +02:00
dac854a129
fix(parser): handle responses with empty body
...
It makes root's value None
2026-01-13 02:35:26 +02:00
35d570120a
feat(spiders): Add option to logging format and date format
2026-01-12 17:11:52 +02:00
3466bcab2c
feat: add hook to run after collecting an item
2026-01-12 15:58:33 +02:00
e0f94cf7d0
feat(spiders): Add a stream mode to collect items in realtime
2026-01-12 15:53:29 +02:00
927cfd85ff
feat(spiders): The option to start sessions lazily
2026-01-12 02:09:50 +02:00
f2c6216968
style: remove unwanted import from spiders main init file
2026-01-12 01:21:26 +02:00
72c3c2792a
feat(spiders): add simple export system for the results
2026-01-12 01:21:05 +02:00
a17bb1d976
feat(browsers): Add option to retry tabs that gives errors
2026-01-11 23:40:26 +02:00
5a11c2466b
fix: make retry_blocked_request take the response as well
2026-01-11 22:54:17 +02:00
736873a7c6
feat(spiders): Add logic to detect blocked requests and retry them
2026-01-11 21:56:57 +02:00
774364b6d3
style(spiders): make the usage of uvloop optional
2026-01-11 19:57:03 +02:00
0bfdab7158
feat: spiders use uvloop/winloop when available
2026-01-11 18:39:42 +02:00
d5b9ed11b2
fix(browsers): Close pages that give error
...
This was causing the browser and the whole code to stand still.
2026-01-11 18:34:13 +02:00
32f4f2adf4
build: Add AnyIO to the dependencies
2026-01-11 17:01:33 +02:00
059a708b6d
feat(spiders system): a prototype of the new spiders system
...
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library like other known alternatives.
- Each spider has its own logger that forces sessions to use it.
- Each spider has functions to override that run before start and after close.
- There's a spider argument to set the logging level and another one to make the spider write to a log file.
- This is only the start. A lot more features are coming in the way.
2026-01-11 16:53:18 +02:00
dd5c30c521
feat(response): Option to set the referer as the previous response url.
2026-01-11 03:24:30 +02:00
1443a469a3
fix: handle a bug with response follow referer flow argument
2026-01-11 03:22:51 +02:00
aa4817f349
feat: Change logger to be flexible enough for the spiders classes
2026-01-11 03:22:20 +02:00
c5bc82deb8
fix(browsers): solve a bug with setting referer on the request level
2026-01-11 03:21:03 +02:00
8e36962177
ops: add local Claude files to gitignore
2026-01-10 20:41:23 +02:00
0634f5796c
feat(spiders): Add follow function to the response
2026-01-10 20:39:43 +02:00
f80e8828f7
refactor: internal API changes
2026-01-08 01:53:48 +02:00
3a360acced
build: pump version up temporary
2026-01-08 01:37:47 +02:00
08f3e6efad
fix: add missing line
2026-01-08 01:01:00 +02:00
6553d90e0f
ops: update workflows to run for this branch
2026-01-08 00:53:29 +02:00
9eb5f68fcb
test: update tests accordingly
2026-01-08 00:52:35 +02:00
6a73f9dcd7
refactor: internal API changes to be easily used as indicators for spiders
2026-01-08 00:52:06 +02:00
Karim shoair