95955af3be
build: Add w3lib to deps
2026-01-20 16:36:34 +02:00
6947066761
feat(spiders): Make graceful shutdown always enabled
2026-01-19 16:40:02 +02:00
25850db92c
fix(spiders): handle errors with manual pause during stream mode
2026-01-19 16:13:03 +02:00
fbc8be5bcc
fix(spiders): fix a bug with session stats counter
2026-01-19 16:00:06 +02:00
4ecf8a64d3
fix(spiders): Fix for when the pausing system has large queue
2026-01-19 15:04:40 +02:00
1f0f86e3a2
feat(spiders): Add pause/resume system for crawls
2026-01-19 03:43:10 +02:00
bf5aa021d4
fix(browsers): solving an issues with leaving playwright loop open when cdp connection fails
...
This has been causing issues with tests for a long time, and now finally found the reason.
2026-01-18 16:46:48 +02:00
02c9dffc77
style(spiders): make parse function abstract
2026-01-18 00:01:04 +02:00
afe6b2e97b
feat(spiders): Change items hook to make it for processing items + add a stat for this
2026-01-17 20:52:49 +02:00
41c7a7e793
fix(spiders engine): Improve logging
2026-01-17 19:57:04 +02:00
dac854a129
fix(parser): handle responses with empty body
...
It makes root's value None
2026-01-13 02:35:26 +02:00
35d570120a
feat(spiders): Add option to logging format and date format
2026-01-12 17:11:52 +02:00
3466bcab2c
feat: add hook to run after collecting an item
2026-01-12 15:58:33 +02:00
e0f94cf7d0
feat(spiders): Add a stream mode to collect items in realtime
2026-01-12 15:53:29 +02:00
927cfd85ff
feat(spiders): The option to start sessions lazily
2026-01-12 02:09:50 +02:00
f2c6216968
style: remove unwanted import from spiders main init file
2026-01-12 01:21:26 +02:00
72c3c2792a
feat(spiders): add simple export system for the results
2026-01-12 01:21:05 +02:00
a17bb1d976
feat(browsers): Add option to retry tabs that gives errors
2026-01-11 23:40:26 +02:00
5a11c2466b
fix: make retry_blocked_request take the response as well
2026-01-11 22:54:17 +02:00
736873a7c6
feat(spiders): Add logic to detect blocked requests and retry them
2026-01-11 21:56:57 +02:00
774364b6d3
style(spiders): make the usage of uvloop optional
2026-01-11 19:57:03 +02:00
0bfdab7158
feat: spiders use uvloop/winloop when available
2026-01-11 18:39:42 +02:00
d5b9ed11b2
fix(browsers): Close pages that give error
...
This was causing the browser and the whole code to stand still.
2026-01-11 18:34:13 +02:00
059a708b6d
feat(spiders system): a prototype of the new spiders system
...
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library like other known alternatives.
- Each spider has its own logger that forces sessions to use it.
- Each spider has functions to override that run before start and after close.
- There's a spider argument to set the logging level and another one to make the spider write to a log file.
- This is only the start. A lot more features are coming in the way.
2026-01-11 16:53:18 +02:00
dd5c30c521
feat(response): Option to set the referer as the previous response url.
2026-01-11 03:24:30 +02:00
1443a469a3
fix: handle a bug with response follow referer flow argument
2026-01-11 03:22:51 +02:00
aa4817f349
feat: Change logger to be flexible enough for the spiders classes
2026-01-11 03:22:20 +02:00
c5bc82deb8
fix(browsers): solve a bug with setting referer on the request level
2026-01-11 03:21:03 +02:00
0634f5796c
feat(spiders): Add follow function to the response
2026-01-10 20:39:43 +02:00
f80e8828f7
refactor: internal API changes
2026-01-08 01:53:48 +02:00
3a360acced
build: pump version up temporary
2026-01-08 01:37:47 +02:00
08f3e6efad
fix: add missing line
2026-01-08 01:01:00 +02:00
6a73f9dcd7
refactor: internal API changes to be easily used as indicators for spiders
2026-01-08 00:52:06 +02:00
c8a2456b81
fix(StealthyFetcher): disable incognito mode to solve #123
2026-01-03 01:50:02 +02:00
30a77de2be
build: pump version up and browserforge version
2026-01-03 01:48:34 +02:00
4947152be2
docs: change docstring style for some methods
2026-01-01 21:06:35 +02:00
3b9a6d021e
fix: correct cli argument
2025-12-31 03:00:06 +02:00
f6f0ed4b37
fix(async stealth): wait the coroutine of the bounding box
2025-12-31 00:19:53 +02:00
0c7e26ae5c
fix(fingerprints): do not create google referer for IPs
2025-12-30 21:58:09 +02:00
ee13af9516
build: update deps
...
and docstrings
2025-12-29 16:08:51 +02:00
5e3845f060
docs: shorten docstrings for some functions
2025-12-28 21:56:28 +02:00
9bfe744840
fix: remove forgotten unused code
2025-12-28 00:14:47 +02:00
1b838eabd0
fix: update the cookies type hint for the mcp server
2025-12-28 00:08:44 +02:00
09bbd416b8
fix: remove old proxy logic
...
When I was caching it with the rest of the data
2025-12-28 00:02:50 +02:00
fb3a1b4c17
style: removing unused code
2025-12-27 20:50:08 +02:00
842718ba94
fix: update shell signatures for the autocompletion
...
Added missing fields and reordered classes as well for easier comparisons
2025-12-27 14:30:48 +02:00
cfc667f7dc
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
...
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
2025-12-26 03:21:13 +02:00
f9f34e6ee9
fix: improve the handling of useragent in chrome fetcher
2025-12-24 23:08:30 +02:00
b40c16a2cd
build: pump version up
2025-12-24 21:35:41 +02:00
85b327a598
fix: Improve stealth of dynamic browser by pinning Chrome's UA version
2025-12-24 21:34:44 +02:00
Karim shoair