35d570120a
feat(spiders): Add option to logging format and date format
2026-01-12 17:11:52 +02:00
3466bcab2c
feat: add hook to run after collecting an item
2026-01-12 15:58:33 +02:00
e0f94cf7d0
feat(spiders): Add a stream mode to collect items in realtime
2026-01-12 15:53:29 +02:00
927cfd85ff
feat(spiders): The option to start sessions lazily
2026-01-12 02:09:50 +02:00
f2c6216968
style: remove unwanted import from spiders main init file
2026-01-12 01:21:26 +02:00
72c3c2792a
feat(spiders): add simple export system for the results
2026-01-12 01:21:05 +02:00
a17bb1d976
feat(browsers): Add option to retry tabs that gives errors
2026-01-11 23:40:26 +02:00
5a11c2466b
fix: make retry_blocked_request take the response as well
2026-01-11 22:54:17 +02:00
736873a7c6
feat(spiders): Add logic to detect blocked requests and retry them
2026-01-11 21:56:57 +02:00
774364b6d3
style(spiders): make the usage of uvloop optional
2026-01-11 19:57:03 +02:00
0bfdab7158
feat: spiders use uvloop/winloop when available
2026-01-11 18:39:42 +02:00
d5b9ed11b2
fix(browsers): Close pages that give error
...
This was causing the browser and the whole code to stand still.
2026-01-11 18:34:13 +02:00
32f4f2adf4
build: Add AnyIO to the dependencies
2026-01-11 17:01:33 +02:00
059a708b6d
feat(spiders system): a prototype of the new spiders system
...
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library like other known alternatives.
- Each spider has its own logger that forces sessions to use it.
- Each spider has functions to override that run before start and after close.
- There's a spider argument to set the logging level and another one to make the spider write to a log file.
- This is only the start. A lot more features are coming in the way.
2026-01-11 16:53:18 +02:00
dd5c30c521
feat(response): Option to set the referer as the previous response url.
2026-01-11 03:24:30 +02:00
1443a469a3
fix: handle a bug with response follow referer flow argument
2026-01-11 03:22:51 +02:00
aa4817f349
feat: Change logger to be flexible enough for the spiders classes
2026-01-11 03:22:20 +02:00
c5bc82deb8
fix(browsers): solve a bug with setting referer on the request level
2026-01-11 03:21:03 +02:00
8e36962177
ops: add local Claude files to gitignore
2026-01-10 20:41:23 +02:00
0634f5796c
feat(spiders): Add follow function to the response
2026-01-10 20:39:43 +02:00
f80e8828f7
refactor: internal API changes
2026-01-08 01:53:48 +02:00
3a360acced
build: pump version up temporary
2026-01-08 01:37:47 +02:00
08f3e6efad
fix: add missing line
2026-01-08 01:01:00 +02:00
6553d90e0f
ops: update workflows to run for this branch
2026-01-08 00:53:29 +02:00
9eb5f68fcb
test: update tests accordingly
2026-01-08 00:52:35 +02:00
6a73f9dcd7
refactor: internal API changes to be easily used as indicators for spiders
2026-01-08 00:52:06 +02:00
de79fe80bb
v0.3.14
2026-01-03 22:29:20 +02:00
2f87507afe
test: update database test to suite windows
2026-01-03 22:02:03 +02:00
f99baf65f7
test: remove the check for incognito
2026-01-03 01:50:46 +02:00
c8a2456b81
fix(StealthyFetcher): disable incognito mode to solve #123
2026-01-03 01:50:02 +02:00
30a77de2be
build: pump version up and browserforge version
2026-01-03 01:48:34 +02:00
6ad2ecb9f2
v0.3.13
2026-01-01 22:07:00 +02:00
b5fdb71b52
docs: improving the API reference section
2026-01-01 21:07:20 +02:00
4947152be2
docs: change docstring style for some methods
2026-01-01 21:06:35 +02:00
9742b239be
docs: fix for rtd build
2026-01-01 17:42:47 +02:00
e275509ece
docs: fix for rtd build
...
test
2026-01-01 17:05:10 +02:00
185a9c594f
docs: fix for rtd build
2026-01-01 17:01:24 +02:00
c6e4482837
docs: Improve docs website
2026-01-01 16:56:57 +02:00
cb5051ca56
docs: update contributing page
2026-01-01 03:18:15 +02:00
4d478eb84c
docs: update the donate page
2026-01-01 03:16:10 +02:00
2d5e2f41c4
docs: update dev articles
2026-01-01 03:14:12 +02:00
0a3bb6cfdb
docs: update the migrating from BS article
2025-12-31 18:32:34 +02:00
55ff20c238
docs: update replacing AI article
2025-12-31 18:28:28 +02:00
7d5b907d6c
docs: adding more data to the adaptive feature page
2025-12-31 16:56:57 +02:00
17202d23a8
docs: Update the MCP server page
2025-12-31 16:32:21 +02:00
ad234165e9
docs: update the cli section
2025-12-31 03:00:26 +02:00
3b9a6d021e
fix: correct cli argument
2025-12-31 03:00:06 +02:00
d4ae6c49c0
docs: update the StealthyFetcher page
2025-12-31 00:22:33 +02:00
f6f0ed4b37
fix(async stealth): wait the coroutine of the bounding box
2025-12-31 00:19:53 +02:00
33c5633c58
test: correcting tests with the new arguments
2025-12-30 22:06:37 +02:00
Karim shoair