059a708b6d
feat(spiders system): a prototype of the new spiders system
...
- A modern spider design that uses AnyIO and asyncio, yet it's very similar to Scrapy spiders API because it's the easiest design for users, and to make it easier for new users.
- Spiders can have multiple sessions per crawl, and users decide which session to use with each request.
- A scheduler system that uses heapq logic.
- The user can set the number of concurrent requests for a spider globally or per domain.
- The user can set a download delay to control the speed of the spider more.
- There's a global function that can be overridden to handle errors for all requests. (Similar to errback in scrapy).
- There's a spider argument to set the allowed domains for the spider to stay in.
- Each spider has a very detailed crawl stats that can be accessed right away from the code after the crawl finishes. Same case with scraped items.
- The whole spider as written as any other script and you just run it. No command-line arguments, and no need to run it from the terminal through the library like other known alternatives.
- Each spider has its own logger that forces sessions to use it.
- Each spider has functions to override that run before start and after close.
- There's a spider argument to set the logging level and another one to make the spider write to a log file.
- This is only the start. A lot more features are coming in the way.
2026-01-11 16:53:18 +02:00
dd5c30c521
feat(response): Option to set the referer as the previous response url.
2026-01-11 03:24:30 +02:00
1443a469a3
fix: handle a bug with response follow referer flow argument
2026-01-11 03:22:51 +02:00
aa4817f349
feat: Change logger to be flexible enough for the spiders classes
2026-01-11 03:22:20 +02:00
c5bc82deb8
fix(browsers): solve a bug with setting referer on the request level
2026-01-11 03:21:03 +02:00
8e36962177
ops: add local Claude files to gitignore
2026-01-10 20:41:23 +02:00
0634f5796c
feat(spiders): Add follow function to the response
2026-01-10 20:39:43 +02:00
f80e8828f7
refactor: internal API changes
2026-01-08 01:53:48 +02:00
3a360acced
build: pump version up temporary
2026-01-08 01:37:47 +02:00
08f3e6efad
fix: add missing line
2026-01-08 01:01:00 +02:00
6553d90e0f
ops: update workflows to run for this branch
2026-01-08 00:53:29 +02:00
9eb5f68fcb
test: update tests accordingly
2026-01-08 00:52:35 +02:00
6a73f9dcd7
refactor: internal API changes to be easily used as indicators for spiders
2026-01-08 00:52:06 +02:00
de79fe80bb
v0.3.14
2026-01-03 22:29:20 +02:00
2f87507afe
test: update database test to suite windows
2026-01-03 22:02:03 +02:00
f99baf65f7
test: remove the check for incognito
2026-01-03 01:50:46 +02:00
c8a2456b81
fix(StealthyFetcher): disable incognito mode to solve #123
2026-01-03 01:50:02 +02:00
30a77de2be
build: pump version up and browserforge version
2026-01-03 01:48:34 +02:00
6ad2ecb9f2
v0.3.13
2026-01-01 22:07:00 +02:00
b5fdb71b52
docs: improving the API reference section
2026-01-01 21:07:20 +02:00
4947152be2
docs: change docstring style for some methods
2026-01-01 21:06:35 +02:00
9742b239be
docs: fix for rtd build
2026-01-01 17:42:47 +02:00
e275509ece
docs: fix for rtd build
...
test
2026-01-01 17:05:10 +02:00
185a9c594f
docs: fix for rtd build
2026-01-01 17:01:24 +02:00
c6e4482837
docs: Improve docs website
2026-01-01 16:56:57 +02:00
cb5051ca56
docs: update contributing page
2026-01-01 03:18:15 +02:00
4d478eb84c
docs: update the donate page
2026-01-01 03:16:10 +02:00
2d5e2f41c4
docs: update dev articles
2026-01-01 03:14:12 +02:00
0a3bb6cfdb
docs: update the migrating from BS article
2025-12-31 18:32:34 +02:00
55ff20c238
docs: update replacing AI article
2025-12-31 18:28:28 +02:00
7d5b907d6c
docs: adding more data to the adaptive feature page
2025-12-31 16:56:57 +02:00
17202d23a8
docs: Update the MCP server page
2025-12-31 16:32:21 +02:00
ad234165e9
docs: update the cli section
2025-12-31 03:00:26 +02:00
3b9a6d021e
fix: correct cli argument
2025-12-31 03:00:06 +02:00
d4ae6c49c0
docs: update the StealthyFetcher page
2025-12-31 00:22:33 +02:00
f6f0ed4b37
fix(async stealth): wait the coroutine of the bounding box
2025-12-31 00:19:53 +02:00
33c5633c58
test: correcting tests with the new arguments
2025-12-30 22:06:37 +02:00
0c7e26ae5c
fix(fingerprints): do not create google referer for IPs
2025-12-30 21:58:09 +02:00
a2b5628a73
docs: update DynamicFetcher page
2025-12-29 22:02:01 +02:00
67e8f4d515
docs: update DynamicFetcher page
2025-12-29 20:48:45 +02:00
e1cac0f69f
docs: update requests page
2025-12-29 20:48:32 +02:00
bc420da19b
docs: update deps
2025-12-29 20:39:13 +02:00
ee13af9516
build: update deps
...
and docstrings
2025-12-29 16:08:51 +02:00
6c25dccb05
docs: updating the Fetcher Basics and the HTTP requests pages
2025-12-28 21:57:21 +02:00
5e3845f060
docs: shorten docstrings for some functions
2025-12-28 21:56:28 +02:00
b1cd5c5a82
test: fix old assert test
2025-12-28 00:16:37 +02:00
9bfe744840
fix: remove forgotten unused code
2025-12-28 00:14:47 +02:00
1b838eabd0
fix: update the cookies type hint for the mcp server
2025-12-28 00:08:44 +02:00
a4ad6b5fce
test: remove old test
2025-12-28 00:03:09 +02:00
09bbd416b8
fix: remove old proxy logic
...
When I was caching it with the rest of the data
2025-12-28 00:02:50 +02:00
Karim shoair