07129ce4b1
feat(deps): move protego to fetchers optional dependency
...
protego is only used by the spider framework for robots.txt compliance.
Moving it from core dependencies to the optional 'fetchers' group reduces
the dependency footprint for users who don't need the spider framework.
for pyproject.toml file
2026-04-03 17:51:00 +02:00
a2bf00c995
Merge branch 'D4Vinci:dev' into dev
2026-04-03 17:38:28 +02:00
eaf7ec2cc0
fix(requests): handle missing type hint returned by curl-cffi
2026-04-03 17:26:52 +02:00
9dcfe47da7
build: pump up deps
2026-04-03 17:18:15 +02:00
132f33c846
test(spiders): add comprehensive test suite for robots.txt compliance
2026-04-03 15:08:34 +02:00
5c40c6a853
feat(spiders): integrate robots.txt compliance into the crawl engine
2026-04-03 15:08:33 +02:00
0bbe62fc7f
feat(spiders): implement RobotsTxtManager with concurrent fetch deduplication
2026-04-03 15:08:33 +02:00
1d15349e07
feat(deps): add protego for robots.txt parsing and fix pyright type error in static.py
2026-04-03 15:08:33 +02:00
cb93c15c5a
Merge branch 'main' into dev
2026-04-02 18:14:51 +02:00
636612e894
fix: proxy rotation page pool leak ( #223 )
2026-04-02 18:04:02 +02:00
6c6aabeb73
fix: proxy rotation page pool leak
2026-04-02 11:57:17 -04:00
11f238bb63
docs: fix
2026-04-02 16:39:37 +02:00
f48418abbc
docs: adding a new sponsor
2026-04-02 16:35:43 +02:00
966e17a1dc
build: pump version up
2026-04-01 17:38:54 +02:00
f614651a97
fix(Proxy Rotation): Fix an MRO issue
...
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.
Possible fix for #215
Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com >
2026-04-01 17:36:35 +02:00
e173f813da
v0.4.3 ( #217 )
2026-03-30 05:50:45 +02:00
b354be06c0
fix(agent): update skill zip file with the latest changes
2026-03-30 03:54:28 +02:00
a403156b2e
docs: style adjustment
2026-03-30 03:53:56 +02:00
1f1e475772
docs: add the browser path to docs
2026-03-30 03:37:38 +02:00
8a4c5ffa3b
feat(browsers): Add a new option to set browser path
...
Solves #202
2026-03-30 03:06:58 +02:00
6598664f55
fix(agent): update skill zip file with the latest changes
2026-03-30 02:55:05 +02:00
3ac7f76f9b
docs(mcp): add section about prompt injection in docs and skill
2026-03-30 02:54:05 +02:00
786093ff36
docs: update pages and skill with the new commandline option
2026-03-30 02:49:32 +02:00
4efbffa1dc
feat(cli): Add an option to make content safe/targets AI
2026-03-30 02:44:14 +02:00
375951bd49
feat(mcp): Protect from Prompt Injection by removing hidden content
...
Solves #214 as well
2026-03-30 02:31:14 +02:00
bcd39d57c0
docs: update the agent skill with the new features
...
Before I forget lol
2026-03-29 23:46:25 +02:00
7f552bed73
docs: Add docs for the new MCP tools
2026-03-29 23:38:54 +02:00
8e89a73014
build(docs): Pump up Zensical version to the latest
2026-03-29 23:14:52 +02:00
61cda587be
docs: update pages with the XHR feature
2026-03-29 23:12:56 +02:00
5c450a3b52
fix: improve type hints for the static checkers
2026-03-29 23:01:02 +02:00
68f7c5c36f
feat(browser sessions): Collect XHR requests done while loading the page
...
Solves #159
2026-03-29 22:53:54 +02:00
0f6dcccf5f
fix(mcp): remove unneeded code and fix type hint for mypy
2026-03-28 17:19:42 +02:00
c458ab65a2
feat(mcp): Add three new tools to control browser sessions
...
Now you can open a browser, keep using it for other requests as you want, and close it when you want.
2026-03-28 00:16:14 +02:00
a356dd2f2b
refactor(mcp)!: Cleaning and unifying functions to async
...
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
2026-03-27 19:30:58 +02:00
8e147db7f8
refactor(cli): Code cleaning for easier maintenance/adding new features
...
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).
- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
2026-03-27 18:02:33 +02:00
d47599f204
Merge branch 'main' into dev
2026-03-26 17:54:49 +02:00
f36ddb4b16
docs: update a sponsor url
2026-03-26 00:04:47 +02:00
8e19f825a4
docs(skill): adjustment
2026-03-25 01:47:52 +02:00
c2c02dc78e
docs: updating the agent skill for Clawhub validation
2026-03-25 01:38:11 +02:00
422b4713ec
build: pump up version
2026-03-25 00:19:58 +02:00
9fee49c928
test: fixes to storage tests
2026-03-25 00:13:42 +02:00
3757cf6d77
test: add save/retrieve round-trip and core storage coverage ( #193 )
2026-03-25 00:11:42 +02:00
30ffe76cfa
Merge branch 'dev' into test/storage-core-coverage
2026-03-24 22:42:58 +02:00
3b4cae6889
Merge branch 'main' into dev
2026-03-24 22:21:07 +02:00
6635a4ad72
docs: adding a new sponsor
2026-03-24 17:13:24 +02:00
718db42ff0
Merge branch 'main' into dev
2026-03-23 16:40:12 +02:00
32cec404f6
docs: adjustment to ad plans
2026-03-23 15:00:26 +02:00
e230403232
docs: removing an old sponsor
2026-03-23 14:56:35 +02:00
df4b1ac7e3
Merge branch 'dev' into test/storage-core-coverage
2026-03-22 18:05:42 +02:00
d7c08073d4
Merge branch 'main' into dev
2026-03-22 18:05:26 +02:00
Abdullah