freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,405 Commits 1 Branch 0 Tags
a2285436e15952b8cf1cdafa9892210de84d4ac8
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair a2285436e1 docs: adding a new sponsor 2026-04-30 19:20:25 +03:00
Karim shoair be9f12f05a docs: adding a new sponsor 2026-04-30 16:24:44 +03:00
Karim shoair 984b90da65 docs: moving up a sponsor 2026-04-28 23:29:45 +03:00
Karim shoair 19d4bff427 docs: remove segment left by mistake 2026-04-24 22:00:09 +03:00
Karim shoair fb6ac7e167 docs: readd a sponsor 2026-04-24 21:58:15 +03:00
Karim shoair 54a080aae1 v0.4.7 (#251) 2026-04-17 23:09:43 +02:00
Karim shoair 2a1e9e22f2 docs(agent): update skill with the latest changes 2026-04-17 22:51:50 +02:00
Karim shoair 77b29a3ebe docs: update with the latest changes 2026-04-17 22:51:20 +02:00
Karim shoair 78e388f75c feat: add new mcp tool to screenshot pages 
Implements #244
 2026-04-17 22:07:44 +02:00
Karim shoair 9619c64fd8 fix(static): exclude block_ads from HTTP session request args (#249) 2026-04-17 20:47:52 +02:00
Karim shoair 65421f0a16 Merge branch 'dev' into fix/block-ads-unexpected-kwarg 2026-04-17 20:45:00 +02:00
Karim shoair 07bbb37bce docs: Add Brazilian Portuguese translation (#250) 2026-04-17 20:36:29 +02:00
Karim shoair 3565318100 Merge branch 'dev' into main 2026-04-17 20:33:26 +02:00
Karim shoair df0531d326 Merge branch 'main' into dev 2026-04-17 20:33:04 +02:00
Karim shoair 42663dcba1 Merge branch 'dev' into main 2026-04-17 20:31:43 +02:00
Karim shoair 689c79e7d3 fix: prevent FetcherSession state corruption and lazy session close crash (#245) 2026-04-17 20:27:48 +02:00
Karim shoair 6b51b977e7 Merge branch 'main' into main 2026-04-17 20:25:10 +02:00
Karim shoair 11e038768b docs: removing a sponsor 2026-04-17 19:24:06 +02:00
Karim shoair d60d7438a5 docs: removing a sponsor 2026-04-16 17:01:21 +02:00
Karim shoair 614d136f8c build: pump up version and deps 2026-04-15 20:44:38 +02:00
Karim shoair f8ec79833f feat(mcp): add optional session_id parameter to open_session (#243) 2026-04-15 02:37:11 +02:00
Karim shoair ced9a8d63c v0.4.6 (#242) 2026-04-13 15:36:33 +02:00
Karim shoair 65c54f3b91 fix: duplicate ID segments in full-path selector generation (#241) 2026-04-13 12:04:20 +02:00
Karim shoair 3c23436448 Merge branch 'dev' into fix/full-path-selector-duplicate-id 2026-04-13 12:03:15 +02:00
Karim shoair c2e66ab753 docs(agent): update skill with the latest changes 2026-04-13 04:39:01 +02:00
Karim shoair 158ff04c96 docs: add the new page_setup feature 2026-04-13 04:38:23 +02:00
Karim shoair 5e13d3ece6 feat(browsers): add a pre-navigation hook to allow page setup 
Solves #238
 2026-04-13 03:57:27 +02:00
Karim shoairandCocoon-Break d19e861e45 fix: correct Seconds type alias to accept float values 
The Seconds type was defined as `Annotated[int, float, Meta(ge=0)]` which per PEP 593 treated float as metadata, not a type. This caused passing float values like wait=1.5 to be rejected. Fixed by using Annotated[float, Meta(ge=0)] since int is a subtype of float.

Correct fix for #240

Co-Authored-By: Cocoon-Break <54054995+kuishou68@users.noreply.github.com>
 2026-04-13 01:08:53 +02:00
Karim shoair d4c36ea125 Merge branch 'main' into dev 2026-04-12 20:42:19 +02:00
Karim shoair 4279ca8cbc docs(agent): update skill with the latest changes 2026-04-12 18:47:26 +02:00
Karim shoair 887eeee4c2 build: pump up version and deps 2026-04-12 18:47:13 +02:00
Karim shoair 9a7fb0b426 docs: adding the new features and updating the translated pages 2026-04-12 18:20:56 +02:00
Karim shoair ad6fd52845 perf: force ad blocking on the MCP server and when the AI mode is activated on CLI 2026-04-12 18:11:02 +02:00
Karim shoair 131b729f22 docs: adding the new features 2026-04-12 18:07:38 +02:00
Karim shoair be28fe16ec feat(browsers): add new feature to enable DNS-over-HTTP to prevent DNS leaks 2026-04-12 18:03:03 +02:00
Karim shoair 0678ed1406 fix(shell): add missing parameters to the shell signature 2026-04-12 18:00:41 +02:00
Karim shoair d952db8ef8 feat(browsers): add a new feature to block ads 
This is working by aborting all requests to known ads domains.
 2026-04-12 17:59:00 +02:00
Karim shoair 2046527575 docs: adding a new sponsor 2026-04-08 13:30:00 +02:00
Karim shoair e5c5e7f50c docs: adding new sponsor 2026-04-08 13:24:43 +02:00
Karim shoair cb449afc81 v0.4.5 (#235) 2026-04-07 06:21:55 +02:00
Karim shoair 664e419751 docs(agent): update skill with the latest changes 2026-04-07 05:39:39 +02:00
Karim shoair 88d0459cd2 docs: adding the new development mode 2026-04-07 05:30:51 +02:00
Karim shoair fc6a0345a8 Merge branch 'main' into dev 2026-04-07 05:09:06 +02:00
Karim shoair aee6888dd5 docs: remove an old sponsor 2026-04-07 05:08:45 +02:00
Karim shoair d1baf1fc46 feat(spiders): add a development mode 2026-04-07 04:08:54 +02:00
Karim shoair fad9efdc35 fix(spider): save checkpoint before cancel_scope.cancel() on force-stop to prevent data loss (#230) 2026-04-05 22:07:56 +02:00
Karim shoair a61113fc29 docs(agent): update skill with the latest changes 2026-04-05 19:42:00 +02:00
Karim shoair d0a19a6fe1 build: pump version up 2026-04-05 19:41:10 +02:00
Karim shoair f756e519f3 docs: update pages with the new changes 2026-04-05 19:40:05 +02:00
Karim shoair e7f9adb40a feat(security): default follow_redirects to "safe" for SSRF protection 
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.

Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
 2026-04-05 18:41:50 +02:00
Karim shoair 065bf1c0e8 v0.4.4 (#229) 2026-04-05 05:37:08 +02:00
Karim shoair 087634ab67 docs: updated the translated files 2026-04-05 05:28:57 +02:00
Karim shoair 4c3ff2f0f1 docs: updating pages with the new feature 2026-04-05 05:27:07 +02:00
Karim shoair c54081ff1a docs(agent): update skill with the latest changes 2026-04-05 05:26:34 +02:00
Karim shoair 9383bec14e style(spider): removing excessive docstrings and unifying the style with the rest of the repo 2026-04-05 03:43:21 +02:00
Karim shoair 070338cf24 fix(spider): only allocate _domain_delays when robots_txt_obey is enabled 2026-04-05 02:36:15 +02:00
Karim shoair 911d3af632 build: pumping up protego minimum version 2026-04-05 02:33:15 +02:00
Karim shoair afaf68e7d5 fix(spider robots): removing dead code 2026-04-05 02:32:33 +02:00
Karim shoair ea2dd7866b refactor(spiders): Make Robots.txt compliance turned off by default 
Scrapy is turning it off by default
 2026-04-05 01:55:17 +02:00
Karim shoair 556a90f645 refactor(spider): prefetch robots.txt from start_urls only 
Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.
 2026-04-05 01:43:06 +02:00
Karim shoair ec487d37e8 fix(spider): Make delay in robots file don't affect user's concurrency settings 2026-04-05 01:39:05 +02:00
Karim shoair af83a11aa7 fix(spider robots): solve multiple issues with cache prefetch 2026-04-05 00:09:17 +02:00
Karim shoair 854daac794 style(spiders robots feat): Adjustments for maintainability 2026-04-04 21:06:35 +02:00
Karim shoair acb49b178c Merge branch 'main' into dev 2026-04-04 16:46:10 +02:00
Karim shoair 2b6251d485 feat(spiders): robots.txt compliance (#226) 2026-04-04 16:45:27 +02:00
Karim shoair fcf930c568 docs: Adding a new sponsor 2026-04-04 13:03:57 +02:00
Karim shoair eaf7ec2cc0 fix(requests): handle missing type hint returned by curl-cffi 2026-04-03 17:26:52 +02:00
Karim shoair 9dcfe47da7 build: pump up deps 2026-04-03 17:18:15 +02:00
Karim shoair cb93c15c5a Merge branch 'main' into dev 2026-04-02 18:14:51 +02:00
Karim shoair 636612e894 fix: proxy rotation page pool leak (#223) 2026-04-02 18:04:02 +02:00
Karim shoair 11f238bb63 docs: fix 2026-04-02 16:39:37 +02:00
Karim shoair f48418abbc docs: adding a new sponsor 2026-04-02 16:35:43 +02:00
Karim shoair 966e17a1dc build: pump version up 2026-04-01 17:38:54 +02:00
Karim shoairandYuval Dinodia f614651a97 fix(Proxy Rotation): Fix an MRO issue 
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.

Possible fix for #215

Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com>
 2026-04-01 17:36:35 +02:00
Karim shoair e173f813da v0.4.3 (#217) 2026-03-30 05:50:45 +02:00
Karim shoair b354be06c0 fix(agent): update skill zip file with the latest changes 2026-03-30 03:54:28 +02:00
Karim shoair a403156b2e docs: style adjustment 2026-03-30 03:53:56 +02:00
Karim shoair 1f1e475772 docs: add the browser path to docs 2026-03-30 03:37:38 +02:00
Karim shoair 8a4c5ffa3b feat(browsers): Add a new option to set browser path 
Solves #202
 2026-03-30 03:06:58 +02:00
Karim shoair 6598664f55 fix(agent): update skill zip file with the latest changes 2026-03-30 02:55:05 +02:00
Karim shoair 3ac7f76f9b docs(mcp): add section about prompt injection in docs and skill 2026-03-30 02:54:05 +02:00
Karim shoair 786093ff36 docs: update pages and skill with the new commandline option 2026-03-30 02:49:32 +02:00
Karim shoair 4efbffa1dc feat(cli): Add an option to make content safe/targets AI 2026-03-30 02:44:14 +02:00
Karim shoair 375951bd49 feat(mcp): Protect from Prompt Injection by removing hidden content 
Solves #214 as well
 2026-03-30 02:31:14 +02:00
Karim shoair bcd39d57c0 docs: update the agent skill with the new features 
Before I forget lol
 2026-03-29 23:46:25 +02:00
Karim shoair 7f552bed73 docs: Add docs for the new MCP tools 2026-03-29 23:38:54 +02:00
Karim shoair 8e89a73014 build(docs): Pump up Zensical version to the latest 2026-03-29 23:14:52 +02:00
Karim shoair 61cda587be docs: update pages with the XHR feature 2026-03-29 23:12:56 +02:00
Karim shoair 5c450a3b52 fix: improve type hints for the static checkers 2026-03-29 23:01:02 +02:00
Karim shoair 68f7c5c36f feat(browser sessions): Collect XHR requests done while loading the page 
Solves #159
 2026-03-29 22:53:54 +02:00
Karim shoair 0f6dcccf5f fix(mcp): remove unneeded code and fix type hint for mypy 2026-03-28 17:19:42 +02:00
Karim shoair c458ab65a2 feat(mcp): Add three new tools to control browser sessions 
Now you can open a browser, keep using it for other requests as you want, and close it when you want.
 2026-03-28 00:16:14 +02:00
Karim shoair a356dd2f2b refactor(mcp)!: Cleaning and unifying functions to async 
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
 2026-03-27 19:30:58 +02:00
Karim shoair 8e147db7f8 refactor(cli): Code cleaning for easier maintenance/adding new features 
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).

- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
 2026-03-27 18:02:33 +02:00
Karim shoair d47599f204 Merge branch 'main' into dev 2026-03-26 17:54:49 +02:00
Karim shoair f36ddb4b16 docs: update a sponsor url 2026-03-26 00:04:47 +02:00
Karim shoair 8e19f825a4 docs(skill): adjustment 2026-03-25 01:47:52 +02:00
Karim shoair c2c02dc78e docs: updating the agent skill for Clawhub validation 2026-03-25 01:38:11 +02:00
Karim shoair 422b4713ec build: pump up version 2026-03-25 00:19:58 +02:00
Karim shoair 9fee49c928 test: fixes to storage tests 2026-03-25 00:13:42 +02:00