freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,373 Commits 1 Branch 0 Tags
158ff04c968a81d6a04f1c06cbdd62572c2408f3
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 158ff04c96 docs: add the new page_setup feature 2026-04-13 04:38:23 +02:00
Karim shoair 5e13d3ece6 feat(browsers): add a pre-navigation hook to allow page setup 
Solves #238
 2026-04-13 03:57:27 +02:00
Karim shoairandCocoon-Break d19e861e45 fix: correct Seconds type alias to accept float values 
The Seconds type was defined as `Annotated[int, float, Meta(ge=0)]` which per PEP 593 treated float as metadata, not a type. This caused passing float values like wait=1.5 to be rejected. Fixed by using Annotated[float, Meta(ge=0)] since int is a subtype of float.

Correct fix for #240

Co-Authored-By: Cocoon-Break <54054995+kuishou68@users.noreply.github.com>
 2026-04-13 01:08:53 +02:00
Karim shoair d4c36ea125 Merge branch 'main' into dev 2026-04-12 20:42:19 +02:00
Karim shoair 4279ca8cbc docs(agent): update skill with the latest changes 2026-04-12 18:47:26 +02:00
Karim shoair 887eeee4c2 build: pump up version and deps 2026-04-12 18:47:13 +02:00
Karim shoair 9a7fb0b426 docs: adding the new features and updating the translated pages 2026-04-12 18:20:56 +02:00
Karim shoair ad6fd52845 perf: force ad blocking on the MCP server and when the AI mode is activated on CLI 2026-04-12 18:11:02 +02:00
Karim shoair 131b729f22 docs: adding the new features 2026-04-12 18:07:38 +02:00
Karim shoair be28fe16ec feat(browsers): add new feature to enable DNS-over-HTTP to prevent DNS leaks 2026-04-12 18:03:03 +02:00
Karim shoair 0678ed1406 fix(shell): add missing parameters to the shell signature 2026-04-12 18:00:41 +02:00
Karim shoair d952db8ef8 feat(browsers): add a new feature to block ads 
This is working by aborting all requests to known ads domains.
 2026-04-12 17:59:00 +02:00
Karim shoair 2046527575 docs: adding a new sponsor 2026-04-08 13:30:00 +02:00
Karim shoair e5c5e7f50c docs: adding new sponsor 2026-04-08 13:24:43 +02:00
Karim shoair cb449afc81 v0.4.5 (#235) 2026-04-07 06:21:55 +02:00
Karim shoair 664e419751 docs(agent): update skill with the latest changes 2026-04-07 05:39:39 +02:00
Karim shoair 88d0459cd2 docs: adding the new development mode 2026-04-07 05:30:51 +02:00
Karim shoair fc6a0345a8 Merge branch 'main' into dev 2026-04-07 05:09:06 +02:00
Karim shoair aee6888dd5 docs: remove an old sponsor 2026-04-07 05:08:45 +02:00
Karim shoair d1baf1fc46 feat(spiders): add a development mode 2026-04-07 04:08:54 +02:00
Karim shoair fad9efdc35 fix(spider): save checkpoint before cancel_scope.cancel() on force-stop to prevent data loss (#230) 2026-04-05 22:07:56 +02:00
Karim shoair a61113fc29 docs(agent): update skill with the latest changes 2026-04-05 19:42:00 +02:00
Karim shoair d0a19a6fe1 build: pump version up 2026-04-05 19:41:10 +02:00
Karim shoair f756e519f3 docs: update pages with the new changes 2026-04-05 19:40:05 +02:00
Karim shoair e7f9adb40a feat(security): default follow_redirects to "safe" for SSRF protection 
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.

Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
 2026-04-05 18:41:50 +02:00
Karim shoair 065bf1c0e8 v0.4.4 (#229) 2026-04-05 05:37:08 +02:00
Karim shoair 087634ab67 docs: updated the translated files 2026-04-05 05:28:57 +02:00
Karim shoair 4c3ff2f0f1 docs: updating pages with the new feature 2026-04-05 05:27:07 +02:00
Karim shoair c54081ff1a docs(agent): update skill with the latest changes 2026-04-05 05:26:34 +02:00
Karim shoair 9383bec14e style(spider): removing excessive docstrings and unifying the style with the rest of the repo 2026-04-05 03:43:21 +02:00
Karim shoair 070338cf24 fix(spider): only allocate _domain_delays when robots_txt_obey is enabled 2026-04-05 02:36:15 +02:00
Karim shoair 911d3af632 build: pumping up protego minimum version 2026-04-05 02:33:15 +02:00
Karim shoair afaf68e7d5 fix(spider robots): removing dead code 2026-04-05 02:32:33 +02:00
Karim shoair ea2dd7866b refactor(spiders): Make Robots.txt compliance turned off by default 
Scrapy is turning it off by default
 2026-04-05 01:55:17 +02:00
Karim shoair 556a90f645 refactor(spider): prefetch robots.txt from start_urls only 
Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.
 2026-04-05 01:43:06 +02:00
Karim shoair ec487d37e8 fix(spider): Make delay in robots file don't affect user's concurrency settings 2026-04-05 01:39:05 +02:00
Karim shoair af83a11aa7 fix(spider robots): solve multiple issues with cache prefetch 2026-04-05 00:09:17 +02:00
Karim shoair 854daac794 style(spiders robots feat): Adjustments for maintainability 2026-04-04 21:06:35 +02:00
Karim shoair acb49b178c Merge branch 'main' into dev 2026-04-04 16:46:10 +02:00
Karim shoair 2b6251d485 feat(spiders): robots.txt compliance (#226) 2026-04-04 16:45:27 +02:00
Karim shoair fcf930c568 docs: Adding a new sponsor 2026-04-04 13:03:57 +02:00
Karim shoair eaf7ec2cc0 fix(requests): handle missing type hint returned by curl-cffi 2026-04-03 17:26:52 +02:00
Karim shoair 9dcfe47da7 build: pump up deps 2026-04-03 17:18:15 +02:00
Karim shoair cb93c15c5a Merge branch 'main' into dev 2026-04-02 18:14:51 +02:00
Karim shoair 636612e894 fix: proxy rotation page pool leak (#223) 2026-04-02 18:04:02 +02:00
Karim shoair 11f238bb63 docs: fix 2026-04-02 16:39:37 +02:00
Karim shoair f48418abbc docs: adding a new sponsor 2026-04-02 16:35:43 +02:00
Karim shoair 966e17a1dc build: pump version up 2026-04-01 17:38:54 +02:00
Karim shoairandYuval Dinodia f614651a97 fix(Proxy Rotation): Fix an MRO issue 
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.

Possible fix for #215

Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com>
 2026-04-01 17:36:35 +02:00
Karim shoair e173f813da v0.4.3 (#217) 2026-03-30 05:50:45 +02:00
Karim shoair b354be06c0 fix(agent): update skill zip file with the latest changes 2026-03-30 03:54:28 +02:00
Karim shoair a403156b2e docs: style adjustment 2026-03-30 03:53:56 +02:00
Karim shoair 1f1e475772 docs: add the browser path to docs 2026-03-30 03:37:38 +02:00
Karim shoair 8a4c5ffa3b feat(browsers): Add a new option to set browser path 
Solves #202
 2026-03-30 03:06:58 +02:00
Karim shoair 6598664f55 fix(agent): update skill zip file with the latest changes 2026-03-30 02:55:05 +02:00
Karim shoair 3ac7f76f9b docs(mcp): add section about prompt injection in docs and skill 2026-03-30 02:54:05 +02:00
Karim shoair 786093ff36 docs: update pages and skill with the new commandline option 2026-03-30 02:49:32 +02:00
Karim shoair 4efbffa1dc feat(cli): Add an option to make content safe/targets AI 2026-03-30 02:44:14 +02:00
Karim shoair 375951bd49 feat(mcp): Protect from Prompt Injection by removing hidden content 
Solves #214 as well
 2026-03-30 02:31:14 +02:00
Karim shoair bcd39d57c0 docs: update the agent skill with the new features 
Before I forget lol
 2026-03-29 23:46:25 +02:00
Karim shoair 7f552bed73 docs: Add docs for the new MCP tools 2026-03-29 23:38:54 +02:00
Karim shoair 8e89a73014 build(docs): Pump up Zensical version to the latest 2026-03-29 23:14:52 +02:00
Karim shoair 61cda587be docs: update pages with the XHR feature 2026-03-29 23:12:56 +02:00
Karim shoair 5c450a3b52 fix: improve type hints for the static checkers 2026-03-29 23:01:02 +02:00
Karim shoair 68f7c5c36f feat(browser sessions): Collect XHR requests done while loading the page 
Solves #159
 2026-03-29 22:53:54 +02:00
Karim shoair 0f6dcccf5f fix(mcp): remove unneeded code and fix type hint for mypy 2026-03-28 17:19:42 +02:00
Karim shoair c458ab65a2 feat(mcp): Add three new tools to control browser sessions 
Now you can open a browser, keep using it for other requests as you want, and close it when you want.
 2026-03-28 00:16:14 +02:00
Karim shoair a356dd2f2b refactor(mcp)!: Cleaning and unifying functions to async 
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
 2026-03-27 19:30:58 +02:00
Karim shoair 8e147db7f8 refactor(cli): Code cleaning for easier maintenance/adding new features 
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).

- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
 2026-03-27 18:02:33 +02:00
Karim shoair d47599f204 Merge branch 'main' into dev 2026-03-26 17:54:49 +02:00
Karim shoair f36ddb4b16 docs: update a sponsor url 2026-03-26 00:04:47 +02:00
Karim shoair 8e19f825a4 docs(skill): adjustment 2026-03-25 01:47:52 +02:00
Karim shoair c2c02dc78e docs: updating the agent skill for Clawhub validation 2026-03-25 01:38:11 +02:00
Karim shoair 422b4713ec build: pump up version 2026-03-25 00:19:58 +02:00
Karim shoair 9fee49c928 test: fixes to storage tests 2026-03-25 00:13:42 +02:00
Karim shoair 3757cf6d77 test: add save/retrieve round-trip and core storage coverage (#193) 2026-03-25 00:11:42 +02:00
Karim shoair 30ffe76cfa Merge branch 'dev' into test/storage-core-coverage 2026-03-24 22:42:58 +02:00
Karim shoair 3b4cae6889 Merge branch 'main' into dev 2026-03-24 22:21:07 +02:00
Karim shoair 6635a4ad72 docs: adding a new sponsor 2026-03-24 17:13:24 +02:00
Karim shoair 718db42ff0 Merge branch 'main' into dev 2026-03-23 16:40:12 +02:00
Karim shoair 32cec404f6 docs: adjustment to ad plans 2026-03-23 15:00:26 +02:00
Karim shoair e230403232 docs: removing an old sponsor 2026-03-23 14:56:35 +02:00
Karim shoair df4b1ac7e3 Merge branch 'dev' into test/storage-core-coverage 2026-03-22 18:05:42 +02:00
Karim shoair d7c08073d4 Merge branch 'main' into dev 2026-03-22 18:05:26 +02:00
Karim shoair a96809a340 docs: removing an old sponsor 2026-03-22 16:48:26 +02:00
Karim shoair bc509e9fcd Merge branch 'main' into dev 2026-03-19 17:01:25 +02:00
Karim shoair 3ed59c2a84 fix(agent): update skill zip file with the latest fix 2026-03-19 17:01:05 +02:00
Karim shoair 2e78b3f8aa fix(skill): Fix broken markdown links in skill references (#204) 
Fixes #203
 2026-03-19 16:58:06 +02:00
Karim shoair 7ca02ebb80 test: add edge case tests for filter, iterancestors, and find_similar (#200) 2026-03-17 22:16:15 +02:00
Karim shoair bf9e2da329 Merge branch 'dev' into test/edge-cases-filter-ancestors-find-similar 2026-03-17 22:15:59 +02:00
Karim shoair 1dc0b7a1bd fix(fetchers/content): increase the default max number of retries and raise error on max retries 
Ref.: https://github.com/D4Vinci/Scrapling/pull/197#issuecomment-4077705587
 2026-03-17 22:14:02 +02:00
Karim shoair 17ea982162 fix(fetchers): add max retry limit to _get_page_content to prevent infinite loop (#197) 2026-03-17 22:09:05 +02:00
Karim shoair 8e4e59e13e Merge branch 'dev' into fix/page-content-infinite-loop 2026-03-17 22:08:47 +02:00
Karim shoair d9932f2308 fix(spider/fetcher): preserve HTTP method across retries in spider session (#201) 2026-03-17 22:06:01 +02:00
Karim shoair 4c07b294ae Merge branch 'dev' into fix/preserve-http-method-on-retry 2026-03-17 22:04:57 +02:00
Karim shoair 27f306259e Merge branch 'dev' into fix/page-content-infinite-loop 2026-03-17 21:55:26 +02:00
Karim shoair 136c389787 fix(Texthandler): Replace get_all with getall to match the Selector class 2026-03-17 21:53:17 +02:00
Karim shoair 980be18a2a test: add coverage for TextHandler regex paths and TextHandlers.re() (#194) 2026-03-17 21:50:05 +02:00
Karim shoair 5123590a15 Merge branch 'dev' into test/custom-types-coverage 2026-03-17 21:42:17 +02:00
Karim shoair cc6c0dbfb9 fix: adjust test to the _restore_from_checkpoint fix 2026-03-17 17:46:47 +02:00