freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,354 Commits 1 Branch 0 Tags
d1baf1fc465602cdbec780afe2a4cb24a1a1af30
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair d1baf1fc46 feat(spiders): add a development mode 2026-04-07 04:08:54 +02:00
Karim shoair fad9efdc35 fix(spider): save checkpoint before cancel_scope.cancel() on force-stop to prevent data loss (#230) 2026-04-05 22:07:56 +02:00
Karim shoair a61113fc29 docs(agent): update skill with the latest changes 2026-04-05 19:42:00 +02:00
Karim shoair d0a19a6fe1 build: pump version up 2026-04-05 19:41:10 +02:00
Karim shoair f756e519f3 docs: update pages with the new changes 2026-04-05 19:40:05 +02:00
Karim shoair e7f9adb40a feat(security): default follow_redirects to "safe" for SSRF protection 
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.

Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
 2026-04-05 18:41:50 +02:00
Karim shoair 065bf1c0e8 v0.4.4 (#229) 2026-04-05 05:37:08 +02:00
Karim shoair 087634ab67 docs: updated the translated files 2026-04-05 05:28:57 +02:00
Karim shoair 4c3ff2f0f1 docs: updating pages with the new feature 2026-04-05 05:27:07 +02:00
Karim shoair c54081ff1a docs(agent): update skill with the latest changes 2026-04-05 05:26:34 +02:00
Karim shoair 9383bec14e style(spider): removing excessive docstrings and unifying the style with the rest of the repo 2026-04-05 03:43:21 +02:00
Karim shoair 070338cf24 fix(spider): only allocate _domain_delays when robots_txt_obey is enabled 2026-04-05 02:36:15 +02:00
Karim shoair 911d3af632 build: pumping up protego minimum version 2026-04-05 02:33:15 +02:00
Karim shoair afaf68e7d5 fix(spider robots): removing dead code 2026-04-05 02:32:33 +02:00
Karim shoair ea2dd7866b refactor(spiders): Make Robots.txt compliance turned off by default 
Scrapy is turning it off by default
 2026-04-05 01:55:17 +02:00
Karim shoair 556a90f645 refactor(spider): prefetch robots.txt from start_urls only 
Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.
 2026-04-05 01:43:06 +02:00
Karim shoair ec487d37e8 fix(spider): Make delay in robots file don't affect user's concurrency settings 2026-04-05 01:39:05 +02:00
Karim shoair af83a11aa7 fix(spider robots): solve multiple issues with cache prefetch 2026-04-05 00:09:17 +02:00
Karim shoair 854daac794 style(spiders robots feat): Adjustments for maintainability 2026-04-04 21:06:35 +02:00
Karim shoair acb49b178c Merge branch 'main' into dev 2026-04-04 16:46:10 +02:00
Karim shoair 2b6251d485 feat(spiders): robots.txt compliance (#226) 2026-04-04 16:45:27 +02:00
Karim shoair fcf930c568 docs: Adding a new sponsor 2026-04-04 13:03:57 +02:00
Karim shoair eaf7ec2cc0 fix(requests): handle missing type hint returned by curl-cffi 2026-04-03 17:26:52 +02:00
Karim shoair 9dcfe47da7 build: pump up deps 2026-04-03 17:18:15 +02:00
Karim shoair cb93c15c5a Merge branch 'main' into dev 2026-04-02 18:14:51 +02:00
Karim shoair 636612e894 fix: proxy rotation page pool leak (#223) 2026-04-02 18:04:02 +02:00
Karim shoair 11f238bb63 docs: fix 2026-04-02 16:39:37 +02:00
Karim shoair f48418abbc docs: adding a new sponsor 2026-04-02 16:35:43 +02:00
Karim shoair 966e17a1dc build: pump version up 2026-04-01 17:38:54 +02:00
Karim shoairandYuval Dinodia f614651a97 fix(Proxy Rotation): Fix an MRO issue 
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.

Possible fix for #215

Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com>
 2026-04-01 17:36:35 +02:00
Karim shoair e173f813da v0.4.3 (#217) 2026-03-30 05:50:45 +02:00
Karim shoair b354be06c0 fix(agent): update skill zip file with the latest changes 2026-03-30 03:54:28 +02:00
Karim shoair a403156b2e docs: style adjustment 2026-03-30 03:53:56 +02:00
Karim shoair 1f1e475772 docs: add the browser path to docs 2026-03-30 03:37:38 +02:00
Karim shoair 8a4c5ffa3b feat(browsers): Add a new option to set browser path 
Solves #202
 2026-03-30 03:06:58 +02:00
Karim shoair 6598664f55 fix(agent): update skill zip file with the latest changes 2026-03-30 02:55:05 +02:00
Karim shoair 3ac7f76f9b docs(mcp): add section about prompt injection in docs and skill 2026-03-30 02:54:05 +02:00
Karim shoair 786093ff36 docs: update pages and skill with the new commandline option 2026-03-30 02:49:32 +02:00
Karim shoair 4efbffa1dc feat(cli): Add an option to make content safe/targets AI 2026-03-30 02:44:14 +02:00
Karim shoair 375951bd49 feat(mcp): Protect from Prompt Injection by removing hidden content 
Solves #214 as well
 2026-03-30 02:31:14 +02:00
Karim shoair bcd39d57c0 docs: update the agent skill with the new features 
Before I forget lol
 2026-03-29 23:46:25 +02:00
Karim shoair 7f552bed73 docs: Add docs for the new MCP tools 2026-03-29 23:38:54 +02:00
Karim shoair 8e89a73014 build(docs): Pump up Zensical version to the latest 2026-03-29 23:14:52 +02:00
Karim shoair 61cda587be docs: update pages with the XHR feature 2026-03-29 23:12:56 +02:00
Karim shoair 5c450a3b52 fix: improve type hints for the static checkers 2026-03-29 23:01:02 +02:00
Karim shoair 68f7c5c36f feat(browser sessions): Collect XHR requests done while loading the page 
Solves #159
 2026-03-29 22:53:54 +02:00
Karim shoair 0f6dcccf5f fix(mcp): remove unneeded code and fix type hint for mypy 2026-03-28 17:19:42 +02:00
Karim shoair c458ab65a2 feat(mcp): Add three new tools to control browser sessions 
Now you can open a browser, keep using it for other requests as you want, and close it when you want.
 2026-03-28 00:16:14 +02:00
Karim shoair a356dd2f2b refactor(mcp)!: Cleaning and unifying functions to async 
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
 2026-03-27 19:30:58 +02:00
Karim shoair 8e147db7f8 refactor(cli): Code cleaning for easier maintenance/adding new features 
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).

- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
 2026-03-27 18:02:33 +02:00
Karim shoair d47599f204 Merge branch 'main' into dev 2026-03-26 17:54:49 +02:00
Karim shoair f36ddb4b16 docs: update a sponsor url 2026-03-26 00:04:47 +02:00
Karim shoair 8e19f825a4 docs(skill): adjustment 2026-03-25 01:47:52 +02:00
Karim shoair c2c02dc78e docs: updating the agent skill for Clawhub validation 2026-03-25 01:38:11 +02:00
Karim shoair 422b4713ec build: pump up version 2026-03-25 00:19:58 +02:00
Karim shoair 9fee49c928 test: fixes to storage tests 2026-03-25 00:13:42 +02:00
Karim shoair 3757cf6d77 test: add save/retrieve round-trip and core storage coverage (#193) 2026-03-25 00:11:42 +02:00
Karim shoair 30ffe76cfa Merge branch 'dev' into test/storage-core-coverage 2026-03-24 22:42:58 +02:00
Karim shoair 3b4cae6889 Merge branch 'main' into dev 2026-03-24 22:21:07 +02:00
Karim shoair 6635a4ad72 docs: adding a new sponsor 2026-03-24 17:13:24 +02:00
Karim shoair 718db42ff0 Merge branch 'main' into dev 2026-03-23 16:40:12 +02:00
Karim shoair 32cec404f6 docs: adjustment to ad plans 2026-03-23 15:00:26 +02:00
Karim shoair e230403232 docs: removing an old sponsor 2026-03-23 14:56:35 +02:00
Karim shoair df4b1ac7e3 Merge branch 'dev' into test/storage-core-coverage 2026-03-22 18:05:42 +02:00
Karim shoair d7c08073d4 Merge branch 'main' into dev 2026-03-22 18:05:26 +02:00
Karim shoair a96809a340 docs: removing an old sponsor 2026-03-22 16:48:26 +02:00
Karim shoair bc509e9fcd Merge branch 'main' into dev 2026-03-19 17:01:25 +02:00
Karim shoair 3ed59c2a84 fix(agent): update skill zip file with the latest fix 2026-03-19 17:01:05 +02:00
Karim shoair 2e78b3f8aa fix(skill): Fix broken markdown links in skill references (#204) 
Fixes #203
 2026-03-19 16:58:06 +02:00
Karim shoair 7ca02ebb80 test: add edge case tests for filter, iterancestors, and find_similar (#200) 2026-03-17 22:16:15 +02:00
Karim shoair bf9e2da329 Merge branch 'dev' into test/edge-cases-filter-ancestors-find-similar 2026-03-17 22:15:59 +02:00
Karim shoair 1dc0b7a1bd fix(fetchers/content): increase the default max number of retries and raise error on max retries 
Ref.: https://github.com/D4Vinci/Scrapling/pull/197#issuecomment-4077705587
 2026-03-17 22:14:02 +02:00
Karim shoair 17ea982162 fix(fetchers): add max retry limit to _get_page_content to prevent infinite loop (#197) 2026-03-17 22:09:05 +02:00
Karim shoair 8e4e59e13e Merge branch 'dev' into fix/page-content-infinite-loop 2026-03-17 22:08:47 +02:00
Karim shoair d9932f2308 fix(spider/fetcher): preserve HTTP method across retries in spider session (#201) 2026-03-17 22:06:01 +02:00
Karim shoair 4c07b294ae Merge branch 'dev' into fix/preserve-http-method-on-retry 2026-03-17 22:04:57 +02:00
Karim shoair 27f306259e Merge branch 'dev' into fix/page-content-infinite-loop 2026-03-17 21:55:26 +02:00
Karim shoair 136c389787 fix(Texthandler): Replace get_all with getall to match the Selector class 2026-03-17 21:53:17 +02:00
Karim shoair 980be18a2a test: add coverage for TextHandler regex paths and TextHandlers.re() (#194) 2026-03-17 21:50:05 +02:00
Karim shoair 5123590a15 Merge branch 'dev' into test/custom-types-coverage 2026-03-17 21:42:17 +02:00
Karim shoair cc6c0dbfb9 fix: adjust test to the _restore_from_checkpoint fix 2026-03-17 17:46:47 +02:00
Karim shoair 1fb7961d42 fix: replace bare raise with return False in _restore_from_checkpoint (#196) 2026-03-17 17:45:52 +02:00
Karim shoair bcd45e47bb Merge branch 'dev' into fix/checkpoint-restore-bare-raise 2026-03-17 17:36:11 +02:00
Karim shoair 5eefbc6c05 Merge branch 'dev' into test/custom-types-coverage 2026-03-17 17:34:12 +02:00
Karim shoair 6699213bdc Merge branch 'dev' into test/storage-core-coverage 2026-03-17 17:31:31 +02:00
Karim shoair 440da11259 Merge branch 'main' into dev 2026-03-17 17:28:38 +02:00
Karim shoair 7bfaf204b7 docs: CJK Latin spacing fixes for the JP README 2026-03-17 17:12:41 +02:00
Karim shoair 74abd54a0b docs: Add translations to the new sponsor 2026-03-17 17:00:32 +02:00
Karim shoair 3d60792cbc docs: update Zensical 2026-03-17 16:53:55 +02:00
Karim shoair 3b543a6bdd docs: add a new sponsor 2026-03-17 16:45:04 +02:00
Karim shoair 012e9f9f5e docs: images adjustments 2026-03-16 16:23:08 +02:00
Karim shoair f25bca75bf docs: fixing broken links 
Also, Fixes #199
 2026-03-16 15:38:23 +02:00
Karim shoair ca6c1f3c1e docs: CJK Latin spacing fixes for the Chinese README 2026-03-16 15:26:25 +02:00
Karim shoair 72a2c8de81 Merge branch 'main' into dev 2026-03-15 22:44:16 +02:00
Karim shoair eb32285c49 docs: add the sponsor translated versions 2026-03-15 22:25:50 +02:00
Karim shoair f286575f31 docs: adding a new sponsor 2026-03-15 22:17:12 +02:00
Karim shoair 1e7704864e Merge branch 'dev' into test/custom-types-coverage 2026-03-15 16:38:48 +02:00
Karim shoair e63e57e6dc test(ai): add _normalize_credentials edge case coverage (#192) 2026-03-15 16:25:57 +02:00
Karim shoair 656dbc7d12 Merge branch 'dev' into test/normalize-credentials-coverage 2026-03-15 16:24:59 +02:00
Karim shoair c7f0db912c docs: move up a new sponsor 2026-03-14 17:08:51 +02:00