freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,307 Commits 1 Branch 0 Tags
6598664f5506460d262519caca5e659a825f0b7d
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 6598664f55 fix(agent): update skill zip file with the latest changes 2026-03-30 02:55:05 +02:00
Karim shoair 3ac7f76f9b docs(mcp): add section about prompt injection in docs and skill 2026-03-30 02:54:05 +02:00
Karim shoair 786093ff36 docs: update pages and skill with the new commandline option 2026-03-30 02:49:32 +02:00
Karim shoair 4efbffa1dc feat(cli): Add an option to make content safe/targets AI 2026-03-30 02:44:14 +02:00
Karim shoair 375951bd49 feat(mcp): Protect from Prompt Injection by removing hidden content 
Solves #214 as well
 2026-03-30 02:31:14 +02:00
Karim shoair bcd39d57c0 docs: update the agent skill with the new features 
Before I forget lol
 2026-03-29 23:46:25 +02:00
Karim shoair 7f552bed73 docs: Add docs for the new MCP tools 2026-03-29 23:38:54 +02:00
Karim shoair 8e89a73014 build(docs): Pump up Zensical version to the latest 2026-03-29 23:14:52 +02:00
Karim shoair 61cda587be docs: update pages with the XHR feature 2026-03-29 23:12:56 +02:00
Karim shoair 5c450a3b52 fix: improve type hints for the static checkers 2026-03-29 23:01:02 +02:00
Karim shoair 68f7c5c36f feat(browser sessions): Collect XHR requests done while loading the page 
Solves #159
 2026-03-29 22:53:54 +02:00
Karim shoair 0f6dcccf5f fix(mcp): remove unneeded code and fix type hint for mypy 2026-03-28 17:19:42 +02:00
Karim shoair c458ab65a2 feat(mcp): Add three new tools to control browser sessions 
Now you can open a browser, keep using it for other requests as you want, and close it when you want.
 2026-03-28 00:16:14 +02:00
Karim shoair a356dd2f2b refactor(mcp)!: Cleaning and unifying functions to async 
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
 2026-03-27 19:30:58 +02:00
Karim shoair 8e147db7f8 refactor(cli): Code cleaning for easier maintenance/adding new features 
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).

- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
 2026-03-27 18:02:33 +02:00
Karim shoair d47599f204 Merge branch 'main' into dev 2026-03-26 17:54:49 +02:00
Karim shoair f36ddb4b16 docs: update a sponsor url 2026-03-26 00:04:47 +02:00
Karim shoair 8e19f825a4 docs(skill): adjustment 2026-03-25 01:47:52 +02:00
Karim shoair c2c02dc78e docs: updating the agent skill for Clawhub validation 2026-03-25 01:38:11 +02:00
Karim shoair 422b4713ec build: pump up version 2026-03-25 00:19:58 +02:00
Karim shoair 9fee49c928 test: fixes to storage tests 2026-03-25 00:13:42 +02:00
Karim shoair 3757cf6d77 test: add save/retrieve round-trip and core storage coverage (#193) 2026-03-25 00:11:42 +02:00
Karim shoair 30ffe76cfa Merge branch 'dev' into test/storage-core-coverage 2026-03-24 22:42:58 +02:00
Karim shoair 3b4cae6889 Merge branch 'main' into dev 2026-03-24 22:21:07 +02:00
Karim shoair 6635a4ad72 docs: adding a new sponsor 2026-03-24 17:13:24 +02:00
Karim shoair 718db42ff0 Merge branch 'main' into dev 2026-03-23 16:40:12 +02:00
Karim shoair 32cec404f6 docs: adjustment to ad plans 2026-03-23 15:00:26 +02:00
Karim shoair e230403232 docs: removing an old sponsor 2026-03-23 14:56:35 +02:00
Karim shoair df4b1ac7e3 Merge branch 'dev' into test/storage-core-coverage 2026-03-22 18:05:42 +02:00
Karim shoair d7c08073d4 Merge branch 'main' into dev 2026-03-22 18:05:26 +02:00
Karim shoair a96809a340 docs: removing an old sponsor 2026-03-22 16:48:26 +02:00
Karim shoair bc509e9fcd Merge branch 'main' into dev 2026-03-19 17:01:25 +02:00
Karim shoair 3ed59c2a84 fix(agent): update skill zip file with the latest fix 2026-03-19 17:01:05 +02:00
Karim shoair 2e78b3f8aa fix(skill): Fix broken markdown links in skill references (#204) 
Fixes #203
 2026-03-19 16:58:06 +02:00
Karim shoair 7ca02ebb80 test: add edge case tests for filter, iterancestors, and find_similar (#200) 2026-03-17 22:16:15 +02:00
Karim shoair bf9e2da329 Merge branch 'dev' into test/edge-cases-filter-ancestors-find-similar 2026-03-17 22:15:59 +02:00
Karim shoair 1dc0b7a1bd fix(fetchers/content): increase the default max number of retries and raise error on max retries 
Ref.: https://github.com/D4Vinci/Scrapling/pull/197#issuecomment-4077705587
 2026-03-17 22:14:02 +02:00
Karim shoair 17ea982162 fix(fetchers): add max retry limit to _get_page_content to prevent infinite loop (#197) 2026-03-17 22:09:05 +02:00
Karim shoair 8e4e59e13e Merge branch 'dev' into fix/page-content-infinite-loop 2026-03-17 22:08:47 +02:00
Karim shoair d9932f2308 fix(spider/fetcher): preserve HTTP method across retries in spider session (#201) 2026-03-17 22:06:01 +02:00
Karim shoair 4c07b294ae Merge branch 'dev' into fix/preserve-http-method-on-retry 2026-03-17 22:04:57 +02:00
Karim shoair 27f306259e Merge branch 'dev' into fix/page-content-infinite-loop 2026-03-17 21:55:26 +02:00
Karim shoair 136c389787 fix(Texthandler): Replace get_all with getall to match the Selector class 2026-03-17 21:53:17 +02:00
Karim shoair 980be18a2a test: add coverage for TextHandler regex paths and TextHandlers.re() (#194) 2026-03-17 21:50:05 +02:00
Karim shoair 5123590a15 Merge branch 'dev' into test/custom-types-coverage 2026-03-17 21:42:17 +02:00
Karim shoair cc6c0dbfb9 fix: adjust test to the _restore_from_checkpoint fix 2026-03-17 17:46:47 +02:00
Karim shoair 1fb7961d42 fix: replace bare raise with return False in _restore_from_checkpoint (#196) 2026-03-17 17:45:52 +02:00
Karim shoair bcd45e47bb Merge branch 'dev' into fix/checkpoint-restore-bare-raise 2026-03-17 17:36:11 +02:00
Karim shoair 5eefbc6c05 Merge branch 'dev' into test/custom-types-coverage 2026-03-17 17:34:12 +02:00
Karim shoair 6699213bdc Merge branch 'dev' into test/storage-core-coverage 2026-03-17 17:31:31 +02:00
Karim shoair 440da11259 Merge branch 'main' into dev 2026-03-17 17:28:38 +02:00
Karim shoair 7bfaf204b7 docs: CJK Latin spacing fixes for the JP README 2026-03-17 17:12:41 +02:00
Karim shoair 74abd54a0b docs: Add translations to the new sponsor 2026-03-17 17:00:32 +02:00
Karim shoair 3d60792cbc docs: update Zensical 2026-03-17 16:53:55 +02:00
Karim shoair 3b543a6bdd docs: add a new sponsor 2026-03-17 16:45:04 +02:00
Karim shoair 012e9f9f5e docs: images adjustments 2026-03-16 16:23:08 +02:00
Karim shoair f25bca75bf docs: fixing broken links 
Also, Fixes #199
 2026-03-16 15:38:23 +02:00
Karim shoair ca6c1f3c1e docs: CJK Latin spacing fixes for the Chinese README 2026-03-16 15:26:25 +02:00
Karim shoair 72a2c8de81 Merge branch 'main' into dev 2026-03-15 22:44:16 +02:00
Karim shoair eb32285c49 docs: add the sponsor translated versions 2026-03-15 22:25:50 +02:00
Karim shoair f286575f31 docs: adding a new sponsor 2026-03-15 22:17:12 +02:00
Karim shoair 1e7704864e Merge branch 'dev' into test/custom-types-coverage 2026-03-15 16:38:48 +02:00
Karim shoair e63e57e6dc test(ai): add _normalize_credentials edge case coverage (#192) 2026-03-15 16:25:57 +02:00
Karim shoair 656dbc7d12 Merge branch 'dev' into test/normalize-credentials-coverage 2026-03-15 16:24:59 +02:00
Karim shoair c7f0db912c docs: move up a new sponsor 2026-03-14 17:08:51 +02:00
Karim shoair 184ba6d0b0 docs: Add Korean (한국어) README translation (#187) 2026-03-11 16:49:52 +02:00
Karim shoair 3836712939 docs: adjustments 2026-03-11 06:13:30 +02:00
Karim shoair 50388bf675 docs: update sponsors page 2026-03-11 05:54:54 +02:00
Karim shoair d67a397ecc docs: add the sponsor to the translated README files 2026-03-11 05:47:32 +02:00
Karim shoair 569e317437 docs: add new sponsor 2026-03-11 05:42:18 +02:00
Karim shoair 79a03ae7f0 docs: move up a sponsor 2026-03-11 01:11:19 +02:00
Karim shoair 1aa150f090 docs: replace the downloads badge 2026-03-11 00:06:05 +02:00
Karim shoair 19c21c17ee v0.4.2 (#183) 2026-03-09 01:37:28 +02:00
Karim shoair 6ec40ad19d docs: remove a sponsor 2026-03-09 00:48:15 +02:00
Karim shoair f80743a97b style: remove unused import 2026-03-09 00:35:06 +02:00
Karim shoair 66e0e7af57 fix(agent): update the agent skill zip file with the new version 2026-03-09 00:31:57 +02:00
Karim shoair 73832f9dfe docs: update the website to reflect the google referer logic 2026-03-09 00:29:54 +02:00
Karim shoair d48ac78c79 test: update tests accordingly 2026-03-09 00:28:32 +02:00
Karim shoair 8e3bf89e6c fix(agent): update the agent skills docs and version 2026-03-09 00:27:10 +02:00
Karim shoair a929de6ca4 fix: update code and docstrings to remove the old google referer logic 2026-03-09 00:25:19 +02:00
Karim shoair b547e04207 ops: update playwright pinned version 2026-03-09 00:05:46 +02:00
Karim shoair 27409927a3 build: pump up deps and browsers versions 2026-03-08 23:57:10 +02:00
Karim shoair aa6f96eb67 fix: return bare Google URL in referer instead of search query (#179) 
Solves #172
 2026-03-08 23:20:43 +02:00
Karim shoair b0c8448f81 Merge branch 'dev' into fix/google-referrer-spoof 2026-03-08 19:01:20 +02:00
Karim shoair 11bc08fbaa fix: Selector.get_all_text() doesn't get all text #167 (#168) 
Fixes #167
 2026-03-08 19:01:04 +02:00
Karim shoair e045d0f73f Merge branch 'dev' into dev 2026-03-08 16:31:25 +02:00
Karim shoair 49db2b643b Merge branch 'main' into dev 2026-03-08 16:27:25 +02:00
Karim shoair b9b5bb0c80 docs: add badges for the agent skill and Clawhub 2026-03-08 06:56:50 +02:00
Karim shoair 68e6c3c3e8 docs: adjustments 2026-03-08 06:39:49 +02:00
Karim shoair d1a0b815f5 docs: update translation files with new sponsor 2026-03-08 06:27:04 +02:00
Karim shoair cb7e9b9ef3 docs: add new platinum sponsor 2026-03-08 06:16:48 +02:00
Karim shoair 25de97f534 docs: update the agent skill readme 2026-03-08 03:30:04 +02:00
Karim shoair 68da2af556 docs: update the agent skill readme 2026-03-08 03:21:23 +02:00
Karim shoair 54f2c96adf Merge branch 'main' into dev 2026-03-08 03:06:17 +02:00
Karim shoair f469fb9b8b feat: Upload the library agent skill 2026-03-08 03:06:01 +02:00
Karim shoair aa3f4b4b0e docs: remove previous versions pages 
The Change log page is sufficient.
 2026-03-08 02:47:53 +02:00
Karim shoair 27db107886 ops: exclude agent skill from workflow triggers 2026-03-08 02:46:54 +02:00
Karim shoair 98be98e875 docs: adjustments 2026-03-07 18:57:53 +02:00
Karim shoair cd73d901ab docs: adjustments 2026-03-07 18:54:29 +02:00
Karim shoair df7ded440c docs: table adjustments 2026-03-07 18:18:01 +02:00