freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,470 Commits 1 Branch 0 Tags
b3408c6f3c70eb8672c4178a55918ec567e2dad1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair b3408c6f3c docs: warn that the default installation doesn't include fetchers 
The quickstart examples import from `scrapling.fetchers`, which raises `ModuleNotFoundError` on a bare install since those dependencies live in the fetchers extra. Make the consequence explicit in the installation section across the docs and all README translations.

Closes #343
 2026-06-07 16:44:18 +03:00
Karim shoair c732c4ef31 docs: fix broken docs link in CONTRIBUTING file (#320) 2026-06-07 16:25:01 +03:00
Karim shoair c84764d627 Merge branch 'dev' into docs/fix-contributing-docs-link 2026-06-07 16:24:20 +03:00
Karim shoair 4ced6e4ac2 build: pump up version, browsers and deps 2026-06-07 16:16:23 +03:00
Karim shoair 1fd1013806 fix: prevent IndexError in adaptive relocation when auto_save is enabled (#340) 2026-06-07 15:54:34 +03:00
Karim shoair 3d96baf284 Merge branch 'dev' into fix/adaptive-autosave-indexerror 2026-06-07 15:53:47 +03:00
Karim shoair 9c0c857245 fix(parser): use max of both attribute counts in similarity scoring denominator 
Candidates with fewer attributes than the original got inflated scores because the denominator counted candidate attributes only, while the extra-attributes penalty direction worked as intended. Using `max()` on both counts fixes the inflation while keeping the penalty.

Closes #322
 2026-06-07 15:51:10 +03:00
Karim shoair f7d1e338e0 fix(spiders): use os.replace for atomic checkpoint/cache writes on Windows (#344) 2026-06-07 15:49:33 +03:00
Karim shoair 4f0a593b7d Merge branch 'dev' into fix/atomic-checkpoint-cache-writes 2026-06-07 15:49:01 +03:00
Karim shoair 2b2abb3810 feat: add --version flag to the CLI (#303) 2026-06-07 15:37:41 +03:00
Karim shoair 507f7f626d Merge branch 'dev' into feat/cli-version-flag 2026-06-07 15:37:23 +03:00
Karim shoair 5d839c5995 Merge branch 'main' into dev 2026-06-07 15:37:12 +03:00
Karim shoair fa5f1477d0 Merge branch 'dev' into feat/cli-version-flag 2026-06-07 15:36:58 +03:00
Karim shoair b9bc17b461 docs: removing a sponsor 2026-06-07 15:11:06 +03:00
Karim shoair 141b389cdb docs: updating the support page title 2026-06-04 17:47:40 +03:00
Karim shoair 3a2dd29255 docs: update a sponsor 2026-06-03 20:43:40 +03:00
Karim shoair 53ef32c723 docs: fix Docker image name in remaining examples (#315) 2026-06-02 02:19:27 +03:00
Karim shoair 6efd50dd17 Merge branch 'dev' into docs/fix-docker-image-name-sweep 2026-06-02 02:17:22 +03:00
Karim shoair 464fba2435 docs: remove a sponsor 2026-06-02 01:39:06 +03:00
Karim shoair a6a57eee44 docs: update a sponsor 2026-05-30 15:37:08 +03:00
Karim shoair 961f659ac8 docs: remove old sponsors 2026-05-30 15:30:05 +03:00
Karim shoair dbd48ffd60 docs: add a new sponsor 2026-05-28 00:53:54 +03:00
Karim shoair b31dc50a06 docs: remove an old sponsor 2026-05-27 04:10:38 +03:00
Karim shoair ed008efcf7 docs: updating a sponsor image 2026-05-18 19:33:49 +03:00
Karim shoair 735af8f221 docs: fix Docker image name in MCP example (#283) 2026-05-16 03:42:13 +03:00
Karim shoair b5d7d3d103 Merge branch 'main' into fix/mcp-docker-image-name 2026-05-16 03:41:48 +03:00
Karim shoair 6380ef0f26 docs: adding a new sponsor 2026-05-11 15:41:27 +03:00
Karim shoair 9ee35014fe v0.4.8 (#277) 2026-05-11 04:59:43 +03:00
Karim shoair 66d8917013 ops: update tests deps 2026-05-11 04:37:18 +03:00
Karim shoair 149d5f0923 Merge branch 'main' into dev 2026-05-11 04:28:34 +03:00
Karim shoair 55931897d6 docs(agent): update skill zip file 2026-05-11 03:34:17 +03:00
Karim shoair 34651abc6b tests: remove old code and update the rest 2026-05-11 03:30:59 +03:00
Karim shoair ebd7e0971c build: update deps and browser useragents 2026-05-11 03:18:26 +03:00
Karim shoair c0ed8949f6 docs: update zensical version 2026-05-11 03:00:11 +03:00
Karim shoair 6a641912cd docs: add docs for the new features 2026-05-11 02:58:00 +03:00
Karim shoair 8a8b2d14ba test: add tests accordingly 2026-05-11 02:33:43 +03:00
Karim shoair f7da15771b feat(spiders): Add SitemapSpider 2026-05-11 02:33:37 +03:00
Karim shoair f093d0cf12 feat(spiders): Add CrawlSpider and CrawlRule 2026-05-10 21:13:16 +03:00
Karim shoair 18e9121135 feat(spiders): Add pure URL discovery primitive 2026-05-10 21:12:36 +03:00
Karim shoair 940036fa8f docs: removing a sponsor 2026-05-10 15:49:54 +03:00
Karim shoair e5d7ed27fb Merge branch 'main' into dev 2026-05-09 03:43:58 +03:00
Karim shoair 07e0c1c4b9 docs: adding a new sponsor 2026-05-06 17:37:48 +03:00
Karim shoair 37c17e8666 docs: adding a new sponsor 2026-05-05 15:38:13 +03:00
Karim shoair 333b6de0b5 fix(parser): change the default threshold and add warning 2026-05-02 19:58:53 +03:00
Karim shoair ea7b4bddd3 Merge branch 'main' into dev 2026-04-30 19:21:46 +03:00
Karim shoair a2285436e1 docs: adding a new sponsor 2026-04-30 19:20:25 +03:00
Karim shoair 90a853d0fd fix(request fp): hash request kwargs and headers correctly (#255) 2026-04-30 18:54:16 +03:00
Karim shoair 809d4782ba Merge branch 'dev' into fix/request-fingerprint 2026-04-30 18:45:25 +03:00
Karim shoair f305580f1d Merge branch 'main' into dev 2026-04-30 16:25:35 +03:00
Karim shoair be9f12f05a docs: adding a new sponsor 2026-04-30 16:24:44 +03:00
Karim shoair 756a59516b Merge branch 'main' into dev 2026-04-28 23:31:47 +03:00
Karim shoair 984b90da65 docs: moving up a sponsor 2026-04-28 23:29:45 +03:00
Karim shoair 835e7ca8c3 docs: update old examples 2026-04-26 02:46:13 +03:00
Karim shoair 82837ee11f Merge branch 'main' into dev 2026-04-25 20:34:39 +03:00
Karim shoair fa41ca6fa9 docs: Updating deps and allow code copy 2026-04-25 20:34:33 +03:00
Karim shoair 19d4bff427 docs: remove segment left by mistake 2026-04-24 22:00:09 +03:00
Karim shoair fb6ac7e167 docs: readd a sponsor 2026-04-24 21:58:15 +03:00
Karim shoair b6f0f2a7f2 build: pump up version and deps 2026-04-22 17:01:11 +02:00
Karim shoair 9af644ef84 docs: improving the code copy-paste experience and use less tokens for the agent skill 2026-04-22 16:52:59 +02:00
Karim shoair b626b4d585 test: adding new tests for the configure function 2026-04-22 15:35:15 +02:00
Karim shoair 35032d6f5f fix: solving a bug with using configure on Fetcher 2026-04-22 15:31:18 +02:00
Karim shoair 54a080aae1 v0.4.7 (#251) 2026-04-17 23:09:43 +02:00
Karim shoair 2a1e9e22f2 docs(agent): update skill with the latest changes 2026-04-17 22:51:50 +02:00
Karim shoair 77b29a3ebe docs: update with the latest changes 2026-04-17 22:51:20 +02:00
Karim shoair 78e388f75c feat: add new mcp tool to screenshot pages 
Implements #244
 2026-04-17 22:07:44 +02:00
Karim shoair 9619c64fd8 fix(static): exclude block_ads from HTTP session request args (#249) 2026-04-17 20:47:52 +02:00
Karim shoair 65421f0a16 Merge branch 'dev' into fix/block-ads-unexpected-kwarg 2026-04-17 20:45:00 +02:00
Karim shoair 07bbb37bce docs: Add Brazilian Portuguese translation (#250) 2026-04-17 20:36:29 +02:00
Karim shoair 3565318100 Merge branch 'dev' into main 2026-04-17 20:33:26 +02:00
Karim shoair df0531d326 Merge branch 'main' into dev 2026-04-17 20:33:04 +02:00
Karim shoair 42663dcba1 Merge branch 'dev' into main 2026-04-17 20:31:43 +02:00
Karim shoair 689c79e7d3 fix: prevent FetcherSession state corruption and lazy session close crash (#245) 2026-04-17 20:27:48 +02:00
Karim shoair 6b51b977e7 Merge branch 'main' into main 2026-04-17 20:25:10 +02:00
Karim shoair 11e038768b docs: removing a sponsor 2026-04-17 19:24:06 +02:00
Karim shoair d60d7438a5 docs: removing a sponsor 2026-04-16 17:01:21 +02:00
Karim shoair 614d136f8c build: pump up version and deps 2026-04-15 20:44:38 +02:00
Karim shoair f8ec79833f feat(mcp): add optional session_id parameter to open_session (#243) 2026-04-15 02:37:11 +02:00
Karim shoair ced9a8d63c v0.4.6 (#242) 2026-04-13 15:36:33 +02:00
Karim shoair 65c54f3b91 fix: duplicate ID segments in full-path selector generation (#241) 2026-04-13 12:04:20 +02:00
Karim shoair 3c23436448 Merge branch 'dev' into fix/full-path-selector-duplicate-id 2026-04-13 12:03:15 +02:00
Karim shoair c2e66ab753 docs(agent): update skill with the latest changes 2026-04-13 04:39:01 +02:00
Karim shoair 158ff04c96 docs: add the new page_setup feature 2026-04-13 04:38:23 +02:00
Karim shoair 5e13d3ece6 feat(browsers): add a pre-navigation hook to allow page setup 
Solves #238
 2026-04-13 03:57:27 +02:00
Karim shoairandCocoon-Break d19e861e45 fix: correct Seconds type alias to accept float values 
The Seconds type was defined as `Annotated[int, float, Meta(ge=0)]` which per PEP 593 treated float as metadata, not a type. This caused passing float values like wait=1.5 to be rejected. Fixed by using Annotated[float, Meta(ge=0)] since int is a subtype of float.

Correct fix for #240

Co-Authored-By: Cocoon-Break <54054995+kuishou68@users.noreply.github.com>
 2026-04-13 01:08:53 +02:00
Karim shoair d4c36ea125 Merge branch 'main' into dev 2026-04-12 20:42:19 +02:00
Karim shoair 4279ca8cbc docs(agent): update skill with the latest changes 2026-04-12 18:47:26 +02:00
Karim shoair 887eeee4c2 build: pump up version and deps 2026-04-12 18:47:13 +02:00
Karim shoair 9a7fb0b426 docs: adding the new features and updating the translated pages 2026-04-12 18:20:56 +02:00
Karim shoair ad6fd52845 perf: force ad blocking on the MCP server and when the AI mode is activated on CLI 2026-04-12 18:11:02 +02:00
Karim shoair 131b729f22 docs: adding the new features 2026-04-12 18:07:38 +02:00
Karim shoair be28fe16ec feat(browsers): add new feature to enable DNS-over-HTTP to prevent DNS leaks 2026-04-12 18:03:03 +02:00
Karim shoair 0678ed1406 fix(shell): add missing parameters to the shell signature 2026-04-12 18:00:41 +02:00
Karim shoair d952db8ef8 feat(browsers): add a new feature to block ads 
This is working by aborting all requests to known ads domains.
 2026-04-12 17:59:00 +02:00
Karim shoair 2046527575 docs: adding a new sponsor 2026-04-08 13:30:00 +02:00
Karim shoair e5c5e7f50c docs: adding new sponsor 2026-04-08 13:24:43 +02:00
Karim shoair cb449afc81 v0.4.5 (#235) 2026-04-07 06:21:55 +02:00
Karim shoair 664e419751 docs(agent): update skill with the latest changes 2026-04-07 05:39:39 +02:00
Karim shoair 88d0459cd2 docs: adding the new development mode 2026-04-07 05:30:51 +02:00
Karim shoair fc6a0345a8 Merge branch 'main' into dev 2026-04-07 05:09:06 +02:00
Karim shoair aee6888dd5 docs: remove an old sponsor 2026-04-07 05:08:45 +02:00