freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1,450 Commits 1 Branch 0 Tags
b9bc17b46109da28be7c7b89ef398904ec15b56b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair b9bc17b461 docs: removing a sponsor 2026-06-07 15:11:06 +03:00
Karim shoair 141b389cdb docs: updating the support page title 2026-06-04 17:47:40 +03:00
Karim shoair 3a2dd29255 docs: update a sponsor 2026-06-03 20:43:40 +03:00
Karim shoair 464fba2435 docs: remove a sponsor 2026-06-02 01:39:06 +03:00
Karim shoair a6a57eee44 docs: update a sponsor 2026-05-30 15:37:08 +03:00
Karim shoair 961f659ac8 docs: remove old sponsors 2026-05-30 15:30:05 +03:00
Karim shoair dbd48ffd60 docs: add a new sponsor 2026-05-28 00:53:54 +03:00
Karim shoair b31dc50a06 docs: remove an old sponsor 2026-05-27 04:10:38 +03:00
Karim shoair ed008efcf7 docs: updating a sponsor image 2026-05-18 19:33:49 +03:00
Karim shoair 735af8f221 docs: fix Docker image name in MCP example (#283) 2026-05-16 03:42:13 +03:00
Karim shoair b5d7d3d103 Merge branch 'main' into fix/mcp-docker-image-name 2026-05-16 03:41:48 +03:00
Karim shoair 6380ef0f26 docs: adding a new sponsor 2026-05-11 15:41:27 +03:00
Karim shoair 9ee35014fe v0.4.8 (#277) 2026-05-11 04:59:43 +03:00
Karim shoair 66d8917013 ops: update tests deps 2026-05-11 04:37:18 +03:00
Karim shoair 149d5f0923 Merge branch 'main' into dev 2026-05-11 04:28:34 +03:00
Karim shoair 55931897d6 docs(agent): update skill zip file 2026-05-11 03:34:17 +03:00
Karim shoair 34651abc6b tests: remove old code and update the rest 2026-05-11 03:30:59 +03:00
Karim shoair ebd7e0971c build: update deps and browser useragents 2026-05-11 03:18:26 +03:00
Karim shoair c0ed8949f6 docs: update zensical version 2026-05-11 03:00:11 +03:00
Karim shoair 6a641912cd docs: add docs for the new features 2026-05-11 02:58:00 +03:00
Karim shoair 8a8b2d14ba test: add tests accordingly 2026-05-11 02:33:43 +03:00
Karim shoair f7da15771b feat(spiders): Add SitemapSpider 2026-05-11 02:33:37 +03:00
Karim shoair f093d0cf12 feat(spiders): Add CrawlSpider and CrawlRule 2026-05-10 21:13:16 +03:00
Karim shoair 18e9121135 feat(spiders): Add pure URL discovery primitive 2026-05-10 21:12:36 +03:00
Karim shoair 940036fa8f docs: removing a sponsor 2026-05-10 15:49:54 +03:00
Karim shoair e5d7ed27fb Merge branch 'main' into dev 2026-05-09 03:43:58 +03:00
Karim shoair 07e0c1c4b9 docs: adding a new sponsor 2026-05-06 17:37:48 +03:00
Karim shoair 37c17e8666 docs: adding a new sponsor 2026-05-05 15:38:13 +03:00
Karim shoair 333b6de0b5 fix(parser): change the default threshold and add warning 2026-05-02 19:58:53 +03:00
Karim shoair ea7b4bddd3 Merge branch 'main' into dev 2026-04-30 19:21:46 +03:00
Karim shoair a2285436e1 docs: adding a new sponsor 2026-04-30 19:20:25 +03:00
Karim shoair 90a853d0fd fix(request fp): hash request kwargs and headers correctly (#255) 2026-04-30 18:54:16 +03:00
Karim shoair 809d4782ba Merge branch 'dev' into fix/request-fingerprint 2026-04-30 18:45:25 +03:00
Karim shoair f305580f1d Merge branch 'main' into dev 2026-04-30 16:25:35 +03:00
Karim shoair be9f12f05a docs: adding a new sponsor 2026-04-30 16:24:44 +03:00
Karim shoair 756a59516b Merge branch 'main' into dev 2026-04-28 23:31:47 +03:00
Karim shoair 984b90da65 docs: moving up a sponsor 2026-04-28 23:29:45 +03:00
Karim shoair 835e7ca8c3 docs: update old examples 2026-04-26 02:46:13 +03:00
Karim shoair 82837ee11f Merge branch 'main' into dev 2026-04-25 20:34:39 +03:00
Karim shoair fa41ca6fa9 docs: Updating deps and allow code copy 2026-04-25 20:34:33 +03:00
Karim shoair 19d4bff427 docs: remove segment left by mistake 2026-04-24 22:00:09 +03:00
Karim shoair fb6ac7e167 docs: readd a sponsor 2026-04-24 21:58:15 +03:00
Karim shoair b6f0f2a7f2 build: pump up version and deps 2026-04-22 17:01:11 +02:00
Karim shoair 9af644ef84 docs: improving the code copy-paste experience and use less tokens for the agent skill 2026-04-22 16:52:59 +02:00
Karim shoair b626b4d585 test: adding new tests for the configure function 2026-04-22 15:35:15 +02:00
Karim shoair 35032d6f5f fix: solving a bug with using configure on Fetcher 2026-04-22 15:31:18 +02:00
Karim shoair 54a080aae1 v0.4.7 (#251) 2026-04-17 23:09:43 +02:00
Karim shoair 2a1e9e22f2 docs(agent): update skill with the latest changes 2026-04-17 22:51:50 +02:00
Karim shoair 77b29a3ebe docs: update with the latest changes 2026-04-17 22:51:20 +02:00
Karim shoair 78e388f75c feat: add new mcp tool to screenshot pages 
Implements #244
 2026-04-17 22:07:44 +02:00
Karim shoair 9619c64fd8 fix(static): exclude block_ads from HTTP session request args (#249) 2026-04-17 20:47:52 +02:00
Karim shoair 65421f0a16 Merge branch 'dev' into fix/block-ads-unexpected-kwarg 2026-04-17 20:45:00 +02:00
Karim shoair 07bbb37bce docs: Add Brazilian Portuguese translation (#250) 2026-04-17 20:36:29 +02:00
Karim shoair 3565318100 Merge branch 'dev' into main 2026-04-17 20:33:26 +02:00
Karim shoair df0531d326 Merge branch 'main' into dev 2026-04-17 20:33:04 +02:00
Karim shoair 42663dcba1 Merge branch 'dev' into main 2026-04-17 20:31:43 +02:00
Karim shoair 689c79e7d3 fix: prevent FetcherSession state corruption and lazy session close crash (#245) 2026-04-17 20:27:48 +02:00
Karim shoair 6b51b977e7 Merge branch 'main' into main 2026-04-17 20:25:10 +02:00
Karim shoair 11e038768b docs: removing a sponsor 2026-04-17 19:24:06 +02:00
Karim shoair d60d7438a5 docs: removing a sponsor 2026-04-16 17:01:21 +02:00
Karim shoair 614d136f8c build: pump up version and deps 2026-04-15 20:44:38 +02:00
Karim shoair f8ec79833f feat(mcp): add optional session_id parameter to open_session (#243) 2026-04-15 02:37:11 +02:00
Karim shoair ced9a8d63c v0.4.6 (#242) 2026-04-13 15:36:33 +02:00
Karim shoair 65c54f3b91 fix: duplicate ID segments in full-path selector generation (#241) 2026-04-13 12:04:20 +02:00
Karim shoair 3c23436448 Merge branch 'dev' into fix/full-path-selector-duplicate-id 2026-04-13 12:03:15 +02:00
Karim shoair c2e66ab753 docs(agent): update skill with the latest changes 2026-04-13 04:39:01 +02:00
Karim shoair 158ff04c96 docs: add the new page_setup feature 2026-04-13 04:38:23 +02:00
Karim shoair 5e13d3ece6 feat(browsers): add a pre-navigation hook to allow page setup 
Solves #238
 2026-04-13 03:57:27 +02:00
Karim shoairandCocoon-Break d19e861e45 fix: correct Seconds type alias to accept float values 
The Seconds type was defined as `Annotated[int, float, Meta(ge=0)]` which per PEP 593 treated float as metadata, not a type. This caused passing float values like wait=1.5 to be rejected. Fixed by using Annotated[float, Meta(ge=0)] since int is a subtype of float.

Correct fix for #240

Co-Authored-By: Cocoon-Break <54054995+kuishou68@users.noreply.github.com>
 2026-04-13 01:08:53 +02:00
Karim shoair d4c36ea125 Merge branch 'main' into dev 2026-04-12 20:42:19 +02:00
Karim shoair 4279ca8cbc docs(agent): update skill with the latest changes 2026-04-12 18:47:26 +02:00
Karim shoair 887eeee4c2 build: pump up version and deps 2026-04-12 18:47:13 +02:00
Karim shoair 9a7fb0b426 docs: adding the new features and updating the translated pages 2026-04-12 18:20:56 +02:00
Karim shoair ad6fd52845 perf: force ad blocking on the MCP server and when the AI mode is activated on CLI 2026-04-12 18:11:02 +02:00
Karim shoair 131b729f22 docs: adding the new features 2026-04-12 18:07:38 +02:00
Karim shoair be28fe16ec feat(browsers): add new feature to enable DNS-over-HTTP to prevent DNS leaks 2026-04-12 18:03:03 +02:00
Karim shoair 0678ed1406 fix(shell): add missing parameters to the shell signature 2026-04-12 18:00:41 +02:00
Karim shoair d952db8ef8 feat(browsers): add a new feature to block ads 
This is working by aborting all requests to known ads domains.
 2026-04-12 17:59:00 +02:00
Karim shoair 2046527575 docs: adding a new sponsor 2026-04-08 13:30:00 +02:00
Karim shoair e5c5e7f50c docs: adding new sponsor 2026-04-08 13:24:43 +02:00
Karim shoair cb449afc81 v0.4.5 (#235) 2026-04-07 06:21:55 +02:00
Karim shoair 664e419751 docs(agent): update skill with the latest changes 2026-04-07 05:39:39 +02:00
Karim shoair 88d0459cd2 docs: adding the new development mode 2026-04-07 05:30:51 +02:00
Karim shoair fc6a0345a8 Merge branch 'main' into dev 2026-04-07 05:09:06 +02:00
Karim shoair aee6888dd5 docs: remove an old sponsor 2026-04-07 05:08:45 +02:00
Karim shoair d1baf1fc46 feat(spiders): add a development mode 2026-04-07 04:08:54 +02:00
Karim shoair fad9efdc35 fix(spider): save checkpoint before cancel_scope.cancel() on force-stop to prevent data loss (#230) 2026-04-05 22:07:56 +02:00
Karim shoair a61113fc29 docs(agent): update skill with the latest changes 2026-04-05 19:42:00 +02:00
Karim shoair d0a19a6fe1 build: pump version up 2026-04-05 19:41:10 +02:00
Karim shoair f756e519f3 docs: update pages with the new changes 2026-04-05 19:40:05 +02:00
Karim shoair e7f9adb40a feat(security): default follow_redirects to "safe" for SSRF protection 
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.

Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
 2026-04-05 18:41:50 +02:00
Karim shoair 065bf1c0e8 v0.4.4 (#229) 2026-04-05 05:37:08 +02:00
Karim shoair 087634ab67 docs: updated the translated files 2026-04-05 05:28:57 +02:00
Karim shoair 4c3ff2f0f1 docs: updating pages with the new feature 2026-04-05 05:27:07 +02:00
Karim shoair c54081ff1a docs(agent): update skill with the latest changes 2026-04-05 05:26:34 +02:00
Karim shoair 9383bec14e style(spider): removing excessive docstrings and unifying the style with the rest of the repo 2026-04-05 03:43:21 +02:00
Karim shoair 070338cf24 fix(spider): only allocate _domain_delays when robots_txt_obey is enabled 2026-04-05 02:36:15 +02:00
Karim shoair 911d3af632 build: pumping up protego minimum version 2026-04-05 02:33:15 +02:00
Karim shoair afaf68e7d5 fix(spider robots): removing dead code 2026-04-05 02:32:33 +02:00
Karim shoair ea2dd7866b refactor(spiders): Make Robots.txt compliance turned off by default 
Scrapy is turning it off by default
 2026-04-05 01:55:17 +02:00