freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
826 Commits 1 Branch 0 Tags
4fea8c2e59ea533d62427110e337744843138bf0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 4fea8c2e59 docs: Update docs accordingly 2025-10-12 04:51:11 +03:00
Karim shoair c34e10faed feat(fetchers): Add argument to control the user data directory 2025-10-12 04:48:59 +03:00
Karim shoair 8c995c936d fix(CF solver): further improve the detection 2025-10-12 04:15:16 +03:00
Karim shoair 073c4910d4 fix(fetchers): More accurate collection of the final response 
This will make the `response.body` return the raw response for text/json responses as well
 2025-10-12 04:12:18 +03:00
Karim shoair 6bfa266c90 fix(camoufox): Fix a typo in the async solver 2025-10-12 03:07:18 +03:00
Karim shoair 1ff09e5b47 Merge branch 'main' into dev 2025-10-06 17:19:14 +03:00
Karim shoair 458ad5b36a docs: remove a sponsor 2025-10-06 17:18:49 +03:00
Karim shoair 0dffa3bbba docs: removing unnecessary docstrings 
They are internal functions, not user-facing ones
 2025-10-06 14:24:16 +03:00
Karim shoair f9303ab275 test: update session tests 2025-10-06 06:53:24 +03:00
Karim shoair 9cae97b177 fix(requests): Fix multiple configuration issues 2025-10-06 06:53:09 +03:00
Karim shoair 390916ba45 tests: change test accordingly 2025-10-06 05:00:58 +03:00
Karim shoair e53ce3f749 refactor: Making the requests logic better and cleaner 
Also, solve some bugs along the way and a cleaner fix for #97
 2025-10-06 05:00:41 +03:00
Karim shoair cc45ded0cf ops: update pre-commit hooks 2025-10-05 04:12:18 +03:00
Karim shoair c01fefadea ops: change docker workflow trigger 2025-10-05 04:11:14 +03:00
Karim shoair f4ce433ff3 build: pump up the version and deps 2025-10-05 04:08:19 +03:00
Karim shoair b80f866434 docs: update an outdated part 2025-10-05 04:04:09 +03:00
Karim shoair debe03256b refactor: Making all the codebase acceptable by PyRight 
Also fixes #97
 2025-10-05 04:03:39 +03:00
Karim shoair e149c715dd feat(CF solver): better implementation to help with all custom implementations of turnstile 2025-10-02 20:05:39 +03:00
Karim shoair db1179790e docs: correcting docker installation instruction 2025-10-01 07:30:49 +03:00
Karim shoair 653fbb0594 ops: fix docker image name 2025-10-01 07:19:48 +03:00
Karim shoair e8b0e72512 v0.3.6 (#95) 2025-10-01 06:40:04 +03:00
Karim shoair a7e29a1662 docs: Update introduction 2025-10-01 06:26:19 +03:00
Karim shoair 5127d386c9 style(Fetcher): correcting type hints and annotation 2025-10-01 06:18:23 +03:00
Karim shoair 849913ad77 style: Correcting return type annotation for Fetcher/AsyncFetcher (#93) 2025-10-01 04:17:59 +03:00
Karim shoair 7aa083053b test: change tests accordingly 2025-10-01 03:50:57 +03:00
Karim shoair eedfa855ab fix: Fixes for the type checking in the main init file 
and a bit of cleaning
 2025-10-01 03:50:39 +03:00
Karim shoair 4135dd86b8 refactor: Restructure the fetchers code to not use more memory than needed 
Also fixes: #92
 2025-10-01 03:48:43 +03:00
Karim shoair ec3809ce40 docs: fix installation section on the website 2025-09-29 17:50:45 +03:00
Karim shoair f6c122b87f style: Removing dead code/docstrings 2025-09-29 05:09:23 +03:00
Karim shoair 8ad7cd6343 docs: Update all pages/docstrings to reflect recent changes 2025-09-29 03:57:40 +03:00
Karim shoair 0cf3bcf3aa ops: improve the layering of the docker image 2025-09-29 02:53:06 +03:00
Karim shoair 840aef6884 build: Pump up version and deps 2025-09-28 22:04:09 +03:00
Karim shoair 2bad72b77a fix: correct http option type in the mcp server 2025-09-28 21:19:51 +03:00
Karim shoair 6f0ce428aa tests: update tests for mcp 2025-09-28 20:14:17 +03:00
Karim shoair 74f20d2a0e refactor: better implementation for the mcp mode 2025-09-28 20:14:04 +03:00
Karim shoair 0b729176bc ops: add workflow to build and push docker images 2025-09-28 04:56:01 +03:00
Karim shoair 8a867fc68e feat: Add docker support 2025-09-28 04:55:00 +03:00
Karim shoair 4a661b4875 feat: Make mcp able to use http transport 2025-09-28 04:54:35 +03:00
Karim shoair 292208e22d build: update bandit rules 2025-09-28 04:54:27 +03:00
Karim shoair 716a5c3572 feat(cf solver): Make solver able to handle Turnstile and interstitial 2025-09-28 02:53:51 +03:00
Karim shoair 3da806210b perf: General code restructure to not use more than needed memory 2025-09-23 18:34:54 +03:00
Karim shoair 63a46a8e0b fix(parser): An encoding issue with converting bytes to string on some encoding types 
Removing that `invalid start byte` annoying bug
 2025-09-23 18:32:56 +03:00
Karim shoair 38e7941b71 docs: add a new sponsor 2025-09-22 16:03:13 +03:00
Karim shoair 61078ca1d2 v0.3.5 (#89) 2025-09-20 15:57:08 +03:00
Karim shoair 82b09620b2 perf: Improving all browser-based fetches by ~15% 2025-09-20 06:20:23 +03:00
Karim shoair 970ad40404 perf: Speed up sync fetches by ignoring number of pages checking 2025-09-20 04:31:57 +03:00
Karim shoair 5164b801ec build: pump version up and update deps 2025-09-20 00:33:07 +03:00
Karim shoair 67aa1a2ae7 tests: Add a test for using stealth and real_chrome arguments together 2025-09-20 00:23:55 +03:00
Karim shoair 67db865aee feat(DynamicFetcher): Replace rebrowser with patchright 2025-09-20 00:23:29 +03:00
Karim shoair 386da62ec9 tests: fix CLI tests according to last changes 2025-09-19 23:17:54 +03:00
Karim shoair 0ee3f93227 Merge branch 'main' into dev 2025-09-19 07:01:53 +03:00
Karim shoair 165dc524f8 docs: add a new sponsor 2025-09-19 06:06:30 +03:00
Karim shoair 2d704b2a8b fix(shell): Fixing a bug with content converting 2025-09-19 04:50:15 +03:00
Karim shoair a9d05cc0ef style: Removing dead code/docstrings and correcting type hints 2025-09-19 04:49:30 +03:00
Karim shoair bf5fe6d451 fix(shell): Fix view command edge case 2025-09-19 04:21:15 +03:00
Karim shoair 504ecdf7dc fix(shell): Fix parsing curl --data-raw 2025-09-19 04:20:16 +03:00
Karim shoair b197f8a5f9 fix(parser): Improve selectors re function 2025-09-18 13:56:37 +03:00
Karim shoair 41c8a2dcca Merge branch 'main' into dev 2025-09-18 13:55:48 +03:00
Karim shoair 189f323904 misc: update sponsor poster 2025-09-18 13:31:17 +03:00
Karim shoair ddf5b52e3b Improve page management and speed up execution (#87) 2025-09-18 03:34:40 +03:00
Karim shoair 6b2675c7dd v0.3.4 (#86) 2025-09-16 15:50:22 +03:00
Karim shoair 1550ba64bb docs: Adding new sponsor 2025-09-16 15:35:22 +03:00
Karim shoair 570303f938 fix(parser): Make html_content and prettify return strings not bytes (depends on the encoding) 2025-09-16 01:58:51 +03:00
Karim shoair 5683965367 fix(sessions): Fixing a bug with max pages counter 2025-09-16 01:47:31 +03:00
Karim shoair b7327466cf build: pump version up 2025-09-15 23:35:08 +03:00
Karim shoair 05c2d9aa00 feat(CLI): Make the session classes available in the shell 2025-09-15 23:34:36 +03:00
Karim shoair 5bcd4c2927 fix: A workaround for Playwright issue with page content retrieving on windows 2025-09-15 21:34:20 +03:00
Karim shoair bdcf96fa9e Merge branch 'main' into dev 2025-09-15 21:33:49 +03:00
Karim shoair 1547366c85 fix: fixing body issue with cli view command 2025-09-15 21:32:57 +03:00
Karim shoair 8a54745e47 v0.3.3 (#84) 2025-09-15 14:06:24 +03:00
Karim shoair f3a1f1ecb4 Merge branch 'main' into dev 2025-09-15 13:52:40 +03:00
Karim shoair a6aabda67c build: Pump version up 2025-09-15 13:52:09 +03:00
Karim shoair 7a7097b775 fix: Stop removing browser default page 
Looks like it's problematic with some systems so it doesn't worth the trouble
 2025-09-15 13:43:47 +03:00
Karim shoair fac92ef260 v0.3.2 (#83) 2025-09-15 04:28:55 +03:00
Karim shoair 3bf06a8521 docs: clarifying updates 2025-09-15 01:39:19 +03:00
Karim shoair f848e6b7f2 fix: Fixes to handle the new extra deps group 2025-09-15 00:19:19 +03:00
Karim shoair c7b9d45232 ops: Make tests workflow doesn't run when the changes isn't code related 2025-09-14 21:35:58 +03:00
Karim shoair d259b85c5b docs: Updating benchmarks with the latest version of all libraries 2025-09-14 21:26:28 +03:00
Karim shoair fdac239fcc build: Move click library to be part of the fetchers extra 2025-09-14 20:32:46 +03:00
Karim shoair eae088a07b tests: Multiple changes to tests 
- Remove all tests for the old encoding logic
- Stop using the `nopecha` test page to test Cloudflare solver
- Remove useless tests like testing for infinite timeout
- Fixes to make the code compatible with new changes
 2025-09-14 20:24:46 +03:00
Karim shoair ce4fc31f2c feat: Make .body return the passed content as it is without any processing 
This makes it possible to download files and deal with non-HTML requests (ex: #81 )
 2025-09-14 20:21:45 +03:00
Karim shoair 67ca139ff9 fix: Fixes for multiple encoding issues (#80 & #81 ) 2025-09-14 20:20:29 +03:00
Karim shoair 68615b2347 tests: remove the cloudflare test from the sync version 
To not put pressure on the website
 2025-09-14 16:31:38 +03:00
Karim shoair 1024ba6916 tests: Update test to be up-to-date with current version of the code 2025-09-14 16:30:59 +03:00
Karim shoair 0b3509a295 fix(shell): Fix a small typo 2025-09-14 05:05:55 +03:00
Karim shoair f604485a09 docs: Update all pages related to last changes 2025-09-14 04:45:56 +03:00
Karim shoair d6844232eb tests: Update all tests according to new update 2025-09-13 16:07:54 +03:00
Karim shoair 13d7e70cb7 refactor: Make all fetchers as an optional dependency group 
+ Removing some dead code
 2025-09-13 16:07:48 +03:00
Karim shoair d7e3deae2a fix: fix invalid dep version 2025-09-13 05:10:56 +03:00
Karim shoair 521366e77e build: bump up deps 2025-09-13 04:29:09 +03:00
Karim shoair 80077953f6 docs: update all pages related to last changes 2025-09-13 04:19:16 +03:00
Karim shoair 8c4d6158d6 feat(browser fetchers): A new option to control page JS 2025-09-13 03:59:12 +03:00
Karim shoair aaec00563c ops: update pre-commit hooks 2025-09-13 03:34:28 +03:00
Karim shoair 330d03559c style: applying the new ruff rules to all files 2025-09-13 03:22:53 +03:00
Karim shoair 60be9dc816 feat: Validate all fetch-level parameters 
So it's validated like the session-level ones. Needs improvement, but that's for later
 2025-09-13 03:22:08 +03:00
Karim shoair 831eafca34 ops: Update ruff pre-commit rules 2025-09-13 03:07:15 +03:00
Karim shoair 0c7db3c9d9 fix: validation keywords 2025-09-13 02:07:04 +03:00
Karim shoair adb82008e5 style: Using keywords for validation 
Less lines of code and more stylish
 2025-09-12 04:33:00 +03:00
Karim shoair e4b1e00e63 build: Pump up the version 2025-09-11 04:14:32 +03:00
Karim shoair 05a714980d docs: Update all docstring according to the new changes 2025-09-11 04:13:36 +03:00