freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
934 Commits 1 Branch 0 Tags
842718ba94dea20f48cd66e6a1711a3195bbaa30
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 842718ba94 fix: update shell signatures for the autocompletion 
Added missing fields and reordered classes as well for easier comparisons
 2025-12-27 14:30:48 +02:00
Karim shoair 2af21e240d test: remove problematic sync tests 
The async version are the same tests
 2025-12-27 14:02:02 +02:00
Karim shoair 4753743f49 fix: pinning the deps version for tox 2025-12-26 14:44:26 +02:00
Karim shoair 5638755558 ops: update tests workflow accordingly 2025-12-26 03:23:28 +02:00
Karim shoair fdfd5bf96b test: update all tests accordingly 2025-12-26 03:22:56 +02:00
Karim shoair 67c8cb7de3 ops: update tox file accordingly 2025-12-26 03:22:28 +02:00
Karim shoair c5f32e495c ops: update docker file accordingly 2025-12-26 03:22:11 +02:00
Karim shoair ccf97803d5 build: update deps 2025-12-26 03:21:40 +02:00
Karim shoair cfc667f7dc refactor(fetchers)!: Replace Camoufox with patchright and many optimizations 
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
 2025-12-26 03:21:13 +02:00
Karim shoair f9f34e6ee9 fix: improve the handling of useragent in chrome fetcher 2025-12-24 23:08:30 +02:00
Karim shoair b40c16a2cd build: pump version up 2025-12-24 21:35:41 +02:00
Karim shoair 85b327a598 fix: Improve stealth of dynamic browser by pinning Chrome's UA version 2025-12-24 21:34:44 +02:00
Karim shoair 2e5152f790 v0.3.12 2025-12-18 02:21:43 +02:00
Karim shoair a7b55d4a74 test: fix for typo in the workflow 2025-12-18 01:48:47 +02:00
Karim shoair e53f1cb363 build: forcing playwright version 
Until the Python driver for patchright 1.57 releases
 2025-12-18 01:45:21 +02:00
Karim shoair 580bc5390f test: Adjusting test workflow for GitHub 2025-12-18 01:44:05 +02:00
Karim shoair 1694b44a2b test: adjusting deps versions for tox 2025-12-18 01:30:00 +02:00
Karim shoair 381c4eab7f test: skip checking camoufox browser version if he cache hits 
To avoid GitHub repeated rate-limit errors
 2025-12-18 01:23:50 +02:00
Karim shoair ecb97d3fd6 docs: add docs for timezone_id argument 2025-12-18 01:16:22 +02:00
Karim shoair 9443ab80ae test: update tests accordingly 2025-12-18 01:14:05 +02:00
Karim shoair 39f544eae4 feat(dynamicSession): add the option to set timezone of the browser 2025-12-18 01:13:31 +02:00
Karim shoair b507e4d4a0 refactor(fetchers): rename internal api 
To make it easier to use to use sessions outside `with` context
 2025-12-17 01:27:40 +02:00
Karim shoair ae719a9d54 fix(parser): Improve response to json conversion 2025-12-17 00:15:55 +02:00
Karim shoair fd9fc83c6c build: pump version up and deps 2025-12-17 00:04:24 +02:00
Karim shoair f6862f8170 docs: Add new sponsor 2025-12-09 12:47:36 +02:00
Karim shoair 465b0f4733 docs: Add new sponsor 2025-12-08 03:35:24 +02:00
Karim shoair 0f9127e975 v0.3.11 2025-12-03 03:52:52 +02:00
Karim shoair 51d347270f build: remove unnecessary reinstallation of deps in tox 2025-12-03 03:43:17 +02:00
Karim shoair aed34e43b5 fix(fetchers): better handling for infinite network_idle 2025-12-03 03:11:30 +02:00
Karim shoair 16ff07a287 build: Pump version up and mcp 2025-12-03 02:20:18 +02:00
Karim shoair ac2d4d3833 fix: use the correct signature for the stealthy_fetch shortcut 2025-12-02 02:15:42 +02:00
Karim shoair ab0be95129 v0.3.10 2025-11-26 19:49:23 +02:00
Karim shoair eb15aae444 docs: update benchmarks with the current results 2025-11-26 02:50:01 +02:00
Karim shoair ab23b71836 tests(fetchers): Update tests to use the new argument naming 2025-11-26 02:23:54 +02:00
Karim shoair 0e1f3b9b05 fix(fetchers): Add backward compatibility for custom_config argument 2025-11-26 02:23:17 +02:00
Karim shoair 9b3309f825 fix(shell): use correct argument name in signatures 2025-11-26 02:22:29 +02:00
Karim shoair f84fbe8814 docs: update the renamed argument 2025-11-26 02:12:50 +02:00
Karim shoair 880b144af0 refactor(browser fetchers): Make all the type hints dynamic + Faster validation 
+ Also renamed `custom_config` to `selector_config` so it matches the session class.
 2025-11-25 22:03:45 +02:00
Karim shoair 8940bdeb55 fix(shell): dynamically build the signature of shortcuts after last changes 2025-11-25 21:23:06 +02:00
Karim shoair 04a612e64e docs: fix docstrings for all requests methods 2025-11-25 21:05:21 +02:00
Karim shoair 57e11cef76 tests: Update tests accordingly 2025-11-25 20:25:23 +02:00
Karim shoair 5376c97996 fix(fetcher): Remove non-keywords arguments 2025-11-25 20:25:09 +02:00
Karim shoair f0b01fc253 refactor(browser fetchers): Make all the type hints dynamic + Faster validation 
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
 2025-11-25 20:18:36 +02:00
Karim shoair 9e85311ff6 fix(validators): Fix overriding session values with request ones 2025-11-25 19:48:09 +02:00
Karim shoair 74f24d60f6 fix(fetcher): Replace vars logic with one that doesn't require dict attribute 2025-11-24 13:04:12 +02:00
Karim shoair abc9aaebfe build: Pump deps up 
msgspec now supports py14
 2025-11-24 13:03:18 +02:00
Karim shoair 088d1912d6 ops: fix tests workflow caching 2025-11-24 12:52:28 +02:00
Karim shoair abaf877eff ops: fix tests workflow caching 2025-11-24 03:55:13 +02:00
Karim shoair b5cb235c53 ops: fix tests workflow caching 2025-11-24 03:30:14 +02:00
Karim shoair c487faf4fd pref(requests): Optimizing memory by specifiying slots 2025-11-23 21:50:54 +02:00
Karim shoair 357b170e1c refactor(requests): Make all the type hints dynamic 
Made the code shorter by about ~500 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
 2025-11-23 20:48:51 +02:00
Karim shoair 2bae3261a0 build: Pumping version and deps up 
This also updates the Maxmind database for camoufox
 2025-11-22 21:41:47 +02:00
Karim shoair 170205599d fix(parser): Better approach for web pages where the encoding is not always correctly declared 
Fixes #110 and avoids defaulting to a specific encoding like #111
 2025-11-22 20:24:55 +02:00
Karim shoair 48313307d0 docs: update ads order 2025-11-21 17:50:18 +02:00
Karim shoair 2a9aef8d48 docs: Make Serpapi as Gold-tier sponsor 2025-11-21 17:30:55 +02:00
Karim shoair 2506f21cf6 docs: fix releases link 2025-11-19 16:23:39 +02:00
Karim shoair e6e65ba360 v0.3.9 2025-11-17 03:38:46 +02:00
Karim shoair 9602ad6986 docs: Add the remove_entities argument 2025-11-17 03:20:49 +02:00
Karim shoair da5c180329 test: improve tox config 2025-11-17 02:50:16 +02:00
Karim shoair 5685e4930c test: fix for previous commit 2025-11-17 02:22:16 +02:00
Karim shoair d47f77d4d1 test: fixes for Github CI issues 2025-11-17 02:12:03 +02:00
Karim shoair fb80beba04 feat(TextHandler): Add argument to clean method to remove html entities 2025-11-17 01:36:06 +02:00
Karim shoair f8a6ec2c69 chore: automatic update of pre-commit hooks 2025-11-17 01:01:33 +02:00
Karim shoair 763363457f docs: Updating the contribution guide 2025-11-17 01:00:10 +02:00
Karim shoair 64aa29cd40 ops: add a workflow to enforce code quality 2025-11-17 00:59:49 +02:00
Karim shoair 2751653190 test: add tests for the new impersonate logic 2025-11-16 16:34:24 +02:00
Karim shoair 3565f9d500 feat(fetcher): Make impersonate able to randomize fingerprint 2025-11-16 16:33:58 +02:00
Karim shoair aa245159d0 Merge branch 'main' into dev 2025-11-15 14:39:22 +02:00
Karim shoair 573ec04b9b docs: removing a sponsor 2025-11-15 14:39:11 +02:00
Karim shoair fc2c4c1ab7 docs: removing a sponsor 2025-11-15 14:37:57 +02:00
Karim shoair ceda6eb0c1 docs: Add AI translations to other languages 2025-11-14 02:56:00 +02:00
Karim shoair c8b9a04e87 docs: making some adjustments 2025-11-14 02:22:59 +02:00
Karim shoair d018d6e438 fix: fixing the response logic 2025-11-10 03:03:05 +02:00
Karim shoair 026161c8f8 docs: update docs to add file downloading examples 2025-11-10 03:02:41 +02:00
Karim shoair ff19dd5bf5 feat: replace raw_response property with smart content detection 2025-11-10 02:39:45 +02:00
Karim shoair 372fc1e051 docs: Style updates and a lot of clarifications 2025-11-09 02:15:43 +02:00
Karim shoair 3b5f0470d2 docs: Update docs accordingly and some adjustments 2025-11-08 20:18:58 +02:00
Karim shoair df073c4680 fix: added raw_response to separate collected response from rendered response 2025-11-08 20:09:40 +02:00
Karim shoair e20882af5d style: removing dead code from the parser 2025-11-08 20:08:23 +02:00
Karim shoair 42a8b43207 build: Pump version up 2025-11-08 20:07:26 +02:00
Karim shoair bad574e0e8 Merge branch 'main' into dev 2025-10-29 14:44:29 +03:00
Karim shoair d02da49865 docs: fix downloads badge 2025-10-29 14:43:43 +03:00
Karim shoair baa5a97a1f docs: fix downloads badge 2025-10-29 14:43:13 +03:00
Karim shoair 6cc48161cd docs: Update docker instructions 2025-10-28 17:20:27 +03:00
Karim shoair 58784b7c90 v0.3.8 2025-10-27 18:08:18 +03:00
Karim shoair 1e4d1e73a0 ops: Make container image pushed to Github registry too 2025-10-27 17:36:44 +03:00
Karim shoair 3c8d5c68e8 docs: updating external links 2025-10-27 16:42:03 +03:00
Karim shoair 33ed7fcf7d Merge branch 'main' into dev 2025-10-27 16:38:48 +03:00
Karim shoair ba7015bc57 docs: Add new sponsor 2025-10-27 16:38:13 +03:00
Karim shoair 37cb5097c0 docs: update external browser tutorial 2025-10-27 06:33:45 +03:00
Karim shoair 339ec43341 docs: adding separate doc for external browser 2025-10-27 06:32:31 +03:00
Karim shoair fc4a809985 fix(cloudflare): Limit the waiting period for websites that captcha doesn't disappear after solving 
Also solves #100
 2025-10-27 05:13:12 +03:00
Karim shoair d4a2507666 refactor(validation): All browser based fetchers are ~8-15% faster now in most cases 2025-10-27 04:48:45 +03:00
Karim shoair 13b9a4668c feat(DynamicSession): New option to add extra browser flags 2025-10-26 16:53:27 +03:00
Karim shoair f9a514cc64 build: Pump up deps 2025-10-26 05:04:53 +03:00
Karim shoair 59ec6be201 refactor(fetchers): Less duplicated code and better handling for unstable websites 
- Websites that never finish loading their requests won't crash the code now if you used `network_idle` with them
- Fixed a typo in `load_dom` in DynamicSession's async_fetch
- Removed dead code
 2025-10-26 05:04:19 +03:00
Karim shoair 74fa45daea fix: Addressing the issue of collecting response after page_action in #100 2025-10-26 00:41:45 +03:00
Karim shoair 43db79d1df docs: adding cloud browser segment 2025-10-25 22:29:07 +03:00
Karim shoair cabf5f963d Merge branch 'main' into dev 2025-10-22 15:39:07 +03:00
Karim shoair c796787863 docs: Update sponsor banner 2025-10-22 15:23:39 +03:00