freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
986 Commits 1 Branch 0 Tags
8e3696217763c5b1656354d6daaefeaa5dda42f5
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 8e36962177 ops: add local Claude files to gitignore 2026-01-10 20:41:23 +02:00
Karim shoair 0634f5796c feat(spiders): Add follow function to the response 2026-01-10 20:39:43 +02:00
Karim shoair f80e8828f7 refactor: internal API changes 2026-01-08 01:53:48 +02:00
Karim shoair 3a360acced build: pump version up temporary 2026-01-08 01:37:47 +02:00
Karim shoair 08f3e6efad fix: add missing line 2026-01-08 01:01:00 +02:00
Karim shoair 6553d90e0f ops: update workflows to run for this branch 2026-01-08 00:53:29 +02:00
Karim shoair 9eb5f68fcb test: update tests accordingly 2026-01-08 00:52:35 +02:00
Karim shoair 6a73f9dcd7 refactor: internal API changes to be easily used as indicators for spiders 2026-01-08 00:52:06 +02:00
Karim shoair de79fe80bb v0.3.14 2026-01-03 22:29:20 +02:00
Karim shoair 2f87507afe test: update database test to suite windows 2026-01-03 22:02:03 +02:00
Karim shoair f99baf65f7 test: remove the check for incognito 2026-01-03 01:50:46 +02:00
Karim shoair c8a2456b81 fix(StealthyFetcher): disable incognito mode to solve #123 2026-01-03 01:50:02 +02:00
Karim shoair 30a77de2be build: pump version up and browserforge version 2026-01-03 01:48:34 +02:00
Karim shoair 6ad2ecb9f2 v0.3.13 2026-01-01 22:07:00 +02:00
Karim shoair b5fdb71b52 docs: improving the API reference section 2026-01-01 21:07:20 +02:00
Karim shoair 4947152be2 docs: change docstring style for some methods 2026-01-01 21:06:35 +02:00
Karim shoair 9742b239be docs: fix for rtd build 2026-01-01 17:42:47 +02:00
Karim shoair e275509ece docs: fix for rtd build 
test
 2026-01-01 17:05:10 +02:00
Karim shoair 185a9c594f docs: fix for rtd build 2026-01-01 17:01:24 +02:00
Karim shoair c6e4482837 docs: Improve docs website 2026-01-01 16:56:57 +02:00
Karim shoair cb5051ca56 docs: update contributing page 2026-01-01 03:18:15 +02:00
Karim shoair 4d478eb84c docs: update the donate page 2026-01-01 03:16:10 +02:00
Karim shoair 2d5e2f41c4 docs: update dev articles 2026-01-01 03:14:12 +02:00
Karim shoair 0a3bb6cfdb docs: update the migrating from BS article 2025-12-31 18:32:34 +02:00
Karim shoair 55ff20c238 docs: update replacing AI article 2025-12-31 18:28:28 +02:00
Karim shoair 7d5b907d6c docs: adding more data to the adaptive feature page 2025-12-31 16:56:57 +02:00
Karim shoair 17202d23a8 docs: Update the MCP server page 2025-12-31 16:32:21 +02:00
Karim shoair ad234165e9 docs: update the cli section 2025-12-31 03:00:26 +02:00
Karim shoair 3b9a6d021e fix: correct cli argument 2025-12-31 03:00:06 +02:00
Karim shoair d4ae6c49c0 docs: update the StealthyFetcher page 2025-12-31 00:22:33 +02:00
Karim shoair f6f0ed4b37 fix(async stealth): wait the coroutine of the bounding box 2025-12-31 00:19:53 +02:00
Karim shoair 33c5633c58 test: correcting tests with the new arguments 2025-12-30 22:06:37 +02:00
Karim shoair 0c7e26ae5c fix(fingerprints): do not create google referer for IPs 2025-12-30 21:58:09 +02:00
Karim shoair a2b5628a73 docs: update DynamicFetcher page 2025-12-29 22:02:01 +02:00
Karim shoair 67e8f4d515 docs: update DynamicFetcher page 2025-12-29 20:48:45 +02:00
Karim shoair e1cac0f69f docs: update requests page 2025-12-29 20:48:32 +02:00
Karim shoair bc420da19b docs: update deps 2025-12-29 20:39:13 +02:00
Karim shoair ee13af9516 build: update deps 
and docstrings
 2025-12-29 16:08:51 +02:00
Karim shoair 6c25dccb05 docs: updating the Fetcher Basics and the HTTP requests pages 2025-12-28 21:57:21 +02:00
Karim shoair 5e3845f060 docs: shorten docstrings for some functions 2025-12-28 21:56:28 +02:00
Karim shoair b1cd5c5a82 test: fix old assert test 2025-12-28 00:16:37 +02:00
Karim shoair 9bfe744840 fix: remove forgotten unused code 2025-12-28 00:14:47 +02:00
Karim shoair 1b838eabd0 fix: update the cookies type hint for the mcp server 2025-12-28 00:08:44 +02:00
Karim shoair a4ad6b5fce test: remove old test 2025-12-28 00:03:09 +02:00
Karim shoair 09bbd416b8 fix: remove old proxy logic 
When I was caching it with the rest of the data
 2025-12-28 00:02:50 +02:00
Karim shoair a8e646160f docs: update the parsing section 2025-12-27 23:48:31 +02:00
Karim shoair a35cc4bdd6 docs: updating overview and benchmarks pages with the new changes 2025-12-27 20:51:28 +02:00
Karim shoair fb3a1b4c17 style: removing unused code 2025-12-27 20:50:08 +02:00
Karim shoair 8ca0646172 docs: make the cli section clearer for README files of all languages 2025-12-27 16:55:27 +02:00
Karim shoair 423de087d4 docs: update README for all languages with the new updates 2025-12-27 16:40:47 +02:00
Karim shoair ab76f1bee7 docs: update the website main page 2025-12-27 16:26:30 +02:00
Karim shoair 423755fe82 fix: remove firefox from test workflow 2025-12-27 14:33:24 +02:00
Karim shoair 842718ba94 fix: update shell signatures for the autocompletion 
Added missing fields and reordered classes as well for easier comparisons
 2025-12-27 14:30:48 +02:00
Karim shoair 2af21e240d test: remove problematic sync tests 
The async version are the same tests
 2025-12-27 14:02:02 +02:00
Karim shoair 4753743f49 fix: pinning the deps version for tox 2025-12-26 14:44:26 +02:00
Karim shoair 5638755558 ops: update tests workflow accordingly 2025-12-26 03:23:28 +02:00
Karim shoair fdfd5bf96b test: update all tests accordingly 2025-12-26 03:22:56 +02:00
Karim shoair 67c8cb7de3 ops: update tox file accordingly 2025-12-26 03:22:28 +02:00
Karim shoair c5f32e495c ops: update docker file accordingly 2025-12-26 03:22:11 +02:00
Karim shoair ccf97803d5 build: update deps 2025-12-26 03:21:40 +02:00
Karim shoair cfc667f7dc refactor(fetchers)!: Replace Camoufox with patchright and many optimizations 
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
 2025-12-26 03:21:13 +02:00
Karim shoair f9f34e6ee9 fix: improve the handling of useragent in chrome fetcher 2025-12-24 23:08:30 +02:00
Karim shoair b40c16a2cd build: pump version up 2025-12-24 21:35:41 +02:00
Karim shoair 85b327a598 fix: Improve stealth of dynamic browser by pinning Chrome's UA version 2025-12-24 21:34:44 +02:00
Karim shoair 2e5152f790 v0.3.12 2025-12-18 02:21:43 +02:00
Karim shoair a7b55d4a74 test: fix for typo in the workflow 2025-12-18 01:48:47 +02:00
Karim shoair e53f1cb363 build: forcing playwright version 
Until the Python driver for patchright 1.57 releases
 2025-12-18 01:45:21 +02:00
Karim shoair 580bc5390f test: Adjusting test workflow for GitHub 2025-12-18 01:44:05 +02:00
Karim shoair 1694b44a2b test: adjusting deps versions for tox 2025-12-18 01:30:00 +02:00
Karim shoair 381c4eab7f test: skip checking camoufox browser version if he cache hits 
To avoid GitHub repeated rate-limit errors
 2025-12-18 01:23:50 +02:00
Karim shoair ecb97d3fd6 docs: add docs for timezone_id argument 2025-12-18 01:16:22 +02:00
Karim shoair 9443ab80ae test: update tests accordingly 2025-12-18 01:14:05 +02:00
Karim shoair 39f544eae4 feat(dynamicSession): add the option to set timezone of the browser 2025-12-18 01:13:31 +02:00
Karim shoair b507e4d4a0 refactor(fetchers): rename internal api 
To make it easier to use to use sessions outside `with` context
 2025-12-17 01:27:40 +02:00
Karim shoair ae719a9d54 fix(parser): Improve response to json conversion 2025-12-17 00:15:55 +02:00
Karim shoair fd9fc83c6c build: pump version up and deps 2025-12-17 00:04:24 +02:00
Karim shoair f6862f8170 docs: Add new sponsor 2025-12-09 12:47:36 +02:00
Karim shoair 465b0f4733 docs: Add new sponsor 2025-12-08 03:35:24 +02:00
Karim shoair 0f9127e975 v0.3.11 2025-12-03 03:52:52 +02:00
Karim shoair 51d347270f build: remove unnecessary reinstallation of deps in tox 2025-12-03 03:43:17 +02:00
Karim shoair aed34e43b5 fix(fetchers): better handling for infinite network_idle 2025-12-03 03:11:30 +02:00
Karim shoair 16ff07a287 build: Pump version up and mcp 2025-12-03 02:20:18 +02:00
Karim shoair ac2d4d3833 fix: use the correct signature for the stealthy_fetch shortcut 2025-12-02 02:15:42 +02:00
Karim shoair ab0be95129 v0.3.10 2025-11-26 19:49:23 +02:00
Karim shoair eb15aae444 docs: update benchmarks with the current results 2025-11-26 02:50:01 +02:00
Karim shoair ab23b71836 tests(fetchers): Update tests to use the new argument naming 2025-11-26 02:23:54 +02:00
Karim shoair 0e1f3b9b05 fix(fetchers): Add backward compatibility for custom_config argument 2025-11-26 02:23:17 +02:00
Karim shoair 9b3309f825 fix(shell): use correct argument name in signatures 2025-11-26 02:22:29 +02:00
Karim shoair f84fbe8814 docs: update the renamed argument 2025-11-26 02:12:50 +02:00
Karim shoair 880b144af0 refactor(browser fetchers): Make all the type hints dynamic + Faster validation 
+ Also renamed `custom_config` to `selector_config` so it matches the session class.
 2025-11-25 22:03:45 +02:00
Karim shoair 8940bdeb55 fix(shell): dynamically build the signature of shortcuts after last changes 2025-11-25 21:23:06 +02:00
Karim shoair 04a612e64e docs: fix docstrings for all requests methods 2025-11-25 21:05:21 +02:00
Karim shoair 57e11cef76 tests: Update tests accordingly 2025-11-25 20:25:23 +02:00
Karim shoair 5376c97996 fix(fetcher): Remove non-keywords arguments 2025-11-25 20:25:09 +02:00
Karim shoair f0b01fc253 refactor(browser fetchers): Make all the type hints dynamic + Faster validation 
Made the code shorter by an additional ~200 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
 2025-11-25 20:18:36 +02:00
Karim shoair 9e85311ff6 fix(validators): Fix overriding session values with request ones 2025-11-25 19:48:09 +02:00
Karim shoair 74f24d60f6 fix(fetcher): Replace vars logic with one that doesn't require dict attribute 2025-11-24 13:04:12 +02:00
Karim shoair abc9aaebfe build: Pump deps up 
msgspec now supports py14
 2025-11-24 13:03:18 +02:00
Karim shoair 088d1912d6 ops: fix tests workflow caching 2025-11-24 12:52:28 +02:00
Karim shoair abaf877eff ops: fix tests workflow caching 2025-11-24 03:55:13 +02:00