freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
600 Commits 1 Branch 0 Tags
07773eb2f30de9aa9de22b6a8f6a108eaced94ef
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair 07773eb2f3 Merge branch 'main' into dev 2025-08-20 15:30:02 +03:00
Karim shoair 4fa92e1073 docs: update how sponsors displayed 2025-08-20 15:20:25 +03:00
Karim shoair 8a2bb0573b chore: replace sponsors banners with logos 2025-08-20 15:04:19 +03:00
Karim shoair 44d2b55f99 test: fix for GH actions 2025-08-20 04:33:20 +03:00
Karim shoair 6b8d861b92 build: Move non-primarily deps to optional extras 2025-08-20 03:58:10 +03:00
Karim shoair 85885d9962 tests: add geoip to Stealth tests 2025-08-20 01:36:55 +03:00
Karim shoair 57a025a34f build: update dependencies and fix geoip issue 2025-08-20 01:10:17 +03:00
Karim shoair 5aea62256b test: adding new tests and updating existing ones 
Code Coverage is now 92%
 2025-08-17 01:03:19 +03:00
Karim shoair 2f402f4835 style: add flags for tests coverage 
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
 2025-08-17 01:02:14 +03:00
Karim shoair 009bbb20c0 refactor(dynamic Fetcher): small optimization 2025-08-16 23:20:54 +03:00
Karim shoair 3e0ca33111 test: adding new tests and updating existing ones 2025-08-16 19:07:08 +03:00
Karim shoair 4c33c47609 test: Update and improve Camo/Dynamic tests 2025-08-16 16:02:17 +03:00
Karim shoair 6c9a960a44 fix(dynamicFetcher): Fix a bug with real_chrome argument 2025-08-16 16:01:32 +03:00
Karim shoair e6e5b3cc80 test: adding new tests and updating existing ones 
The coverage is now 78%
 2025-08-16 14:58:11 +03:00
Karim shoair 84337ef1ac fix(parser): count all nested children of ignored tags in get_all_text 2025-08-16 14:24:51 +03:00
Karim shoair 8bfbcec2d7 test: remove the test for install command 
GitHub is already reinstalls the library with it with every test
 2025-08-15 06:01:53 +03:00
Karim shoair 0e3358007f test: adding new tests and updating existing ones 2025-08-15 04:52:51 +03:00
Karim shoair 13700e2692 fix: improve error handling 2025-08-15 04:52:11 +03:00
Karim shoair ba81890daf test: adding new tests and updating existing ones 2025-08-03 16:37:47 +03:00
Karim shoair 825896c6a8 fix: improve checking for valid proxy and valid CDP URL 2025-08-03 16:31:16 +03:00
Karim shoair 740ae815c3 perf: speeding up find_by_text and find_by_regex by 3% 2025-08-02 19:46:18 +03:00
Karim shoair 67658f1723 perf: Speeding up below_elements and relocate by 3% 2025-08-02 19:45:43 +03:00
Karim shoair d1a2ecd341 perf: optimize get_all_text and adaptive logic by another 10% 2025-08-02 04:08:14 +03:00
Karim shoair 1ec6f0e0f0 perf: optimizing find_similar method 2025-08-02 03:39:05 +03:00
Karim shoair aec4889d25 perf: Optimizing next and previous properties 2025-08-02 03:37:48 +03:00
Karim shoair 548a40d850 perf: speed up get_all_text function by another 20% 2025-08-02 03:36:59 +03:00
Karim shoair 4c4202daae perf(parser): Speeding up css_first and xpath_first than normal ones 2025-08-01 16:41:09 +03:00
Karim shoair 83a19f3b17 perf(parser): A lot of optimizations to speed things up 2025-08-01 06:28:52 +03:00
Karim shoair f93348b915 style: remove unused code 2025-08-01 05:43:31 +03:00
Karim shoair df48662c00 perf(parser): A lot of optimizations to speed things up 2025-08-01 05:42:32 +03:00
Karim shoair af5f3688c1 fix: moving types to use Union again 2025-07-30 05:39:25 +03:00
Karim shoair 27658b33ff fix: fix invalid return type 2025-07-30 03:39:33 +03:00
Karim shoair 32cb76604c style: Adjustments to the translator 2025-07-30 03:28:16 +03:00
Karim shoair 5bb1266fa5 style: using isinstance function as the main way for type checking 2025-07-30 03:14:23 +03:00
Karim shoair ae9ccaec79 style: A lot of type hints correction 
Since we are using Py3.10 as minimum version now, we remove Union when possible
 2025-07-30 02:46:57 +03:00
Karim shoair ca12a11b7e style(utils): optimize imports 2025-07-30 02:08:01 +03:00
Karim shoair db781dcc32 style: Removing dead code 2025-07-30 02:03:39 +03:00
Karim shoair 3c5de8e0f2 perf: Speed up clean functions 2025-07-30 02:03:27 +03:00
Karim shoair 9415acccce fix: shortcuts for backward compatibility 2025-07-30 01:50:48 +03:00
Karim shoair 7300efa77e style(parser): optimize selectors instances creation 2025-07-30 01:45:26 +03:00
Karim shoair e7cdd39695 style: replacing os with Pathlib and small optimizations 2025-07-30 01:15:28 +03:00
Karim shoair 18660f8132 style: type hints corrections and docstrings 2025-07-30 00:32:39 +03:00
Karim shoair ba585c0adc perf: imports optimizing 2025-07-30 00:22:06 +03:00
Karim shoair 8ca940d768 style: removing dead code 2025-07-29 23:43:31 +03:00
Karim shoair 9bcb9e9d93 style: General type hints fixes and imports optimizing 2025-07-29 23:43:06 +03:00
Karim shoair 3d07a1533e refactor: cleaner code for the fetcher's lazy loader 2025-07-29 22:43:12 +03:00
Karim shoair 54cba6db45 ops: fix benchmarks script and make it more accurate 2025-07-29 22:36:47 +03:00
Karim shoair 9ce3279488 fix(TextHandler): Increase speed of clean method 5 times 2025-07-29 22:31:03 +03:00
Karim shoair 81a9eb068a ops(tests workflow): Cache browsers on GitHub 2025-07-29 21:03:14 +03:00
Karim shoair 08cae510a6 ops: Cache keys adjustment for tests 2025-07-29 16:40:48 +03:00
Karim shoair 715dfb4243 feat: add length property to Selectors to write less code 2025-07-29 06:19:45 +03:00
Karim shoair 29b77a96c8 refactor(parser): optimize imports 2025-07-29 06:18:12 +03:00
Karim shoair b9c7a5af2e refactor: replace's Selector inpt (text/body) with 1 argument called content 2025-07-29 06:08:15 +03:00
Karim shoair 9e9ba9ab10 docs: update roadmap 2025-07-29 04:20:43 +03:00
Karim shoair 264ae02aa7 refactor: huge change, many features/class got a better naming 
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
 2025-07-29 04:20:23 +03:00
Karim shoair 0c649987f8 test: remove irrelevant test case 2025-07-28 15:10:23 +03:00
Karim shoair a5e4b91653 refactor: remove clean up function for Adaptor + make adaptor attributes accessible directly 2025-07-28 14:48:22 +03:00
Karim shoair 297e14230b refactor(parser): Multiple optimizations and fixes 2025-07-28 03:32:22 +03:00
Karim shoair e35bade804 build: update bandit rules 2025-07-28 03:32:17 +03:00
Karim shoair d220050160 refactor(parser): Make get_all_text method 40% faster 2025-07-28 03:29:13 +03:00
Karim shoair 7f11b6f59b fix(storage): possible threading issue with recursion + optimizations 2025-07-28 00:16:58 +03:00
Karim shoair 3b0237e402 docs: improve All storage classes doc strings 2025-07-27 23:37:47 +03:00
Karim shoair b60fcbb884 refactor(Adaptor): Cleaner approach to find_similar method 
This code is slower than before by about 2-5μs, but it's worth it.
 2025-07-27 23:34:22 +03:00
Karim shoair ffde7e8ad9 fix(Adaptor): Add cleanup function to handle possible memory leak 2025-07-27 23:31:55 +03:00
Karim shoair 6ae1810405 docs: improve All Adaptor class doc strings 2025-07-27 22:41:37 +03:00
Karim shoair 93bd131bb6 fix(parser): Solve the ignored elements children issue while keeping speed 
solves #61
 2025-07-27 05:34:48 +03:00
Karim shoair d1aa0be6e4 refactor(StealthyFetcher): Remove virtual mode and use persistent context 
Solves #64 completely too
 2025-07-27 03:41:28 +03:00
Karim shoair 172a5b4a0a feat: Add an mcp server 2025-07-26 22:57:08 +03:00
Karim shoair 9d3b1335a9 build: update deps 2025-07-26 19:00:38 +03:00
Karim shoair 1076001e29 refactor: changes to be used by the mcp server 2025-07-14 05:38:29 +03:00
Karim shoair f7e1b7261e ops: adjust vermin to use 3.10 as the new minimum Python version 2025-07-14 05:38:16 +03:00
Karim shoair ecef751468 fix: small fix for fetcher 2025-07-14 02:14:54 +03:00
Karim shoair 0e5fd502d2 Merge branch 'main' into dev 2025-07-11 02:44:10 +03:00
Karim shoair 80e3a97990 docs: Update README.md 2025-07-10 21:19:00 +03:00
Karim shoair 51b5c4dff8 Merge branch 'main' into dev 2025-07-10 21:17:39 +03:00
Karim shoair 377ad92485 docs: update PetroSky promo 2025-07-10 21:14:34 +03:00
Karim shoair 6eb9626865 docs: add PetroSky as sponsor 2025-07-10 21:12:54 +03:00
Karim shoair 8af4833c64 chore: add new sponsor poster 2025-07-10 21:00:06 +03:00
Karim shoair 96be8af1b3 chore: add new sponsor poster 2025-07-10 20:56:20 +03:00
Karim shoair 1c6a960f17 Merge branch 'main' into dev 2025-07-09 19:39:08 +03:00
Karim shoair 119e00e16d chore: Delete the "others" issue template 2025-07-09 19:30:06 +03:00
Karim shoair fe8b11155e test: remove tests for Python 3.9 2025-07-09 16:31:05 +03:00
Karim shoair 4bba8598c2 build: update deps 2025-07-09 16:20:21 +03:00
Karim shoair 403f7bfe72 build: Updating PyPI classifiers 2025-07-07 00:10:20 +03:00
Karim shoair 939b78f3cf build: adding mcp as dependency 2025-07-07 00:09:54 +03:00
Karim shoair 3184dd8581 build: set minimum version to Python 3.10 2025-07-07 00:09:02 +03:00
Karim shoair 2b73fdbfd7 fix(DynamicFetcher): Remove old argument 2025-07-06 05:02:16 +03:00
Karim shoair 2ecf9b0c20 build: Improve description 2025-07-06 05:01:19 +03:00
Karim shoair 63a9db21a1 refactor(DynamicSession): Optimization + Removing max_pages from sync version 2025-07-06 04:59:22 +03:00
Karim shoair 6d79927233 refactor(Fetcher): Fix the issue of caching impersonation state + Less code duplication 
It now creates a session with each request, but at the same time, it's still faster than the fetcher in v0.2.99 by 20% with more features enabled.
 2025-07-06 02:21:15 +03:00
Karim shoair 18e56a8c50 refactor: Optimizing extract command and cleaning code 2025-06-29 19:40:03 +03:00
Karim shoair 29e485bdf2 refactor: Optimizations to CLI 2025-06-29 19:10:52 +03:00
Karim shoair b28f90e854 build: Update deps 2025-06-29 18:58:36 +03:00
Karim shoair 1c9e48b1c6 feat(extract): Adding new command to CLI options + Optimizations 
Users can now fetch websites directly without code and extract full/selected HTML content as HTML, Markdown, or extract text content.
 2025-06-29 18:56:09 +03:00
Karim shoair ce034f9f79 test: remove problematic test 2025-06-28 18:24:08 +03:00
Karim shoair b02f0bd236 ops: Adjust tox file to avoid CI issues 2025-06-28 18:12:38 +03:00
Karim shoair ac3db69c47 ops: new approach to CI tests workflow 2025-06-28 17:40:19 +03:00
Karim shoair ee1a0828a2 ops: Correcting cache paths 2025-06-27 17:16:35 +03:00
Karim shoair d603c15212 ops: rewrite tests logic to cache downloaded browsers before testing 2025-06-27 16:02:16 +03:00
Karim shoair 4ee32af67b ops: fix for GitHub's CI 2025-06-27 14:50:06 +03:00