freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
398 Commits 1 Branch 0 Tags
c36baf7b4ce1895c56fb31d08eb187947d4bfcdb
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Karim shoair c36baf7b4c test: Update tests to use current configuration 2025-03-26 21:56:04 +02:00
Karim shoair 8551d7eeea docs: Update README reflect current version accurately 2025-03-26 19:25:49 +02:00
Karim shoair 811cd2d9b0 refactor: make auto_match disabled by default 2025-03-26 19:25:19 +02:00
Karim shoair fec8215631 docs: Clearer docs for recent changes 2025-03-26 05:56:49 +02:00
Karim shoair 7025a7910e build: Pump up the version 2025-03-26 05:44:45 +02:00
Karim shoair 8a2ac94243 docs: Updating the benchmarks table with current numbers 
All libraries are updated to the latest version
 2025-03-26 05:41:03 +02:00
Karim shoair d33e738c53 feat(StealthyFetcher): The ability to pass additional arguments to Camoufox 2025-03-26 05:36:05 +02:00
Karim shoair a0952dca8f docs: Updating the README to reflect the new changes 2025-03-26 05:28:28 +02:00
Karim shoair 7ca8aaf2df feat(custom_config): The ability to pass custom parser config to requests 2025-03-26 03:56:56 +02:00
Karim shoair d7ef555d99 docs: Updating the README to reflect the new changes 2025-03-26 03:01:42 +02:00
Karim shoair 416bd919ba fix: backward-compatibility with the new import logic 2025-03-26 02:49:02 +02:00
Karim shoair 7c0c23fd33 feat(unified import): Make one import for all fetchers 
Why I didn't think of that from the start? IDK
 2025-03-26 02:48:34 +02:00
Karim shoair c004e388c4 build: fix auto-release workflow 
Escape special chars by using a cleaner approach
 2025-03-17 15:33:07 +02:00
Karim shoair e60d0cb4db Merge pull request #51 from D4Vinci/dev 
v0.2.98
 2025-03-17 15:24:09 +02:00
Karim shoair 9b0515b1cd fix(defaults): Lazy Loading fetchers to lower memory usage on importing 2025-03-17 06:56:34 +02:00
Karim shoair 94f16ca1af build: Pumping up the version 2025-03-17 06:47:43 +02:00
Karim shoair ff3962b1ad fix: Adjusting cache size in all the library 2025-03-17 06:46:41 +02:00
Karim shoair 18d2d29fa8 docs: use a better import to reflect the new changes 2025-03-17 06:29:32 +02:00
Karim shoair bbc05ca164 fix(importing): Lazy Loading submodules to lower memory usage on importing 
Load what you use instead of all submodules together
 2025-03-17 06:24:42 +02:00
Karim shoair 68ac9d4d05 fix(parser): Lowering memory with replacing body property with html_content 2025-03-17 05:58:40 +02:00
Karim shoair e5a022ec6a fix(parser): Improving CSS selecting memory usage by ~144% 
and speed by ~2%
 2025-03-16 22:33:46 +02:00
Karim shoair ed89161cd9 build: fixing release workflow 2025-03-15 03:26:40 +02:00
Karim shoair 01bd9a3d28 Merge pull request #49 from D4Vinci/dev 
v0.2.97
 2025-03-15 03:19:35 +02:00
Karim shoair b85cd5c607 build: update publish.yml 2025-03-15 02:46:26 +02:00
Karim shoair 51f2cf6b47 docs: updating FUNDING.yml 2025-03-15 02:43:55 +02:00
Karim shoair 257b740e1d build: Pumping up the version 2025-03-15 02:38:46 +02:00
Karim shoair 55802bac97 build: New workflow to automatically create releases from PRs on main 2025-03-15 02:37:50 +02:00
Karim shoair e05ff06b92 build: Updating the publish workflow 2025-03-15 02:32:09 +02:00
Karim shoair 7bef44ea4c fix: Limit caching in all the library 
To lower memory usage with high numbers of requests. Possibly improve case #48
 2025-03-15 02:01:16 +02:00
Karim shoair 4f75827641 refactor(fetchers): Clearer more accurate type hints 2025-03-15 01:47:41 +02:00
Karim shoair b5c7a8c36d refactor/fix(Fetcher engine): Improvements and fixes 
- Added the retries argument to the async methods, I don't understand how I missed it but that's why the code is rewritten now for easier maintenance.
 2025-03-15 01:46:38 +02:00
Karim shoair 0c31fb7c0e refactor/fix(Playwright engine): Improvements and fixes 2025-03-15 01:45:12 +02:00
Karim shoair a2123fe91c refactor/fix(Camo engine): Improvements and fixes 2025-03-15 01:44:49 +02:00
Karim shoair 94909099a9 fix(Parser): Change default text content to empty string instead of None 
For compatibility with previous fix/commit
 2025-03-15 00:56:19 +02:00
Karim shoair bf6a3a9c20 fix(TextHandler): auto handling non-string type variables 
Fix for #45
 2025-03-08 01:42:58 +02:00
Karim shoair 6906b13500 docs: update Scrapeless promo 2025-03-05 15:22:50 +02:00
Karim shoair 26c10d5373 docs: updating scrapeless banner 2025-03-05 15:18:00 +02:00
Karim shoair 3ca4ea1d45 Merge pull request #43 from D4Vinci/dev 
v0.2.96
 2025-03-05 03:43:19 +02:00
Karim shoair f8662bee29 feat(cli): Adding the -f flag to install command 2025-03-05 03:32:43 +02:00
Karim shoair 087a85d615 fix(cli): Fix issue where install drops user into Python Shell 2025-03-05 03:28:07 +02:00
Karim shoair 7ed189aa4e docs: Updating the README 2025-03-05 03:15:11 +02:00
Karim shoair 5f06a34a2f build: Pumping version up 2025-03-05 03:13:47 +02:00
Karim shoair e9344a7384 fix(TextHandler): Fixing an issue with slicing 
Slicing was returning `TextHandlers` instead of `TextHandler`. This solves #41 and corrects the type hint
 2025-03-05 03:11:46 +02:00
Karim shoair 573bfe030c Merge pull request #40 from D4Vinci/dev 
v0.2.95
 2025-02-26 00:27:21 +02:00
Karim shoair 1dc79d449b docs: update library description 2025-02-26 00:19:54 +02:00
Karim shoair dc702d2d7c build: pumping version up 2025-02-25 23:57:31 +02:00
Karim shoair c596a13901 docs: reflect updates on README 2025-02-25 23:54:23 +02:00
Karim shoair 5f4ec3ba3a fix(Fetcher): make stealthy_headers don't overwrite user headers 
Fix for #39
 2025-02-25 23:14:59 +02:00
Karim shoair 5251abbd1a Merge pull request #38 from D4Vinci/dev 
v0.2.94
 2025-02-22 18:55:16 +02:00
Karim shoair 7f7b8e080b docs(README): add history to the page to reflect new changes 2025-02-22 18:47:02 +02:00
Karim shoair aa4d365901 fix(StealthyFetcher): no need to disable coop now 
Camoufox can now click inside coop iframes without it
 2025-02-22 18:46:02 +02:00
Karim shoair 10ecd41444 feat(StealthyFetcher): Add the redirections history 2025-02-22 18:45:17 +02:00
Karim shoair 8049ac0f28 feat(PlayWrightFetcher): Add the redirections history 2025-02-22 17:47:27 +02:00
Karim shoair 3d2418b67a feat(Fetcher): Add the redirections history to Fetcher 
Feature requested in #32
 2025-02-22 16:58:54 +02:00
Karim shoair f0db07a31f build: pumping up deps and version to 0.2.94 2025-02-22 15:25:35 +02:00
Karim shoair 390bd51cb1 fix(TextHandler): correcting case_sensitive argument logic 
This is awkward, I can't believe how this slipped from me!
 2025-02-22 15:20:11 +02:00
Karim shoair db068f4eaa build: fix classifiers 2025-01-31 03:53:28 +02:00
Karim shoair e40a1e1f73 Merge pull request #33 from D4Vinci/dev 
v0.2.93
 2025-01-31 03:43:05 +02:00
Karim shoair 7c35341d85 refactor(parser/get_all_text): Cleaner and a bit faster implementation 2025-01-31 02:47:49 +02:00
Karim shoair a6fd25f64c fix(parser): fix traverse selectors 2025-01-31 01:57:19 +02:00
Karim shoair a8d475afb0 feat(parser): Make all returned html as TextHandler 
So you can do regex on raw html if wanted etc...
 2025-01-31 01:38:48 +02:00
Karim shoair 3330e9f475 refactor(parser/automatch): Cleaner and faster implementation 
~5% speed boost
 2025-01-31 01:34:02 +02:00
Karim shoair 37a31a120e refactor(parser/find_by_regex): Better implementation for ~60% speed boost 2025-01-31 01:19:11 +02:00
Karim shoair 2c6142d96d refactor(parser/find_by_text): Better implementation for ~20% speed boost 2025-01-31 01:12:00 +02:00
Karim shoair 9ea245eab4 doc(README): be more specific about what the regex match 2025-01-31 00:48:59 +02:00
Karim shoair 3f23f18473 style(TextHandler): fix re overload 2025-01-31 00:48:27 +02:00
Karim shoair 1962c11f64 fix(parser/find_all): Logic issues made conditions used sometimes in a (and) fashion and other times (OR) 
Also, the speed boost is 6-14% not 95% as I said first :(
 2025-01-31 00:23:54 +02:00
Karim shoair 57090f90a4 refactor(parser/find_all): about ~95% speed increase 
Boom!
 2025-01-30 23:39:46 +02:00
Karim shoair f37538320f feat(parser): adding below_elements method 2025-01-30 23:05:10 +02:00
Karim shoair ae60c2c0ad fix(playwright engine): bug in checking nstbrowser_config type 2025-01-30 20:07:19 +02:00
Karim shoair 148103f3c8 fix(translator): increase CSS to XPath convertor cache limit 2025-01-30 20:01:25 +02:00
Karim shoair c0d912d16a fix(asyncFetcher): A typo made async version of put do post request instead 
Awkward moment
 2025-01-30 19:53:13 +02:00
Karim shoair 7882cf2527 style(TextHandler): Setting overload on check_match argument for auto completion 2025-01-30 02:44:57 +02:00
Karim shoair e507778c48 feat(Parser): Set regex methods new return types as TextHandlers 2025-01-30 02:27:38 +02:00
Karim shoair 93be416572 feat(TextHandler): Now regex methods return TextHandler/TextHandlers 2025-01-30 02:15:09 +02:00
Karim shoair 4e2e7a4b22 fix(parser): More on correcting type hints and unifying return type 2025-01-30 02:03:09 +02:00
Karim shoair 7ee0f6114d fix(parser): Code restructure for speed boost & better type hints 
- Now return types are consistent across all the parser engine
- Parser got a 5-30% performance boost across different methods.
- Renamed some of the internal methods for clearer code.
- A lot better auto-completion experience after a lot of adjustments.
 2025-01-30 01:46:33 +02:00
Karim shoair 1d5fcc060d feat(TextHandler): Make slicing return TextHandlers + autocompletion fixes 
and access by index return `TextHandler`
 2025-01-29 14:22:47 +02:00
Karim shoair 70ffdd09e2 feat(TextHandler): Make split string method return Texthandlers 2025-01-29 01:22:07 +02:00
Karim shoair 5047e886a5 style(autocompletion): Improving type hints for custom types 
This will provide a better autocompletion experience inside IDEs, commit affects:
- TextHandler
- AttributesHandler
 2025-01-29 00:58:14 +02:00
Karim shoair 85f98708b2 feat(StealthyFetcher): Use memory cache by default for performance boost 
This is instead of the default disk-based cache.
 2025-01-28 23:55:24 +02:00
Karim shoair e9e606bca5 feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
 2025-01-28 23:53:12 +02:00
Karim shoair 675c0110e6 feat(StealthyFetcher): Make ads enabled by default for performance boost 2025-01-28 23:51:31 +02:00
Karim shoair 52b66c98ec feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
 2025-01-28 23:47:47 +02:00
Karim shoair 1332be7e63 fix(autocompletion): Fix the defaults import 
It runs autocompletion on some IDEs
 2025-01-28 23:44:58 +02:00
Karim shoair c2447e069e build: Pumping version up 2025-01-28 23:41:48 +02:00
Karim shoair 99fe842bc9 build: updating dependencies 2025-01-28 23:40:56 +02:00
Karim shoair 21291958f1 docs(README file): Better more-friendly description 2025-01-28 23:10:56 +02:00
Karim shoair 198b76f5d5 Test: disable stealth parameter 
It causes issues with Github Actions
 2025-01-28 13:43:36 +02:00
Karim shoair 7776e191e8 Update test_playwright.py 2025-01-28 13:42:11 +02:00
Karim shoair 75a965e802 Update FUNDING.yml 2025-01-27 21:14:50 +02:00
Karim shoair 0e1dcd4f29 Adding Scrapeless sponsor 2025-01-06 11:59:19 +02:00
Karim shoair 32d96606ea Merge pull request #27 from D4Vinci/dev 
v0.2.92
 2024-12-26 20:03:24 +02:00
Karim shoair c5d83a11ad build: Pumping version up 2024-12-26 19:57:25 +02:00
Karim shoair ef284234e6 docs: update README file to reflect new installation instructions 2024-12-26 19:56:37 +02:00
Karim shoair 03b52aa74c feat(cli): adding terminal command scrapling install 
Instead of making users install browsers by themself
 2024-12-26 19:46:22 +02:00
Karim shoair 2006be2173 fix(StealthyFetcher): Use more dependable response data 2024-12-26 00:29:42 +02:00
Karim shoair f9b85cfbac fix(PlaywrightFetcher): Use more dependable response data 2024-12-26 00:27:13 +02:00
Karim shoair 3cca3fd56b style(parser): more accurate type hints 2024-12-21 16:17:02 +02:00
Karim shoair ee599141fe Merge pull request #26 from D4Vinci/dev 
v0.2.91
 2024-12-19 13:51:53 +02:00