freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
395 Commits 1 Branch 0 Tags
fec82156312842a52ac5956e55891fa4ddbec4df
Commit Graph

107 Commits

Author SHA1 Message Date
Karim shoair fec8215631 docs: Clearer docs for recent changes 2025-03-26 05:56:49 +02:00
Karim shoair d33e738c53 feat(StealthyFetcher): The ability to pass additional arguments to Camoufox 2025-03-26 05:36:05 +02:00
Karim shoair 7c0c23fd33 feat(unified import): Make one import for all fetchers 
Why I didn't think of that from the start? IDK
 2025-03-26 02:48:34 +02:00
Karim shoair ff3962b1ad fix: Adjusting cache size in all the library 2025-03-17 06:46:41 +02:00
Karim shoair 7bef44ea4c fix: Limit caching in all the library 
To lower memory usage with high numbers of requests. Possibly improve case #48
 2025-03-15 02:01:16 +02:00
Karim shoair b5c7a8c36d refactor/fix(Fetcher engine): Improvements and fixes 
- Added the retries argument to the async methods, I don't understand how I missed it but that's why the code is rewritten now for easier maintenance.
 2025-03-15 01:46:38 +02:00
Karim shoair 0c31fb7c0e refactor/fix(Playwright engine): Improvements and fixes 2025-03-15 01:45:12 +02:00
Karim shoair a2123fe91c refactor/fix(Camo engine): Improvements and fixes 2025-03-15 01:44:49 +02:00
Karim shoair 5f4ec3ba3a fix(Fetcher): make stealthy_headers don't overwrite user headers 
Fix for #39
 2025-02-25 23:14:59 +02:00
Karim shoair aa4d365901 fix(StealthyFetcher): no need to disable coop now 
Camoufox can now click inside coop iframes without it
 2025-02-22 18:46:02 +02:00
Karim shoair 10ecd41444 feat(StealthyFetcher): Add the redirections history 2025-02-22 18:45:17 +02:00
Karim shoair 8049ac0f28 feat(PlayWrightFetcher): Add the redirections history 2025-02-22 17:47:27 +02:00
Karim shoair 3d2418b67a feat(Fetcher): Add the redirections history to Fetcher 
Feature requested in #32
 2025-02-22 16:58:54 +02:00
Karim shoair ae60c2c0ad fix(playwright engine): bug in checking nstbrowser_config type 2025-01-30 20:07:19 +02:00
Karim shoair 85f98708b2 feat(StealthyFetcher): Use memory cache by default for performance boost 
This is instead of the default disk-based cache.
 2025-01-28 23:55:24 +02:00
Karim shoair e9e606bca5 feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
 2025-01-28 23:53:12 +02:00
Karim shoair 675c0110e6 feat(StealthyFetcher): Make ads enabled by default for performance boost 2025-01-28 23:51:31 +02:00
Karim shoair 52b66c98ec feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
 2025-01-28 23:47:47 +02:00
Karim shoair 2006be2173 fix(StealthyFetcher): Use more dependable response data 2024-12-26 00:29:42 +02:00
Karim shoair f9b85cfbac fix(PlaywrightFetcher): Use more dependable response data 2024-12-26 00:27:13 +02:00
Karim shoair 5be8c2a95e style(Fetchers): Setting correct type hint for wait_selector_state argument 2024-12-19 13:22:12 +02:00
Karim shoair 94745b1555 fix(StealthyFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
 2024-12-19 13:08:41 +02:00
Karim shoair 844cb0c6a2 fix(PlaywrightFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright's default behavior.
 2024-12-19 13:00:19 +02:00
Karim shoair 6652b8ac18 fix: solve the response log issue so now you get log for every response from Fetchers 2024-12-19 12:29:34 +02:00
Karim shoair c282cb3cb7 fix: Stopped the log spamming that happens with multiple instances creation 2024-12-16 13:35:48 +02:00
Karim shoair d66ebc19a5 fix(Fetchers/disable_resources): Fixing the logic for intercepting resources 2024-12-16 12:35:25 +02:00
Karim shoair eceef48ef5 fix(Fetchers/page_action): Fixing logic 2024-12-16 12:34:23 +02:00
Karim shoair 79a911e24e feat(StealthyFetcher): Add async fetch support 2024-12-16 00:59:19 +02:00
Karim shoair af4f2c0f74 feat(PlaywrightFetcher): Add async support for PlaywrightFetcher 2024-12-16 00:15:47 +02:00
Karim shoair 889c111855 refactor(Playwright Engine): Separate what we can for cleaner code and the async function later 
The caching will give a slight performance increase with bulk requests
 2024-12-15 23:25:02 +02:00
Karim shoair 6c17bd886e style: using better data structures for constants 
This will cause a slight performance increase
 2024-12-15 23:19:47 +02:00
Karim shoair b10cfd3643 feat: Adding AsyncFetcher class version of Fetcher 
The first step in fully supporting async
 2024-12-15 16:51:58 +02:00
Karim shoair faf728a794 style: moving repeated arguments from inside the functions to __init__ 
Might give a slight performance increase too
 2024-12-15 16:49:48 +02:00
Karim shoair 445af3c702 perf: Give repeated usage of Fetcher a slight performance increase 
By caching the `StaticEngine` class instance
 2024-12-15 16:13:08 +02:00
Karim shoair 299793af3f feat: adding the retries argument for all methods of Fetcher class 2024-12-15 15:59:15 +02:00
Karim shoair bfe906315e feat: adding geoip parameter to the StealthyFetcher 2024-12-12 12:43:49 +02:00
Karim shoair e254341974 fix: forgot to stage it with last commit 2024-12-11 21:43:36 +02:00
Karim shoair 193827e27b refactor(api)!: Unifying log under 1 logger and removing debug parameter 
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
 2024-12-11 21:41:37 +02:00
Karim shoair 9f0001af26 feat: logging for response status 2024-12-11 20:14:32 +02:00
Karim shoair b4f90615c3 Adding keep_cdata argument for Adaptor and Response classes 
This will force lxml to keep cdata while parsing html if you want
 2024-12-10 22:06:08 +02:00
Karim shoair d0f18951ac fix: Enable WebGL by default 2024-12-04 23:38:30 +02:00
Karim shoair 1a17b2ce46 Sorting all library imports with isort 2024-11-30 14:02:06 +02:00
Karim shoair 97fecc9d5f Style adjustments by flake8 2024-11-29 21:14:52 +02:00
Karim shoair 06a47f9644 Small fix for the ResponseEncoding class 2024-11-26 22:55:22 +02:00
Karim shoair 28b783e86b Improving the stealth mode a bit for Playwright 2024-11-26 22:41:04 +02:00
Karim shoair 14ffdbd6fc Handling problematic encodings 2024-11-26 22:27:05 +02:00
Karim shoair dbd674aa7c Adding proxy support for basic requests 2024-11-26 21:24:41 +02:00
Karim shoair 1cf7cee1a2 Fetchers - Wait for page to be stable again if the user wanted to wait for a selector 2024-11-26 20:26:54 +02:00
Karim shoair d6a59774ea StealthyFetcher - The option to disable ads with ublock origin addon 2024-11-26 20:23:02 +02:00
Karim shoair 60d551d839 PlaywrightFetcher - Add the option to set locale of the browser 2024-11-26 17:39:03 +02:00