freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
454 Commits 1 Branch 0 Tags
fcac10fa6d83edd5e6d07d33f8d3327f103dee4f
Commit Graph

53 Commits

Author SHA1 Message Date
Karim shoair c46ca8873f refactor(fetchers): Optimizing fetchers + making PlayWrightFetcher 10% faster 
Check out the Discord server for full details
 2025-05-25 21:14:24 +03:00
Karim shoair bf72678480 feat(cookies): The ability to pass cookies to browser fetchers 2025-05-10 19:36:53 +03:00
Karim shoair c84143129e fix(fetchers): Adjusting all cookies returned from each fetcher 
The cookies returned now can be passed again to each fetcher's engine without further validation by us.
 2025-05-10 19:16:34 +03:00
Karim shoair bd47c070aa fix(StealthyFetcher): Adjustments 2025-05-10 17:49:03 +03:00
Karim shoair fd33e35ff8 feat(StealthyFetcher): adding the option to solve Cloudflare Turnstile 2025-05-08 01:08:04 +03:00
Karim shoair 0c8dd63f87 chore: migrating to ruff and updating pre-commit hooks 2025-04-13 17:32:00 +02:00
Karim shoair a79bf3cc9d feat(StealthyFetcher): adding the wait option 
This will make fetcher sleep after fetch before closing the page
 2025-04-08 03:08:48 +02:00
Karim shoair 8f5d66f678 fix(StealthyFetcher): fix redirection issue with async_fetch 2025-04-08 02:58:10 +02:00
Karim shoair eb4ca2c6a4 fix(StealthyFetcher): change additional_arguments to a dictionary argument 2025-04-08 02:24:43 +02:00
Karim shoair d33e738c53 feat(StealthyFetcher): The ability to pass additional arguments to Camoufox 2025-03-26 05:36:05 +02:00
Karim shoair a2123fe91c refactor/fix(Camo engine): Improvements and fixes 2025-03-15 01:44:49 +02:00
Karim shoair aa4d365901 fix(StealthyFetcher): no need to disable coop now 
Camoufox can now click inside coop iframes without it
 2025-02-22 18:46:02 +02:00
Karim shoair 10ecd41444 feat(StealthyFetcher): Add the redirections history 2025-02-22 18:45:17 +02:00
Karim shoair 85f98708b2 feat(StealthyFetcher): Use memory cache by default for performance boost 
This is instead of the default disk-based cache.
 2025-01-28 23:55:24 +02:00
Karim shoair e9e606bca5 feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
 2025-01-28 23:53:12 +02:00
Karim shoair 675c0110e6 feat(StealthyFetcher): Make ads enabled by default for performance boost 2025-01-28 23:51:31 +02:00
Karim shoair 52b66c98ec feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default 
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
 2025-01-28 23:47:47 +02:00
Karim shoair 2006be2173 fix(StealthyFetcher): Use more dependable response data 2024-12-26 00:29:42 +02:00
Karim shoair 5be8c2a95e style(Fetchers): Setting correct type hint for wait_selector_state argument 2024-12-19 13:22:12 +02:00
Karim shoair 94745b1555 fix(StealthyFetcher): Wait for the real response status code 
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
 2024-12-19 13:08:41 +02:00
Karim shoair eceef48ef5 fix(Fetchers/page_action): Fixing logic 2024-12-16 12:34:23 +02:00
Karim shoair 79a911e24e feat(StealthyFetcher): Add async fetch support 2024-12-16 00:59:19 +02:00
Karim shoair af4f2c0f74 feat(PlaywrightFetcher): Add async support for PlaywrightFetcher 2024-12-16 00:15:47 +02:00
Karim shoair bfe906315e feat: adding geoip parameter to the StealthyFetcher 2024-12-12 12:43:49 +02:00
Karim shoair 193827e27b refactor(api)!: Unifying log under 1 logger and removing debug parameter 
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
 2024-12-11 21:41:37 +02:00
Karim shoair d0f18951ac fix: Enable WebGL by default 2024-12-04 23:38:30 +02:00
Karim shoair 1a17b2ce46 Sorting all library imports with isort 2024-11-30 14:02:06 +02:00
Karim shoair 1cf7cee1a2 Fetchers - Wait for page to be stable again if the user wanted to wait for a selector 2024-11-26 20:26:54 +02:00
Karim shoair d6a59774ea StealthyFetcher - The option to disable ads with ublock origin addon 2024-11-26 20:23:02 +02:00
Karim shoair 073ab2c7e3 Universally handling different response encoding and charsets for all responses 
Thanks for @andyfcx
 2024-11-23 15:40:41 +02:00
Karim shoair 869412761a Fix for the wait_selector argument resolving to more than 1 element 2024-11-22 22:15:52 +02:00
andyfcx 63112bfc50 fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes 2024-11-21 14:54:38 +08:00
Karim shoair 19ad82caef Calculate status text manually if it's not returned by PlayWright API 2024-11-20 13:27:18 +02:00
Karim shoair 4c74d9bc97 Fixing a bug with reading response bytes in playwright/camoufox when network_idle is used 
PlayWright doesn't provide a way to get the response in bytes after all wait ends like `page.content()` so that's more efficient to do anyway
 2024-11-20 12:33:54 +02:00
Karim shoair 572df6b3b4 Fixing the way Response object handles sub items in some edge cases 
+ Do it with the least impact on performance
 2024-11-16 13:54:55 +02:00
Karim shoair 90e38af552 Correcting the return string for the Response object in all functions/methods 2024-11-15 16:44:47 +02:00
Karim shoair 5e8275c3eb Adding the option to randomize the OS fingerprints with the StealthyFetcher 2024-11-15 16:36:32 +02:00
Karim shoair 2abb70218f Adding the proxy support to browser-based Fetchers 2024-11-15 14:43:03 +02:00
Karim shoair eb100e45f7 Corrections and rephrasing 2024-11-11 16:40:21 +02:00
Karim shoair 239e2c73bf Correctly get page content after JS Execution 
- Thanks for @AbdullahY36 for the heads-up.

**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
 2024-11-10 16:11:46 +02:00
Karim shoair 43045e3eb2 Get page content after JS Execution 
Thanks for @AbdullahY36
 2024-11-10 15:54:44 +02:00
Karim shoair ee8f78feb1 Update camo.py 2024-11-09 00:48:11 +02:00
Karim shoair fd34cd9afa Adding arguments google_search and extra_headers to StealthyFetcher 2024-11-09 00:43:24 +02:00
Karim shoair 530b54d964 Clearer doc-string 2024-11-09 00:17:41 +02:00
Karim shoair 1b015bbd07 Type correction 2024-11-08 23:59:12 +02:00
Karim shoair a0a3e62075 Adding addons and humanize arguments to StealthyFetcher 2024-11-06 20:14:42 +02:00
Karim shoair a60662af4a Update camo.py 2024-11-05 11:38:35 +02:00
Karim shoair bd612bb0e2 Updating the disable_resources doc-string 2024-11-04 20:16:04 +02:00
Karim shoair 54cacb8a4e Camoufox Engine - Doc strings and type annotations stuff 2024-11-03 22:20:29 +02:00
Karim shoair b32e1203f2 StealthyFetcher - The option to drop unnecessary resources requests for speed boost 2024-11-03 16:27:32 +02:00