a79bf3cc9d
feat(StealthyFetcher): adding the wait option
...
This will make fetcher sleep after fetch before closing the page
2025-04-08 03:08:48 +02:00
8f5d66f678
fix(StealthyFetcher): fix redirection issue with async_fetch
2025-04-08 02:58:10 +02:00
eb4ca2c6a4
fix(StealthyFetcher): change additional_arguments to a dictionary argument
2025-04-08 02:24:43 +02:00
d33e738c53
feat(StealthyFetcher): The ability to pass additional arguments to Camoufox
2025-03-26 05:36:05 +02:00
a2123fe91c
refactor/fix(Camo engine): Improvements and fixes
2025-03-15 01:44:49 +02:00
aa4d365901
fix(StealthyFetcher): no need to disable coop now
...
Camoufox can now click inside coop iframes without it
2025-02-22 18:46:02 +02:00
10ecd41444
feat(StealthyFetcher): Add the redirections history
2025-02-22 18:45:17 +02:00
85f98708b2
feat(StealthyFetcher): Use memory cache by default for performance boost
...
This is instead of the default disk-based cache.
2025-01-28 23:55:24 +02:00
e9e606bca5
feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch
...
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
2025-01-28 23:53:12 +02:00
675c0110e6
feat(StealthyFetcher): Make ads enabled by default for performance boost
2025-01-28 23:51:31 +02:00
52b66c98ec
feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default
...
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
2025-01-28 23:47:47 +02:00
2006be2173
fix(StealthyFetcher): Use more dependable response data
2024-12-26 00:29:42 +02:00
5be8c2a95e
style(Fetchers): Setting correct type hint for wait_selector_state argument
2024-12-19 13:22:12 +02:00
94745b1555
fix(StealthyFetcher): Wait for the real response status code
...
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
2024-12-19 13:08:41 +02:00
eceef48ef5
fix(Fetchers/page_action): Fixing logic
2024-12-16 12:34:23 +02:00
79a911e24e
feat(StealthyFetcher): Add async fetch support
2024-12-16 00:59:19 +02:00
af4f2c0f74
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
2024-12-16 00:15:47 +02:00
bfe906315e
feat: adding geoip parameter to the StealthyFetcher
2024-12-12 12:43:49 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
d0f18951ac
fix: Enable WebGL by default
2024-12-04 23:38:30 +02:00
1a17b2ce46
Sorting all library imports with isort
2024-11-30 14:02:06 +02:00
1cf7cee1a2
Fetchers - Wait for page to be stable again if the user wanted to wait for a selector
2024-11-26 20:26:54 +02:00
d6a59774ea
StealthyFetcher - The option to disable ads with ublock origin addon
2024-11-26 20:23:02 +02:00
073ab2c7e3
Universally handling different response encoding and charsets for all responses
...
Thanks for @andyfcx
2024-11-23 15:40:41 +02:00
869412761a
Fix for the wait_selector argument resolving to more than 1 element
2024-11-22 22:15:52 +02:00
63112bfc50
fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes
2024-11-21 14:54:38 +08:00
19ad82caef
Calculate status text manually if it's not returned by PlayWright API
2024-11-20 13:27:18 +02:00
4c74d9bc97
Fixing a bug with reading response bytes in playwright/camoufox when network_idle is used
...
PlayWright doesn't provide a way to get the response in bytes after all wait ends like `page.content()` so that's more efficient to do anyway
2024-11-20 12:33:54 +02:00
572df6b3b4
Fixing the way Response object handles sub items in some edge cases
...
+ Do it with the least impact on performance
2024-11-16 13:54:55 +02:00
90e38af552
Correcting the return string for the Response object in all functions/methods
2024-11-15 16:44:47 +02:00
5e8275c3eb
Adding the option to randomize the OS fingerprints with the StealthyFetcher
2024-11-15 16:36:32 +02:00
2abb70218f
Adding the proxy support to browser-based Fetchers
2024-11-15 14:43:03 +02:00
eb100e45f7
Corrections and rephrasing
2024-11-11 16:40:21 +02:00
239e2c73bf
Correctly get page content after JS Execution
...
- Thanks for @AbdullahY36 for the heads-up.
**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
2024-11-10 16:11:46 +02:00
43045e3eb2
Get page content after JS Execution
...
Thanks for @AbdullahY36
2024-11-10 15:54:44 +02:00
ee8f78feb1
Update camo.py
2024-11-09 00:48:11 +02:00
fd34cd9afa
Adding arguments google_search and extra_headers to StealthyFetcher
2024-11-09 00:43:24 +02:00
530b54d964
Clearer doc-string
2024-11-09 00:17:41 +02:00
1b015bbd07
Type correction
2024-11-08 23:59:12 +02:00
a0a3e62075
Adding addons and humanize arguments to StealthyFetcher
2024-11-06 20:14:42 +02:00
a60662af4a
Update camo.py
2024-11-05 11:38:35 +02:00
bd612bb0e2
Updating the disable_resources doc-string
2024-11-04 20:16:04 +02:00
54cacb8a4e
Camoufox Engine - Doc strings and type annotations stuff
2024-11-03 22:20:29 +02:00
b32e1203f2
StealthyFetcher - The option to drop unnecessary resources requests for speed boost
2024-11-03 16:27:32 +02:00
e7300ecba0
StealthFetcher - Add the option to allow webgl in the browser
2024-11-03 14:40:19 +02:00
145c03daff
Big structure changes (check commit description)
...
- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
2024-11-03 01:04:02 +02:00
8f8d08b568
Adding browsing engines
2024-10-31 02:11:33 +03:00
Karim shoair