 Karim shoair
|
e9e606bca5
|
feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default for async_fetch
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
.
Forgot to add it with the other commit
|
2025-01-28 23:53:12 +02:00 |
|
|
Karim shoair
|
675c0110e6
|
feat(StealthyFetcher): Make ads enabled by default for performance boost
|
2025-01-28 23:51:31 +02:00 |
|
|
Karim shoair
|
52b66c98ec
|
feat(StealthyFetcher): Disable Cross-Origin-Opener-Policy by default
This allows elements in cross-origin iframes, such as the Turnstile checkbox, to be clicked.
|
2025-01-28 23:47:47 +02:00 |
|
|
Karim shoair
|
1332be7e63
|
fix(autocompletion): Fix the defaults import
It runs autocompletion on some IDEs
|
2025-01-28 23:44:58 +02:00 |
|
|
Karim shoair
|
c2447e069e
|
build: Pumping version up
|
2025-01-28 23:41:48 +02:00 |
|
|
Karim shoair
|
c5d83a11ad
|
build: Pumping version up
|
2024-12-26 19:57:25 +02:00 |
|
|
Karim shoair
|
03b52aa74c
|
feat(cli): adding terminal command scrapling install
Instead of making users install browsers by themself
|
2024-12-26 19:46:22 +02:00 |
|
|
Karim shoair
|
2006be2173
|
fix(StealthyFetcher): Use more dependable response data
|
2024-12-26 00:29:42 +02:00 |
|
|
Karim shoair
|
f9b85cfbac
|
fix(PlaywrightFetcher): Use more dependable response data
|
2024-12-26 00:27:13 +02:00 |
|
|
Karim shoair
|
3cca3fd56b
|
style(parser): more accurate type hints
|
2024-12-21 16:17:02 +02:00 |
|
|
Karim shoair
|
f74db648bc
|
build: Pumping up the version to 0.2.91
|
2024-12-19 13:29:45 +02:00 |
|
|
Karim shoair
|
5be8c2a95e
|
style(Fetchers): Setting correct type hint for wait_selector_state argument
|
2024-12-19 13:22:12 +02:00 |
|
|
Karim shoair
|
94745b1555
|
fix(StealthyFetcher): Wait for the real response status code
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright API's default behavior.
|
2024-12-19 13:08:41 +02:00 |
|
|
Karim shoair
|
844cb0c6a2
|
fix(PlaywrightFetcher): Wait for the real response status code
This will return the real status response after redirection for websites that have Cloudflare instead of Playwright's default behavior.
|
2024-12-19 13:00:19 +02:00 |
|
|
Karim shoair
|
6652b8ac18
|
fix: solve the response log issue so now you get log for every response from Fetchers
|
2024-12-19 12:29:34 +02:00 |
|
|
Karim shoair
|
838b8b1196
|
style: fixing a typo in logging
|
2024-12-16 13:39:30 +02:00 |
|
|
Karim shoair
|
c282cb3cb7
|
fix: Stopped the log spamming that happens with multiple instances creation
|
2024-12-16 13:35:48 +02:00 |
|
|
Karim shoair
|
05c6eeb387
|
fix: add AsyncFetcher to top-level shortcuts
|
2024-12-16 12:37:29 +02:00 |
|
|
Karim shoair
|
d66ebc19a5
|
fix(Fetchers/disable_resources): Fixing the logic for intercepting resources
|
2024-12-16 12:35:25 +02:00 |
|
|
Karim shoair
|
eceef48ef5
|
fix(Fetchers/page_action): Fixing logic
|
2024-12-16 12:34:23 +02:00 |
|
|
Karim shoair
|
79a911e24e
|
feat(StealthyFetcher): Add async fetch support
|
2024-12-16 00:59:19 +02:00 |
|
|
Karim shoair
|
af4f2c0f74
|
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
|
2024-12-16 00:15:47 +02:00 |
|
|
Karim shoair
|
889c111855
|
refactor(Playwright Engine): Separate what we can for cleaner code and the async function later
The caching will give a slight performance increase with bulk requests
|
2024-12-15 23:25:02 +02:00 |
|
|
Karim shoair
|
6c17bd886e
|
style: using better data structures for constants
This will cause a slight performance increase
|
2024-12-15 23:19:47 +02:00 |
|
|
Karim shoair
|
b10cfd3643
|
feat: Adding AsyncFetcher class version of Fetcher
The first step in fully supporting async
|
2024-12-15 16:51:58 +02:00 |
|
|
Karim shoair
|
faf728a794
|
style: moving repeated arguments from inside the functions to __init__
Might give a slight performance increase too
|
2024-12-15 16:49:48 +02:00 |
|
|
Karim shoair
|
445af3c702
|
perf: Give repeated usage of Fetcher a slight performance increase
By caching the `StaticEngine` class instance
|
2024-12-15 16:13:08 +02:00 |
|
|
Karim shoair
|
299793af3f
|
feat: adding the retries argument for all methods of Fetcher class
|
2024-12-15 15:59:15 +02:00 |
|
|
Karim shoair
|
bfe906315e
|
feat: adding geoip parameter to the StealthyFetcher
|
2024-12-12 12:43:49 +02:00 |
|
|
Karim shoair
|
e254341974
|
fix: forgot to stage it with last commit
|
2024-12-11 21:43:36 +02:00 |
|
|
Karim shoair
|
193827e27b
|
refactor(api)!: Unifying log under 1 logger and removing debug parameter
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
|
2024-12-11 21:41:37 +02:00 |
|
|
Karim shoair
|
dcf51876a5
|
fix: Adaptor.body returns raw HTML without processing
If possible, otherwise returns `Adaptor.html_content`
|
2024-12-11 20:18:05 +02:00 |
|
|
Karim shoair
|
9f0001af26
|
feat: logging for response status
|
2024-12-11 20:14:32 +02:00 |
|
|
Karim shoair
|
ba11c433ff
|
Preparing to release 0.2.9 soon
|
2024-12-10 22:56:51 +02:00 |
|
|
Karim shoair
|
b4f90615c3
|
Adding keep_cdata argument for Adaptor and Response classes
This will force lxml to keep cdata while parsing html if you want
|
2024-12-10 22:06:08 +02:00 |
|
|
Karim shoair
|
5f9c3980dc
|
Adding urljoin method to Adaptors and Responses
|
2024-12-10 22:02:43 +02:00 |
|
|
Karim shoair
|
d0f18951ac
|
fix: Enable WebGL by default
|
2024-12-04 23:38:30 +02:00 |
|
|
Karim shoair
|
45e86f50a3
|
Adding empty methods (get/get_all/extract/extract_all)
For easy copy-paste from Scrapy/parsel code when needed :)
|
2024-12-04 14:28:37 +02:00 |
|
|
Karim shoair
|
31e838ce9b
|
Make standard string methods return TextHandler again instead of str
|
2024-12-04 14:27:36 +02:00 |
|
|
Karim shoair
|
c481a1c8ad
|
Pumping version up to 0.2.8
|
2024-11-30 18:13:19 +02:00 |
|
|
Karim shoair
|
1a17b2ce46
|
Sorting all library imports with isort
|
2024-11-30 14:02:06 +02:00 |
|
|
Karim shoair
|
97fecc9d5f
|
Style adjustments by flake8
|
2024-11-29 21:14:52 +02:00 |
|
|
Karim shoair
|
06a47f9644
|
Small fix for the ResponseEncoding class
|
2024-11-26 22:55:22 +02:00 |
|
|
Karim shoair
|
28b783e86b
|
Improving the stealth mode a bit for Playwright
|
2024-11-26 22:41:04 +02:00 |
|
|
Karim shoair
|
14ffdbd6fc
|
Handling problematic encodings
|
2024-11-26 22:27:05 +02:00 |
|
|
Karim shoair
|
d3fbcab3df
|
Pumping version up to 0.2.7
|
2024-11-26 21:27:22 +02:00 |
|
|
Karim shoair
|
dbd674aa7c
|
Adding proxy support for basic requests
|
2024-11-26 21:24:41 +02:00 |
|
|
Karim shoair
|
1cf7cee1a2
|
Fetchers - Wait for page to be stable again if the user wanted to wait for a selector
|
2024-11-26 20:26:54 +02:00 |
|
|
Karim shoair
|
d6a59774ea
|
StealthyFetcher - The option to disable ads with ublock origin addon
|
2024-11-26 20:23:02 +02:00 |
|
|
Karim shoair
|
60d551d839
|
PlaywrightFetcher - Add the option to set locale of the browser
|
2024-11-26 17:39:03 +02:00 |
|