 Karim shoair
|
79a911e24e
|
feat(StealthyFetcher): Add async fetch support
|
2024-12-16 00:59:19 +02:00 |
|
|
Karim shoair
|
af4f2c0f74
|
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
|
2024-12-16 00:15:47 +02:00 |
|
|
Karim shoair
|
889c111855
|
refactor(Playwright Engine): Separate what we can for cleaner code and the async function later
The caching will give a slight performance increase with bulk requests
|
2024-12-15 23:25:02 +02:00 |
|
|
Karim shoair
|
6c17bd886e
|
style: using better data structures for constants
This will cause a slight performance increase
|
2024-12-15 23:19:47 +02:00 |
|
|
Karim shoair
|
b10cfd3643
|
feat: Adding AsyncFetcher class version of Fetcher
The first step in fully supporting async
|
2024-12-15 16:51:58 +02:00 |
|
|
Karim shoair
|
faf728a794
|
style: moving repeated arguments from inside the functions to __init__
Might give a slight performance increase too
|
2024-12-15 16:49:48 +02:00 |
|
|
Karim shoair
|
445af3c702
|
perf: Give repeated usage of Fetcher a slight performance increase
By caching the `StaticEngine` class instance
|
2024-12-15 16:13:08 +02:00 |
|
|
Karim shoair
|
299793af3f
|
feat: adding the retries argument for all methods of Fetcher class
|
2024-12-15 15:59:15 +02:00 |
|
|
Karim shoair
|
bfe906315e
|
feat: adding geoip parameter to the StealthyFetcher
|
2024-12-12 12:43:49 +02:00 |
|
|
Karim shoair
|
e254341974
|
fix: forgot to stage it with last commit
|
2024-12-11 21:43:36 +02:00 |
|
|
Karim shoair
|
193827e27b
|
refactor(api)!: Unifying log under 1 logger and removing debug parameter
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
|
2024-12-11 21:41:37 +02:00 |
|
|
Karim shoair
|
9f0001af26
|
feat: logging for response status
|
2024-12-11 20:14:32 +02:00 |
|
|
Karim shoair
|
b4f90615c3
|
Adding keep_cdata argument for Adaptor and Response classes
This will force lxml to keep cdata while parsing html if you want
|
2024-12-10 22:06:08 +02:00 |
|
|
Karim shoair
|
d0f18951ac
|
fix: Enable WebGL by default
|
2024-12-04 23:38:30 +02:00 |
|
|
Karim shoair
|
1a17b2ce46
|
Sorting all library imports with isort
|
2024-11-30 14:02:06 +02:00 |
|
|
Karim shoair
|
97fecc9d5f
|
Style adjustments by flake8
|
2024-11-29 21:14:52 +02:00 |
|
|
Karim shoair
|
06a47f9644
|
Small fix for the ResponseEncoding class
|
2024-11-26 22:55:22 +02:00 |
|
|
Karim shoair
|
28b783e86b
|
Improving the stealth mode a bit for Playwright
|
2024-11-26 22:41:04 +02:00 |
|
|
Karim shoair
|
14ffdbd6fc
|
Handling problematic encodings
|
2024-11-26 22:27:05 +02:00 |
|
|
Karim shoair
|
dbd674aa7c
|
Adding proxy support for basic requests
|
2024-11-26 21:24:41 +02:00 |
|
|
Karim shoair
|
1cf7cee1a2
|
Fetchers - Wait for page to be stable again if the user wanted to wait for a selector
|
2024-11-26 20:26:54 +02:00 |
|
|
Karim shoair
|
d6a59774ea
|
StealthyFetcher - The option to disable ads with ublock origin addon
|
2024-11-26 20:23:02 +02:00 |
|
|
Karim shoair
|
60d551d839
|
PlaywrightFetcher - Add the option to set locale of the browser
|
2024-11-26 17:39:03 +02:00 |
|
|
Karim shoair
|
0176bc0523
|
PlaywrightFetcher - Make proxies available without enabling stealth
Thanks for @AbdullahY36
|
2024-11-26 16:48:35 +02:00 |
|
|
Karim shoair
|
e96e7d4440
|
Adding the option to use real chrome browser directly
|
2024-11-24 15:14:42 +02:00 |
|
|
Karim shoair
|
088117441d
|
Making the type hints more accurate before update
|
2024-11-24 13:49:31 +02:00 |
|
|
Karim shoair
|
15b7e3acee
|
Stop using generated headers by browserforge
It turns out its headers format is pretty outdated, which made Google reCaptcha not work sometimes
|
2024-11-24 13:47:21 +02:00 |
|
|
Karim shoair
|
7f330efc52
|
Turn-off the hide_canvas option by default
|
2024-11-24 13:44:16 +02:00 |
|
|
Karim shoair
|
58151c12e8
|
Update the minimum browser version for headers generated
|
2024-11-24 13:41:06 +02:00 |
|
|
Karim shoair
|
1929d9bebb
|
Fix for docs
|
2024-11-23 17:48:05 +02:00 |
|
|
Karim shoair
|
e51556f80c
|
Merge branch 'main' into docs
|
2024-11-23 16:48:22 +02:00 |
|
|
Karim shoair
|
073ab2c7e3
|
Universally handling different response encoding and charsets for all responses
Thanks for @andyfcx
|
2024-11-23 15:40:41 +02:00 |
|
|
Karim shoair
|
869412761a
|
Fix for the wait_selector argument resolving to more than 1 element
|
2024-11-22 22:15:52 +02:00 |
|
|
andyfcx
|
63112bfc50
|
fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes
|
2024-11-21 14:54:38 +08:00 |
|
|
Karim shoair
|
3f39bd52d4
|
Merge branch 'main' into docs
|
2024-11-20 14:36:30 +02:00 |
|
|
Karim shoair
|
19ad82caef
|
Calculate status text manually if it's not returned by PlayWright API
|
2024-11-20 13:27:18 +02:00 |
|
|
Karim shoair
|
4c74d9bc97
|
Fixing a bug with reading response bytes in playwright/camoufox when network_idle is used
PlayWright doesn't provide a way to get the response in bytes after all wait ends like `page.content()` so that's more efficient to do anyway
|
2024-11-20 12:33:54 +02:00 |
|
|
Karim shoair
|
f443d774c1
|
Fixing the doc-string to match Sphinx
|
2024-11-19 14:33:54 +02:00 |
|
|
Karim shoair
|
b60ac7f56a
|
Change debug mode to be off by default
|
2024-11-16 21:11:58 +02:00 |
|
|
Karim shoair
|
572df6b3b4
|
Fixing the way Response object handles sub items in some edge cases
+ Do it with the least impact on performance
|
2024-11-16 13:54:55 +02:00 |
|
|
Karim shoair
|
90e38af552
|
Correcting the return string for the Response object in all functions/methods
|
2024-11-15 16:44:47 +02:00 |
|
|
Karim shoair
|
5e8275c3eb
|
Adding the option to randomize the OS fingerprints with the StealthyFetcher
|
2024-11-15 16:36:32 +02:00 |
|
|
Karim shoair
|
2abb70218f
|
Adding the proxy support to browser-based Fetchers
|
2024-11-15 14:43:03 +02:00 |
|
|
Karim shoair
|
095d50cd75
|
Making the Response returned from all fetchers same as Adaptor object
but with added attributes
|
2024-11-15 02:23:14 +02:00 |
|
|
Karim shoair
|
ab7b0d45ad
|
StaticEngine - Fix the headers bug
|
2024-11-13 23:13:39 +02:00 |
|
|
Karim shoair
|
b3418f7f50
|
Corrections and rephrasing
|
2024-11-11 16:40:35 +02:00 |
|
|
Karim shoair
|
d6b454dc43
|
Corrections and rephrasing
|
2024-11-11 16:40:30 +02:00 |
|
|
Karim shoair
|
eb100e45f7
|
Corrections and rephrasing
|
2024-11-11 16:40:21 +02:00 |
|
|
Karim shoair
|
239e2c73bf
|
Correctly get page content after JS Execution
- Thanks for @AbdullahY36 for the heads-up.
**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
|
2024-11-10 16:11:46 +02:00 |
|
|
Karim shoair
|
43045e3eb2
|
Get page content after JS Execution
Thanks for @AbdullahY36
|
2024-11-10 15:54:44 +02:00 |
|