14ffdbd6fc
Handling problematic encodings
2024-11-26 22:27:05 +02:00
d3fbcab3df
Pumping version up to 0.2.7
2024-11-26 21:27:22 +02:00
dbd674aa7c
Adding proxy support for basic requests
2024-11-26 21:24:41 +02:00
1cf7cee1a2
Fetchers - Wait for page to be stable again if the user wanted to wait for a selector
2024-11-26 20:26:54 +02:00
d6a59774ea
StealthyFetcher - The option to disable ads with ublock origin addon
2024-11-26 20:23:02 +02:00
60d551d839
PlaywrightFetcher - Add the option to set locale of the browser
2024-11-26 17:39:03 +02:00
0176bc0523
PlaywrightFetcher - Make proxies available without enabling stealth
...
Thanks for @AbdullahY36
2024-11-26 16:48:35 +02:00
438867e0be
Pumping the version up to 0.2.6
2024-11-24 15:21:58 +02:00
12d638ac2e
Updating the doc to reflect new changes
2024-11-24 15:20:18 +02:00
a8c576ad28
Adding the real_chrome option to the fetcher
2024-11-24 15:15:08 +02:00
e96e7d4440
Adding the option to use real chrome browser directly
2024-11-24 15:14:42 +02:00
088117441d
Making the type hints more accurate before update
2024-11-24 13:49:31 +02:00
15b7e3acee
Stop using generated headers by browserforge
...
It turns out its headers format is pretty outdated, which made Google reCaptcha not work sometimes
2024-11-24 13:47:21 +02:00
7f330efc52
Turn-off the hide_canvas option by default
2024-11-24 13:44:16 +02:00
58151c12e8
Update the minimum browser version for headers generated
2024-11-24 13:41:06 +02:00
1929d9bebb
Fix for docs
2024-11-23 17:48:05 +02:00
e51556f80c
Merge branch 'main' into docs
2024-11-23 16:48:22 +02:00
2b925b4a33
Pumping version up to 0.2.5
2024-11-23 16:06:11 +02:00
073ab2c7e3
Universally handling different response encoding and charsets for all responses
...
Thanks for @andyfcx
2024-11-23 15:40:41 +02:00
869412761a
Fix for the wait_selector argument resolving to more than 1 element
2024-11-22 22:15:52 +02:00
63112bfc50
fix: make sure to default utf-8 as encoding and strip unwanted single or double quotes
2024-11-21 14:54:38 +08:00
3f39bd52d4
Merge branch 'main' into docs
2024-11-20 14:36:30 +02:00
17d7934484
Pump version to 0.2.4
2024-11-20 13:27:39 +02:00
19ad82caef
Calculate status text manually if it's not returned by PlayWright API
2024-11-20 13:27:18 +02:00
4c74d9bc97
Fixing a bug with reading response bytes in playwright/camoufox when network_idle is used
...
PlayWright doesn't provide a way to get the response in bytes after all wait ends like `page.content()` so that's more efficient to do anyway
2024-11-20 12:33:54 +02:00
526c516693
Pump version up to 0.2.3
2024-11-20 01:28:01 +02:00
f443d774c1
Fixing the doc-string to match Sphinx
2024-11-19 14:33:54 +02:00
b60ac7f56a
Change debug mode to be off by default
2024-11-16 21:11:58 +02:00
422c135aae
Adding a file import initialized fetchers for cleaner looking code
...
and lazy people like me :)
2024-11-16 21:10:26 +02:00
cefc95cc3d
Pump version up to 0.2.2
2024-11-16 13:57:33 +02:00
572df6b3b4
Fixing the way Response object handles sub items in some edge cases
...
+ Do it with the least impact on performance
2024-11-16 13:54:55 +02:00
90e38af552
Correcting the return string for the Response object in all functions/methods
2024-11-15 16:44:47 +02:00
5e8275c3eb
Adding the option to randomize the OS fingerprints with the StealthyFetcher
2024-11-15 16:36:32 +02:00
2abb70218f
Adding the proxy support to browser-based Fetchers
2024-11-15 14:43:03 +02:00
7dd9d3090d
Pumping the version up to 0.2.1
2024-11-15 02:43:29 +02:00
095d50cd75
Making the Response returned from all fetchers same as Adaptor object
...
but with added attributes
2024-11-15 02:23:14 +02:00
b0c2b1818b
Return to the default text behaviour
...
The logic to remove comment tags before returning text was causing errors sometimes so from now on leave `keep_comments` set to True better as it is.
2024-11-14 22:21:45 +02:00
eaa7da27c6
Better logic to handle json responses
2024-11-14 00:03:26 +02:00
d4b896b4e9
Parser - New logic to handle JSON responses passed from Fetchers
2024-11-13 23:14:27 +02:00
ab7b0d45ad
StaticEngine - Fix the headers bug
2024-11-13 23:13:39 +02:00
b2993d2fcf
Update and rename py.typeda to py.typed
2024-11-13 21:56:02 +13:00
067809974e
Rename py.typed to py.typeda
2024-11-13 21:55:51 +13:00
bd38fdb347
Corrections and rephrasing
2024-11-11 16:40:46 +02:00
b3418f7f50
Corrections and rephrasing
2024-11-11 16:40:35 +02:00
d6b454dc43
Corrections and rephrasing
2024-11-11 16:40:30 +02:00
eb100e45f7
Corrections and rephrasing
2024-11-11 16:40:21 +02:00
3d4b427168
Rework Selectors generation
...
- Renamed both `css_selector` and `xpath_selector` both to `generate_css_selector` and `generate_xpath_selector` for clarity and not interrupting the auto-completion while coding.
- Added `generate_full_css_selector` and `generate_full_xpath_selector` methods
2024-11-10 18:19:48 +02:00
239e2c73bf
Correctly get page content after JS Execution
...
- Thanks for @AbdullahY36 for the heads-up.
**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
2024-11-10 16:11:46 +02:00
43045e3eb2
Get page content after JS Execution
...
Thanks for @AbdullahY36
2024-11-10 15:54:44 +02:00
4b2814404a
Fixing the functions-based filtering logic
2024-11-09 22:32:05 +02:00
Karim shoair