03091beb8b
Small clarification
2024-11-09 13:40:50 +02:00
fd7d787ab0
Add extra_headers argument to PW and isolate google referer feature from stealth mode as google_search
2024-11-09 00:54:11 +02:00
ee8f78feb1
Update camo.py
2024-11-09 00:48:11 +02:00
fd34cd9afa
Adding arguments google_search and extra_headers to StealthyFetcher
2024-11-09 00:43:24 +02:00
530b54d964
Clearer doc-string
2024-11-09 00:17:41 +02:00
1b015bbd07
Type correction
2024-11-08 23:59:12 +02:00
b919536527
Updating the keywords whitelisting logic in find/find_all functions
2024-11-07 12:56:32 +02:00
91f6b46e6d
Adding new filter and search functions to parser
2024-11-07 00:31:14 +02:00
b25d2ee79b
Adding new find_all and find functions to parser
2024-11-07 00:01:14 +02:00
6b40af426f
Better logic for css_first and xpath_first
2024-11-06 22:14:33 +02:00
5e848ef046
Adding new class type TextHandlers
2024-11-06 21:03:19 +02:00
7a84511b7a
Making Adaptor and Adaptor re/re_first arguments consistent with the TextHandler ones
2024-11-06 20:47:21 +02:00
7af75f78a5
Better logic for Adaptors re_first method
2024-11-06 20:24:30 +02:00
a1fd5ddc7e
Correcting headless argument type
2024-11-06 20:18:44 +02:00
a0a3e62075
Adding addons and humanize arguments to StealthyFetcher
2024-11-06 20:14:42 +02:00
5ad2126030
Small speed boost for fetching by some caching
2024-11-06 19:56:39 +02:00
5548941cf1
Correcting return type for css_first and xpath_first
2024-11-06 12:28:43 +02:00
10bcdfb596
Moving bypasses folder deeper
2024-11-06 12:19:26 +02:00
6cf506cb22
PlaywrightEngine - Validate CDP URLs in all cases
2024-11-06 11:24:56 +02:00
6497b1b559
Type correction
2024-11-06 02:20:33 +02:00
3622040440
Adding css_first and xpath_first for easier usage
2024-11-05 23:41:51 +02:00
009c2914aa
Fixing re_first logic in Adaptors type
2024-11-05 22:59:47 +02:00
a60662af4a
Update camo.py
2024-11-05 11:38:35 +02:00
bd612bb0e2
Updating the disable_resources doc-string
2024-11-04 20:16:04 +02:00
544d23b696
Toolbelt fingerprint functions - Doc strings and type annotations stuff
2024-11-03 23:31:09 +02:00
e270b00811
Toolbelt subpackage custom functions general adjustments
2024-11-03 23:17:20 +02:00
2ff0fbb0e5
Engines utils - Navigation functions
...
- Adding doc strings and type annotations stuff.
- Making error messages more detailed.
- Renaming the `construct_websocket_url` function for more clarity.
2024-11-03 22:46:40 +02:00
b8afb390ae
Renaming fetcher file to fetchers for clarity
2024-11-03 22:43:16 +02:00
282b130864
Static Engine - Doc strings and type annotations stuff
2024-11-03 22:25:34 +02:00
3af174526f
PlayWright Engine - Doc strings and type annotations stuff
2024-11-03 22:24:44 +02:00
54cacb8a4e
Camoufox Engine - Doc strings and type annotations stuff
2024-11-03 22:20:29 +02:00
61fd592a6a
Adding documentation to PlaywrightFetcher class
2024-11-03 21:04:11 +02:00
c1575b8e2e
Updating doc-strings for more clarity
2024-11-03 16:49:33 +02:00
1d2c68125a
Update fetcher.py
2024-11-03 16:40:42 +02:00
3556d11fb2
StealthyFetcher - Add disable_resources option and its explanation
2024-11-03 16:33:08 +02:00
711de81106
PlaywrightFetcher - The option to drop unnecessary resources requests for speed boost
2024-11-03 16:27:55 +02:00
b32e1203f2
StealthyFetcher - The option to drop unnecessary resources requests for speed boost
2024-11-03 16:27:32 +02:00
c8c494e73f
Adding a function to intercept and drop unwanted requests
2024-11-03 16:25:55 +02:00
e7e499d64b
Add documentation for StealthyFetcher class + add webgl option
2024-11-03 14:45:20 +02:00
e7300ecba0
StealthFetcher - Add the option to allow webgl in the browser
2024-11-03 14:40:19 +02:00
2528d4e023
Add documentation for the Fetcher class
2024-11-03 14:39:24 +02:00
be350f21dd
Adding documentation to the base fetcher...
...
+ Configure logger from here.
+ Enable auto_match by default.
2024-11-03 13:45:13 +02:00
eaae5908c9
Make auto_match enabled by default
2024-11-03 13:33:24 +02:00
1ff1c93bcf
Give response's body bytes priority over response's text
2024-11-03 13:04:02 +02:00
afe53332b8
Convert the request headers to dictionary to unify response object shape across engines
2024-11-03 13:01:48 +02:00
942aa08c85
Handling JSON responses better
2024-11-03 13:00:23 +02:00
145c03daff
Big structure changes (check commit description)
...
- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
2024-11-03 01:04:02 +02:00
78c718c1d2
I don't know how that happened!
2024-11-02 21:17:56 +02:00
be2847ffb8
Bumping the library for version 0.2 for now and more adjustments
2024-10-31 02:14:41 +03:00
3cb1696d6c
Adding the main fetcher file to use for fetching pages
2024-10-31 02:13:35 +03:00
Karim shoair