6e837f6a12
refactor(StealthyFetcher): Improve OS name extraction
2025-06-14 18:57:31 +03:00
178eed7533
feat/fix(Fetcher): Adding http3 support + Fix stealthy_headers
...
And some fixes here and there to the docs
2025-05-31 03:35:46 +03:00
c46ca8873f
refactor(fetchers): Optimizing fetchers + making PlayWrightFetcher 10% faster
...
Check out the Discord server for full details
2025-05-25 21:14:24 +03:00
c84143129e
fix(fetchers): Adjusting all cookies returned from each fetcher
...
The cookies returned now can be passed again to each fetcher's engine without further validation by us.
2025-05-10 19:16:34 +03:00
0c8dd63f87
chore: migrating to ruff and updating pre-commit hooks
2025-04-13 17:32:00 +02:00
811cd2d9b0
refactor: make auto_match disabled by default
2025-03-26 19:25:19 +02:00
fec8215631
docs: Clearer docs for recent changes
2025-03-26 05:56:49 +02:00
7c0c23fd33
feat(unified import): Make one import for all fetchers
...
Why I didn't think of that from the start? IDK
2025-03-26 02:48:34 +02:00
ff3962b1ad
fix: Adjusting cache size in all the library
2025-03-17 06:46:41 +02:00
7bef44ea4c
fix: Limit caching in all the library
...
To lower memory usage with high numbers of requests. Possibly improve case #48
2025-03-15 02:01:16 +02:00
3d2418b67a
feat(Fetcher): Add the redirections history to Fetcher
...
Feature requested in #32
2025-02-22 16:58:54 +02:00
6652b8ac18
fix: solve the response log issue so now you get log for every response from Fetchers
2024-12-19 12:29:34 +02:00
c282cb3cb7
fix: Stopped the log spamming that happens with multiple instances creation
2024-12-16 13:35:48 +02:00
d66ebc19a5
fix(Fetchers/disable_resources): Fixing the logic for intercepting resources
2024-12-16 12:35:25 +02:00
af4f2c0f74
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
2024-12-16 00:15:47 +02:00
889c111855
refactor(Playwright Engine): Separate what we can for cleaner code and the async function later
...
The caching will give a slight performance increase with bulk requests
2024-12-15 23:25:02 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
9f0001af26
feat: logging for response status
2024-12-11 20:14:32 +02:00
b4f90615c3
Adding keep_cdata argument for Adaptor and Response classes
...
This will force lxml to keep cdata while parsing html if you want
2024-12-10 22:06:08 +02:00
1a17b2ce46
Sorting all library imports with isort
2024-11-30 14:02:06 +02:00
97fecc9d5f
Style adjustments by flake8
2024-11-29 21:14:52 +02:00
06a47f9644
Small fix for the ResponseEncoding class
2024-11-26 22:55:22 +02:00
14ffdbd6fc
Handling problematic encodings
2024-11-26 22:27:05 +02:00
58151c12e8
Update the minimum browser version for headers generated
2024-11-24 13:41:06 +02:00
1929d9bebb
Fix for docs
2024-11-23 17:48:05 +02:00
073ab2c7e3
Universally handling different response encoding and charsets for all responses
...
Thanks for @andyfcx
2024-11-23 15:40:41 +02:00
19ad82caef
Calculate status text manually if it's not returned by PlayWright API
2024-11-20 13:27:18 +02:00
b60ac7f56a
Change debug mode to be off by default
2024-11-16 21:11:58 +02:00
572df6b3b4
Fixing the way Response object handles sub items in some edge cases
...
+ Do it with the least impact on performance
2024-11-16 13:54:55 +02:00
2abb70218f
Adding the proxy support to browser-based Fetchers
2024-11-15 14:43:03 +02:00
095d50cd75
Making the Response returned from all fetchers same as Adaptor object
...
but with added attributes
2024-11-15 02:23:14 +02:00
239e2c73bf
Correctly get page content after JS Execution
...
- Thanks for @AbdullahY36 for the heads-up.
**Update reference** https://developer.mozilla.org/en-US/docs/Web/API/Document/DOMContentLoaded_event
2024-11-10 16:11:46 +02:00
b90c8adb02
Adding the argument automatch_domain to all fetchers
2024-11-09 15:38:27 +02:00
5ad2126030
Small speed boost for fetching by some caching
2024-11-06 19:56:39 +02:00
10bcdfb596
Moving bypasses folder deeper
2024-11-06 12:19:26 +02:00
6cf506cb22
PlaywrightEngine - Validate CDP URLs in all cases
2024-11-06 11:24:56 +02:00
544d23b696
Toolbelt fingerprint functions - Doc strings and type annotations stuff
2024-11-03 23:31:09 +02:00
e270b00811
Toolbelt subpackage custom functions general adjustments
2024-11-03 23:17:20 +02:00
2ff0fbb0e5
Engines utils - Navigation functions
...
- Adding doc strings and type annotations stuff.
- Making error messages more detailed.
- Renaming the `construct_websocket_url` function for more clarity.
2024-11-03 22:46:40 +02:00
c8c494e73f
Adding a function to intercept and drop unwanted requests
2024-11-03 16:25:55 +02:00
be350f21dd
Adding documentation to the base fetcher...
...
+ Configure logger from here.
+ Enable auto_match by default.
2024-11-03 13:45:13 +02:00
1ff1c93bcf
Give response's body bytes priority over response's text
2024-11-03 13:04:02 +02:00
145c03daff
Big structure changes (check commit description)
...
- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
2024-11-03 01:04:02 +02:00
Karim shoair