c8b9a04e87
docs: making some adjustments
2025-11-14 02:22:59 +02:00
d018d6e438
fix: fixing the response logic
2025-11-10 03:03:05 +02:00
ff19dd5bf5
feat: replace raw_response property with smart content detection
2025-11-10 02:39:45 +02:00
df073c4680
fix: added raw_response to separate collected response from rendered response
2025-11-08 20:09:40 +02:00
59ec6be201
refactor(fetchers): Less duplicated code and better handling for unstable websites
...
- Websites that never finish loading their requests won't crash the code now if you used `network_idle` with them
- Fixed a typo in `load_dom` in DynamicSession's async_fetch
- Removed dead code
2025-10-26 05:04:19 +03:00
74fa45daea
fix: Addressing the issue of collecting response after page_action in #100
2025-10-26 00:41:45 +03:00
073c4910d4
fix(fetchers): More accurate collection of the final response
...
This will make the `response.body` return the raw response for text/json responses as well
2025-10-12 04:12:18 +03:00
debe03256b
refactor: Making all the codebase acceptable by PyRight
...
Also fixes #97
2025-10-05 04:03:39 +03:00
a9d05cc0ef
style: Removing dead code/docstrings and correcting type hints
2025-09-19 04:49:30 +03:00
f848e6b7f2
fix: Fixes to handle the new extra deps group
2025-09-15 00:19:19 +03:00
67ca139ff9
fix: Fixes for multiple encoding issues ( #80 & #81 )
2025-09-14 20:20:29 +03:00
13d7e70cb7
refactor: Make all fetchers as an optional dependency group
...
+ Removing some dead code
2025-09-13 16:07:48 +03:00
330d03559c
style: applying the new ruff rules to all files
2025-09-13 03:22:53 +03:00
967c0956a1
feat(DynamicSession): Improve stealth mode
2025-08-31 02:57:12 +03:00
2f402f4835
style: add flags for tests coverage
...
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
2025-08-17 01:02:14 +03:00
825896c6a8
fix: improve checking for valid proxy and valid CDP URL
2025-08-03 16:31:16 +03:00
5bb1266fa5
style: using isinstance function as the main way for type checking
2025-07-30 03:14:23 +03:00
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
e7cdd39695
style: replacing os with Pathlib and small optimizations
2025-07-30 01:15:28 +03:00
8ca940d768
style: removing dead code
2025-07-29 23:43:31 +03:00
b9c7a5af2e
refactor: replace's Selector inpt (text/body) with 1 argument called content
2025-07-29 06:08:15 +03:00
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
8d44e9c51c
fix(proxy): fix proxy unpacking
2025-06-26 15:49:53 +03:00
3754547e58
refactor: removing unused code
2025-06-23 03:20:49 +03:00
9bfc900140
feat(navigation tools): Improvements
2025-06-19 04:07:45 +03:00
2bdee60004
fix(fingerprints): Fixing headers generation for requests
2025-06-15 03:37:22 +03:00
7efbcd33c3
refactor(fingerprints): Remove unused function and improve another
2025-06-15 03:16:27 +03:00
6e837f6a12
refactor(StealthyFetcher): Improve OS name extraction
2025-06-14 18:57:31 +03:00
178eed7533
feat/fix(Fetcher): Adding http3 support + Fix stealthy_headers
...
And some fixes here and there to the docs
2025-05-31 03:35:46 +03:00
c46ca8873f
refactor(fetchers): Optimizing fetchers + making PlayWrightFetcher 10% faster
...
Check out the Discord server for full details
2025-05-25 21:14:24 +03:00
c84143129e
fix(fetchers): Adjusting all cookies returned from each fetcher
...
The cookies returned now can be passed again to each fetcher's engine without further validation by us.
2025-05-10 19:16:34 +03:00
0c8dd63f87
chore: migrating to ruff and updating pre-commit hooks
2025-04-13 17:32:00 +02:00
811cd2d9b0
refactor: make auto_match disabled by default
2025-03-26 19:25:19 +02:00
fec8215631
docs: Clearer docs for recent changes
2025-03-26 05:56:49 +02:00
7c0c23fd33
feat(unified import): Make one import for all fetchers
...
Why I didn't think of that from the start? IDK
2025-03-26 02:48:34 +02:00
ff3962b1ad
fix: Adjusting cache size in all the library
2025-03-17 06:46:41 +02:00
7bef44ea4c
fix: Limit caching in all the library
...
To lower memory usage with high numbers of requests. Possibly improve case #48
2025-03-15 02:01:16 +02:00
3d2418b67a
feat(Fetcher): Add the redirections history to Fetcher
...
Feature requested in #32
2025-02-22 16:58:54 +02:00
6652b8ac18
fix: solve the response log issue so now you get log for every response from Fetchers
2024-12-19 12:29:34 +02:00
c282cb3cb7
fix: Stopped the log spamming that happens with multiple instances creation
2024-12-16 13:35:48 +02:00
d66ebc19a5
fix(Fetchers/disable_resources): Fixing the logic for intercepting resources
2024-12-16 12:35:25 +02:00
af4f2c0f74
feat(PlaywrightFetcher): Add async support for PlaywrightFetcher
2024-12-16 00:15:47 +02:00
889c111855
refactor(Playwright Engine): Separate what we can for cleaner code and the async function later
...
The caching will give a slight performance increase with bulk requests
2024-12-15 23:25:02 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
9f0001af26
feat: logging for response status
2024-12-11 20:14:32 +02:00
b4f90615c3
Adding keep_cdata argument for Adaptor and Response classes
...
This will force lxml to keep cdata while parsing html if you want
2024-12-10 22:06:08 +02:00
1a17b2ce46
Sorting all library imports with isort
2024-11-30 14:02:06 +02:00
97fecc9d5f
Style adjustments by flake8
2024-11-29 21:14:52 +02:00
06a47f9644
Small fix for the ResponseEncoding class
2024-11-26 22:55:22 +02:00
14ffdbd6fc
Handling problematic encodings
2024-11-26 22:27:05 +02:00
Karim shoair