f80e8828f7
refactor: internal API changes
2026-01-08 01:53:48 +02:00
6a73f9dcd7
refactor: internal API changes to be easily used as indicators for spiders
2026-01-08 00:52:06 +02:00
04a612e64e
docs: fix docstrings for all requests methods
2025-11-25 21:05:21 +02:00
5376c97996
fix(fetcher): Remove non-keywords arguments
2025-11-25 20:25:09 +02:00
74f24d60f6
fix(fetcher): Replace vars logic with one that doesn't require dict attribute
2025-11-24 13:04:12 +02:00
c487faf4fd
pref(requests): Optimizing memory by specifiying slots
2025-11-23 21:50:54 +02:00
357b170e1c
refactor(requests): Make all the type hints dynamic
...
Made the code shorter by about ~500 lines and easier to maintain in return for making the arguments autocompletion bad for shells that don't check for dynamic type hints like IPython.
2025-11-23 20:48:51 +02:00
3565f9d500
feat(fetcher): Make impersonate able to randomize fingerprint
2025-11-16 16:33:58 +02:00
0dffa3bbba
docs: removing unnecessary docstrings
...
They are internal functions, not user-facing ones
2025-10-06 14:24:16 +03:00
9cae97b177
fix(requests): Fix multiple configuration issues
2025-10-06 06:53:09 +03:00
e53ce3f749
refactor: Making the requests logic better and cleaner
...
Also, solve some bugs along the way and a cleaner fix for #97
2025-10-06 05:00:41 +03:00
debe03256b
refactor: Making all the codebase acceptable by PyRight
...
Also fixes #97
2025-10-05 04:03:39 +03:00
5127d386c9
style(Fetcher): correcting type hints and annotation
2025-10-01 06:18:23 +03:00
849913ad77
style: Correcting return type annotation for Fetcher/AsyncFetcher ( #93 )
2025-10-01 04:17:59 +03:00
a9d05cc0ef
style: Removing dead code/docstrings and correcting type hints
2025-09-19 04:49:30 +03:00
f848e6b7f2
fix: Fixes to handle the new extra deps group
2025-09-15 00:19:19 +03:00
13d7e70cb7
refactor: Make all fetchers as an optional dependency group
...
+ Removing some dead code
2025-09-13 16:07:48 +03:00
330d03559c
style: applying the new ruff rules to all files
2025-09-13 03:22:53 +03:00
2f402f4835
style: add flags for tests coverage
...
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
2025-08-17 01:02:14 +03:00
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
ecef751468
fix: small fix for fetcher
2025-07-14 02:14:54 +03:00
6d79927233
refactor(Fetcher): Fix the issue of caching impersonation state + Less code duplication
...
It now creates a session with each request, but at the same time, it's still faster than the fetcher in v0.2.99 by 20% with more features enabled.
2025-07-06 02:21:15 +03:00
724e3a648e
fix(FetcherSession): Use Sentinel pattern to solve arguments precedence issue
2025-06-25 03:53:54 +03:00
9c44ad1930
fix(AsyncFetcher): Fix stealthy_headers issue
2025-06-22 03:54:36 +03:00
d049f63404
fix(fetcher): Fix impersonate and headers generation conflict
2025-06-20 03:29:05 +03:00
1f34ac22f6
style(fetcher): Move the default UA line
2025-06-15 03:21:21 +03:00
178eed7533
feat/fix(Fetcher): Adding http3 support + Fix stealthy_headers
...
And some fixes here and there to the docs
2025-05-31 03:35:46 +03:00
9e68e60161
feat(Fetcher): Replacing httpx + Adding FetcherSession
...
Check out the discord server for details
2025-05-25 21:10:20 +03:00
c84143129e
fix(fetchers): Adjusting all cookies returned from each fetcher
...
The cookies returned now can be passed again to each fetcher's engine without further validation by us.
2025-05-10 19:16:34 +03:00
1240ee3eb0
fix(Fetcher): Fix the issue of passing cookies
2025-04-29 03:01:19 +03:00
0c8dd63f87
chore: migrating to ruff and updating pre-commit hooks
2025-04-13 17:32:00 +02:00
ff3962b1ad
fix: Adjusting cache size in all the library
2025-03-17 06:46:41 +02:00
7bef44ea4c
fix: Limit caching in all the library
...
To lower memory usage with high numbers of requests. Possibly improve case #48
2025-03-15 02:01:16 +02:00
b5c7a8c36d
refactor/fix(Fetcher engine): Improvements and fixes
...
- Added the retries argument to the async methods, I don't understand how I missed it but that's why the code is rewritten now for easier maintenance.
2025-03-15 01:46:38 +02:00
5f4ec3ba3a
fix(Fetcher): make stealthy_headers don't overwrite user headers
...
Fix for #39
2025-02-25 23:14:59 +02:00
3d2418b67a
feat(Fetcher): Add the redirections history to Fetcher
...
Feature requested in #32
2025-02-22 16:58:54 +02:00
b10cfd3643
feat: Adding AsyncFetcher class version of Fetcher
...
The first step in fully supporting async
2024-12-15 16:51:58 +02:00
faf728a794
style: moving repeated arguments from inside the functions to __init__
...
Might give a slight performance increase too
2024-12-15 16:49:48 +02:00
445af3c702
perf: Give repeated usage of Fetcher a slight performance increase
...
By caching the `StaticEngine` class instance
2024-12-15 16:13:08 +02:00
299793af3f
feat: adding the retries argument for all methods of Fetcher class
2024-12-15 15:59:15 +02:00
e254341974
fix: forgot to stage it with last commit
2024-12-11 21:43:36 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
1a17b2ce46
Sorting all library imports with isort
2024-11-30 14:02:06 +02:00
dbd674aa7c
Adding proxy support for basic requests
2024-11-26 21:24:41 +02:00
f443d774c1
Fixing the doc-string to match Sphinx
2024-11-19 14:33:54 +02:00
572df6b3b4
Fixing the way Response object handles sub items in some edge cases
...
+ Do it with the least impact on performance
2024-11-16 13:54:55 +02:00
90e38af552
Correcting the return string for the Response object in all functions/methods
2024-11-15 16:44:47 +02:00
ab7b0d45ad
StaticEngine - Fix the headers bug
2024-11-13 23:13:39 +02:00
b3418f7f50
Corrections and rephrasing
2024-11-11 16:40:35 +02:00
Karim shoair