330d03559c
style: applying the new ruff rules to all files
2025-09-13 03:22:53 +03:00
2f402f4835
style: add flags for tests coverage
...
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
2025-08-17 01:02:14 +03:00
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
ecef751468
fix: small fix for fetcher
2025-07-14 02:14:54 +03:00
6d79927233
refactor(Fetcher): Fix the issue of caching impersonation state + Less code duplication
...
It now creates a session with each request, but at the same time, it's still faster than the fetcher in v0.2.99 by 20% with more features enabled.
2025-07-06 02:21:15 +03:00
724e3a648e
fix(FetcherSession): Use Sentinel pattern to solve arguments precedence issue
2025-06-25 03:53:54 +03:00
9c44ad1930
fix(AsyncFetcher): Fix stealthy_headers issue
2025-06-22 03:54:36 +03:00
d049f63404
fix(fetcher): Fix impersonate and headers generation conflict
2025-06-20 03:29:05 +03:00
1f34ac22f6
style(fetcher): Move the default UA line
2025-06-15 03:21:21 +03:00
178eed7533
feat/fix(Fetcher): Adding http3 support + Fix stealthy_headers
...
And some fixes here and there to the docs
2025-05-31 03:35:46 +03:00
9e68e60161
feat(Fetcher): Replacing httpx + Adding FetcherSession
...
Check out the discord server for details
2025-05-25 21:10:20 +03:00
c84143129e
fix(fetchers): Adjusting all cookies returned from each fetcher
...
The cookies returned now can be passed again to each fetcher's engine without further validation by us.
2025-05-10 19:16:34 +03:00
1240ee3eb0
fix(Fetcher): Fix the issue of passing cookies
2025-04-29 03:01:19 +03:00
0c8dd63f87
chore: migrating to ruff and updating pre-commit hooks
2025-04-13 17:32:00 +02:00
ff3962b1ad
fix: Adjusting cache size in all the library
2025-03-17 06:46:41 +02:00
7bef44ea4c
fix: Limit caching in all the library
...
To lower memory usage with high numbers of requests. Possibly improve case #48
2025-03-15 02:01:16 +02:00
b5c7a8c36d
refactor/fix(Fetcher engine): Improvements and fixes
...
- Added the retries argument to the async methods, I don't understand how I missed it but that's why the code is rewritten now for easier maintenance.
2025-03-15 01:46:38 +02:00
5f4ec3ba3a
fix(Fetcher): make stealthy_headers don't overwrite user headers
...
Fix for #39
2025-02-25 23:14:59 +02:00
3d2418b67a
feat(Fetcher): Add the redirections history to Fetcher
...
Feature requested in #32
2025-02-22 16:58:54 +02:00
b10cfd3643
feat: Adding AsyncFetcher class version of Fetcher
...
The first step in fully supporting async
2024-12-15 16:51:58 +02:00
faf728a794
style: moving repeated arguments from inside the functions to __init__
...
Might give a slight performance increase too
2024-12-15 16:49:48 +02:00
445af3c702
perf: Give repeated usage of Fetcher a slight performance increase
...
By caching the `StaticEngine` class instance
2024-12-15 16:13:08 +02:00
299793af3f
feat: adding the retries argument for all methods of Fetcher class
2024-12-15 15:59:15 +02:00
e254341974
fix: forgot to stage it with last commit
2024-12-11 21:43:36 +02:00
193827e27b
refactor(api)!: Unifying log under 1 logger and removing debug parameter
...
So now you control the logging and the debugging from the shell through the logger with the name 'scrapling'
2024-12-11 21:41:37 +02:00
1a17b2ce46
Sorting all library imports with isort
2024-11-30 14:02:06 +02:00
dbd674aa7c
Adding proxy support for basic requests
2024-11-26 21:24:41 +02:00
f443d774c1
Fixing the doc-string to match Sphinx
2024-11-19 14:33:54 +02:00
572df6b3b4
Fixing the way Response object handles sub items in some edge cases
...
+ Do it with the least impact on performance
2024-11-16 13:54:55 +02:00
90e38af552
Correcting the return string for the Response object in all functions/methods
2024-11-15 16:44:47 +02:00
ab7b0d45ad
StaticEngine - Fix the headers bug
2024-11-13 23:13:39 +02:00
b3418f7f50
Corrections and rephrasing
2024-11-11 16:40:35 +02:00
282b130864
Static Engine - Doc strings and type annotations stuff
2024-11-03 22:25:34 +02:00
afe53332b8
Convert the request headers to dictionary to unify response object shape across engines
2024-11-03 13:01:48 +02:00
145c03daff
Big structure changes (check commit description)
...
- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
2024-11-03 01:04:02 +02:00
8f8d08b568
Adding browsing engines
2024-10-31 02:11:33 +03:00
Karim shoair