 Karim shoair
|
ecef751468
|
fix: small fix for fetcher
|
2025-07-14 02:14:54 +03:00 |
|
|
Karim shoair
|
0e5fd502d2
|
Merge branch 'main' into dev
|
2025-07-11 02:44:10 +03:00 |
|
|
Karim shoair
|
80e3a97990
|
docs: Update README.md
|
2025-07-10 21:19:00 +03:00 |
|
|
Karim shoair
|
51b5c4dff8
|
Merge branch 'main' into dev
|
2025-07-10 21:17:39 +03:00 |
|
|
Karim shoair
|
377ad92485
|
docs: update PetroSky promo
|
2025-07-10 21:14:34 +03:00 |
|
|
Karim shoair
|
6eb9626865
|
docs: add PetroSky as sponsor
|
2025-07-10 21:12:54 +03:00 |
|
|
Karim shoair
|
8af4833c64
|
chore: add new sponsor poster
|
2025-07-10 21:00:06 +03:00 |
|
|
Karim shoair
|
96be8af1b3
|
chore: add new sponsor poster
|
2025-07-10 20:56:20 +03:00 |
|
|
Karim shoair
|
1c6a960f17
|
Merge branch 'main' into dev
|
2025-07-09 19:39:08 +03:00 |
|
|
Karim shoair
|
119e00e16d
|
chore: Delete the "others" issue template
|
2025-07-09 19:30:06 +03:00 |
|
|
Karim shoair
|
fe8b11155e
|
test: remove tests for Python 3.9
|
2025-07-09 16:31:05 +03:00 |
|
|
Karim shoair
|
4bba8598c2
|
build: update deps
|
2025-07-09 16:20:21 +03:00 |
|
|
Karim shoair
|
403f7bfe72
|
build: Updating PyPI classifiers
|
2025-07-07 00:10:20 +03:00 |
|
|
Karim shoair
|
939b78f3cf
|
build: adding mcp as dependency
|
2025-07-07 00:09:54 +03:00 |
|
|
Karim shoair
|
3184dd8581
|
build: set minimum version to Python 3.10
|
2025-07-07 00:09:02 +03:00 |
|
|
Karim shoair
|
2b73fdbfd7
|
fix(DynamicFetcher): Remove old argument
|
2025-07-06 05:02:16 +03:00 |
|
|
Karim shoair
|
2ecf9b0c20
|
build: Improve description
|
2025-07-06 05:01:19 +03:00 |
|
|
Karim shoair
|
63a9db21a1
|
refactor(DynamicSession): Optimization + Removing max_pages from sync version
|
2025-07-06 04:59:22 +03:00 |
|
|
Karim shoair
|
6d79927233
|
refactor(Fetcher): Fix the issue of caching impersonation state + Less code duplication
It now creates a session with each request, but at the same time, it's still faster than the fetcher in v0.2.99 by 20% with more features enabled.
|
2025-07-06 02:21:15 +03:00 |
|
|
Karim shoair
|
18e56a8c50
|
refactor: Optimizing extract command and cleaning code
|
2025-06-29 19:40:03 +03:00 |
|
|
Karim shoair
|
29e485bdf2
|
refactor: Optimizations to CLI
|
2025-06-29 19:10:52 +03:00 |
|
|
Karim shoair
|
b28f90e854
|
build: Update deps
|
2025-06-29 18:58:36 +03:00 |
|
|
Karim shoair
|
1c9e48b1c6
|
feat(extract): Adding new command to CLI options + Optimizations
Users can now fetch websites directly without code and extract full/selected HTML content as HTML, Markdown, or extract text content.
|
2025-06-29 18:56:09 +03:00 |
|
|
Karim shoair
|
ce034f9f79
|
test: remove problematic test
|
2025-06-28 18:24:08 +03:00 |
|
|
Karim shoair
|
b02f0bd236
|
ops: Adjust tox file to avoid CI issues
|
2025-06-28 18:12:38 +03:00 |
|
|
Karim shoair
|
ac3db69c47
|
ops: new approach to CI tests workflow
|
2025-06-28 17:40:19 +03:00 |
|
|
Karim shoair
|
ee1a0828a2
|
ops: Correcting cache paths
|
2025-06-27 17:16:35 +03:00 |
|
|
Karim shoair
|
d603c15212
|
ops: rewrite tests logic to cache downloaded browsers before testing
|
2025-06-27 16:02:16 +03:00 |
|
|
Karim shoair
|
4ee32af67b
|
ops: fix for GitHub's CI
|
2025-06-27 14:50:06 +03:00 |
|
|
Karim shoair
|
3578a07c0a
|
ops: fix for test deps
|
2025-06-27 14:35:12 +03:00 |
|
|
Karim shoair
|
94edb92bc3
|
ops: update hashing for the cache
|
2025-06-27 14:19:07 +03:00 |
|
|
Karim shoair
|
941ed20a15
|
ops: switch tests to use MacOS
|
2025-06-27 03:19:58 +03:00 |
|
|
Karim shoair
|
d6f8926d7a
|
ops: switch tests to Windows system instead of Linux
|
2025-06-27 03:08:42 +03:00 |
|
|
Karim shoair
|
dbf7aa7761
|
feat(DynamicFetcher): Use persistent context by default for better stealth
And might solve the sandbox issue with GitHub
|
2025-06-27 03:02:46 +03:00 |
|
|
Karim shoair
|
76248eabf6
|
ops: possible fix for sandbox issue with GitHub's CI
|
2025-06-27 03:01:27 +03:00 |
|
|
Karim shoair
|
9b62b59490
|
ops: possible fix for sandbox issue with GitHub's CI
|
2025-06-27 02:26:35 +03:00 |
|
|
Karim shoair
|
e535a3fa16
|
ops: fix sandbox issue with GitHub's CI
|
2025-06-27 02:14:53 +03:00 |
|
|
Karim shoair
|
5895edef57
|
ops: fix test workflow for GitHub
|
2025-06-27 01:42:22 +03:00 |
|
|
Karim shoair
|
8395db9121
|
fix(controllers): Fix issue with async adding stealth scripts
|
2025-06-27 01:39:32 +03:00 |
|
|
Karim shoair
|
228d294afc
|
test: Enable Dynamic stealth tests again
It's problematic with GitHub but letsgo
|
2025-06-27 01:15:06 +03:00 |
|
|
Karim shoair
|
37e842b796
|
fix(controllers): Fix issue with adding stealth scripts
|
2025-06-27 01:04:58 +03:00 |
|
|
Karim shoair
|
53318f3a2a
|
fix(StealthyFetcher): Fix passed proxy type
|
2025-06-26 16:36:30 +03:00 |
|
|
Karim shoair
|
8d44e9c51c
|
fix(proxy): fix proxy unpacking
|
2025-06-26 15:49:53 +03:00 |
|
|
Karim shoair
|
38851698ad
|
refactor: optimize imports and docstrings correction
|
2025-06-25 23:15:32 +03:00 |
|
|
Karim shoair
|
a41be48047
|
refactor: optimize imports and docstrings correction
|
2025-06-25 21:36:53 +03:00 |
|
|
Karim shoair
|
a3a5e34408
|
refactor: optimize imports and docstrings correction
|
2025-06-25 21:36:43 +03:00 |
|
|
Karim shoair
|
724e3a648e
|
fix(FetcherSession): Use Sentinel pattern to solve arguments precedence issue
|
2025-06-25 03:53:54 +03:00 |
|
|
Karim shoair
|
63ff12bcd6
|
docs: correcting a small mistake
|
2025-06-23 03:38:34 +03:00 |
|
|
Karim shoair
|
78b81c537f
|
fix(fetchers): Fix the bug of referer and google_search argument conflict
|
2025-06-23 03:36:11 +03:00 |
|
|
Karim shoair
|
560ae95281
|
test: Adjusting tests with the correct timeout
|
2025-06-23 03:24:13 +03:00 |
|
|
Karim shoair
|
69b983b42f
|
feat(fetchers): Improve StealthyFetcher + Adding StealthySession/AsyncStealthySession classes
|
2025-06-23 03:23:28 +03:00 |
|
|
Karim shoair
|
3754547e58
|
refactor: removing unused code
|
2025-06-23 03:20:49 +03:00 |
|
|
Karim shoair
|
fc530677de
|
refactor(DynamicFetcher): Cleaner exit for fetch
|
2025-06-23 02:58:01 +03:00 |
|
|
Karim shoair
|
22490db85f
|
fix(DynamicFetcher): Set number of tabs to 1
There is no logical use for it here; seek the session classes to use it.
|
2025-06-23 02:42:43 +03:00 |
|
|
Karim shoair
|
460b4443c2
|
refactor(controllers): Improve type validation
|
2025-06-22 19:59:40 +03:00 |
|
|
Karim shoair
|
0fe04e499f
|
refactor(validators): Optimize imports
|
2025-06-22 18:55:13 +03:00 |
|
|
Karim shoair
|
fc42487a5f
|
refactor(controllers): Optimize imports
|
2025-06-22 18:53:44 +03:00 |
|
|
Karim shoair
|
4769a43525
|
test: Fix for Github CI
|
2025-06-22 04:02:19 +03:00 |
|
|
Karim shoair
|
9c44ad1930
|
fix(AsyncFetcher): Fix stealthy_headers issue
|
2025-06-22 03:54:36 +03:00 |
|
|
Karim shoair
|
8100a2a865
|
test: fix for Github CI
|
2025-06-22 03:07:46 +03:00 |
|
|
Karim shoair
|
8b46e3a3a5
|
tests: Updating tests according to the new changes
|
2025-06-22 02:12:39 +03:00 |
|
|
Karim shoair
|
f28b4d3653
|
docs: Updating Readme accordingly to the new naming
|
2025-06-22 02:12:03 +03:00 |
|
|
Karim shoair
|
b77c5b7abf
|
refactor: updating top level shortcuts
|
2025-06-22 02:11:26 +03:00 |
|
|
Karim shoair
|
8a272cf19b
|
feat/refactor(fetchers): Replacing PlayWrightFetcher with DynamicFetcher and adding session classes
|
2025-06-22 02:10:45 +03:00 |
|
|
Karim shoair
|
1e51fc9738
|
refactor: Removing defaults as said
|
2025-06-22 02:07:34 +03:00 |
|
|
Karim shoair
|
0a570a7ca7
|
docs: Updating doc strings
|
2025-06-22 01:59:38 +03:00 |
|
|
Karim shoair
|
d049f63404
|
fix(fetcher): Fix impersonate and headers generation conflict
|
2025-06-20 03:29:05 +03:00 |
|
|
Karim shoair
|
194ce24201
|
build: update deps
|
2025-06-19 15:19:23 +03:00 |
|
|
Karim shoair
|
9bfc900140
|
feat(navigation tools): Improvements
|
2025-06-19 04:07:45 +03:00 |
|
|
Karim shoair
|
145b710960
|
feat(fetchers): Adding the foundation of the new browser-based fetchers logic
|
2025-06-19 04:06:52 +03:00 |
|
|
Karim shoair
|
23b883f05a
|
build: Add msgspec to the deps
It will be used very soon
|
2025-06-19 04:04:25 +03:00 |
|
|
Karim shoair
|
3e63fa2523
|
fix(shell): Hide the automatic tips and solve the namespace issue
|
2025-06-19 04:03:13 +03:00 |
|
|
Karim shoair
|
2bdee60004
|
fix(fingerprints): Fixing headers generation for requests
|
2025-06-15 03:37:22 +03:00 |
|
|
Karim shoair
|
1f34ac22f6
|
style(fetcher): Move the default UA line
|
2025-06-15 03:21:21 +03:00 |
|
|
Karim shoair
|
7efbcd33c3
|
refactor(fingerprints): Remove unused function and improve another
|
2025-06-15 03:16:27 +03:00 |
|
|
Karim shoair
|
fcac10fa6d
|
build: update deps
|
2025-06-14 22:05:57 +03:00 |
|
|
Karim shoair
|
6e837f6a12
|
refactor(StealthyFetcher): Improve OS name extraction
|
2025-06-14 18:57:31 +03:00 |
|
|
Karim shoair
|
3068bb1356
|
build: update deps
|
2025-06-14 18:56:31 +03:00 |
|
|
Karim shoair
|
b55d97500b
|
build: Updating playwright deps
|
2025-06-01 00:06:08 +03:00 |
|
|
Karim shoair
|
bbdac4d967
|
build(setup): Replacing old setup style with new TOML format
|
2025-05-31 23:24:41 +03:00 |
|
|
Karim shoair
|
178eed7533
|
feat/fix(Fetcher): Adding http3 support + Fix stealthy_headers
And some fixes here and there to the docs
|
2025-05-31 03:35:46 +03:00 |
|
|
Karim shoair
|
af643e4c01
|
test: Updating old httpx tests for curl_cffi
|
2025-05-25 21:18:03 +03:00 |
|
|
Karim shoair
|
c46ca8873f
|
refactor(fetchers): Optimizing fetchers + making PlayWrightFetcher 10% faster
Check out the Discord server for full details
|
2025-05-25 21:14:24 +03:00 |
|
|
Karim shoair
|
9e68e60161
|
feat(Fetcher): Replacing httpx + Adding FetcherSession
Check out the discord server for details
|
2025-05-25 21:10:20 +03:00 |
|
|
Karim shoair
|
5c420b7384
|
Merge branch 'main' into dev
|
2025-05-23 20:29:05 +03:00 |
|
|
Karim shoair
|
561c8b0fcf
|
Chore: adding Swiftproxy ad
|
2025-05-23 14:28:02 +03:00 |
|
|
Karim shoair
|
f28e9c2b1e
|
Chore: adding Swiftproxy image
|
2025-05-23 14:19:08 +03:00 |
|
|
Karim shoair
|
f105518fd9
|
Merge branch 'main' into dev
|
2025-05-18 13:55:29 +03:00 |
|
|
Karim shoair
|
e7f8e5794a
|
fix(tests): fix for GitHub actions
|
2025-05-13 15:49:52 +03:00 |
|
|
Karim shoair
|
a941c02e1c
|
Update README.md
|
2025-05-13 15:42:02 +03:00 |
|
|
Karim shoair
|
43e3642719
|
Update Scrapeless segment
|
2025-05-13 15:40:49 +03:00 |
|
|
Karim shoair
|
fc535fa208
|
perf: drop w3lib dependency
|
2025-05-11 20:22:15 +03:00 |
|
|
Karim shoair
|
3268617240
|
test(cookies): Adjust cookies unit tests to the new changes
|
2025-05-10 20:08:31 +03:00 |
|
|
Karim shoair
|
bf72678480
|
feat(cookies): The ability to pass cookies to browser fetchers
|
2025-05-10 19:36:53 +03:00 |
|
|
Karim shoair
|
c84143129e
|
fix(fetchers): Adjusting all cookies returned from each fetcher
The cookies returned now can be passed again to each fetcher's engine without further validation by us.
|
2025-05-10 19:16:34 +03:00 |
|
|
Karim shoair
|
bd47c070aa
|
fix(StealthyFetcher): Adjustments
|
2025-05-10 17:49:03 +03:00 |
|
|
Karim shoair
|
cd3321a6f0
|
fix(tests): fix for GitHub actions
|
2025-05-08 03:11:01 +03:00 |
|
|
Karim shoair
|
fd33e35ff8
|
feat(StealthyFetcher): adding the option to solve Cloudflare Turnstile
|
2025-05-08 01:08:04 +03:00 |
|
|
Karim shoair
|
6c0b786dea
|
docs(playwrightfetcher): improving docstring
|
2025-05-08 01:04:08 +03:00 |
|
|
Karim shoair
|
74fa1bfbed
|
feat(shell): Add support to curl -b argument
|
2025-04-30 04:23:05 +03:00 |
|