Karim shoair
9bcb9e9d93
style: General type hints fixes and imports optimizing
2025-07-29 23:43:06 +03:00
Karim shoair
3d07a1533e
refactor: cleaner code for the fetcher's lazy loader
2025-07-29 22:43:12 +03:00
Karim shoair
54cba6db45
ops: fix benchmarks script and make it more accurate
2025-07-29 22:36:47 +03:00
Karim shoair
9ce3279488
fix(TextHandler): Increase speed of clean method 5 times
2025-07-29 22:31:03 +03:00
Karim shoair
81a9eb068a
ops(tests workflow): Cache browsers on GitHub
2025-07-29 21:03:14 +03:00
Karim shoair
08cae510a6
ops: Cache keys adjustment for tests
2025-07-29 16:40:48 +03:00
Karim shoair
715dfb4243
feat: add length property to Selectors to write less code
2025-07-29 06:19:45 +03:00
Karim shoair
29b77a96c8
refactor(parser): optimize imports
2025-07-29 06:18:12 +03:00
Karim shoair
b9c7a5af2e
refactor: replace's Selector inpt (text/body) with 1 argument called content
2025-07-29 06:08:15 +03:00
Karim shoair
9e9ba9ab10
docs: update roadmap
2025-07-29 04:20:43 +03:00
Karim shoair
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
Karim shoair
0c649987f8
test: remove irrelevant test case
2025-07-28 15:10:23 +03:00
Karim shoair
a5e4b91653
refactor: remove clean up function for Adaptor + make adaptor attributes accessible directly
2025-07-28 14:48:22 +03:00
Karim shoair
297e14230b
refactor(parser): Multiple optimizations and fixes
2025-07-28 03:32:22 +03:00
Karim shoair
e35bade804
build: update bandit rules
2025-07-28 03:32:17 +03:00
Karim shoair
d220050160
refactor(parser): Make get_all_text method 40% faster
2025-07-28 03:29:13 +03:00
Karim shoair
7f11b6f59b
fix(storage): possible threading issue with recursion + optimizations
2025-07-28 00:16:58 +03:00
Karim shoair
3b0237e402
docs: improve All storage classes doc strings
2025-07-27 23:37:47 +03:00
Karim shoair
b60fcbb884
refactor(Adaptor): Cleaner approach to find_similar method
...
This code is slower than before by about 2-5μs, but it's worth it.
2025-07-27 23:34:22 +03:00
Karim shoair
ffde7e8ad9
fix(Adaptor): Add cleanup function to handle possible memory leak
2025-07-27 23:31:55 +03:00
Karim shoair
6ae1810405
docs: improve All Adaptor class doc strings
2025-07-27 22:41:37 +03:00
Karim shoair
93bd131bb6
fix(parser): Solve the ignored elements children issue while keeping speed
...
solves #61
2025-07-27 05:34:48 +03:00
Karim shoair
d1aa0be6e4
refactor(StealthyFetcher): Remove virtual mode and use persistent context
...
Solves #64 completely too
2025-07-27 03:41:28 +03:00
Karim shoair
172a5b4a0a
feat: Add an mcp server
2025-07-26 22:57:08 +03:00
Karim shoair
9d3b1335a9
build: update deps
2025-07-26 19:00:38 +03:00
Karim shoair
1076001e29
refactor: changes to be used by the mcp server
2025-07-14 05:38:29 +03:00
Karim shoair
f7e1b7261e
ops: adjust vermin to use 3.10 as the new minimum Python version
2025-07-14 05:38:16 +03:00
Karim shoair
ecef751468
fix: small fix for fetcher
2025-07-14 02:14:54 +03:00
Karim shoair
0e5fd502d2
Merge branch 'main' into dev
2025-07-11 02:44:10 +03:00
Karim shoair
80e3a97990
docs: Update README.md
2025-07-10 21:19:00 +03:00
Karim shoair
51b5c4dff8
Merge branch 'main' into dev
2025-07-10 21:17:39 +03:00
Karim shoair
377ad92485
docs: update PetroSky promo
2025-07-10 21:14:34 +03:00
Karim shoair
6eb9626865
docs: add PetroSky as sponsor
2025-07-10 21:12:54 +03:00
Karim shoair
8af4833c64
chore: add new sponsor poster
2025-07-10 21:00:06 +03:00
Karim shoair
96be8af1b3
chore: add new sponsor poster
2025-07-10 20:56:20 +03:00
Karim shoair
1c6a960f17
Merge branch 'main' into dev
2025-07-09 19:39:08 +03:00
Karim shoair
119e00e16d
chore: Delete the "others" issue template
2025-07-09 19:30:06 +03:00
Karim shoair
fe8b11155e
test: remove tests for Python 3.9
2025-07-09 16:31:05 +03:00
Karim shoair
4bba8598c2
build: update deps
2025-07-09 16:20:21 +03:00
Karim shoair
403f7bfe72
build: Updating PyPI classifiers
2025-07-07 00:10:20 +03:00
Karim shoair
939b78f3cf
build: adding mcp as dependency
2025-07-07 00:09:54 +03:00
Karim shoair
3184dd8581
build: set minimum version to Python 3.10
2025-07-07 00:09:02 +03:00
Karim shoair
2b73fdbfd7
fix(DynamicFetcher): Remove old argument
2025-07-06 05:02:16 +03:00
Karim shoair
2ecf9b0c20
build: Improve description
2025-07-06 05:01:19 +03:00
Karim shoair
63a9db21a1
refactor(DynamicSession): Optimization + Removing max_pages from sync version
2025-07-06 04:59:22 +03:00
Karim shoair
6d79927233
refactor(Fetcher): Fix the issue of caching impersonation state + Less code duplication
...
It now creates a session with each request, but at the same time, it's still faster than the fetcher in v0.2.99 by 20% with more features enabled.
2025-07-06 02:21:15 +03:00
Karim shoair
18e56a8c50
refactor: Optimizing extract command and cleaning code
2025-06-29 19:40:03 +03:00
Karim shoair
29e485bdf2
refactor: Optimizations to CLI
2025-06-29 19:10:52 +03:00
Karim shoair
b28f90e854
build: Update deps
2025-06-29 18:58:36 +03:00
Karim shoair
1c9e48b1c6
feat(extract): Adding new command to CLI options + Optimizations
...
Users can now fetch websites directly without code and extract full/selected HTML content as HTML, Markdown, or extract text content.
2025-06-29 18:56:09 +03:00
Karim shoair
ce034f9f79
test: remove problematic test
2025-06-28 18:24:08 +03:00
Karim shoair
b02f0bd236
ops: Adjust tox file to avoid CI issues
2025-06-28 18:12:38 +03:00
Karim shoair
ac3db69c47
ops: new approach to CI tests workflow
2025-06-28 17:40:19 +03:00
Karim shoair
ee1a0828a2
ops: Correcting cache paths
2025-06-27 17:16:35 +03:00
Karim shoair
d603c15212
ops: rewrite tests logic to cache downloaded browsers before testing
2025-06-27 16:02:16 +03:00
Karim shoair
4ee32af67b
ops: fix for GitHub's CI
2025-06-27 14:50:06 +03:00
Karim shoair
3578a07c0a
ops: fix for test deps
2025-06-27 14:35:12 +03:00
Karim shoair
94edb92bc3
ops: update hashing for the cache
2025-06-27 14:19:07 +03:00
Karim shoair
941ed20a15
ops: switch tests to use MacOS
2025-06-27 03:19:58 +03:00
Karim shoair
d6f8926d7a
ops: switch tests to Windows system instead of Linux
2025-06-27 03:08:42 +03:00
Karim shoair
dbf7aa7761
feat(DynamicFetcher): Use persistent context by default for better stealth
...
And might solve the sandbox issue with GitHub
2025-06-27 03:02:46 +03:00
Karim shoair
76248eabf6
ops: possible fix for sandbox issue with GitHub's CI
2025-06-27 03:01:27 +03:00
Karim shoair
9b62b59490
ops: possible fix for sandbox issue with GitHub's CI
2025-06-27 02:26:35 +03:00
Karim shoair
e535a3fa16
ops: fix sandbox issue with GitHub's CI
2025-06-27 02:14:53 +03:00
Karim shoair
5895edef57
ops: fix test workflow for GitHub
2025-06-27 01:42:22 +03:00
Karim shoair
8395db9121
fix(controllers): Fix issue with async adding stealth scripts
2025-06-27 01:39:32 +03:00
Karim shoair
228d294afc
test: Enable Dynamic stealth tests again
...
It's problematic with GitHub but letsgo
2025-06-27 01:15:06 +03:00
Karim shoair
37e842b796
fix(controllers): Fix issue with adding stealth scripts
2025-06-27 01:04:58 +03:00
Karim shoair
53318f3a2a
fix(StealthyFetcher): Fix passed proxy type
2025-06-26 16:36:30 +03:00
Karim shoair
8d44e9c51c
fix(proxy): fix proxy unpacking
2025-06-26 15:49:53 +03:00
Karim shoair
38851698ad
refactor: optimize imports and docstrings correction
2025-06-25 23:15:32 +03:00
Karim shoair
a41be48047
refactor: optimize imports and docstrings correction
2025-06-25 21:36:53 +03:00
Karim shoair
a3a5e34408
refactor: optimize imports and docstrings correction
2025-06-25 21:36:43 +03:00
Karim shoair
724e3a648e
fix(FetcherSession): Use Sentinel pattern to solve arguments precedence issue
2025-06-25 03:53:54 +03:00
Karim shoair
63ff12bcd6
docs: correcting a small mistake
2025-06-23 03:38:34 +03:00
Karim shoair
78b81c537f
fix(fetchers): Fix the bug of referer and google_search argument conflict
2025-06-23 03:36:11 +03:00
Karim shoair
560ae95281
test: Adjusting tests with the correct timeout
2025-06-23 03:24:13 +03:00
Karim shoair
69b983b42f
feat(fetchers): Improve StealthyFetcher + Adding StealthySession/AsyncStealthySession classes
2025-06-23 03:23:28 +03:00
Karim shoair
3754547e58
refactor: removing unused code
2025-06-23 03:20:49 +03:00
Karim shoair
fc530677de
refactor(DynamicFetcher): Cleaner exit for fetch
2025-06-23 02:58:01 +03:00
Karim shoair
22490db85f
fix(DynamicFetcher): Set number of tabs to 1
...
There is no logical use for it here; seek the session classes to use it.
2025-06-23 02:42:43 +03:00
Karim shoair
460b4443c2
refactor(controllers): Improve type validation
2025-06-22 19:59:40 +03:00
Karim shoair
0fe04e499f
refactor(validators): Optimize imports
2025-06-22 18:55:13 +03:00
Karim shoair
fc42487a5f
refactor(controllers): Optimize imports
2025-06-22 18:53:44 +03:00
Karim shoair
4769a43525
test: Fix for Github CI
2025-06-22 04:02:19 +03:00
Karim shoair
9c44ad1930
fix(AsyncFetcher): Fix stealthy_headers issue
2025-06-22 03:54:36 +03:00
Karim shoair
8100a2a865
test: fix for Github CI
2025-06-22 03:07:46 +03:00
Karim shoair
8b46e3a3a5
tests: Updating tests according to the new changes
2025-06-22 02:12:39 +03:00
Karim shoair
f28b4d3653
docs: Updating Readme accordingly to the new naming
2025-06-22 02:12:03 +03:00
Karim shoair
b77c5b7abf
refactor: updating top level shortcuts
2025-06-22 02:11:26 +03:00
Karim shoair
8a272cf19b
feat/refactor(fetchers): Replacing PlayWrightFetcher with DynamicFetcher and adding session classes
2025-06-22 02:10:45 +03:00
Karim shoair
1e51fc9738
refactor: Removing defaults as said
2025-06-22 02:07:34 +03:00
Karim shoair
0a570a7ca7
docs: Updating doc strings
2025-06-22 01:59:38 +03:00
Karim shoair
d049f63404
fix(fetcher): Fix impersonate and headers generation conflict
2025-06-20 03:29:05 +03:00
Karim shoair
194ce24201
build: update deps
2025-06-19 15:19:23 +03:00
Karim shoair
9bfc900140
feat(navigation tools): Improvements
2025-06-19 04:07:45 +03:00
Karim shoair
145b710960
feat(fetchers): Adding the foundation of the new browser-based fetchers logic
2025-06-19 04:06:52 +03:00
Karim shoair
23b883f05a
build: Add msgspec to the deps
...
It will be used very soon
2025-06-19 04:04:25 +03:00
Karim shoair
3e63fa2523
fix(shell): Hide the automatic tips and solve the namespace issue
2025-06-19 04:03:13 +03:00
Karim shoair
2bdee60004
fix(fingerprints): Fixing headers generation for requests
2025-06-15 03:37:22 +03:00
Karim shoair