Karim shoair
47dd2ed28f
Merge branch 'main' into dev
2025-08-30 22:35:25 +03:00
Karim shoair
76721f906e
docs(cli): Correcting docstrings
2025-08-30 22:12:33 +03:00
Karim shoair
9dfca4c6f2
misc: Add ko-fi username to FUNDING.yml
2025-08-30 22:08:25 +03:00
Karim shoair
8edcd611d3
docs: replace the test website
2025-08-30 18:51:31 +03:00
Karim shoair
c66c48e3ae
fix(extract): Correcting docs for the css-selector option
2025-08-29 21:34:39 +03:00
Karim shoair
cd388059e8
docs: Add a page about the interactive shell
2025-08-29 20:31:00 +03:00
Karim shoair
badc3b21e1
docs: update article A Free Alternative to AI for Robust Web Scraping
2025-08-28 23:18:56 +03:00
Karim shoair
6d419e0f4d
docs: update BS article
2025-08-28 05:01:38 +03:00
Karim shoair
e14522f57a
docs: update the StealthyFetcher page
2025-08-28 04:53:22 +03:00
Karim shoair
1f87210b93
docs: Add max_pages explanation to DynamicFetcher
2025-08-28 04:36:49 +03:00
Karim shoair
31d555687f
docs: update the DynamicFetcher page
2025-08-28 03:59:00 +03:00
Karim shoair
18f9e1efb0
docs: update the fetcher page
2025-08-28 02:07:37 +03:00
Karim shoair
2435d00f16
docs: update fetchers choosing page
2025-08-28 02:07:22 +03:00
Karim shoair
d65a30e6b6
docs: update fetchers choosing page
2025-08-27 16:21:46 +03:00
Karim shoair
c6088f641b
docs: update Readme's downloads badge
2025-08-26 20:43:22 +03:00
Karim shoair
9120fe14bf
docs: update fetchers choosing page
2025-08-26 04:54:00 +03:00
Karim shoair
bac3ffefcf
docs: update fetchers page in the API reference
2025-08-25 06:15:20 +03:00
Karim shoair
41aceb2c43
docs: update storage dev tutorial
2025-08-25 06:10:46 +03:00
Karim shoair
cf57af588e
docs: update doc strings with correct naming
2025-08-25 06:08:15 +03:00
Karim shoair
6ddabaa593
docs: replace automatch page with adaptive + updating
2025-08-25 05:57:29 +03:00
Karim shoair
d5d4003277
docs: update the sponsors page
2025-08-25 00:42:31 +03:00
Karim shoair
a917899e4f
docs: replace adaptor with selector in the API reference
2025-08-25 00:42:01 +03:00
Karim shoair
545e03229d
docs: Update the 'main classes' page
2025-08-25 00:40:24 +03:00
Karim shoair
8eb51f5653
docs: updating 'Querying elements' page
2025-08-24 22:36:00 +03:00
Karim shoair
040246177f
docs: Update all benchmarks
2025-08-24 16:06:07 +03:00
Karim shoair
dc0107666f
docs: small update to main page
2025-08-24 16:03:17 +03:00
Karim shoair
eaf3751f76
docs: Update overview page
2025-08-24 16:02:21 +03:00
Karim shoair
b467cd025f
docs: update website main page
2025-08-24 04:00:16 +03:00
Karim shoair
0c63f95d07
docs: update the README
2025-08-24 03:04:52 +03:00
Karim shoair
a8f5c55f06
Merge branch 'main' into dev
2025-08-23 23:33:10 +03:00
Karim shoair
bdbc877b4b
[ImgBot] Optimize images ( #72 )
2025-08-23 23:20:32 +03:00
Karim shoair
377a5ceb09
Move docs to the main branch ( #71 )
2025-08-23 23:17:00 +03:00
Karim shoair
f8fbd4b0e0
ops: combine the auto-release/publish workflows
2025-08-23 22:43:03 +03:00
Karim shoair
637580e130
tests: stop testing geoip
...
The DB is downloaded from GitHub releases using public API so we get rate limited a lot
2025-08-23 22:20:51 +03:00
Karim shoair
b50c85fa60
docs: Update the README file to reflect the new version changes
2025-08-23 19:22:35 +03:00
Karim shoair
5389b5c7ff
docs: add PetroSky as a top sponsor
2025-08-21 04:12:13 +03:00
Karim shoair
dd4fcc47c8
test: remove geoip from the async camoufox test
...
to lower errors of rate limiting caused by downloading the geo db from GitHub
2025-08-20 16:36:58 +03:00
Karim shoair
293206b466
Update index.md
2025-08-20 16:12:12 +03:00
Karim shoair
9d880f2731
Update index.md
2025-08-20 16:11:02 +03:00
Karim shoair
df3ddb1c78
docs: Update how sponsors displayed on the website
2025-08-20 15:36:01 +03:00
Karim shoair
cc84fa2f54
Merge branch 'main' into docs
2025-08-20 15:30:16 +03:00
Karim shoair
07773eb2f3
Merge branch 'main' into dev
2025-08-20 15:30:02 +03:00
Karim shoair
4fa92e1073
docs: update how sponsors displayed
2025-08-20 15:20:25 +03:00
Karim shoair
8a2bb0573b
chore: replace sponsors banners with logos
2025-08-20 15:04:19 +03:00
Karim shoair
44d2b55f99
test: fix for GH actions
2025-08-20 04:33:20 +03:00
Karim shoair
6b8d861b92
build: Move non-primarily deps to optional extras
2025-08-20 03:58:10 +03:00
Karim shoair
85885d9962
tests: add geoip to Stealth tests
2025-08-20 01:36:55 +03:00
Karim shoair
57a025a34f
build: update dependencies and fix geoip issue
2025-08-20 01:10:17 +03:00
Karim shoair
5aea62256b
test: adding new tests and updating existing ones
...
Code Coverage is now 92%
2025-08-17 01:03:19 +03:00
Karim shoair
2f402f4835
style: add flags for tests coverage
...
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
2025-08-17 01:02:14 +03:00
Karim shoair
009bbb20c0
refactor(dynamic Fetcher): small optimization
2025-08-16 23:20:54 +03:00
Karim shoair
3e0ca33111
test: adding new tests and updating existing ones
2025-08-16 19:07:08 +03:00
Karim shoair
4c33c47609
test: Update and improve Camo/Dynamic tests
2025-08-16 16:02:17 +03:00
Karim shoair
6c9a960a44
fix(dynamicFetcher): Fix a bug with real_chrome argument
2025-08-16 16:01:32 +03:00
Karim shoair
e6e5b3cc80
test: adding new tests and updating existing ones
...
The coverage is now 78%
2025-08-16 14:58:11 +03:00
Karim shoair
84337ef1ac
fix(parser): count all nested children of ignored tags in get_all_text
2025-08-16 14:24:51 +03:00
Karim shoair
8bfbcec2d7
test: remove the test for install command
...
GitHub is already reinstalls the library with it with every test
2025-08-15 06:01:53 +03:00
Karim shoair
0e3358007f
test: adding new tests and updating existing ones
2025-08-15 04:52:51 +03:00
Karim shoair
13700e2692
fix: improve error handling
2025-08-15 04:52:11 +03:00
Karim shoair
ba81890daf
test: adding new tests and updating existing ones
2025-08-03 16:37:47 +03:00
Karim shoair
825896c6a8
fix: improve checking for valid proxy and valid CDP URL
2025-08-03 16:31:16 +03:00
Karim shoair
740ae815c3
perf: speeding up find_by_text and find_by_regex by 3%
2025-08-02 19:46:18 +03:00
Karim shoair
67658f1723
perf: Speeding up below_elements and relocate by 3%
2025-08-02 19:45:43 +03:00
Karim shoair
d1a2ecd341
perf: optimize get_all_text and adaptive logic by another 10%
2025-08-02 04:08:14 +03:00
Karim shoair
1ec6f0e0f0
perf: optimizing find_similar method
2025-08-02 03:39:05 +03:00
Karim shoair
aec4889d25
perf: Optimizing next and previous properties
2025-08-02 03:37:48 +03:00
Karim shoair
548a40d850
perf: speed up get_all_text function by another 20%
2025-08-02 03:36:59 +03:00
Karim shoair
4c4202daae
perf(parser): Speeding up css_first and xpath_first than normal ones
2025-08-01 16:41:09 +03:00
Karim shoair
83a19f3b17
perf(parser): A lot of optimizations to speed things up
2025-08-01 06:28:52 +03:00
Karim shoair
f93348b915
style: remove unused code
2025-08-01 05:43:31 +03:00
Karim shoair
df48662c00
perf(parser): A lot of optimizations to speed things up
2025-08-01 05:42:32 +03:00
Karim shoair
af5f3688c1
fix: moving types to use Union again
2025-07-30 05:39:25 +03:00
Karim shoair
27658b33ff
fix: fix invalid return type
2025-07-30 03:39:33 +03:00
Karim shoair
32cb76604c
style: Adjustments to the translator
2025-07-30 03:28:16 +03:00
Karim shoair
5bb1266fa5
style: using isinstance function as the main way for type checking
2025-07-30 03:14:23 +03:00
Karim shoair
ae9ccaec79
style: A lot of type hints correction
...
Since we are using Py3.10 as minimum version now, we remove Union when possible
2025-07-30 02:46:57 +03:00
Karim shoair
ca12a11b7e
style(utils): optimize imports
2025-07-30 02:08:01 +03:00
Karim shoair
db781dcc32
style: Removing dead code
2025-07-30 02:03:39 +03:00
Karim shoair
3c5de8e0f2
perf: Speed up clean functions
2025-07-30 02:03:27 +03:00
Karim shoair
9415acccce
fix: shortcuts for backward compatibility
2025-07-30 01:50:48 +03:00
Karim shoair
7300efa77e
style(parser): optimize selectors instances creation
2025-07-30 01:45:26 +03:00
Karim shoair
e7cdd39695
style: replacing os with Pathlib and small optimizations
2025-07-30 01:15:28 +03:00
Karim shoair
18660f8132
style: type hints corrections and docstrings
2025-07-30 00:32:39 +03:00
Karim shoair
ba585c0adc
perf: imports optimizing
2025-07-30 00:22:06 +03:00
Karim shoair
8ca940d768
style: removing dead code
2025-07-29 23:43:31 +03:00
Karim shoair
9bcb9e9d93
style: General type hints fixes and imports optimizing
2025-07-29 23:43:06 +03:00
Karim shoair
3d07a1533e
refactor: cleaner code for the fetcher's lazy loader
2025-07-29 22:43:12 +03:00
Karim shoair
54cba6db45
ops: fix benchmarks script and make it more accurate
2025-07-29 22:36:47 +03:00
Karim shoair
9ce3279488
fix(TextHandler): Increase speed of clean method 5 times
2025-07-29 22:31:03 +03:00
Karim shoair
81a9eb068a
ops(tests workflow): Cache browsers on GitHub
2025-07-29 21:03:14 +03:00
Karim shoair
08cae510a6
ops: Cache keys adjustment for tests
2025-07-29 16:40:48 +03:00
Karim shoair
715dfb4243
feat: add length property to Selectors to write less code
2025-07-29 06:19:45 +03:00
Karim shoair
29b77a96c8
refactor(parser): optimize imports
2025-07-29 06:18:12 +03:00
Karim shoair
b9c7a5af2e
refactor: replace's Selector inpt (text/body) with 1 argument called content
2025-07-29 06:08:15 +03:00
Karim shoair
9e9ba9ab10
docs: update roadmap
2025-07-29 04:20:43 +03:00
Karim shoair
264ae02aa7
refactor: huge change, many features/class got a better naming
...
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
2025-07-29 04:20:23 +03:00
Karim shoair
0c649987f8
test: remove irrelevant test case
2025-07-28 15:10:23 +03:00
Karim shoair
a5e4b91653
refactor: remove clean up function for Adaptor + make adaptor attributes accessible directly
2025-07-28 14:48:22 +03:00
Karim shoair
297e14230b
refactor(parser): Multiple optimizations and fixes
2025-07-28 03:32:22 +03:00
Karim shoair
e35bade804
build: update bandit rules
2025-07-28 03:32:17 +03:00
Karim shoair