be28fe16ec
feat(browsers): add new feature to enable DNS-over-HTTP to prevent DNS leaks
2026-04-12 18:03:03 +02:00
d952db8ef8
feat(browsers): add a new feature to block ads
...
This is working by aborting all requests to known ads domains.
2026-04-12 17:59:00 +02:00
4efbffa1dc
feat(cli): Add an option to make content safe/targets AI
2026-03-30 02:44:14 +02:00
8e147db7f8
refactor(cli): Code cleaning for easier maintenance/adding new features
...
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).
- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
2026-03-27 18:02:33 +02:00
90c52c45c7
feat(parser): replacing tldextract with tld library
...
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).
In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
2026-01-23 00:00:48 +02:00
3b9a6d021e
fix: correct cli argument
2025-12-31 03:00:06 +02:00
cfc667f7dc
refactor(fetchers)!: Replace Camoufox with patchright and many optimizations
...
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
2025-12-26 03:21:13 +02:00
fd9fc83c6c
build: pump version up and deps
2025-12-17 00:04:24 +02:00
3565f9d500
feat(fetcher): Make impersonate able to randomize fingerprint
2025-11-16 16:33:58 +02:00
8ad7cd6343
docs: Update all pages/docstrings to reflect recent changes
2025-09-29 03:57:40 +03:00
2bad72b77a
fix: correct http option type in the mcp server
2025-09-28 21:19:51 +03:00
74f20d2a0e
refactor: better implementation for the mcp mode
2025-09-28 20:14:04 +03:00
4a661b4875
feat: Make mcp able to use http transport
2025-09-28 04:54:35 +03:00
3da806210b
perf: General code restructure to not use more than needed memory
2025-09-23 18:34:54 +03:00
2d704b2a8b
fix(shell): Fixing a bug with content converting
2025-09-19 04:50:15 +03:00
f848e6b7f2
fix: Fixes to handle the new extra deps group
2025-09-15 00:19:19 +03:00
fdac239fcc
build: Move click library to be part of the fetchers extra
2025-09-14 20:32:46 +03:00
13d7e70cb7
refactor: Make all fetchers as an optional dependency group
...
+ Removing some dead code
2025-09-13 16:07:48 +03:00
330d03559c
style: applying the new ruff rules to all files
2025-09-13 03:22:53 +03:00
76721f906e
docs(cli): Correcting docstrings
2025-08-30 22:12:33 +03:00
c66c48e3ae
fix(extract): Correcting docs for the css-selector option
2025-08-29 21:34:39 +03:00
2f402f4835
style: add flags for tests coverage
...
- Some are already tested but the coverage report can't see it.
- Some are not necessary to test or too hard to test on GitHub's CI
2025-08-17 01:02:14 +03:00
13700e2692
fix: improve error handling
2025-08-15 04:52:11 +03:00
18660f8132
style: type hints corrections and docstrings
2025-07-30 00:32:39 +03:00
d1aa0be6e4
refactor(StealthyFetcher): Remove virtual mode and use persistent context
...
Solves #64 completely too
2025-07-27 03:41:28 +03:00
172a5b4a0a
feat: Add an mcp server
2025-07-26 22:57:08 +03:00
18e56a8c50
refactor: Optimizing extract command and cleaning code
2025-06-29 19:40:03 +03:00
29e485bdf2
refactor: Optimizations to CLI
2025-06-29 19:10:52 +03:00
1c9e48b1c6
feat(extract): Adding new command to CLI options + Optimizations
...
Users can now fetch websites directly without code and extract full/selected HTML content as HTML, Markdown, or extract text content.
2025-06-29 18:56:09 +03:00
a3a5e34408
refactor: optimize imports and docstrings correction
2025-06-25 21:36:43 +03:00
98ad50d5aa
feat(cli): Adding two new commands (uncurl/curl2fetcher)
2025-04-29 02:45:50 +03:00
a9dded34f4
fix(install): Fix error with spaces in Python's path ( #57 )
2025-04-26 04:10:38 +03:00
49b7ae13f6
feat(cli): Adding Scrapling Shell feature
2025-04-22 05:15:57 +02:00
0c8dd63f87
chore: migrating to ruff and updating pre-commit hooks
2025-04-13 17:32:00 +02:00
f8662bee29
feat(cli): Adding the -f flag to install command
2025-03-05 03:32:43 +02:00
087a85d615
fix(cli): Fix issue where install drops user into Python Shell
2025-03-05 03:28:07 +02:00
03b52aa74c
feat(cli): adding terminal command scrapling install
...
Instead of making users install browsers by themself
2024-12-26 19:46:22 +02:00
Karim shoair