- Add a `--version` flag to the main CLI group using click's `version_option`
- Prints `Scrapling, version <version>` and exits, sourcing the version from `scrapling.__version__`
- Add a CLI test asserting the flag's output
Closes#299
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).
- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).
In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
- DynamicFetcher became 20% faster
- StealthyFetcher became 99% faster
- Scrapling size decreased
- Code became ~400 lines shorter
- Most importantly, scrapling is more stable and reliable now.
- Less confusing for new users.
- More...
ETM-Code