Commit Graph

1078 Commits

Author SHA1 Message Date
Karim shoair 2dbd402000 docs: update links to use the current branch 2026-02-10 21:51:02 +02:00
Karim shoair f350800fd7 docs: updating Dynamic websites with protection page and some corrections 2026-02-10 21:48:09 +02:00
Karim shoair dba3e9d53e docs: updating dynamic websites page and some corrections 2026-02-10 21:23:14 +02:00
Karim shoair 8b1704a87e docs: use better import for ProxyRotator class 2026-02-10 21:20:43 +02:00
Karim shoair 4e3829fced style: expose ProxyRotator class to the public API 2026-02-10 21:20:07 +02:00
Karim shoair 8df74dc57b docs: updating http requests page and some corrections 2026-02-10 19:07:09 +02:00
Karim shoair f25fe64d3c docs: updating fetcher choosing page and some corrections 2026-02-10 19:01:50 +02:00
Karim shoair 87b99a407b docs: updating adaptive page and some corrections 2026-02-10 19:00:36 +02:00
Karim shoair 11bd4e4295 docs: updating main classes page and some corrections 2026-02-10 18:58:58 +02:00
Karim shoair 44e2c42eaf docs: updating selection page and some corrections 2026-02-10 18:42:37 +02:00
Karim shoair 989fdca201 docs: updating benchmarks page 2026-02-10 18:28:23 +02:00
Karim shoair a5abf1f3ba docs: updating overview page and some corrections 2026-02-10 18:25:31 +02:00
Karim shoair 8505c4a91c docs: possible fix for RTD zensical build 2026-02-09 01:50:22 +02:00
Karim shoair 33eb2ff56b docs: possible fix for RTD zensical build 2026-02-09 01:41:50 +02:00
Karim shoair 9b46cb35ab docs: possible fix for RTD zensical build 2026-02-09 01:16:04 +02:00
Karim shoair dc071ee6b8 docs: possible fix for RTD zensical build 2026-02-09 01:10:00 +02:00
Karim shoair 8ed71e4864 V4 (#130) 2026-02-09 00:55:21 +02:00
Karim shoair e688093974 docs: update the translated readme files with the new content 2026-02-09 00:46:58 +02:00
Karim shoair 3dc4bdcb4a docs: small adjustments to the readme file 2026-02-09 00:36:52 +02:00
Karim shoair 2a4d008927 docs: full update for main readme 2026-02-09 00:28:24 +02:00
Karim shoair 1f9eb17414 ops: update benchmarks script 2026-02-08 23:33:30 +02:00
Karim shoair 0c86d70307 docs: update the website main page 2026-02-08 23:33:10 +02:00
Karim shoair ca46070a1b docs: Improve README.md description formatting
Reformatted the description of Scrapling for clarity and readability.
2026-02-08 22:43:00 +02:00
Karim shoair 8d82d3c90d docs: update readme file 2026-02-08 22:39:50 +02:00
Karim shoair 9a108160c2 docs: move translated README files to docs folder again 2026-02-08 22:18:36 +02:00
Karim shoair 12b731b14b docs: move readme files to the root directory 2026-02-08 21:04:15 +02:00
Karim shoair 09cd5e6183 docs(readme): update translation links 2026-02-08 20:54:26 +02:00
Karim shoair 2ae11d21da docs: full switch from mkdocs to zensical 2026-02-08 20:46:38 +02:00
Karim shoair db41a6c4b6 docs: update pages with the new media for testing zensical 2026-02-08 20:46:14 +02:00
Karim shoair 0e2e9fb2ed docs: moving readme files to be outside of the website build 2026-02-08 20:45:03 +02:00
Karim shoair 219db79195 docs: update logo and media 2026-02-08 20:44:04 +02:00
Karim shoair 3eaa7ffc25 ops: exclude site directory from git 2026-02-08 20:42:35 +02:00
Karim shoair b1ce0b414b ops: Add mypy and pyright to the code quality workflow 2026-02-07 17:28:44 +02:00
Karim shoair 16852cb1ac style: Use shorter and more accurate naming for constants 2026-02-07 16:37:47 +02:00
Karim shoair ec7c7f8153 Merge branch 'main' into v4 2026-02-07 16:30:37 +02:00
Karim shoair e777342741 build: pump up version and deps 2026-02-07 16:30:32 +02:00
Karim shoair 5ec929435b style: Fix all mypy errors and add type hints to untyped function bodies
**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.**

`scrapling/core/_types.py`

  - Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11

`scrapling/core/storage.py`

  - Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes

`scrapling/core/custom_types.py`

  - split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str])
  - format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature)
  - AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None
  - json_string property: Added -> bytes return type

`scrapling/core/mixins.py`

  - Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-subclass mixins)
  - Added Dict[str, int] annotation for counter variable
  - Removed unused TYPE_CHECKING / Selector imports

`scrapling/parser.py (~30 errors)`

  - Added body: str | bytes pre-annotation for dual-type if/elif assignment
  - Used Dict[str, Any] kwargs dict for HTMLParser(...) to bypass incomplete lxml stubs missing default_doctype
  - Changed base_url=url or None → base_url=url or "" (avoids str | None vs str | bytes)
  - bool(adaptive) to guarantee bool type for __adaptive_enabled
  - Declared __text: Optional[TextHandler], __tag: Optional[str], __attributes: Optional[AttributesHandler] at top of __init__
  - cast(List, ...) for all XPath() call results (_find_all_elements, _find_all_elements_with_spaces)
  - Added Dict[float, List[Any]] for score_table, Dict[str, Any] for attributes
  - Changed score, checks = 0, 0 → score: float = 0; checks: int = 0 (two locations)
  - Renamed target → target_element in save() to avoid variable redefinition with different types
  - Wrapped node_text.clean() / .lower() in TextHandler(...) to preserve type

`scrapling/engines/_browsers/_page.py`

  - Added PageInfo[SyncPage] | PageInfo[AsyncPage] union type annotation to page_info variable

`scrapling/engines/_browsers/_validators.py`

  - Convert method_kwargs (TypedDict) to plain Dict[str, Any] before dynamic key access

`scrapling/engines/_browsers/_base.py`

  - Added _config declaration to BaseSessionMixin
  - Used cast(StealthConfig, self._config) in __generate_stealth_options to access stealth-only attributes
  - Added Tuple[str, ...] annotation for flags
  - Removed redundant narrower StealthConfig type annotation on self._config in StealthySessionMixin.__validate__
  - Widened SyncSession and AsyncSession fields (playwright, context, browser) to Any to support both playwright and patchright types
  - Added -> None to both start() methods

`scrapling/engines/_browsers/_stealth.py`

  - Added Optional, ProxyType imports
  - Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
  - Annotated outer_box: Any at first declaration, removed duplicate type annotations in subsequent branches
  - Added -> None to sync and async start()
  - Added config: Any parameter type to _initialize_context
  - Removed redundant self.context: AsyncBrowserContext re-annotations in conditional branches

`scrapling/engines/_browsers/_controllers.py`

  - Added Optional, ProxyType imports
  - Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
  - Added -> None to async start()
  - Removed redundant self.context: AsyncBrowserContext re-annotations

`scrapling/spiders/request.py`

  - Added Optional import, typed _fp: Optional[bytes] = None
  - Removed redundant body: bytes re-annotation

`scrapling/spiders/session.py`

  - Used separate client variable instead of reassigning session = session._client (avoids type incompatibility and fixes a bug where session._make_request was called instead of client._make_request)
  - Added -> None to SessionManager.__init__

`scrapling/engines/toolbelt/convertor.py`

  - Added list[Response] annotation for history in both sync/async methods

`scrapling/engines/static.py`

  - FetcherClient.__init__ and AsyncFetcherClient.__init__: Added **kwargs: Any and -> None

`scrapling/core/shell.py`

  - Wrapped re_sub(...) result in TextHandler(...) to maintain correct type
  - Added -> None to CurlParser.__init__
  - Added full type signature to create_wrapper, replaced wrapper.__signature__ = ... with setattr(wrapper, "__signature__", ...) to satisfy mypy
  - Added Callable to imports
2026-02-07 16:30:00 +02:00
Karim shoair 5f557a7f9f test: add tests for the new feature 2026-02-07 01:28:58 +02:00
Karim shoair 66476d42be feat(browsers): Add option to block requests to specific domains 2026-02-07 01:26:21 +02:00
Karim shoair d00c34ee7f style(parser): Improve the type hint for find_by_text and find_by_regex 2026-02-06 03:19:00 +02:00
Karim shoair 05f62fba81 fix(response): Force the body of the response to always be bytes 2026-02-06 03:14:23 +02:00
Karim shoair 9acc8194c5 test: update the tests accordingly 2026-02-06 02:45:13 +02:00
Karim shoair aa7a95fb70 feat(parser)!: Make all selection return selector objects by default
- The strings/Texthandlers are only returned by `get`/`getall`/`extract`/`extract_first`. This makes the type checking/autocompletion experience consistent.
- Removed `css_first` and `xpath_first` since it doesn't make sense to leave them now.
- Made the type hints more accurate in multiple places.
2026-02-06 02:43:51 +02:00
Karim shoair d8fe909c8b docs: make translation urls absolute
For PyPI, Docker, and other external websites featuring this
2026-02-05 02:44:12 +02:00
Karim shoair 151e4e7a7c docs: adding new sponsor 2026-02-05 02:39:51 +02:00
Karim shoair f88502718f feat(proxy control): Force a proxy at request level at any given point
And merge request's meta with response's meta
2026-02-02 14:26:52 +02:00
Karim shoair 6e0277f23b test: add new tests for the proxy rotation logic 2026-02-02 00:16:40 +02:00
Karim shoair ef8c5bc7d6 feat(spiders/fetchers): Adding proxy rotation logic and change retry logic
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.

- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.

- Normal requests use what you pass without issues, of course.

- All errors are retried now, and proxies are rotated on retries automatically.
2026-02-02 00:16:22 +02:00
Karim shoair 65f1a23358 fix(browsers): Improve stealth and speed by adjusting flags 2026-01-29 00:58:04 +02:00
Karim shoair 0c3d359b46 fix(browsers): disable autoplay for all browsers 2026-01-29 00:57:30 +02:00