Address review feedback: In full-path XPath generation, elements with
IDs were producing bare predicates like `[@id='x']` which creates
invalid XPath steps like `//body/[@id='main']`. Now emits `*[@id='x']`
for full-path mode (e.g. `//body/*[@id='main']/*[@id='target']`).
Short-path XPath mode unchanged — still uses `//*[@id='x']` prefix.
Also added XPath evaluation assertion to the regression test to verify
the generated selector actually selects the correct element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When generating full-path CSS/XPath selectors, elements with id
attributes had their selector appended twice — once in the id branch
(line 30) and again unconditionally (line 50).
This produced selectors like 'body > #main > #main > #target > #target'
instead of the correct 'body > #main > #target'.
Move the append into the else branch so it only fires for elements
without an id (elements with id already append in the if branch).
Includes 2 regression tests.
**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.**
`scrapling/core/_types.py`
- Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11
`scrapling/core/storage.py`
- Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes
`scrapling/core/custom_types.py`
- split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str])
- format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature)
- AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None
- json_string property: Added -> bytes return type
`scrapling/core/mixins.py`
- Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-subclass mixins)
- Added Dict[str, int] annotation for counter variable
- Removed unused TYPE_CHECKING / Selector imports
`scrapling/parser.py (~30 errors)`
- Added body: str | bytes pre-annotation for dual-type if/elif assignment
- Used Dict[str, Any] kwargs dict for HTMLParser(...) to bypass incomplete lxml stubs missing default_doctype
- Changed base_url=url or None → base_url=url or "" (avoids str | None vs str | bytes)
- bool(adaptive) to guarantee bool type for __adaptive_enabled
- Declared __text: Optional[TextHandler], __tag: Optional[str], __attributes: Optional[AttributesHandler] at top of __init__
- cast(List, ...) for all XPath() call results (_find_all_elements, _find_all_elements_with_spaces)
- Added Dict[float, List[Any]] for score_table, Dict[str, Any] for attributes
- Changed score, checks = 0, 0 → score: float = 0; checks: int = 0 (two locations)
- Renamed target → target_element in save() to avoid variable redefinition with different types
- Wrapped node_text.clean() / .lower() in TextHandler(...) to preserve type
`scrapling/engines/_browsers/_page.py`
- Added PageInfo[SyncPage] | PageInfo[AsyncPage] union type annotation to page_info variable
`scrapling/engines/_browsers/_validators.py`
- Convert method_kwargs (TypedDict) to plain Dict[str, Any] before dynamic key access
`scrapling/engines/_browsers/_base.py`
- Added _config declaration to BaseSessionMixin
- Used cast(StealthConfig, self._config) in __generate_stealth_options to access stealth-only attributes
- Added Tuple[str, ...] annotation for flags
- Removed redundant narrower StealthConfig type annotation on self._config in StealthySessionMixin.__validate__
- Widened SyncSession and AsyncSession fields (playwright, context, browser) to Any to support both playwright and patchright types
- Added -> None to both start() methods
`scrapling/engines/_browsers/_stealth.py`
- Added Optional, ProxyType imports
- Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
- Annotated outer_box: Any at first declaration, removed duplicate type annotations in subsequent branches
- Added -> None to sync and async start()
- Added config: Any parameter type to _initialize_context
- Removed redundant self.context: AsyncBrowserContext re-annotations in conditional branches
`scrapling/engines/_browsers/_controllers.py`
- Added Optional, ProxyType imports
- Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
- Added -> None to async start()
- Removed redundant self.context: AsyncBrowserContext re-annotations
`scrapling/spiders/request.py`
- Added Optional import, typed _fp: Optional[bytes] = None
- Removed redundant body: bytes re-annotation
`scrapling/spiders/session.py`
- Used separate client variable instead of reassigning session = session._client (avoids type incompatibility and fixes a bug where session._make_request was called instead of client._make_request)
- Added -> None to SessionManager.__init__
`scrapling/engines/toolbelt/convertor.py`
- Added list[Response] annotation for history in both sync/async methods
`scrapling/engines/static.py`
- FetcherClient.__init__ and AsyncFetcherClient.__init__: Added **kwargs: Any and -> None
`scrapling/core/shell.py`
- Wrapped re_sub(...) result in TextHandler(...) to maintain correct type
- Added -> None to CurlParser.__init__
- Added full type signature to create_wrapper, replaced wrapper.__signature__ = ... with setattr(wrapper, "__signature__", ...) to satisfy mypy
- Added Callable to imports
- The strings/Texthandlers are only returned by `get`/`getall`/`extract`/`extract_first`. This makes the type checking/autocompletion experience consistent.
- Removed `css_first` and `xpath_first` since it doesn't make sense to leave them now.
- Made the type hints more accurate in multiple places.
- Renamed both `css_selector` and `xpath_selector` both to `generate_css_selector` and `generate_xpath_selector` for clarity and not interrupting the auto-completion while coding.
- Added `generate_full_css_selector` and `generate_full_xpath_selector` methods
- Moved most of the parser functions/files to the core package.
- Converted tools file to a package and made separate files for similar functions.
- Now all fetcher engines return a Response object
- Instead of selecting an engine to use and passing config to it, we have separate fetcher classes so the user can choose what to use while importing.
- I added a new custom fetcher so the user can create and use an engine.
- More...
sjhddh