freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

style: Fix all mypy errors and add type hints to untyped function bodies

Browse Source 
**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.**

`scrapling/core/_types.py`

  - Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11

`scrapling/core/storage.py`

  - Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes

`scrapling/core/custom_types.py`

  - split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str])
  - format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature)
  - AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None
  - json_string property: Added -> bytes return type

`scrapling/core/mixins.py`

  - Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-subclass mixins)
  - Added Dict[str, int] annotation for counter variable
  - Removed unused TYPE_CHECKING / Selector imports

`scrapling/parser.py (~30 errors)`

  - Added body: str | bytes pre-annotation for dual-type if/elif assignment
  - Used Dict[str, Any] kwargs dict for HTMLParser(...) to bypass incomplete lxml stubs missing default_doctype
  - Changed base_url=url or None → base_url=url or "" (avoids str | None vs str | bytes)
  - bool(adaptive) to guarantee bool type for __adaptive_enabled
  - Declared __text: Optional[TextHandler], __tag: Optional[str], __attributes: Optional[AttributesHandler] at top of __init__
  - cast(List, ...) for all XPath() call results (_find_all_elements, _find_all_elements_with_spaces)
  - Added Dict[float, List[Any]] for score_table, Dict[str, Any] for attributes
  - Changed score, checks = 0, 0 → score: float = 0; checks: int = 0 (two locations)
  - Renamed target → target_element in save() to avoid variable redefinition with different types
  - Wrapped node_text.clean() / .lower() in TextHandler(...) to preserve type

`scrapling/engines/_browsers/_page.py`

  - Added PageInfo[SyncPage] | PageInfo[AsyncPage] union type annotation to page_info variable

`scrapling/engines/_browsers/_validators.py`

  - Convert method_kwargs (TypedDict) to plain Dict[str, Any] before dynamic key access

`scrapling/engines/_browsers/_base.py`

  - Added _config declaration to BaseSessionMixin
  - Used cast(StealthConfig, self._config) in __generate_stealth_options to access stealth-only attributes
  - Added Tuple[str, ...] annotation for flags
  - Removed redundant narrower StealthConfig type annotation on self._config in StealthySessionMixin.__validate__
  - Widened SyncSession and AsyncSession fields (playwright, context, browser) to Any to support both playwright and patchright types
  - Added -> None to both start() methods

`scrapling/engines/_browsers/_stealth.py`

  - Added Optional, ProxyType imports
  - Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
  - Annotated outer_box: Any at first declaration, removed duplicate type annotations in subsequent branches
  - Added -> None to sync and async start()
  - Added config: Any parameter type to _initialize_context
  - Removed redundant self.context: AsyncBrowserContext re-annotations in conditional branches

`scrapling/engines/_browsers/_controllers.py`

  - Added Optional, ProxyType imports
  - Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
  - Added -> None to async start()
  - Removed redundant self.context: AsyncBrowserContext re-annotations

`scrapling/spiders/request.py`

  - Added Optional import, typed _fp: Optional[bytes] = None
  - Removed redundant body: bytes re-annotation

`scrapling/spiders/session.py`

  - Used separate client variable instead of reassigning session = session._client (avoids type incompatibility and fixes a bug where session._make_request was called instead of client._make_request)
  - Added -> None to SessionManager.__init__

`scrapling/engines/toolbelt/convertor.py`

  - Added list[Response] annotation for history in both sync/async methods

`scrapling/engines/static.py`

  - FetcherClient.__init__ and AsyncFetcherClient.__init__: Added **kwargs: Any and -> None

`scrapling/core/shell.py`

  - Wrapped re_sub(...) result in TextHandler(...) to maintain correct type
  - Added -> None to CurlParser.__init__
  - Added full type signature to create_wrapper, replaced wrapper.__signature__ = ... with setattr(wrapper, "__signature__", ...) to satisfy mypy
  - Added Callable to imports
This commit is contained in:
Karim shoair
2026-02-07 16:30:00 +02:00
parent 5f557a7f9f
commit 5ec929435b
15 changed files with 133 additions and 149 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scrapling/core/_types.py
+1 -21
View File
@@ -32,6 +32,7 @@ from typing import (
Coroutine,
SupportsIndex,
)
from typing_extensions import Self, Unpack
# Proxy can be a string URL or a dict (Playwright format: {"server": "...", "username": "...", "password": "..."})
ProxyType = Union[str, Dict[str, str]]
@@ -41,27 +42,6 @@ PageLoadStates = Literal["commit", "domcontentloaded", "load", "networkidle"]
extraction_types = Literal["text", "html", "markdown"]
StrOrBytes = Union[str, bytes]
if TYPE_CHECKING: # pragma: no cover
from typing_extensions import Unpack
else: # pragma: no cover
class _Unpack:
@staticmethod
def __getitem__(*args, **kwargs):
pass
Unpack = _Unpack()
try:
# Python 3.11+
from typing import Self # novermin
except ImportError: # pragma: no cover
try:
from typing_extensions import Self # Backport
except ImportError:
Self = object
# Copied from `playwright._impl._api_structures.SetCookieParam`
class SetCookieParam(TypedDict, total=False):
scrapling/core/custom_types.py
+5 -7
View File
@@ -35,9 +35,7 @@ class TextHandler(str):
lst = super().__getitem__(key)
return TextHandler(lst)
def split(
self, sep: str | None = None, maxsplit: SupportsIndex = -1
) -> Union[List, "TextHandlers"]: # pragma: no cover
def split(self, sep: str | None = None, maxsplit: SupportsIndex = -1) -> list[Any]: # pragma: no cover
return TextHandlers([TextHandler(s) for s in super().split(sep, maxsplit)])
def strip(self, chars: str | None = None) -> Union[str, "TextHandler"]: # pragma: no cover
@@ -61,7 +59,7 @@ class TextHandler(str):
def expandtabs(self, tabsize: SupportsIndex = 8) -> Union[str, "TextHandler"]: # pragma: no cover
return TextHandler(super().expandtabs(tabsize))
def format(self, *args: object, **kwargs: str) -> Union[str, "TextHandler"]: # pragma: no cover
def format(self, *args: object, **kwargs: object) -> Union[str, "TextHandler"]: # pragma: no cover
return TextHandler(super().format(*args, **kwargs))
def format_map(self, mapping) -> Union[str, "TextHandler"]: # pragma: no cover
@@ -291,7 +289,7 @@ class AttributesHandler(Mapping[str, _TextHandlerType]):
__slots__ = ("_data",)
def __init__(self, mapping=None, **kwargs):
def __init__(self, mapping: Any = None, **kwargs: Any) -> None:
mapping = (
{key: TextHandler(value) if isinstance(value, str) else value for key, value in mapping.items()}
if mapping is not None
@@ -324,8 +322,8 @@ class AttributesHandler(Mapping[str, _TextHandlerType]):
yield AttributesHandler({key: value})
@property
def json_string(self):
"""Convert current attributes to JSON string if the attributes are JSON serializable otherwise throws error"""
def json_string(self) -> bytes:
"""Convert current attributes to JSON bytes if the attributes are JSON serializable otherwise throws error"""
return dumps(dict(self._data))
def __getitem__(self, key: str) -> _TextHandlerType:
scrapling/core/mixins.py
+11 -10
View File
@@ -1,7 +1,4 @@
from scrapling.core._types import TYPE_CHECKING
if TYPE_CHECKING:
from scrapling.parser import Selector
from scrapling.core._types import Any, Dict
class SelectorsGeneration:
@@ -11,7 +8,11 @@ class SelectorsGeneration:
Inspiration: https://searchfox.org/mozilla-central/source/devtools/shared/inspector/css-logic.js#591
"""
def _general_selection(self: "Selector", selection: str = "css", full_path: bool = False) -> str: # type: ignore[name-defined]
# Note: This is a mixin class meant to be used with Selector.
# The methods access Selector attributes (._root, .parent, .attrib, .tag, etc.)
# through self, which will be a Selector instance at runtime.
def _general_selection(self: Any, selection: str = "css", full_path: bool = False) -> str:
"""Generate a selector for the current element.
:return: A string of the generated selector.
"""
@@ -36,7 +37,7 @@ class SelectorsGeneration:
# if classes and css:
# part += f".{'.'.join(classes)}"
# else:
counter = {}
counter: Dict[str, int] = {}
for child in target.parent.children:
counter.setdefault(child.tag, 0)
counter[child.tag] += 1
@@ -56,28 +57,28 @@ class SelectorsGeneration:
return " > ".join(reversed(selectorPath)) if css else "//" + "/".join(reversed(selectorPath))
@property
def generate_css_selector(self: "Selector") -> str: # type: ignore[name-defined]
def generate_css_selector(self: Any) -> str:
"""Generate a CSS selector for the current element
:return: A string of the generated selector.
"""
return self._general_selection()
@property
def generate_full_css_selector(self: "Selector") -> str: # type: ignore[name-defined]
def generate_full_css_selector(self: Any) -> str:
"""Generate a complete CSS selector for the current element
:return: A string of the generated selector.
"""
return self._general_selection(full_path=True)
@property
def generate_xpath_selector(self: "Selector") -> str: # type: ignore[name-defined]
def generate_xpath_selector(self: Any) -> str:
"""Generate an XPath selector for the current element
:return: A string of the generated selector.
"""
return self._general_selection("xpath")
@property
def generate_full_xpath_selector(self: "Selector") -> str: # type: ignore[name-defined]
def generate_full_xpath_selector(self: Any) -> str:
"""Generate a complete XPath selector for the current element
:return: A string of the generated selector.
"""
scrapling/core/shell.py
+9 -6
View File
@@ -30,6 +30,7 @@ from scrapling.core.custom_types import TextHandler
from scrapling.engines.toolbelt.custom import Response
from scrapling.core.utils._shell import _ParseHeaders, _CookieParser
from scrapling.core._types import (
Callable,
Dict,
Any,
cast,
@@ -82,7 +83,7 @@ class NoExitArgumentParser(ArgumentParser): # pragma: no cover
class CurlParser:
"""Builds the argument parser for relevant curl flags from DevTools."""
def __init__(self):
def __init__(self) -> None:
from scrapling.fetchers import Fetcher as __Fetcher
self.__fetcher = __Fetcher
@@ -467,19 +468,21 @@ Type 'exit' or press Ctrl+D to exit.
return result
def create_wrapper(self, func, get_signature=True, signature_name=None):
def create_wrapper(
self, func: Callable, get_signature: bool = True, signature_name: Optional[str] = None
) -> Callable:
"""Create a wrapper that preserves function signature but updates page"""
@wraps(func)
def wrapper(*args, **kwargs):
def wrapper(*args: Any, **kwargs: Any) -> Any:
result = func(*args, **kwargs)
return self.update_page(result)
if get_signature:
# Explicitly preserve and unpack signature for IPython introspection and autocompletion
wrapper.__signature__ = _unpack_signature(func, signature_name) # pyright: ignore
setattr(wrapper, "__signature__", _unpack_signature(func, signature_name))
else:
wrapper.__signature__ = signature(func) # pyright: ignore
setattr(wrapper, "__signature__", signature(func))
return wrapper
@@ -601,7 +604,7 @@ class Convertor:
" ",
):
# Remove consecutive white-spaces
txt_content = re_sub(f"[{s}]+", s, txt_content)
txt_content = TextHandler(re_sub(f"[{s}]+", s, txt_content))
yield txt_content
yield ""
scrapling/core/storage.py
+4 -5
View File
@@ -63,12 +63,11 @@ class StorageSystemMixin(ABC): # pragma: no cover
def _get_hash(identifier: str) -> str:
"""If you want to hash identifier in your storage system, use this safer"""
_identifier = identifier.lower().strip()
if isinstance(_identifier, str):
# Hash functions have to take bytes
_identifier = _identifier.encode("utf-8")
# Hash functions have to take bytes
_identifier_bytes = _identifier.encode("utf-8")
hash_value = sha256(_identifier).hexdigest()
return f"{hash_value}_{len(_identifier)}" # Length to reduce collision chance
hash_value = sha256(_identifier_bytes).hexdigest()
return f"{hash_value}_{len(_identifier_bytes)}" # Length to reduce collision chance
@lru_cache(1, typed=True)