`ResponseFactory.__extract_browser_encoding` matched the charset with
`charset=([\w-]+)`, which stops at a quote character. RFC 7231 permits the
charset value to be a quoted-string (e.g. `content-type: text/html;
charset="ISO-8859-1"`), so for any quoted charset the regex failed to match
and the function silently fell back to the `utf-8` default. A page served as
quoted ISO-8859-1 / windows-1252 / Shift_JIS would then be decoded as UTF-8,
producing mojibake.
Allow an optional surrounding quote in the pattern (`charset=["']?([\w-]+)`)
so the value is captured without the quote. Unquoted headers are unaffected.
The existing `content_type_map` fixture in tests/fetchers/test_utils.py was
unused; add focused tests covering unquoted, quoted, and missing charsets.
block_ads is a browser-engine parameter (used by PlayWright/Camoufox
fetchers for ad-domain blocking) and is not recognised by curl_cffi's
Session.request(). When the CLI's --ai-targeted flag sets block_ads=True,
_merge_request_args forwards it unfiltered, causing:
TypeError: Session.request() got an unexpected keyword argument 'block_ads'
Add block_ads to the skip_keys set so it is stripped before the dict
reaches Session.request(), consistent with existing entries for
extra_headers and google_search.
Fixes#247
The Seconds type was defined as `Annotated[int, float, Meta(ge=0)]` which per PEP 593 treated float as metadata, not a type. This caused passing float values like wait=1.5 to be rejected. Fixed by using Annotated[float, Meta(ge=0)] since int is a subtype of float.
Correct fix for #240
Co-Authored-By: Cocoon-Break <54054995+kuishou68@users.noreply.github.com>
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.
Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.
Possible fix for #215
Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com>
Both _get_page_content and _get_async_page_content use a while-True
loop that retries page.content() on PlaywrightError with no upper
bound. If the page is in a permanently broken state (crashed tab,
closed context), this loops forever and hangs the process.
Replace with a bounded for-loop (default 10 retries = 5s), returning
an empty string if all attempts fail. This preserves the existing
retry behavior for the transient Windows issue (playwright#16108)
while preventing hangs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Real browsers send `https://www.google.com/` as the Referer header
when clicking search results, not the full search URL with query
parameters. The previous format was a fingerprinting signal that
the referer was spoofed.
Closes#172
It can be considered the simplest form of round robin since we don't have weights or anything (as was originally planned), but let's change it to avoid confusion or useless debates. Here goes nothing
**Resolved all 65 mypy errors across 14 files and added type annotations to all previously untyped function bodies. Final result: 0 errors with --check-untyped-defs enabled, all 454 tests pass.**
`scrapling/core/_types.py`
- Removed broken Self = object fallback — now requires typing_extensions for Python < 3.11
`scrapling/core/storage.py`
- Fixed str/bytes mismatch in _get_hash() — used separate _identifier_bytes variable instead of reassigning from str to bytes
`scrapling/core/custom_types.py`
- split() return type: Union[List, "TextHandlers"] → list[Any] (avoids LSP violation with parent list[str])
- format() kwargs: **kwargs: str → **kwargs: object (matches parent str.format signature)
- AttributesHandler.__init__: Added mapping: Any = None, **kwargs: Any and -> None
- json_string property: Added -> bytes return type
`scrapling/core/mixins.py`
- Changed self: "Selector" to self: Any on all mixin methods (mypy can't handle forward-reference self types on non-subclass mixins)
- Added Dict[str, int] annotation for counter variable
- Removed unused TYPE_CHECKING / Selector imports
`scrapling/parser.py (~30 errors)`
- Added body: str | bytes pre-annotation for dual-type if/elif assignment
- Used Dict[str, Any] kwargs dict for HTMLParser(...) to bypass incomplete lxml stubs missing default_doctype
- Changed base_url=url or None → base_url=url or "" (avoids str | None vs str | bytes)
- bool(adaptive) to guarantee bool type for __adaptive_enabled
- Declared __text: Optional[TextHandler], __tag: Optional[str], __attributes: Optional[AttributesHandler] at top of __init__
- cast(List, ...) for all XPath() call results (_find_all_elements, _find_all_elements_with_spaces)
- Added Dict[float, List[Any]] for score_table, Dict[str, Any] for attributes
- Changed score, checks = 0, 0 → score: float = 0; checks: int = 0 (two locations)
- Renamed target → target_element in save() to avoid variable redefinition with different types
- Wrapped node_text.clean() / .lower() in TextHandler(...) to preserve type
`scrapling/engines/_browsers/_page.py`
- Added PageInfo[SyncPage] | PageInfo[AsyncPage] union type annotation to page_info variable
`scrapling/engines/_browsers/_validators.py`
- Convert method_kwargs (TypedDict) to plain Dict[str, Any] before dynamic key access
`scrapling/engines/_browsers/_base.py`
- Added _config declaration to BaseSessionMixin
- Used cast(StealthConfig, self._config) in __generate_stealth_options to access stealth-only attributes
- Added Tuple[str, ...] annotation for flags
- Removed redundant narrower StealthConfig type annotation on self._config in StealthySessionMixin.__validate__
- Widened SyncSession and AsyncSession fields (playwright, context, browser) to Any to support both playwright and patchright types
- Added -> None to both start() methods
`scrapling/engines/_browsers/_stealth.py`
- Added Optional, ProxyType imports
- Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
- Annotated outer_box: Any at first declaration, removed duplicate type annotations in subsequent branches
- Added -> None to sync and async start()
- Added config: Any parameter type to _initialize_context
- Removed redundant self.context: AsyncBrowserContext re-annotations in conditional branches
`scrapling/engines/_browsers/_controllers.py`
- Added Optional, ProxyType imports
- Annotated proxy: Optional[ProxyType] in both sync/async fetch loops
- Added -> None to async start()
- Removed redundant self.context: AsyncBrowserContext re-annotations
`scrapling/spiders/request.py`
- Added Optional import, typed _fp: Optional[bytes] = None
- Removed redundant body: bytes re-annotation
`scrapling/spiders/session.py`
- Used separate client variable instead of reassigning session = session._client (avoids type incompatibility and fixes a bug where session._make_request was called instead of client._make_request)
- Added -> None to SessionManager.__init__
`scrapling/engines/toolbelt/convertor.py`
- Added list[Response] annotation for history in both sync/async methods
`scrapling/engines/static.py`
- FetcherClient.__init__ and AsyncFetcherClient.__init__: Added **kwargs: Any and -> None
`scrapling/core/shell.py`
- Wrapped re_sub(...) result in TextHandler(...) to maintain correct type
- Added -> None to CurlParser.__init__
- Added full type signature to create_wrapper, replaced wrapper.__signature__ = ... with setattr(wrapper, "__signature__", ...) to satisfy mypy
- Added Callable to imports
- User passes a single proxy to the browser session, and it will keep using the same context. Pass a proxy manager that automatically refreshes the IP with each proxy, and you get speed but sacrifice some stealth.
- User imports the proxy rotator class and passes proxies to it and a rotation strategy (Round robin by default). Then pass the instance to the session class, which will create a context and a tab for each proxy returned by the rotator. This way, you sacrifice speed a bit, but you get maximum stealth since each context is created with the proxy that will be used.
- Normal requests use what you pass without issues, of course.
- All errors are retried now, and proxies are rotated on retries automatically.
This might break the adaptive data users have for websites BUT:
1. tld uses ~3.7x less memory during extraction operations (1.5 MB vs 5.7 MB).
2. tld uses ~56% less memory on import (5.2 MB vs 11.9 MB).
3. Zero dependencies (vs 3 for tldextract).
In return, it's 30ms slower for extracting 5000 URLs, which is negligible. Also, the type hints aren't always accurate, but it's fine; I corrected them.
Karim shoair