v0.4.9 (#345)

2026-06-07 21:41:55 +03:00
parent b9bc17b461 74c5848060
commit 149050685e
39 changed files with 268 additions and 57 deletions
@@ -73,7 +73,7 @@ jobs:
    - name: Install all browsers dependencies
      run: |
        python3 -m pip install --upgrade pip
-        python3 -m pip install playwright==1.59.0 patchright==1.59.1
+        python3 -m pip install playwright==1.60.0 patchright==1.60.1

    - name: Get Playwright version
      id: playwright-version
@@ -11,7 +11,7 @@ There are many ways to contribute to Scrapling. Here are some of them:
 - Report bugs and request features using the [GitHub issues](https://github.com/D4Vinci/Scrapling/issues). Please follow the issue template to help us resolve your issue quickly.
 - Blog about Scrapling. Tell the world how you’re using Scrapling. This will help newcomers with more examples and increase the Scrapling project's visibility.
 - Join the [Discord community](https://discord.gg/EMgGbDceNQ) and share your ideas on how to improve Scrapling. We’re always open to suggestions.
- If you are not a developer, perhaps you would like to help with translating the [documentation](https://github.com/D4Vinci/Scrapling/tree/docs)?
+- If you are not a developer, perhaps you would like to help with translating the [documentation](https://github.com/D4Vinci/Scrapling/tree/dev/docs)?

 ## Making a Pull Request
 To ensure that your PR gets accepted, please make sure that your PR is based on the latest changes from the dev branch and that it satisfies the following requirements:
@@ -209,7 +209,7 @@ MySpider().start()

 <!-- /sponsors -->

-<i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci) and choose the tier that suites you!</sub></i>
+<i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci) and choose the tier that suits you!</sub></i>

 ---

@@ -471,7 +471,8 @@ Scrapling requires Python 3.10 or higher:
 pip install scrapling
 ```

-This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies.
+> [!IMPORTANT]
+> This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies. So importing anything from `scrapling.fetchers` or `scrapling.spiders`, like in the examples above, will raise `ModuleNotFoundError` with this installation alone. If you are going to use any of the fetchers or spiders, install the fetchers' dependencies first as shown below.

 ### Optional Dependencies

@@ -1,7 +1,7 @@
 ---
 name: scrapling-official
 description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
-version: "0.4.8"
+version: "0.4.9"
 license: Complete terms in LICENSE.txt
 metadata:
  homepage: "https://scrapling.readthedocs.io/en/latest/index.html"
@@ -40,7 +40,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo

 Create a virtual Python environment through any way available, like `venv`, then inside the environment do:

-`pip install "scrapling[all]>=0.4.8"`
+`pip install "scrapling[all]>=0.4.9"`

 Then do this to download all the browsers' dependencies:

@@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.
 Make sure Scrapling is installed:

 ```bash
-pip install "scrapling[all]>=0.4.8"
+pip install "scrapling[all]>=0.4.9"
 scrapling install --force
 ```

@@ -208,7 +208,7 @@ Docker alternative:

 ```bash
 docker pull pyd4vinci/scrapling
-docker run -i --rm scrapling mcp
+docker run -i --rm pyd4vinci/scrapling mcp
 ```

 The MCP server name when registering with a client is `ScraplingServer`. The command is the path to the `scrapling` binary and the argument is `mcp`.
@@ -467,7 +467,8 @@ Scrapling ليس قوياً فحسب - بل هو أيضاً سريع بشكل م
 pip install scrapling
 ```

-يتضمن هذا التثبيت فقط محرك المحلل وتبعياته، بدون أي جوالب أو تبعيات سطر الأوامر.
+> [!IMPORTANT]
+> يتضمن هذا التثبيت فقط محرك المحلل وتبعياته، بدون أي جوالب أو تبعيات سطر الأوامر. لذلك، فإن استيراد أي شيء من `scrapling.fetchers` أو `scrapling.spiders`، كما في الأمثلة أعلاه، سيؤدي إلى خطأ `ModuleNotFoundError` مع هذا التثبيت وحده. إذا كنت ستستخدم أيًا من الجوالب أو العناكب، فقم أولًا بتثبيت تبعيات الجوالب كما هو موضح أدناه.

 ### التبعيات الاختيارية

@@ -467,7 +467,8 @@ Scrapling 需要 Python 3.10 或更高版本：
 pip install scrapling
 ```

-此安装仅包括解析器引擎及其依赖项，没有任何 Fetcher 或命令行依赖项。
+> [!IMPORTANT]
+> 此安装仅包括解析器引擎及其依赖项，没有任何 Fetcher 或命令行依赖项。 因此，仅使用此安装时，像上面的示例那样从 `scrapling.fetchers` 或 `scrapling.spiders` 导入任何内容都会引发 `ModuleNotFoundError`。如果要使用任何 Fetcher 或 Spider，请先按照下面的说明安装 Fetcher 的依赖项。

 ### 可选依赖项

@@ -467,7 +467,8 @@ Scrapling erfordert Python 3.10 oder höher:
 pip install scrapling
 ```

-Diese Installation enthält nur die Parser-Engine und ihre Abhängigkeiten, ohne Fetcher oder Kommandozeilenabhängigkeiten.
+> [!IMPORTANT]
+> Diese Installation enthält nur die Parser-Engine und ihre Abhängigkeiten, ohne Fetcher oder Kommandozeilenabhängigkeiten. Daher führt der Import von allem aus `scrapling.fetchers` oder `scrapling.spiders`, wie in den Beispielen oben, mit dieser Installation allein zu einem `ModuleNotFoundError`. Wenn Sie einen der Fetcher oder Spider verwenden möchten, installieren Sie zuerst die Fetcher-Abhängigkeiten wie unten gezeigt.

 ### Optionale Abhängigkeiten

@@ -467,7 +467,8 @@ Scrapling requiere Python 3.10 o superior:
 pip install scrapling
 ```

-Esta instalación solo incluye el motor de análisis y sus dependencias, sin ningún fetcher ni dependencias de línea de comandos.
+> [!IMPORTANT]
+> Esta instalación solo incluye el motor de análisis y sus dependencias, sin ningún fetcher ni dependencias de línea de comandos. Por lo tanto, importar cualquier cosa desde `scrapling.fetchers` o `scrapling.spiders`, como en los ejemplos anteriores, lanzará un `ModuleNotFoundError` solo con esta instalación. Si va a usar alguno de los fetchers o spiders, instale primero las dependencias de los fetchers como se muestra a continuación.

 ### Dependencias Opcionales

@@ -467,7 +467,8 @@ Scrapling nécessite Python 3.10 ou supérieur :
 pip install scrapling
 ```

-Cette installation n'inclut que le moteur de parsing et ses dépendances, sans aucun fetcher ni dépendance en ligne de commande.
+> [!IMPORTANT]
+> Cette installation n'inclut que le moteur de parsing et ses dépendances, sans aucun fetcher ni dépendance en ligne de commande. Importer quoi que ce soit depuis `scrapling.fetchers` ou `scrapling.spiders`, comme dans les exemples ci-dessus, lèvera donc une `ModuleNotFoundError` avec cette seule installation. Si vous comptez utiliser l'un des fetchers ou spiders, installez d'abord les dépendances des fetchers comme indiqué ci-dessous.

 ### Dépendances optionnelles

@@ -467,7 +467,8 @@ Scrapling には Python 3.10 以上が必要です：
 pip install scrapling
 ```

-このインストールにはパーサーエンジンとその依存関係のみが含まれており、Fetcher やコマンドライン依存関係は含まれていません。
+> [!IMPORTANT]
+> このインストールにはパーサーエンジンとその依存関係のみが含まれており、Fetcher やコマンドライン依存関係は含まれていません。 そのため、このインストールのみでは、上記の例のように `scrapling.fetchers` や `scrapling.spiders` から何かをインポートすると `ModuleNotFoundError` が発生します。Fetcher や Spider を使用する場合は、以下のように、まず Fetcher の依存関係をインストールしてください。

 ### オプションの依存関係

@@ -467,7 +467,8 @@ Scrapling은 Python 3.10 이상이 필요합니다:
 pip install scrapling
 ```

-이 설치에는 파서 엔진과 의존성만 포함되며, Fetcher나 커맨드라인 의존성은 포함되지 않습니다.
+> [!IMPORTANT]
+> 이 설치에는 파서 엔진과 의존성만 포함되며, Fetcher나 커맨드라인 의존성은 포함되지 않습니다. 따라서 이 설치만으로는 위 예제처럼 `scrapling.fetchers`나 `scrapling.spiders`에서 무언가를 임포트하면 `ModuleNotFoundError`가 발생합니다. Fetcher나 Spider를 사용하려면 아래와 같이 먼저 Fetcher 의존성을 설치하세요.

 ### 선택적 의존성

@@ -469,7 +469,8 @@ O Scrapling requer Python 3.10 ou superior:
 pip install scrapling
 ```

-Esta instalação inclui apenas o motor de parsing e suas dependências, sem fetchers nem dependências de linha de comando.
+> [!IMPORTANT]
+> Esta instalação inclui apenas o motor de parsing e suas dependências, sem fetchers nem dependências de linha de comando. Portanto, importar qualquer coisa de `scrapling.fetchers` ou `scrapling.spiders`, como nos exemplos acima, lançará um `ModuleNotFoundError` apenas com esta instalação. Se você for usar algum dos fetchers ou spiders, instale primeiro as dependências dos fetchers como mostrado abaixo.

 ### Dependências Opcionais

@@ -470,7 +470,8 @@ Scrapling требует Python 3.10 или выше:
 pip install scrapling
 ```

-Эта установка включает только движок парсера и его зависимости, без каких-либо Fetcher'ов или зависимостей командной строки.
+> [!IMPORTANT]
+> Эта установка включает только движок парсера и его зависимости, без каких-либо Fetcher'ов или зависимостей командной строки. Поэтому импорт чего-либо из `scrapling.fetchers` или `scrapling.spiders`, как в примерах выше, вызовет `ModuleNotFoundError` при такой установке. Если вы собираетесь использовать какие-либо Fetcher'ы или Spider'ы, сначала установите зависимости Fetcher'ов, как показано ниже.

 ### Опциональные зависимости

@@ -50,7 +50,7 @@ The extract command is a set of simple terminal tools that:
    scrapling extract get "https://example.com" content.txt
  
    # Or use the Docker image with something like this:
-    docker run -v $(pwd)/output:/output scrapling extract get "https://blog.example.com" /output/article.md 
+    docker run -v $(pwd)/output:/output pyd4vinci/scrapling extract get "https://blog.example.com" /output/article.md 
    ```

 - **Extract Specific Content**
@@ -180,7 +180,9 @@ Scrapling requires Python 3.10 or higher:
 pip install scrapling
 ```

-This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies.
+!!! warning
+
+    This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies. So importing anything from `scrapling.fetchers` or `scrapling.spiders`, like in the examples above, will raise `ModuleNotFoundError` with this installation alone. If you are going to use any of the fetchers or spiders, install the fetchers' dependencies first as shown below.

 ### Optional Dependencies

@@ -1,6 +1,6 @@
-zensical>=0.0.41
+zensical>=0.0.44
 mkdocstrings>=1.0.4
-mkdocstrings-python>=2.0.3
+mkdocstrings-python>=2.0.4
 griffe-inherited-docstrings>=1.1.3
 griffe-runtime-objects>=0.3.1
 griffe-sphinx>=0.2.1
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "scrapling"
 # Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
-version = "0.4.8"
+version = "0.4.9"
 description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
 readme = {file = "README.md", content-type = "text/markdown"}
 license = {file = "LICENSE"}
@@ -61,7 +61,7 @@ classifiers = [
    "Typing :: Typed",
 ]
 dependencies = [
-    "lxml>=6.1.0",
+    "lxml>=6.1.1",
    "cssselect>=1.4.0",
    "orjson>=3.11.8",
    "tld>=0.13.2",
@@ -73,8 +73,8 @@ dependencies = [
 fetchers = [
    "click>=8.3.0",
    "curl_cffi>=0.15.0",
-    "playwright==1.59.0",
-    "patchright==1.59.1",
+    "playwright==1.60.0",
+    "patchright==1.60.1",
    "browserforge>=1.2.4",
    "apify-fingerprint-datapoints>=0.13.0",
    "msgspec>=0.21.1",
@@ -1,5 +1,5 @@
 __author__ = "Karim Shoair (karim.shoair@pm.me)"
-__version__ = "0.4.8"
+__version__ = "0.4.9"
 __copyright__ = "Copyright (c) 2024 Karim Shoair"

 from typing import Any, TYPE_CHECKING
@@ -2,6 +2,7 @@ from pathlib import Path
 from subprocess import check_output
 from sys import executable as python_executable

+from scrapling import __version__
 from scrapling.core.utils import log
 from scrapling.engines.toolbelt.custom import Response
 from scrapling.core.utils._shell import _CookieParser, _ParseHeaders
@@ -10,7 +11,7 @@ from scrapling.core._types import List, Optional, Dict, Tuple, Any, Callable
 from orjson import loads as json_loads, JSONDecodeError

 try:
-    from click import command, option, Choice, group, argument
+    from click import command, option, Choice, group, argument, version_option
 except (ImportError, ModuleNotFoundError) as e:
    raise ModuleNotFoundError(
        "You need to install scrapling with any of the extras to enable Shell commands. See: https://scrapling.readthedocs.io/en/latest/#installation"
@@ -650,6 +651,7 @@ def stealthy_fetch(


@group()
+@version_option(version=__version__, prog_name="Scrapling")
 def main():
    pass

@@ -19,6 +19,7 @@ from scrapling.core._types import (
    Unpack,
    Optional,
    Awaitable,
+    ProxyType,
    SUPPORTED_HTTP_METHODS,
    FollowRedirects,
 )
@@ -244,10 +245,11 @@ class _SyncSessionLogic(_ConfigurationLogic):

        try:
            for attempt in range(max_retries):
+                proxy: Optional[ProxyType]
                if self._proxy_rotator and static_proxy is None:
                    proxy = self._proxy_rotator.get_proxy()
                else:
-                    proxy = static_proxy
+                    proxy = static_proxy or self._default_proxy

                request_args = self._merge_request_args(stealth=stealth, proxy=proxy, **kwargs)
                try:
@@ -461,10 +463,11 @@ class _ASyncSessionLogic(_ConfigurationLogic):
        try:
            # Determine if we should use proxy rotation
            for attempt in range(max_retries):
+                proxy: Optional[ProxyType]
                if self._proxy_rotator and static_proxy is None:
                    proxy = self._proxy_rotator.get_proxy()
                else:
-                    proxy = static_proxy
+                    proxy = static_proxy or self._default_proxy

                request_args = self._merge_request_args(stealth=stealth, proxy=proxy, **kwargs)
                try:
@@ -10,7 +10,7 @@ from scrapling.core.utils import log
 from .custom import Response, StatusText
 from scrapling.core._types import Dict, List, Optional

-__CHARSET_RE__ = re_compile(r"charset=([\w-]+)")
+__CHARSET_RE__ = re_compile(r"""charset=["']?([\w-]+)""")


 class ResponseFactory:
@@ -13,8 +13,8 @@ from scrapling.core._types import Dict, Literal, Tuple
 __OS_NAME__ = platform_system()
 OSName = Literal["linux", "macos", "windows"]
 # Current versions hardcoded for now (Playwright doesn't allow to know the version of a browser without launching it)
-chromium_version = 147
-chrome_version = 147
+chromium_version = 148
+chrome_version = 148


@lru_cache(1, typed=True)
@@ -671,7 +671,7 @@ class Selector(SelectorsGeneration):
                    element_data = self.retrieve(identifier or selector)
                    if element_data:
                        elements = self.relocate(element_data, percentage)
-                        if elements is not None and auto_save:
+                        if elements and auto_save:
                            self.save(elements[0], identifier or selector)

                return self.__handle_elements(elements)
@@ -991,7 +991,9 @@ class Selector(SelectorsGeneration):
                SequenceMatcher(None, v, candidate_attributes.get(k, "")).ratio()
                for k, v in original_attributes.items()
            )
-            checks += len(candidate_attributes)
+            # Using `max` so candidates with extra attributes are penalized and candidates
+            # with fewer attributes don't get inflated scores from a smaller denominator
+            checks += max(len(original_attributes), len(candidate_attributes))
        else:
            if not candidate_attributes:
                # Both don't have attributes, this must mean something
@@ -64,7 +64,7 @@ class ResponseCacheManager:
            async with await anyio.open_file(temp_path, "wb") as f:
                await f.write(serialized)

-            await temp_path.rename(self._cache_path(fingerprint))
+            await temp_path.replace(self._cache_path(fingerprint))
        except Exception as e:
            if await temp_path.exists():
                await temp_path.unlink()
@@ -50,7 +50,7 @@ class CheckpointManager:
            async with await anyio.open_file(temp_path, "wb") as f:
                await f.write(serialized)

-            await temp_path.rename(self._checkpoint_path)
+            await temp_path.replace(self._checkpoint_path)

            log.info(f"Checkpoint saved: {len(data.requests)} requests, {len(data.seen)} seen URLs")
        except Exception as e:
@@ -14,12 +14,12 @@
      "mimeType": "image/png"
    }
  ],
-  "version": "0.4.8",
+  "version": "0.4.9",
  "packages": [
    {
      "registryType": "pypi",
      "identifier": "scrapling",
-      "version": "0.4.8",
+      "version": "0.4.9",
      "runtimeHint": "uvx",
      "packageArguments": [
        {
@@ -1,6 +1,6 @@
 [metadata]
 name = scrapling
-version = 0.4.8
+version = 0.4.9
 author = Karim Shoair
 author_email = karim.shoair@pm.me
 description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!
@@ -4,8 +4,9 @@ from unittest.mock import patch, MagicMock
 import pytest_httpbin

 from scrapling.parser import Selector
+from scrapling import __version__
 from scrapling.cli import (
-    shell, mcp, get, post, put, delete, fetch, stealthy_fetch
+    main, shell, mcp, get, post, put, delete, fetch, stealthy_fetch
 )


@@ -32,6 +33,12 @@ class TestCLI:
    def runner(self):
        return CliRunner()

+    def test_version_flag(self, runner):
+        """Test that the --version flag prints the Scrapling version and exits"""
+        result = runner.invoke(main, ['--version'])
+        assert result.exit_code == 0
+        assert result.output.strip() == f'Scrapling, version {__version__}'
+
    def test_shell_command(self, runner):
        """Test shell command"""
        with patch('scrapling.core.shell.CustomShell') as mock_shell:
@@ -1,6 +1,9 @@
+import pytest
+from unittest.mock import patch, MagicMock, AsyncMock
+from curl_cffi.curl import CurlError

-
-from scrapling.engines.static import AsyncFetcherClient
+from scrapling.engines.static import _ASyncSessionLogic as AsyncFetcherSession, AsyncFetcherClient
+from scrapling.engines.toolbelt import ProxyRotator


 class TestFetcherSession:
@@ -13,3 +16,47 @@ class TestFetcherSession:
        # Should not have context manager methods
        assert client.__aenter__ is None
        assert client.__aexit__ is None
+
+    @pytest.mark.asyncio
+    async def test_session_level_proxy_is_applied(self):
+        """Session-level proxy must reach the request, not be silently dropped (#295)"""
+        proxy = "http://10.255.255.1:9999"
+
+        async with AsyncFetcherSession(proxy=proxy) as session:
+            with (
+                patch.object(session._async_curl_session, "request", new=AsyncMock()) as mocked_request,
+                patch("scrapling.engines.static.ResponseFactory.from_http_request", return_value=MagicMock()),
+            ):
+                await session.get("http://example.com")
+
+            assert mocked_request.call_args.kwargs["proxy"] == proxy
+
+    @pytest.mark.asyncio
+    async def test_per_request_proxy_overrides_session_proxy(self):
+        """A per-request proxy must take precedence over the session-level proxy"""
+        request_proxy = "http://10.255.255.2:9999"
+
+        async with AsyncFetcherSession(proxy="http://10.255.255.1:9999") as session:
+            with (
+                patch.object(session._async_curl_session, "request", new=AsyncMock()) as mocked_request,
+                patch("scrapling.engines.static.ResponseFactory.from_http_request", return_value=MagicMock()),
+            ):
+                await session.get("http://example.com", proxy=request_proxy)
+
+            assert mocked_request.call_args.kwargs["proxy"] == request_proxy
+
+    @pytest.mark.asyncio
+    async def test_proxy_rotates_per_retry_attempt(self):
+        """With a rotator, every retry attempt must pull a fresh proxy"""
+        rotator = ProxyRotator(["http://p1:8080", "http://p2:8080"])
+
+        async with AsyncFetcherSession(proxy_rotator=rotator, retries=2, retry_delay=0) as session:
+            with (
+                patch.object(session._async_curl_session, "request", new=AsyncMock()) as mocked_request,
+                patch("scrapling.engines.static.ResponseFactory.from_http_request", return_value=MagicMock()),
+            ):
+                mocked_request.side_effect = [CurlError("transient"), MagicMock()]
+                await session.get("http://example.com")
+
+            proxies_used = [call.kwargs["proxy"] for call in mocked_request.call_args_list]
+            assert proxies_used == ["http://p1:8080", "http://p2:8080"]
@@ -1,7 +1,9 @@
 import pytest
-
+from unittest.mock import patch, MagicMock
+from curl_cffi.curl import CurlError

 from scrapling.engines.static import _SyncSessionLogic as FetcherSession, FetcherClient
+from scrapling.engines.toolbelt import ProxyRotator


 class TestFetcherSession:
@@ -9,11 +11,7 @@ class TestFetcherSession:

    def test_fetcher_session_creation(self):
        """Test FetcherSession creation"""
-        session = FetcherSession(
-            timeout=30,
-            retries=3,
-            stealthy_headers=True
-        )
+        session = FetcherSession(timeout=30, retries=3, stealthy_headers=True)

        assert session._default_timeout == 30
        assert session._default_retries == 3
@@ -43,3 +41,44 @@ class TestFetcherSession:
        # Should not have context manager methods
        assert client.__enter__ is None
        assert client.__exit__ is None
+
+    def test_session_level_proxy_is_applied(self):
+        """Session-level proxy must reach the request, not be silently dropped (#295)"""
+        proxy = "http://10.255.255.1:9999"
+
+        with FetcherSession(proxy=proxy) as session:
+            with (
+                patch.object(session._curl_session, "request") as mocked_request,
+                patch("scrapling.engines.static.ResponseFactory.from_http_request", return_value=MagicMock()),
+            ):
+                session.get("http://example.com")
+
+            assert mocked_request.call_args.kwargs["proxy"] == proxy
+
+    def test_per_request_proxy_overrides_session_proxy(self):
+        """A per-request proxy must take precedence over the session-level proxy"""
+        request_proxy = "http://10.255.255.2:9999"
+
+        with FetcherSession(proxy="http://10.255.255.1:9999") as session:
+            with (
+                patch.object(session._curl_session, "request") as mocked_request,
+                patch("scrapling.engines.static.ResponseFactory.from_http_request", return_value=MagicMock()),
+            ):
+                session.get("http://example.com", proxy=request_proxy)
+
+            assert mocked_request.call_args.kwargs["proxy"] == request_proxy
+
+    def test_proxy_rotates_per_retry_attempt(self):
+        """With a rotator, every retry attempt must pull a fresh proxy"""
+        rotator = ProxyRotator(["http://p1:8080", "http://p2:8080"])
+
+        with FetcherSession(proxy_rotator=rotator, retries=2, retry_delay=0) as session:
+            with (
+                patch.object(session._curl_session, "request") as mocked_request,
+                patch("scrapling.engines.static.ResponseFactory.from_http_request", return_value=MagicMock()),
+            ):
+                mocked_request.side_effect = [CurlError("transient"), MagicMock()]
+                session.get("http://example.com")
+
+            proxies_used = [call.kwargs["proxy"] for call in mocked_request.call_args_list]
+            assert proxies_used == ["http://p1:8080", "http://p2:8080"]
@@ -1,5 +1,6 @@
 import pytest

+from scrapling.engines.toolbelt.convertor import ResponseFactory
 from scrapling.engines.toolbelt.custom import StatusText, Response
 from scrapling.engines.toolbelt.navigation import (
    construct_proxy_dict,
@@ -139,6 +140,32 @@ def test_unknown_status_code():
    assert StatusText.get(1000) == "Unknown Status Code"


+# The private classmethod is name-mangled; resolve it once for the tests below.
+_extract_encoding = getattr(ResponseFactory, "_ResponseFactory__extract_browser_encoding")
+
+
+def test_browser_encoding_unquoted_charset():
+    """A charset declared without quotes is returned verbatim."""
+    assert _extract_encoding("text/html; charset=utf-8") == "utf-8"
+    assert _extract_encoding("text/html; charset=ISO-8859-1") == "ISO-8859-1"
+    assert _extract_encoding("text/html;charset=windows-1252") == "windows-1252"
+
+
+def test_browser_encoding_quoted_charset():
+    """A quoted charset value (RFC 7231 allows quoting) is unwrapped, not dropped."""
+    assert _extract_encoding('text/html; charset="utf-8"') == "utf-8"
+    assert _extract_encoding('text/html; charset="ISO-8859-1"') == "ISO-8859-1"
+    assert _extract_encoding("text/html; charset='Shift_JIS'") == "Shift_JIS"
+    assert _extract_encoding('text/plain; charset="windows-1252"; boundary=x') == "windows-1252"
+
+
+def test_browser_encoding_defaults_when_missing():
+    """Fall back to the default when no charset is present or the header is empty."""
+    assert _extract_encoding("text/html") == "utf-8"
+    assert _extract_encoding("") == "utf-8"
+    assert _extract_encoding(None) == "utf-8"
+
+
 class TestConstructProxyDict:
    """Test proxy dictionary construction"""

@@ -56,6 +56,32 @@ class TestParserAdaptive:
        assert relocated[0].has_class("new-class")
        assert relocated[0].css(".new-description")[0].text == "Description 1"

+    def test_relocation_auto_save_no_match_above_threshold(self):
+        """Adaptive relocation with `auto_save=True` must not crash when no element
+        clears the `percentage` threshold (relocate() returns an empty list)."""
+        original_html = """
+                <div class="container">
+                    <article class="product" id="target">
+                        <h3>Widget</h3>
+                        <p class="desc">A widget</p>
+                    </article>
+                </div>
+                """
+        # Unrelated structure so nothing can match a high threshold
+        changed_html = "<html><body><span>totally unrelated content</span></body></html>"
+
+        old_page = Selector(original_html, url="example.com", adaptive=True)
+        new_page = Selector(changed_html, url="example.com", adaptive=True)
+
+        old_page.css("#target", identifier="target", auto_save=True)
+
+        # Before the fix this raised `IndexError: list index out of range` because the
+        # guard checked `elements is not None` but relocate() returns [] (never None).
+        result = new_page.css(
+            "#target", identifier="target", adaptive=True, auto_save=True, percentage=95
+        )
+        assert list(result) == []
+
    @pytest.mark.asyncio
    async def test_element_relocation_async(self):
        """Test relocating element after structure change in async mode"""
@@ -2,6 +2,7 @@
 Tests for Selector.find_similar() with non-default parameters.
 Target file: tests/parser/test_general.py (append to TestSimilarElements class)
 """
+
 import pytest
 from scrapling import Selector

@@ -61,14 +62,10 @@ class TestFindSimilarAdvanced:
        first = product_page.css("div.product")[0]
        # Ignore both data-price and data-category → only class matters → all 3 divs match
        ignore_all_data = first.find_similar(
-            similarity_threshold=0.2,
-            ignore_attributes=["data-price", "data-category"]
+            similarity_threshold=0.2, ignore_attributes=["data-price", "data-category"]
        )
        # Ignore nothing → data-category difference (fruit vs veggie) may reduce matches
-        ignore_nothing = first.find_similar(
-            similarity_threshold=0.9,
-            ignore_attributes=[]
-        )
+        ignore_nothing = first.find_similar(similarity_threshold=0.9, ignore_attributes=[])
        assert len(ignore_all_data) >= len(ignore_nothing)

    def test_find_similar_on_text_node_returns_empty(self, product_page):
@@ -76,3 +73,31 @@ class TestFindSimilarAdvanced:
        text_node = product_page.css(".name::text")[0]
        result = text_node.find_similar()
        assert len(result) == 0
+
+    def test_find_similar_attribute_count_mismatch_scoring(self):
+        """The similarity denominator uses max() of both attribute counts, so candidates
+        with fewer attributes don't get inflated scores and candidates with extra
+        attributes stay penalized."""
+        html = """
+        <html><body>
+            <div class="cards">
+                <div class="card" data-kind="primary" data-color="red" data-size="large">Alpha</div>
+                <div class="card">Beta</div>
+                <div class="card" data-kind="primary" data-color="red" data-size="large" data-id="x">Gamma</div>
+                <div class="card" data-kind="primary" data-color="red" data-size="large">Delta</div>
+            </div>
+        </body></html>
+        """
+        page = Selector(html, adaptive=False)
+        first = page.css("div.card")[0]  # Alpha
+
+        similar = first.find_similar(similarity_threshold=0.9, ignore_attributes=[])
+        texts = {el.text for el in similar}
+
+        # An exact attribute match must pass
+        assert "Delta" in texts
+        # Beta matches 1 of Alpha's 4 attributes; the old denominator counted candidate
+        # attributes only, inflating it to a perfect score (1.0 / 1)
+        assert "Beta" not in texts
+        # Gamma's extra attribute dilutes the score (4.0 / 5) - the intentional penalty
+        assert "Gamma" not in texts
@@ -1,6 +1,6 @@
 pytest>=2.8.0,<9
 pytest-cov
-playwright==1.59.0
+playwright==1.60.0
 werkzeug<3.0.0
 pytest-httpbin==2.1.0
 pytest-asyncio
@@ -49,6 +49,27 @@ class TestResponseCacheManager:
            assert dict(restored.headers) == dict(original.headers)
            assert dict(restored.request_headers) == dict(original.request_headers)

+    @pytest.mark.anyio
+    async def test_put_overwrites_existing_entry(self):
+        """Re-caching the same fingerprint must replace the stored response.
+
+        Regression test for a Windows-only failure: ``Path.rename`` cannot
+        overwrite an existing destination on Windows (raising ``WinError 183``),
+        so the second ``put`` was caught by the error handler, the temp file was
+        removed, and ``get`` kept returning the stale body. ``Path.replace``
+        overwrites atomically on every platform.
+        """
+        with tempfile.TemporaryDirectory() as tmpdir:
+            cache = ResponseCacheManager(tmpdir)
+            fp = b"\x05" * 20
+
+            await cache.put(fp, _make_response(body=b"<html>first</html>"), "GET")
+            await cache.put(fp, _make_response(body=b"<html>second</html>"), "GET")
+
+            restored = await cache.get(fp)
+            assert restored is not None
+            assert restored.body == b"<html>second</html>"
+
    @pytest.mark.anyio
    async def test_get_cache_miss(self):
        with tempfile.TemporaryDirectory() as tmpdir:
@@ -10,8 +10,8 @@ envlist = pre-commit,py{310,311,312,313}
 usedevelop = True
 changedir = tests
 deps =
-    playwright==1.59.0
-    patchright==1.59.1
+    playwright==1.60.0
+    patchright==1.60.1
    -r{toxinidir}/tests/requirements.txt
 extras = ai,shell
 commands =