diff --git a/README.md b/README.md index dfb7d7f..ea56b15 100644 --- a/README.md +++ b/README.md @@ -185,7 +185,6 @@ MySpider().start() - diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip index 0bbd64c..fb76762 100644 Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ diff --git a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md index a87fdf6..1a4c96d 100644 --- a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md +++ b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md @@ -44,7 +44,7 @@ Instead of launching a browser locally (Chromium/Google Chrome), you can connect **Notes:** * There was a `stealth` option here, but it was moved to the `StealthyFetcher` class, as explained on the next page, with additional features since version 0.3.13. -* This makes it less confusing for new users, easier to maintain, and provides other benefits, as explained on the [StealthyFetcher page](fetching/stealthy.md). +* This makes it less confusing for new users, easier to maintain, and provides other benefits, as explained on the [StealthyFetcher page](stealthy.md). ## Full list of arguments All arguments for `DynamicFetcher` and its session classes: diff --git a/agent-skill/Scrapling-Skill/references/spiders/architecture.md b/agent-skill/Scrapling-Skill/references/spiders/architecture.md index 9e7586f..de95d24 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/architecture.md +++ b/agent-skill/Scrapling-Skill/references/spiders/architecture.md @@ -11,8 +11,8 @@ Here's what happens step by step when you run a spider: 1. The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic. 2. The **Scheduler** receives requests and places them in a priority queue, and creates fingerprints for them. Higher-priority requests are dequeued first. 3. The **Crawler Engine** asks the **Scheduler** to dequeue the next request, respecting concurrency limits (global and per-domain) and download delays. Once the **Crawler Engine** receives the request, it passes it to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID). -4. The **session** fetches the page and returns a [Response](fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized. -5. The **Crawler Engine** passes the [Response](fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing. +4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized. +5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing. 6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused. 7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state. @@ -50,9 +50,9 @@ A priority queue with built-in URL deduplication. Requests are fingerprinted bas Manages one or more named session instances. Each session is one of: -- [FetcherSession](fetching/static.md) -- [AsyncDynamicSession](fetching/dynamic.md) -- [AsyncStealthySession](fetching/stealthy.md) +- [FetcherSession](../fetching/static.md) +- [AsyncDynamicSession](../fetching/dynamic.md) +- [AsyncStealthySession](../fetching/stealthy.md) When a request comes in, the Session Manager routes it to the correct session based on the request's `sid` field. Sessions can be started with the spider start (default) or lazily (started on the first use). diff --git a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md index 1446779..2a9e6c8 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md +++ b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md @@ -25,7 +25,7 @@ Every spider needs three things: 2. **`start_urls`** — A list of URLs to start crawling from. 3. **`parse()`** — An async generator method that processes each response and yields results. -The `parse()` method processes each response. You use the same selection methods you'd use with Scrapling's [Selector](parsing/main_classes.md#selector)/[Response](fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items. +The `parse()` method processes each response. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items. ## Running the Spider diff --git a/agent-skill/Scrapling-Skill/references/spiders/sessions.md b/agent-skill/Scrapling-Skill/references/spiders/sessions.md index 82a21ac..761cea6 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/sessions.md +++ b/agent-skill/Scrapling-Skill/references/spiders/sessions.md @@ -6,14 +6,14 @@ A spider can use multiple fetcher sessions simultaneously — for example, a fas A session is a pre-configured fetcher instance that stays alive for the duration of the crawl. Instead of creating a new connection or browser for every request, the spider reuses sessions, which is faster and more resource-efficient. -By default, every spider creates a single [FetcherSession](fetching/static.md). You can add more sessions or swap the default by overriding the `configure_sessions()` method, but you have to use the async version of each session only, as the table shows below: +By default, every spider creates a single [FetcherSession](../fetching/static.md). You can add more sessions or swap the default by overriding the `configure_sessions()` method, but you have to use the async version of each session only, as the table shows below: | Session Type | Use Case | |-------------------------------------------------|------------------------------------------| -| [FetcherSession](fetching/static.md) | Fast HTTP requests, no JavaScript | -| [AsyncDynamicSession](fetching/dynamic.md) | Browser automation, JavaScript rendering | -| [AsyncStealthySession](fetching/stealthy.md) | Anti-bot bypass, Cloudflare, etc. | +| [FetcherSession](../fetching/static.md) | Fast HTTP requests, no JavaScript | +| [AsyncDynamicSession](../fetching/dynamic.md) | Browser automation, JavaScript rendering | +| [AsyncStealthySession](../fetching/stealthy.md) | Anti-bot bypass, Cloudflare, etc. | ## Configuring Sessions diff --git a/docs/README_AR.md b/docs/README_AR.md index 7573a10..68c6a51 100644 --- a/docs/README_AR.md +++ b/docs/README_AR.md @@ -180,7 +180,6 @@ MySpider().start() - diff --git a/docs/README_CN.md b/docs/README_CN.md index be91908..469b703 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -180,7 +180,6 @@ MySpider().start() - diff --git a/docs/README_DE.md b/docs/README_DE.md index 88ebd0a..d5c30b9 100644 --- a/docs/README_DE.md +++ b/docs/README_DE.md @@ -180,7 +180,6 @@ MySpider().start() - diff --git a/docs/README_ES.md b/docs/README_ES.md index 959bc05..d32cb69 100644 --- a/docs/README_ES.md +++ b/docs/README_ES.md @@ -180,7 +180,6 @@ MySpider().start() - diff --git a/docs/README_FR.md b/docs/README_FR.md index 9d4212a..09180d6 100644 --- a/docs/README_FR.md +++ b/docs/README_FR.md @@ -180,7 +180,6 @@ MySpider().start() - diff --git a/docs/README_JP.md b/docs/README_JP.md index dde6f5b..4b6a0c0 100644 --- a/docs/README_JP.md +++ b/docs/README_JP.md @@ -180,7 +180,6 @@ MySpider().start() - diff --git a/docs/README_KR.md b/docs/README_KR.md index dad94ae..5b29439 100644 --- a/docs/README_KR.md +++ b/docs/README_KR.md @@ -180,7 +180,6 @@ MySpider().start() - diff --git a/docs/README_RU.md b/docs/README_RU.md index 99b347a..6a97f17 100644 --- a/docs/README_RU.md +++ b/docs/README_RU.md @@ -183,7 +183,6 @@ MySpider().start() - diff --git a/images/rapidproxy.jpg b/images/rapidproxy.jpg deleted file mode 100644 index dee0b35..0000000 Binary files a/images/rapidproxy.jpg and /dev/null differ diff --git a/scrapling/core/custom_types.py b/scrapling/core/custom_types.py index d06ac29..53255be 100644 --- a/scrapling/core/custom_types.py +++ b/scrapling/core/custom_types.py @@ -112,10 +112,10 @@ class TextHandler(str): def get(self, default=None): # pragma: no cover return self - def get_all(self): # pragma: no cover + def getall(self): # pragma: no cover return self - extract = get_all + extract = getall extract_first = get def json(self) -> Dict: @@ -279,7 +279,7 @@ class TextHandlers(List[TextHandler]): return self extract_first = get - get_all = extract + getall = extract class AttributesHandler(Mapping[str, _TextHandlerType]): diff --git a/scrapling/engines/toolbelt/convertor.py b/scrapling/engines/toolbelt/convertor.py index 1a8d799..8606003 100644 --- a/scrapling/engines/toolbelt/convertor.py +++ b/scrapling/engines/toolbelt/convertor.py @@ -187,34 +187,34 @@ class ResponseFactory: return history @classmethod - def _get_page_content(cls, page: SyncPage) -> str: + def _get_page_content(cls, page: SyncPage, max_retries: int = 20) -> str: """ A workaround for the Playwright issue with `page.content()` on Windows. Ref.: https://github.com/microsoft/playwright/issues/16108 :param page: The page to extract content from. + :param max_retries: Maximum number of retry attempts before raising `RuntimeError`. :return: """ - while True: + for _ in range(max_retries): try: return page.content() or "" except PlaywrightError: page.wait_for_timeout(500) - continue - return "" # pyright: ignore + raise RuntimeError(f"Failed to retrieve the page content after retrying for {max_retries * 500}ms.") @classmethod - async def _get_async_page_content(cls, page: AsyncPage) -> str: + async def _get_async_page_content(cls, page: AsyncPage, max_retries: int = 20) -> str: """ A workaround for the Playwright issue with `page.content()` on Windows. Ref.: https://github.com/microsoft/playwright/issues/16108 :param page: The page to extract content from. + :param max_retries: Maximum number of retry attempts before raising `RuntimeError`. :return: """ - while True: + for _ in range(max_retries): try: return (await page.content()) or "" except PlaywrightError: await page.wait_for_timeout(500) - continue - return "" # pyright: ignore + raise RuntimeError(f"Failed to retrieve the page content after retrying for {max_retries * 500}ms.") @classmethod async def from_async_playwright_response( diff --git a/scrapling/spiders/engine.py b/scrapling/spiders/engine.py index 416911d..d77f838 100644 --- a/scrapling/spiders/engine.py +++ b/scrapling/spiders/engine.py @@ -205,7 +205,7 @@ class CrawlerEngine: Returns True if successfully restored, False otherwise. """ if not self._checkpoint_system_enabled: - raise + return False data = await self._checkpoint_manager.load() if data is None: diff --git a/scrapling/spiders/session.py b/scrapling/spiders/session.py index cc07042..536be6d 100644 --- a/scrapling/spiders/session.py +++ b/scrapling/spiders/session.py @@ -112,10 +112,12 @@ class SessionManager: client = session._client if isinstance(client, _ASyncSessionLogic): + kwargs = request._session_kwargs.copy() + method = cast(SUPPORTED_HTTP_METHODS, kwargs.pop("method", "GET")) response = await client._make_request( - method=cast(SUPPORTED_HTTP_METHODS, request._session_kwargs.pop("method", "GET")), + method=method, url=request.url, - **request._session_kwargs, + **kwargs, ) else: # Sync session or other types - shouldn't happen in async context diff --git a/tests/parser/test_ancestor_navigation.py b/tests/parser/test_ancestor_navigation.py new file mode 100644 index 0000000..1814ec5 --- /dev/null +++ b/tests/parser/test_ancestor_navigation.py @@ -0,0 +1,66 @@ +""" +Tests for Selector.iterancestors() and Selector.find_ancestor() methods. +Target file: tests/parser/test_general.py (append to TestElementNavigation class) +""" +import pytest +from scrapling import Selector + + +@pytest.fixture +def nested_page(): + html = """ + +
+
+
+

deep text

+
+
+
+ + """ + return Selector(html, adaptive=False) + + +class TestAncestorNavigation: + def test_iterancestors_returns_all_ancestors(self, nested_page): + """iterancestors() should yield every ancestor up to """ + target = nested_page.css("#target")[0] + ancestor_tags = [a.tag for a in target.iterancestors()] + # Expected order: p → article → section → div → body → html + assert ancestor_tags[:4] == ["p", "article", "section", "div"] + assert "body" in ancestor_tags + assert "html" in ancestor_tags + + def test_iterancestors_order_is_bottom_up(self, nested_page): + """iterancestors() should start from the immediate parent, not the root""" + target = nested_page.css("#target")[0] + first_ancestor = next(target.iterancestors()) + assert first_ancestor.attrib.get("id") == "level4" + + def test_find_ancestor_returns_first_match(self, nested_page): + """find_ancestor() should return the closest ancestor matching the predicate""" + target = nested_page.css("#target")[0] + # Looking for the nearest ancestor with class "card" + result = target.find_ancestor(lambda el: el.has_class("card")) + assert result is not None + assert result.attrib.get("id") == "level3" + + def test_find_ancestor_returns_none_when_not_found(self, nested_page): + """find_ancestor() should return None if no ancestor matches""" + target = nested_page.css("#target")[0] + result = target.find_ancestor(lambda el: el.has_class("nonexistent-class")) + assert result is None + + def test_iterancestors_on_text_node_is_empty(self, nested_page): + """iterancestors() on a text node should yield nothing (not raise)""" + text_node = nested_page.css("#target::text")[0] + ancestors = list(text_node.iterancestors()) + assert ancestors == [] + + def test_find_ancestor_on_root_element_returns_none(self, nested_page): + """find_ancestor() on the root element should return None gracefully""" + # html element has no ancestors + html_el = nested_page.css("html")[0] + result = html_el.find_ancestor(lambda el: True) + assert result is None diff --git a/tests/parser/test_find_similar_advanced.py b/tests/parser/test_find_similar_advanced.py new file mode 100644 index 0000000..480f155 --- /dev/null +++ b/tests/parser/test_find_similar_advanced.py @@ -0,0 +1,78 @@ +""" +Tests for Selector.find_similar() with non-default parameters. +Target file: tests/parser/test_general.py (append to TestSimilarElements class) +""" +import pytest +from scrapling import Selector + + +@pytest.fixture +def product_page(): + html = """ + +
+
+ Apple +
+
+ Banana +
+
+ Carrot +
+ +
+ Grape +
+
+ + """ + return Selector(html, adaptive=False) + + +class TestFindSimilarAdvanced: + def test_find_similar_default_finds_same_tag_siblings(self, product_page): + """find_similar() with defaults should find div.product siblings, not the section""" + first = product_page.css("div.product")[0] + similar = first.find_similar() + tags = [el.tag for el in similar] + assert all(t == "div" for t in tags), "Should only return
elements" + assert len(similar) == 2 # Banana and Carrot, not Grape (section) + + def test_find_similar_high_threshold_filters_more(self, product_page): + """A higher similarity_threshold should return fewer (or equal) results""" + first = product_page.css("div.product")[0] + low_threshold = first.find_similar(similarity_threshold=0.1) + high_threshold = first.find_similar(similarity_threshold=0.9) + assert len(high_threshold) <= len(low_threshold) + + def test_find_similar_match_text_excludes_different_text(self, product_page): + """match_text=True should factor in text content during similarity scoring""" + first = product_page.css("div.product")[0] # Apple + # With match_text=True and a high threshold, "Apple" vs "Banana"/"Carrot" text + # should reduce similarity scores — result count may drop + with_text = first.find_similar(similarity_threshold=0.8, match_text=True) + without_text = first.find_similar(similarity_threshold=0.8, match_text=False) + # match_text=True is stricter when text differs, so result should be <= without_text + assert len(with_text) <= len(without_text) + + def test_find_similar_ignore_attributes_affects_matching(self, product_page): + """Ignoring data-price should make more elements qualify as similar""" + first = product_page.css("div.product")[0] + # Ignore both data-price and data-category → only class matters → all 3 divs match + ignore_all_data = first.find_similar( + similarity_threshold=0.2, + ignore_attributes=["data-price", "data-category"] + ) + # Ignore nothing → data-category difference (fruit vs veggie) may reduce matches + ignore_nothing = first.find_similar( + similarity_threshold=0.9, + ignore_attributes=[] + ) + assert len(ignore_all_data) >= len(ignore_nothing) + + def test_find_similar_on_text_node_returns_empty(self, product_page): + """find_similar() on a text node should return empty Selectors without raising""" + text_node = product_page.css(".name::text")[0] + result = text_node.find_similar() + assert len(result) == 0 diff --git a/tests/parser/test_parser_advanced.py b/tests/parser/test_parser_advanced.py index f34ba5c..39829bb 100644 --- a/tests/parser/test_parser_advanced.py +++ b/tests/parser/test_parser_advanced.py @@ -250,6 +250,68 @@ class TestTextHandlerAdvanced: matches = text3.re(r"He l lo", clean_match=True, case_sensitive=False) assert len(matches) == 1 + def test_text_handler_regex_check_match(self): + """Test TextHandler.re() with check_match=True returns bool""" + text = TextHandler("Price: $10.99") + assert text.re(r"\$[\d.]+", check_match=True) is True + assert text.re(r"no-match-pattern", check_match=True) is False + + def test_text_handler_regex_replace_entities_false(self): + """Test TextHandler.re() with replace_entities=False preserves entities""" + text = TextHandler("Hello & World") + results = text.re(r"&", replace_entities=False) + assert len(results) == 1 + assert results[0] == "&" + + def test_text_handler_regex_with_groups(self): + """Test TextHandler.re() with capture groups flattens results""" + text = TextHandler("name=Alice age=30 name=Bob age=25") + results = text.re(r"name=(\w+) age=(\d+)") + assert len(results) == 4 + assert "Alice" in results + assert "30" in results + + def test_text_handler_re_first_with_default(self): + """Test TextHandler.re_first() returns default when no match""" + text = TextHandler("no numbers here") + result = text.re_first(r"\d+", default="N/A") + assert result == "N/A" + + def test_text_handler_re_first_returns_first_match(self): + """Test TextHandler.re_first() returns first match""" + text = TextHandler("a1 b2 c3") + result = text.re_first(r"\d") + assert result == "1" + assert isinstance(result, TextHandler) + + def test_text_handler_clean_with_entities(self): + """Test TextHandler.clean() with remove_entities=True""" + text = TextHandler("Hello\t&\nWorld") + cleaned = text.clean(remove_entities=True) + assert "&" not in cleaned + assert "&" in cleaned + assert "\t" not in cleaned + assert "\n" not in cleaned + + def test_text_handler_clean_without_entities(self): + """Test TextHandler.clean() preserves entities by default""" + text = TextHandler("Hello\t&\nWorld") + cleaned = text.clean(remove_entities=False) + assert "&" in cleaned + + def test_text_handler_json_valid(self): + """Test TextHandler.json() with valid JSON""" + text = TextHandler('{"key": "value", "num": 42}') + data = text.json() + assert data["key"] == "value" + assert data["num"] == 42 + + def test_text_handler_json_invalid(self): + """Test TextHandler.json() raises on invalid JSON""" + text = TextHandler("not json") + with pytest.raises(Exception): + text.json() + def test_text_handlers_operations(self): """Test TextHandlers list operations""" handlers = TextHandlers([ @@ -266,6 +328,37 @@ class TestTextHandlerAdvanced: assert handlers.get("default") == "First" assert TextHandlers([]).get("default") == "default" + def test_text_handlers_re(self): + """Test TextHandlers.re() flattens results across all elements""" + handlers = TextHandlers([ + TextHandler("a1 b2"), + TextHandler("c3 d4"), + ]) + results = handlers.re(r"[a-z]\d") + assert isinstance(results, TextHandlers) + assert len(results) == 4 + assert results[0] == "a1" + assert results[3] == "d4" + + def test_text_handlers_re_empty(self): + """Test TextHandlers.re() on empty list""" + handlers = TextHandlers([]) + results = handlers.re(r"\d+") + assert isinstance(results, TextHandlers) + assert len(results) == 0 + + def test_text_handlers_re_no_matches(self): + """Test TextHandlers.re() when no element matches""" + handlers = TextHandlers([TextHandler("abc"), TextHandler("def")]) + results = handlers.re(r"\d+") + assert len(results) == 0 + + def test_text_handlers_extract(self): + """Test TextHandlers.extract() returns self""" + handlers = TextHandlers([TextHandler("a"), TextHandler("b")]) + assert handlers.extract() is handlers + assert handlers.getall() is handlers + class TestSelectorsAdvanced: """Test advanced Selectors functionality""" diff --git a/tests/parser/test_selectors_filter.py b/tests/parser/test_selectors_filter.py new file mode 100644 index 0000000..9612abc --- /dev/null +++ b/tests/parser/test_selectors_filter.py @@ -0,0 +1,64 @@ +""" +Tests for Selectors.filter() method edge cases. +Target file: tests/parser/test_parser_advanced.py (append to TestAdvancedSelectors class) +""" +import pytest +from scrapling import Selector, Selectors + + +@pytest.fixture +def page(): + html = """ + + + + """ + return Selector(html, adaptive=False) + + +class TestSelectorsFilter: + def test_filter_basic(self, page): + """filter() should return only elements matching the predicate""" + items = page.css("li.item") + expensive = items.filter(lambda el: int(el.attrib.get("data-value", 0)) >= 10) + assert len(expensive) == 2 + texts = expensive.getall() + assert any("Apple" in t for t in texts) + assert any("Cherry" in t for t in texts) + + def test_filter_returns_empty_selectors_when_no_match(self, page): + """filter() should return an empty Selectors (not None/exception) when nothing matches""" + items = page.css("li.item") + result = items.filter(lambda el: int(el.attrib.get("data-value", 0)) > 9999) + assert isinstance(result, Selectors) + assert len(result) == 0 + assert result.first is None + + def test_filter_all_pass(self, page): + """filter() with always-True predicate should return all elements""" + items = page.css("li.item") + result = items.filter(lambda el: True) + assert len(result) == len(items) + + def test_filter_chained(self, page): + """filter() should be chainable — apply two filters in sequence""" + items = page.css("li.item") + # First: value > 0, then: not disabled + result = ( + items + .filter(lambda el: int(el.attrib.get("data-value", 0)) > 0) + .filter(lambda el: not el.has_class("disabled")) + ) + assert len(result) == 3 # Apple, Banana, Cherry (Durian is disabled AND value=0) + + def test_filter_on_empty_selectors(self): + """filter() on an already-empty Selectors should not raise""" + empty = Selectors() + result = empty.filter(lambda el: True) + assert isinstance(result, Selectors) + assert len(result) == 0 diff --git a/tests/spiders/test_engine.py b/tests/spiders/test_engine.py index ca768bf..b7bfd0f 100644 --- a/tests/spiders/test_engine.py +++ b/tests/spiders/test_engine.py @@ -613,8 +613,7 @@ class TestCheckpointMethods: @pytest.mark.asyncio async def test_restore_from_checkpoint_raises_when_disabled(self): engine = _make_engine() # no crawldir → checkpoint disabled - with pytest.raises(RuntimeError): - await engine._restore_from_checkpoint() + assert (await engine._restore_from_checkpoint()) is False # --------------------------------------------------------------------------- diff --git a/tests/spiders/test_session.py b/tests/spiders/test_session.py index c1eed5d..0c422ef 100644 --- a/tests/spiders/test_session.py +++ b/tests/spiders/test_session.py @@ -1,9 +1,12 @@ """Tests for the SessionManager class.""" +from unittest.mock import AsyncMock, PropertyMock + from scrapling.core._types import Any import pytest from scrapling.spiders.session import SessionManager +from scrapling.spiders.request import Request class MockSession: # type: ignore[type-arg] @@ -350,3 +353,57 @@ class TestSessionManagerIntegration: # After close - all inactive await manager.close() assert all(not s._is_alive for s in sessions) + + +class TestSessionManagerFetch: + """Test SessionManager fetch behavior.""" + + @pytest.mark.asyncio + async def test_fetch_preserves_request_method(self): + """Test that fetch does not mutate request._session_kwargs. + + Previously, fetch() used pop("method") which removed the method + key from the original request dict. This caused retried requests + (via request.copy()) to lose their HTTP method and fall back to GET. + """ + from scrapling.engines.static import _ASyncSessionLogic + from scrapling.fetchers import FetcherSession + from scrapling.engines.toolbelt.custom import Response + + mock_response = Response( + url="https://example.com", + content=b"ok", + status=200, + reason="OK", + cookies={}, + headers={"content-type": "text/html"}, + request_headers={}, + ) + mock_response.meta = {} + + mock_client = AsyncMock(spec=_ASyncSessionLogic) + mock_client._make_request = AsyncMock(return_value=mock_response) + + mock_session = AsyncMock(spec=FetcherSession) + mock_session._client = mock_client + mock_session._is_alive = True + + manager = SessionManager() + manager._sessions["default"] = mock_session + manager._default_session_id = "default" + manager._started = True + + request = Request("https://example.com", method="POST", data={"key": "value"}) + + assert request._session_kwargs["method"] == "POST" + + await manager.fetch(request) + + # method must still be present after fetch + assert "method" in request._session_kwargs + assert request._session_kwargs["method"] == "POST" + + # verify the correct method was passed to _make_request + mock_client._make_request.assert_called_once() + call_kwargs = mock_client._make_request.call_args + assert call_kwargs.kwargs["method"] == "POST"