fix: preserve HTTP method across retries in spider session

SessionManager.fetch() pops `method` from `_session_kwargs`,
which mutates the original request dict. When the engine retries
a blocked request via request.copy(), the copy no longer has
`method`, so it defaults to GET.

Steps to reproduce:
1. Yield Request(url, method="POST", data=...)
2. Target returns a response that triggers is_blocked()
3. Engine retries via request.copy() → second fetch uses GET

Fix: copy the kwargs dict before popping, so the original
request stays intact.
This commit is contained in:
karesansui
2026-03-17 00:53:52 +09:00
parent 72a2c8de81
commit 5bf921b308
2 changed files with 61 additions and 2 deletions
+4 -2
View File
@@ -112,10 +112,12 @@ class SessionManager:
client = session._client
if isinstance(client, _ASyncSessionLogic):
kwargs = request._session_kwargs.copy()
method = cast(SUPPORTED_HTTP_METHODS, kwargs.pop("method", "GET"))
response = await client._make_request(
method=cast(SUPPORTED_HTTP_METHODS, request._session_kwargs.pop("method", "GET")),
method=method,
url=request.url,
**request._session_kwargs,
**kwargs,
)
else:
# Sync session or other types - shouldn't happen in async context