fix: preserve HTTP method across retries in spider session
SessionManager.fetch() pops `method` from `_session_kwargs`, which mutates the original request dict. When the engine retries a blocked request via request.copy(), the copy no longer has `method`, so it defaults to GET. Steps to reproduce: 1. Yield Request(url, method="POST", data=...) 2. Target returns a response that triggers is_blocked() 3. Engine retries via request.copy() → second fetch uses GET Fix: copy the kwargs dict before popping, so the original request stays intact.
This commit is contained in:
@@ -112,10 +112,12 @@ class SessionManager:
|
||||
client = session._client
|
||||
|
||||
if isinstance(client, _ASyncSessionLogic):
|
||||
kwargs = request._session_kwargs.copy()
|
||||
method = cast(SUPPORTED_HTTP_METHODS, kwargs.pop("method", "GET"))
|
||||
response = await client._make_request(
|
||||
method=cast(SUPPORTED_HTTP_METHODS, request._session_kwargs.pop("method", "GET")),
|
||||
method=method,
|
||||
url=request.url,
|
||||
**request._session_kwargs,
|
||||
**kwargs,
|
||||
)
|
||||
else:
|
||||
# Sync session or other types - shouldn't happen in async context
|
||||
|
||||
Reference in New Issue
Block a user