Commit Graph

10 Commits

Author SHA1 Message Date
Abdullah 132f33c846 test(spiders): add comprehensive test suite for robots.txt compliance 2026-04-03 15:08:34 +02:00
Karim shoair 4c07b294ae Merge branch 'dev' into fix/preserve-http-method-on-retry 2026-03-17 22:04:57 +02:00
Karim shoair cc6c0dbfb9 fix: adjust test to the _restore_from_checkpoint fix 2026-03-17 17:46:47 +02:00
karesansui 5bf921b308 fix: preserve HTTP method across retries in spider session
SessionManager.fetch() pops `method` from `_session_kwargs`,
which mutates the original request dict. When the engine retries
a blocked request via request.copy(), the copy no longer has
`method`, so it defaults to GET.

Steps to reproduce:
1. Yield Request(url, method="POST", data=...)
2. Target returns a response that triggers is_blocked()
3. Engine retries via request.copy() → second fetch uses GET

Fix: copy the kwargs dict before popping, so the original
request stays intact.
2026-03-17 00:53:52 +09:00
Karim shoair e35ba3455c test: add tests for spiders engine
Automated tests written by AI
2026-02-15 04:35:28 +02:00
Karim shoair d0a5a01299 test: update spider check 2026-02-14 02:51:39 +02:00
Karim shoair d90189e675 test: update tests accordingly 2026-01-20 20:22:47 +02:00
Karim shoair 0ec78e2335 test: update tests accordingly 2026-01-19 16:40:18 +02:00
Karim shoair 94bc759c6a test: updates tests to include new logic 2026-01-19 16:13:29 +02:00
Karim shoair d247b5b691 test: add tests for the spiders system
It's generated by Opus on Claude Code. It's very good as per my review and instructions, but I will have another look later.
2026-01-19 03:46:00 +02:00