SessionManager.fetch() pops `method` from `_session_kwargs`,
which mutates the original request dict. When the engine retries
a blocked request via request.copy(), the copy no longer has
`method`, so it defaults to GET.
Steps to reproduce:
1. Yield Request(url, method="POST", data=...)
2. Target returns a response that triggers is_blocked()
3. Engine retries via request.copy() → second fetch uses GET
Fix: copy the kwargs dict before popping, so the original
request stays intact.
Several critical code paths in custom_types.py lacked test coverage:
- TextHandler.re(check_match=True): returns bool, not TextHandlers
- TextHandler.re(replace_entities=False): entity preservation path
- TextHandler.re() with capture groups: flatten behavior
- TextHandler.re_first() default value when no match
- TextHandler.clean(remove_entities=True): entity replacement path
- TextHandler.json() valid and invalid input
- TextHandlers.re(): list-level regex with result flattening
- TextHandlers.extract()/get_all(): identity return
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The adaptive scraping feature relies on SQLiteStorageSystem to persist
and relocate elements, but the existing tests only verified object
creation — not the actual save/retrieve workflow. This adds:
- _get_base_url: None, empty, valid URL, and case-normalization paths
- _get_hash: determinism, uniqueness, strip/lowercase, length suffix
- save/retrieve round-trip: basic, overwrite (upsert), nonexistent key
- URL-based isolation between different websites
- element_to_dict: with/without text, attributes, whitespace filtering
- _get_element_path: nested and root element paths
- Thread safety: 20 concurrent saves with result verification
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Real browsers send `https://www.google.com/` as the Referer header
when clicking search results, not the full search URL with query
parameters. The previous format was a fingerprinting signal that
the referer was spoofed.
Closes#172
It can be considered the simplest form of round robin since we don't have weights or anything (as was originally planned), but let's change it to avoid confusion or useless debates. Here goes nothing
Karim shoair