The per-request proxy resolution never fell back to the session default, so FetcherSession(proxy=...) was silently ignored, and requests went direct. Same fix in the sync and async paths, with regression tests asserting on the proxy that reaches curl_cffi.
Closes#295
`ResponseFactory.__extract_browser_encoding` matched the charset with
`charset=([\w-]+)`, which stops at a quote character. RFC 7231 permits the
charset value to be a quoted-string (e.g. `content-type: text/html;
charset="ISO-8859-1"`), so for any quoted charset the regex failed to match
and the function silently fell back to the `utf-8` default. A page served as
quoted ISO-8859-1 / windows-1252 / Shift_JIS would then be decoded as UTF-8,
producing mojibake.
Allow an optional surrounding quote in the pattern (`charset=["']?([\w-]+)`)
so the value is captured without the quote. Unquoted headers are unaffected.
The existing `content_type_map` fixture in tests/fetchers/test_utils.py was
unused; add focused tests covering unquoted, quoted, and missing charsets.
block_ads is a browser-engine parameter (used by PlayWright/Camoufox
fetchers for ad-domain blocking) and is not recognised by curl_cffi's
Session.request(). When the CLI's --ai-targeted flag sets block_ads=True,
_merge_request_args forwards it unfiltered, causing:
TypeError: Session.request() got an unexpected keyword argument 'block_ads'
Add block_ads to the skip_keys set so it is stripped before the dict
reaches Session.request(), consistent with existing entries for
extra_headers and google_search.
Fixes#247
Real browsers send `https://www.google.com/` as the Referer header
when clicking search results, not the full search URL with query
parameters. The previous format was a fingerprinting signal that
the referer was spoofed.
Closes#172
It can be considered the simplest form of round robin since we don't have weights or anything (as was originally planned), but let's change it to avoid confusion or useless debates. Here goes nothing
Close pages immediately after tasks finish to reduce resource usage and
improve overall performance. Update corresponding test files to reflect
the new page lifecycle management.
- Remove all tests for the old encoding logic
- Stop using the `nopecha` test page to test Cloudflare solver
- Remove useless tests like testing for infinite timeout
- Fixes to make the code compatible with new changes
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
Karim shoair