fix: parse quoted charset values in content-type headers
`ResponseFactory.__extract_browser_encoding` matched the charset with `charset=([\w-]+)`, which stops at a quote character. RFC 7231 permits the charset value to be a quoted-string (e.g. `content-type: text/html; charset="ISO-8859-1"`), so for any quoted charset the regex failed to match and the function silently fell back to the `utf-8` default. A page served as quoted ISO-8859-1 / windows-1252 / Shift_JIS would then be decoded as UTF-8, producing mojibake. Allow an optional surrounding quote in the pattern (`charset=["']?([\w-]+)`) so the value is captured without the quote. Unquoted headers are unaffected. The existing `content_type_map` fixture in tests/fetchers/test_utils.py was unused; add focused tests covering unquoted, quoted, and missing charsets.
This commit is contained in:
@@ -10,7 +10,7 @@ from scrapling.core.utils import log
|
||||
from .custom import Response, StatusText
|
||||
from scrapling.core._types import Dict, List, Optional
|
||||
|
||||
__CHARSET_RE__ = re_compile(r"charset=([\w-]+)")
|
||||
__CHARSET_RE__ = re_compile(r"""charset=["']?([\w-]+)""")
|
||||
|
||||
|
||||
class ResponseFactory:
|
||||
|
||||
Reference in New Issue
Block a user