The per-request proxy resolution never fell back to the session default, so FetcherSession(proxy=...) was silently ignored, and requests went direct. Same fix in the sync and async paths, with regression tests asserting on the proxy that reaches curl_cffi.
Closes#295
Candidates with fewer attributes than the original got inflated scores because the denominator counted candidate attributes only, while the extra-attributes penalty direction worked as intended. Using `max()` on both counts fixes the inflation while keeping the penalty.
Closes#322
CheckpointManager.save() and ResponseCacheManager.put() write to a temp
file and then move it into place with Path.rename(). On Windows, os.rename
cannot overwrite an existing destination and raises FileExistsError
(WinError 183), so every write after the first one fails: checkpoint
saving raises and breaks resume, while the development response cache
swallows the error and keeps returning the stale entry.
Path.replace() (os.replace) overwrites the destination atomically on every
platform and behaves identically to rename() on POSIX, so this is a no-op
on Linux and macOS and only fixes the broken overwrite on Windows.
Add a regression test for the cache overwrite path; the checkpoint
overwrite is already covered by test_multiple_saves_overwrite.
When `css()`/`xpath()` are called with both `adaptive=True` and
`auto_save=True`, the relocation branch guarded the re-save with
`if elements is not None`. However `relocate()` returns an empty
list (never `None`) when no candidate clears the `percentage`
threshold, so the guard always passed and `self.save(elements[0], ...)`
raised `IndexError: list index out of range`.
This crashes exactly when adaptive resilience is needed most: the page
structure changed enough that nothing matches above the threshold.
Fix: use a truthiness check (`if elements and auto_save`) so the
re-save is skipped when relocation yields nothing. The successful
relocation path (which re-saves the relocated element) is unchanged.
Added a regression test that fails before the fix (IndexError) and
passes after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`ResponseFactory.__extract_browser_encoding` matched the charset with
`charset=([\w-]+)`, which stops at a quote character. RFC 7231 permits the
charset value to be a quoted-string (e.g. `content-type: text/html;
charset="ISO-8859-1"`), so for any quoted charset the regex failed to match
and the function silently fell back to the `utf-8` default. A page served as
quoted ISO-8859-1 / windows-1252 / Shift_JIS would then be decoded as UTF-8,
producing mojibake.
Allow an optional surrounding quote in the pattern (`charset=["']?([\w-]+)`)
so the value is captured without the quote. Unquoted headers are unaffected.
The existing `content_type_map` fixture in tests/fetchers/test_utils.py was
unused; add focused tests covering unquoted, quoted, and missing charsets.
- Add a `--version` flag to the main CLI group using click's `version_option`
- Prints `Scrapling, version <version>` and exits, sourcing the version from `scrapling.__version__`
- Add a CLI test asserting the flag's output
Closes#299
block_ads is a browser-engine parameter (used by PlayWright/Camoufox
fetchers for ad-domain blocking) and is not recognised by curl_cffi's
Session.request(). When the CLI's --ai-targeted flag sets block_ads=True,
_merge_request_args forwards it unfiltered, causing:
TypeError: Session.request() got an unexpected keyword argument 'block_ads'
Add block_ads to the skip_keys set so it is stripped before the dict
reaches Session.request(), consistent with existing entries for
extra_headers and google_search.
Fixes#247
Allow users to specify a custom session_id when opening a browser session,
rather than always generating a random UUID. Useful for naming sessions
for easier management across multiple tool calls.
- Add session_id: Optional[str] = None parameter
- Validate session_id doesn't already exist before starting browser
- Fall back to uuid4().hex[:12] if not provided
- Add tests for custom session_id and duplicate detection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The Seconds type was defined as `Annotated[int, float, Meta(ge=0)]` which per PEP 593 treated float as metadata, not a type. This caused passing float values like wait=1.5 to be rejected. Fixed by using Annotated[float, Meta(ge=0)] since int is a subtype of float.
Correct fix for #240
Co-Authored-By: Cocoon-Break <54054995+kuishou68@users.noreply.github.com>
Address review feedback: In full-path XPath generation, elements with
IDs were producing bare predicates like `[@id='x']` which creates
invalid XPath steps like `//body/[@id='main']`. Now emits `*[@id='x']`
for full-path mode (e.g. `//body/*[@id='main']/*[@id='target']`).
Short-path XPath mode unchanged — still uses `//*[@id='x']` prefix.
Also added XPath evaluation assertion to the regression test to verify
the generated selector actually selects the correct element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When generating full-path CSS/XPath selectors, elements with id
attributes had their selector appended twice — once in the id branch
(line 30) and again unconditionally (line 50).
This produced selectors like 'body > #main > #main > #target > #target'
instead of the correct 'body > #main > #target'.
Move the append into the else branch so it only fires for elements
without an id (elements with id already append in the if branch).
Includes 2 regression tests.
On force-stop (second Ctrl+C), cancel_scope.cancel() was called BEFORE
_save_checkpoint(). Since cancel_scope.cancel() causes all subsequent
awaits within the scope to raise Cancelled, the checkpoint write was
silently aborted:
1. _save_checkpoint() uses anyio.open_file + rename — both are await
checkpoints that get cancelled immediately
2. self.paused never gets set to True (code after the aborted save)
3. The finally block sees 'not self.paused' and calls cleanup() which
DELETES the previous checkpoint file
Result: a user who ran a long crawl, pressed Ctrl+C twice to force-stop,
loses their entire checkpoint irrecoverably. The old checkpoint (from
periodic saves or a previous graceful pause) is deleted, and the new
one was never written.
Fix: move the cancel_scope.cancel() call AFTER the checkpoint save.
The save completes normally, self.paused is set to True, and only then
does the scope get cancelled to abort in-flight tasks. The finally
block correctly sees paused=True and skips cleanup.
Adds 6 regression tests covering:
- Force-stop checkpoint preservation (core regression)
- Graceful pause still works
- Force-stop checkpoint is loadable
- Normal completion cleanup still works
- Force-stop without checkpoint system
- Existing checkpoint not deleted on force-stop
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.
Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.
robots_txt_obey now defaults to True. Spiders must explicitly opt out
with robots_txt_obey = False rather than opt in, making ethical
crawling the default behaviour.
File: scrapling/spiders/spider.py
Previously robots.txt was fetched lazily on the first request per
domain, causing early concurrent requests to each stall waiting for
the same network fetch. The cache is now warmed before the crawl loop
starts, making all subsequent robots.txt lookups a local read.
- RobotsTxtManager gains a prefetch(urls, sid) method that fetches all domains concurrently via a task group
- CrawlerEngine._prefetch_robots_txt() is called after on_start():
uses allowed_domains if configured, otherwise falls back to unique
domains extracted from start_urls
- Mid-crawl domain discovery (not covered by prefetch) still fetches
lazily; two concurrent callbacks on the same new domain can each
trigger a fetch — accepted tradeoff, documented in _get_domain_delay
Files: scrapling/spiders/robotstxt.py, scrapling/spiders/engine.py, tests/spiders/test_engine.py
Karim shoair