Address review feedback: In full-path XPath generation, elements with
IDs were producing bare predicates like `[@id='x']` which creates
invalid XPath steps like `//body/[@id='main']`. Now emits `*[@id='x']`
for full-path mode (e.g. `//body/*[@id='main']/*[@id='target']`).
Short-path XPath mode unchanged — still uses `//*[@id='x']` prefix.
Also added XPath evaluation assertion to the regression test to verify
the generated selector actually selects the correct element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When generating full-path CSS/XPath selectors, elements with id
attributes had their selector appended twice — once in the id branch
(line 30) and again unconditionally (line 50).
This produced selectors like 'body > #main > #main > #target > #target'
instead of the correct 'body > #main > #target'.
Move the append into the else branch so it only fires for elements
without an id (elements with id already append in the if branch).
Includes 2 regression tests.
On force-stop (second Ctrl+C), cancel_scope.cancel() was called BEFORE
_save_checkpoint(). Since cancel_scope.cancel() causes all subsequent
awaits within the scope to raise Cancelled, the checkpoint write was
silently aborted:
1. _save_checkpoint() uses anyio.open_file + rename — both are await
checkpoints that get cancelled immediately
2. self.paused never gets set to True (code after the aborted save)
3. The finally block sees 'not self.paused' and calls cleanup() which
DELETES the previous checkpoint file
Result: a user who ran a long crawl, pressed Ctrl+C twice to force-stop,
loses their entire checkpoint irrecoverably. The old checkpoint (from
periodic saves or a previous graceful pause) is deleted, and the new
one was never written.
Fix: move the cancel_scope.cancel() call AFTER the checkpoint save.
The save completes normally, self.paused is set to True, and only then
does the scope get cancelled to abort in-flight tasks. The finally
block correctly sees paused=True and skips cleanup.
Adds 6 regression tests covering:
- Force-stop checkpoint preservation (core regression)
- Graceful pause still works
- Force-stop checkpoint is loadable
- Normal completion cleanup still works
- Force-stop without checkpoint system
- Existing checkpoint not deleted on force-stop
curl_cffi v0.15.0 introduced CurlFollow.SAFE, which follows redirects but rejects those targeting internal/private IPs (loopback, private networks, link-local). This is now the default for all HTTP fetchers, the MCP server, and the shell curl converter.
Added FollowRedirects type alias supporting all curl_cffi redirect
modes: bool, "safe", "all", "obeycode", "firstonly".
Stop using allowed_domains for robots.txt prefetch since bare domain strings have no scheme info.
Domains discovered mid-crawl via requests still fetch robots.txt lazily.
robots_txt_obey now defaults to True. Spiders must explicitly opt out
with robots_txt_obey = False rather than opt in, making ethical
crawling the default behaviour.
File: scrapling/spiders/spider.py
Previously robots.txt was fetched lazily on the first request per
domain, causing early concurrent requests to each stall waiting for
the same network fetch. The cache is now warmed before the crawl loop
starts, making all subsequent robots.txt lookups a local read.
- RobotsTxtManager gains a prefetch(urls, sid) method that fetches all domains concurrently via a task group
- CrawlerEngine._prefetch_robots_txt() is called after on_start():
uses allowed_domains if configured, otherwise falls back to unique
domains extracted from start_urls
- Mid-crawl domain discovery (not covered by prefetch) still fetches
lazily; two concurrent callbacks on the same new domain can each
trigger a fetch — accepted tradeoff, documented in _get_domain_delay
Files: scrapling/spiders/robotstxt.py, scrapling/spiders/engine.py, tests/spiders/test_engine.py
robots.txt is a domain-level document and does not vary by session.
Keying the cache by (domain, sid) was both wasteful and incorrect —
it caused redundant fetches when the same domain was accessed by different sessions.
- Cache is now keyed by domain string only; all sessions share one entry
- Removed asyncio.Event inflight-deduplication mechanism (superseded by the prefetch approach added in the next commit)
- clear_cache() loses the `sid` parameter (breaking change); clearing a domain now evicts the single shared entry for all sessions
- Updated tests to reflect shared-cache semantics
Files: scrapling/spiders/robotstxt.py, tests/spiders/test_robotstxt.py
The stub shadows the real implementation, and proxy rotation always hits NotImplementedError.
Possible fix for #215
Co-Authored-By: Yuval Dinodia <102706514+yetval@users.noreply.github.com>
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).
- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
SessionManager.fetch() pops `method` from `_session_kwargs`,
which mutates the original request dict. When the engine retries
a blocked request via request.copy(), the copy no longer has
`method`, so it defaults to GET.
Steps to reproduce:
1. Yield Request(url, method="POST", data=...)
2. Target returns a response that triggers is_blocked()
3. Engine retries via request.copy() → second fetch uses GET
Fix: copy the kwargs dict before popping, so the original
request stays intact.
Both _get_page_content and _get_async_page_content use a while-True
loop that retries page.content() on PlaywrightError with no upper
bound. If the page is in a permanently broken state (crashed tab,
closed context), this loops forever and hangs the process.
Replace with a bounded for-loop (default 10 retries = 5s), returning
an empty string if all attempts fail. This preserves the existing
retry behavior for the transient Windows issue (playwright#16108)
while preventing hangs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When _checkpoint_system_enabled is False, the method uses a bare
`raise` with no active exception, which causes RuntimeError at
runtime. The method's docstring says it returns False when restoration
is not possible, so return False is the correct behavior.
The caller in crawl() currently guards with `if
self._checkpoint_system_enabled`, but the method's own contract
should be self-consistent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Real browsers send `https://www.google.com/` as the Referer header
when clicking search results, not the full search URL with query
parameters. The previous format was a fingerprinting signal that
the referer was spoofed.
Closes#172
sjhddh