Candidates with fewer attributes than the original got inflated scores because the denominator counted candidate attributes only, while the extra-attributes penalty direction worked as intended. Using `max()` on both counts fixes the inflation while keeping the penalty.
Closes#322
CheckpointManager.save() and ResponseCacheManager.put() write to a temp
file and then move it into place with Path.rename(). On Windows, os.rename
cannot overwrite an existing destination and raises FileExistsError
(WinError 183), so every write after the first one fails: checkpoint
saving raises and breaks resume, while the development response cache
swallows the error and keeps returning the stale entry.
Path.replace() (os.replace) overwrites the destination atomically on every
platform and behaves identically to rename() on POSIX, so this is a no-op
on Linux and macOS and only fixes the broken overwrite on Windows.
Add a regression test for the cache overwrite path; the checkpoint
overwrite is already covered by test_multiple_saves_overwrite.
When `css()`/`xpath()` are called with both `adaptive=True` and
`auto_save=True`, the relocation branch guarded the re-save with
`if elements is not None`. However `relocate()` returns an empty
list (never `None`) when no candidate clears the `percentage`
threshold, so the guard always passed and `self.save(elements[0], ...)`
raised `IndexError: list index out of range`.
This crashes exactly when adaptive resilience is needed most: the page
structure changed enough that nothing matches above the threshold.
Fix: use a truthiness check (`if elements and auto_save`) so the
re-save is skipped when relocation yields nothing. The successful
relocation path (which re-saves the relocated element) is unchanged.
Added a regression test that fails before the fix (IndexError) and
passes after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`ResponseFactory.__extract_browser_encoding` matched the charset with
`charset=([\w-]+)`, which stops at a quote character. RFC 7231 permits the
charset value to be a quoted-string (e.g. `content-type: text/html;
charset="ISO-8859-1"`), so for any quoted charset the regex failed to match
and the function silently fell back to the `utf-8` default. A page served as
quoted ISO-8859-1 / windows-1252 / Shift_JIS would then be decoded as UTF-8,
producing mojibake.
Allow an optional surrounding quote in the pattern (`charset=["']?([\w-]+)`)
so the value is captured without the quote. Unquoted headers are unaffected.
The existing `content_type_map` fixture in tests/fetchers/test_utils.py was
unused; add focused tests covering unquoted, quoted, and missing charsets.
- Add a `--version` flag to the main CLI group using click's `version_option`
- Prints `Scrapling, version <version>` and exits, sourcing the version from `scrapling.__version__`
- Add a CLI test asserting the flag's output
Closes#299
block_ads is a browser-engine parameter (used by PlayWright/Camoufox
fetchers for ad-domain blocking) and is not recognised by curl_cffi's
Session.request(). When the CLI's --ai-targeted flag sets block_ads=True,
_merge_request_args forwards it unfiltered, causing:
TypeError: Session.request() got an unexpected keyword argument 'block_ads'
Add block_ads to the skip_keys set so it is stripped before the dict
reaches Session.request(), consistent with existing entries for
extra_headers and google_search.
Fixes#247
Allow users to specify a custom session_id when opening a browser session,
rather than always generating a random UUID. Useful for naming sessions
for easier management across multiple tool calls.
- Add session_id: Optional[str] = None parameter
- Validate session_id doesn't already exist before starting browser
- Fall back to uuid4().hex[:12] if not provided
- Add tests for custom session_id and duplicate detection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Address review feedback: In full-path XPath generation, elements with
IDs were producing bare predicates like `[@id='x']` which creates
invalid XPath steps like `//body/[@id='main']`. Now emits `*[@id='x']`
for full-path mode (e.g. `//body/*[@id='main']/*[@id='target']`).
Short-path XPath mode unchanged — still uses `//*[@id='x']` prefix.
Also added XPath evaluation assertion to the regression test to verify
the generated selector actually selects the correct element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When generating full-path CSS/XPath selectors, elements with id
attributes had their selector appended twice — once in the id branch
(line 30) and again unconditionally (line 50).
This produced selectors like 'body > #main > #main > #target > #target'
instead of the correct 'body > #main > #target'.
Move the append into the else branch so it only fires for elements
without an id (elements with id already append in the if branch).
Includes 2 regression tests.
On force-stop (second Ctrl+C), cancel_scope.cancel() was called BEFORE
_save_checkpoint(). Since cancel_scope.cancel() causes all subsequent
awaits within the scope to raise Cancelled, the checkpoint write was
silently aborted:
1. _save_checkpoint() uses anyio.open_file + rename — both are await
checkpoints that get cancelled immediately
2. self.paused never gets set to True (code after the aborted save)
3. The finally block sees 'not self.paused' and calls cleanup() which
DELETES the previous checkpoint file
Result: a user who ran a long crawl, pressed Ctrl+C twice to force-stop,
loses their entire checkpoint irrecoverably. The old checkpoint (from
periodic saves or a previous graceful pause) is deleted, and the new
one was never written.
Fix: move the cancel_scope.cancel() call AFTER the checkpoint save.
The save completes normally, self.paused is set to True, and only then
does the scope get cancelled to abort in-flight tasks. The finally
block correctly sees paused=True and skips cleanup.
Adds 6 regression tests covering:
- Force-stop checkpoint preservation (core regression)
- Graceful pause still works
- Force-stop checkpoint is loadable
- Normal completion cleanup still works
- Force-stop without checkpoint system
- Existing checkpoint not deleted on force-stop
Previously robots.txt was fetched lazily on the first request per
domain, causing early concurrent requests to each stall waiting for
the same network fetch. The cache is now warmed before the crawl loop
starts, making all subsequent robots.txt lookups a local read.
- RobotsTxtManager gains a prefetch(urls, sid) method that fetches all domains concurrently via a task group
- CrawlerEngine._prefetch_robots_txt() is called after on_start():
uses allowed_domains if configured, otherwise falls back to unique
domains extracted from start_urls
- Mid-crawl domain discovery (not covered by prefetch) still fetches
lazily; two concurrent callbacks on the same new domain can each
trigger a fetch — accepted tradeoff, documented in _get_domain_delay
Files: scrapling/spiders/robotstxt.py, scrapling/spiders/engine.py, tests/spiders/test_engine.py
robots.txt is a domain-level document and does not vary by session.
Keying the cache by (domain, sid) was both wasteful and incorrect —
it caused redundant fetches when the same domain was accessed by different sessions.
- Cache is now keyed by domain string only; all sessions share one entry
- Removed asyncio.Event inflight-deduplication mechanism (superseded by the prefetch approach added in the next commit)
- clear_cache() loses the `sid` parameter (breaking change); clearing a domain now evicts the single shared entry for all sessions
- Updated tests to reflect shared-cache semantics
Files: scrapling/spiders/robotstxt.py, tests/spiders/test_robotstxt.py
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
SessionManager.fetch() pops `method` from `_session_kwargs`,
which mutates the original request dict. When the engine retries
a blocked request via request.copy(), the copy no longer has
`method`, so it defaults to GET.
Steps to reproduce:
1. Yield Request(url, method="POST", data=...)
2. Target returns a response that triggers is_blocked()
3. Engine retries via request.copy() → second fetch uses GET
Fix: copy the kwargs dict before popping, so the original
request stays intact.
Several critical code paths in custom_types.py lacked test coverage:
- TextHandler.re(check_match=True): returns bool, not TextHandlers
- TextHandler.re(replace_entities=False): entity preservation path
- TextHandler.re() with capture groups: flatten behavior
- TextHandler.re_first() default value when no match
- TextHandler.clean(remove_entities=True): entity replacement path
- TextHandler.json() valid and invalid input
- TextHandlers.re(): list-level regex with result flattening
- TextHandlers.extract()/get_all(): identity return
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The adaptive scraping feature relies on SQLiteStorageSystem to persist
and relocate elements, but the existing tests only verified object
creation — not the actual save/retrieve workflow. This adds:
- _get_base_url: None, empty, valid URL, and case-normalization paths
- _get_hash: determinism, uniqueness, strip/lowercase, length suffix
- save/retrieve round-trip: basic, overwrite (upsert), nonexistent key
- URL-based isolation between different websites
- element_to_dict: with/without text, attributes, whitespace filtering
- _get_element_path: nested and root element paths
- Thread safety: 20 concurrent saves with result verification
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Karim shoair