When `css()`/`xpath()` are called with both `adaptive=True` and
`auto_save=True`, the relocation branch guarded the re-save with
`if elements is not None`. However `relocate()` returns an empty
list (never `None`) when no candidate clears the `percentage`
threshold, so the guard always passed and `self.save(elements[0], ...)`
raised `IndexError: list index out of range`.
This crashes exactly when adaptive resilience is needed most: the page
structure changed enough that nothing matches above the threshold.
Fix: use a truthiness check (`if elements and auto_save`) so the
re-save is skipped when relocation yields nothing. The successful
relocation path (which re-saves the relocated element) is unchanged.
Added a regression test that fails before the fix (IndexError) and
passes after.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Address review feedback: In full-path XPath generation, elements with
IDs were producing bare predicates like `[@id='x']` which creates
invalid XPath steps like `//body/[@id='main']`. Now emits `*[@id='x']`
for full-path mode (e.g. `//body/*[@id='main']/*[@id='target']`).
Short-path XPath mode unchanged — still uses `//*[@id='x']` prefix.
Also added XPath evaluation assertion to the regression test to verify
the generated selector actually selects the correct element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When generating full-path CSS/XPath selectors, elements with id
attributes had their selector appended twice — once in the id branch
(line 30) and again unconditionally (line 50).
This produced selectors like 'body > #main > #main > #target > #target'
instead of the correct 'body > #main > #target'.
Move the append into the else branch so it only fires for elements
without an id (elements with id already append in the if branch).
Includes 2 regression tests.
Several critical code paths in custom_types.py lacked test coverage:
- TextHandler.re(check_match=True): returns bool, not TextHandlers
- TextHandler.re(replace_entities=False): entity preservation path
- TextHandler.re() with capture groups: flatten behavior
- TextHandler.re_first() default value when no match
- TextHandler.clean(remove_entities=True): entity replacement path
- TextHandler.json() valid and invalid input
- TextHandlers.re(): list-level regex with result flattening
- TextHandlers.extract()/get_all(): identity return
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove all tests for the old encoding logic
- Stop using the `nopecha` test page to test Cloudflare solver
- Remove useless tests like testing for infinite timeout
- Fixes to make the code compatible with new changes
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
- Renamed both `css_selector` and `xpath_selector` both to `generate_css_selector` and `generate_xpath_selector` for clarity and not interrupting the auto-completion while coding.
- Added `generate_full_css_selector` and `generate_full_xpath_selector` methods
Mubashir Rahim