Candidates with fewer attributes than the original got inflated scores because the denominator counted candidate attributes only, while the extra-attributes penalty direction worked as intended. Using `max()` on both counts fixes the inflation while keeping the penalty.
Closes#322
Address review feedback: In full-path XPath generation, elements with
IDs were producing bare predicates like `[@id='x']` which creates
invalid XPath steps like `//body/[@id='main']`. Now emits `*[@id='x']`
for full-path mode (e.g. `//body/*[@id='main']/*[@id='target']`).
Short-path XPath mode unchanged — still uses `//*[@id='x']` prefix.
Also added XPath evaluation assertion to the regression test to verify
the generated selector actually selects the correct element.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When generating full-path CSS/XPath selectors, elements with id
attributes had their selector appended twice — once in the id branch
(line 30) and again unconditionally (line 50).
This produced selectors like 'body > #main > #main > #target > #target'
instead of the correct 'body > #main > #target'.
Move the append into the else branch so it only fires for elements
without an id (elements with id already append in the if branch).
Includes 2 regression tests.
Several critical code paths in custom_types.py lacked test coverage:
- TextHandler.re(check_match=True): returns bool, not TextHandlers
- TextHandler.re(replace_entities=False): entity preservation path
- TextHandler.re() with capture groups: flatten behavior
- TextHandler.re_first() default value when no match
- TextHandler.clean(remove_entities=True): entity replacement path
- TextHandler.json() valid and invalid input
- TextHandlers.re(): list-level regex with result flattening
- TextHandlers.extract()/get_all(): identity return
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove all tests for the old encoding logic
- Stop using the `nopecha` test page to test Cloudflare solver
- Remove useless tests like testing for infinite timeout
- Fixes to make the code compatible with new changes
- `Adaptor` became `Selector`
- `Adaptors` became `Selectors`
- `auto_match` argument/feature became `adaptive`
- `adaptor_arguments` argument became `selector_config`
- `automatch_domain` argument became `adaptive_domain`
- `additional_arguments` argument became `additional_args`
- `storage_adaptors` file became just `storage`
- Renamed both `css_selector` and `xpath_selector` both to `generate_css_selector` and `generate_xpath_selector` for clarity and not interrupting the auto-completion while coding.
- Added `generate_full_css_selector` and `generate_full_xpath_selector` methods
Karim shoair