freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: prevent IndexError in adaptive relocation with auto_save

Browse Source 
When `css()`/`xpath()` are called with both `adaptive=True` and
`auto_save=True`, the relocation branch guarded the re-save with
`if elements is not None`. However `relocate()` returns an empty
list (never `None`) when no candidate clears the `percentage`
threshold, so the guard always passed and `self.save(elements[0], ...)`
raised `IndexError: list index out of range`.

This crashes exactly when adaptive resilience is needed most: the page
structure changed enough that nothing matches above the threshold.

Fix: use a truthiness check (`if elements and auto_save`) so the
re-save is skipped when relocation yields nothing. The successful
relocation path (which re-saves the relocated element) is unchanged.

Added a regression test that fails before the fix (IndexError) and
passes after.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Mubashir Rahim
2026-06-04 14:53:10 +05:00
parent 53ef32c723
commit cd4cdc69e6
2 changed files with 27 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scrapling/parser.py
+1 -1
View File
@@ -671,7 +671,7 @@ class Selector(SelectorsGeneration):
element_data = self.retrieve(identifier or selector)
if element_data:
elements = self.relocate(element_data, percentage)
if elements is not None and auto_save:
if elements and auto_save:
self.save(elements[0], identifier or selector)
return self.__handle_elements(elements)
tests/parser/test_adaptive.py
+26
View File
@@ -56,6 +56,32 @@ class TestParserAdaptive:
assert relocated[0].has_class("new-class")
assert relocated[0].css(".new-description")[0].text == "Description 1"
def test_relocation_auto_save_no_match_above_threshold(self):
"""Adaptive relocation with `auto_save=True` must not crash when no element
clears the `percentage` threshold (relocate() returns an empty list)."""
original_html = """
<div class="container">
<article class="product" id="target">
<h3>Widget</h3>
<p class="desc">A widget</p>
</article>
</div>
"""
# Unrelated structure so nothing can match a high threshold
changed_html = "<html><body><span>totally unrelated content</span></body></html>"
old_page = Selector(original_html, url="example.com", adaptive=True)
new_page = Selector(changed_html, url="example.com", adaptive=True)
old_page.css("#target", identifier="target", auto_save=True)
# Before the fix this raised `IndexError: list index out of range` because the
# guard checked `elements is not None` but relocate() returns [] (never None).
result = new_page.css(
"#target", identifier="target", adaptive=True, auto_save=True, percentage=95
)
assert list(result) == []
@pytest.mark.asyncio
async def test_element_relocation_async(self):
"""Test relocating element after structure change in async mode"""