Scrapling

Files

T

haosenwang1018andClaude Opus 4.6 d3c251c1ab fix: add max retry limit to _get_page_content to prevent infinite loop

Both _get_page_content and _get_async_page_content use a while-True
loop that retries page.content() on PlaywrightError with no upper
bound. If the page is in a permanently broken state (crashed tab,
closed context), this loops forever and hangs the process.

Replace with a bounded for-loop (default 10 retries = 5s), returning
an empty string if all attempts fail. This preserves the existing
retry behavior for the transient Windows issue (playwright#16108)
while preventing hangs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-15 17:53:23 +08:00

core

fix: update code and docstrings to remove the old google referer logic

2026-03-09 00:25:19 +02:00

engines

fix: add max retry limit to _get_page_content to prevent infinite loop

2026-03-15 17:53:23 +08:00

fetchers

fix: update code and docstrings to remove the old google referer logic

2026-03-09 00:25:19 +02:00

spiders

fix(checkpoint): correct error message