422b4713ec
build: pump up version
2026-03-25 00:19:58 +02:00
9fee49c928
test: fixes to storage tests
2026-03-25 00:13:42 +02:00
3757cf6d77
test: add save/retrieve round-trip and core storage coverage ( #193 )
2026-03-25 00:11:42 +02:00
30ffe76cfa
Merge branch 'dev' into test/storage-core-coverage
2026-03-24 22:42:58 +02:00
3b4cae6889
Merge branch 'main' into dev
2026-03-24 22:21:07 +02:00
6635a4ad72
docs: adding a new sponsor
2026-03-24 17:13:24 +02:00
718db42ff0
Merge branch 'main' into dev
2026-03-23 16:40:12 +02:00
32cec404f6
docs: adjustment to ad plans
2026-03-23 15:00:26 +02:00
e230403232
docs: removing an old sponsor
2026-03-23 14:56:35 +02:00
df4b1ac7e3
Merge branch 'dev' into test/storage-core-coverage
2026-03-22 18:05:42 +02:00
d7c08073d4
Merge branch 'main' into dev
2026-03-22 18:05:26 +02:00
a96809a340
docs: removing an old sponsor
2026-03-22 16:48:26 +02:00
bc509e9fcd
Merge branch 'main' into dev
2026-03-19 17:01:25 +02:00
3ed59c2a84
fix(agent): update skill zip file with the latest fix
2026-03-19 17:01:05 +02:00
2e78b3f8aa
fix(skill): Fix broken markdown links in skill references ( #204 )
...
Fixes #203
2026-03-19 16:58:06 +02:00
a4f31ae205
simple md fix
2026-03-19 10:37:02 -04:00
7ca02ebb80
test: add edge case tests for filter, iterancestors, and find_similar ( #200 )
2026-03-17 22:16:15 +02:00
bf9e2da329
Merge branch 'dev' into test/edge-cases-filter-ancestors-find-similar
2026-03-17 22:15:59 +02:00
1dc0b7a1bd
fix(fetchers/content): increase the default max number of retries and raise error on max retries
...
Ref.: https://github.com/D4Vinci/Scrapling/pull/197#issuecomment-4077705587
2026-03-17 22:14:02 +02:00
17ea982162
fix(fetchers): add max retry limit to _get_page_content to prevent infinite loop ( #197 )
2026-03-17 22:09:05 +02:00
8e4e59e13e
Merge branch 'dev' into fix/page-content-infinite-loop
2026-03-17 22:08:47 +02:00
d9932f2308
fix(spider/fetcher): preserve HTTP method across retries in spider session ( #201 )
2026-03-17 22:06:01 +02:00
4c07b294ae
Merge branch 'dev' into fix/preserve-http-method-on-retry
2026-03-17 22:04:57 +02:00
27f306259e
Merge branch 'dev' into fix/page-content-infinite-loop
2026-03-17 21:55:26 +02:00
136c389787
fix(Texthandler): Replace get_all with getall to match the Selector class
2026-03-17 21:53:17 +02:00
980be18a2a
test: add coverage for TextHandler regex paths and TextHandlers.re() ( #194 )
2026-03-17 21:50:05 +02:00
5123590a15
Merge branch 'dev' into test/custom-types-coverage
2026-03-17 21:42:17 +02:00
cc6c0dbfb9
fix: adjust test to the _restore_from_checkpoint fix
2026-03-17 17:46:47 +02:00
1fb7961d42
fix: replace bare raise with return False in _restore_from_checkpoint ( #196 )
2026-03-17 17:45:52 +02:00
bcd45e47bb
Merge branch 'dev' into fix/checkpoint-restore-bare-raise
2026-03-17 17:36:11 +02:00
5eefbc6c05
Merge branch 'dev' into test/custom-types-coverage
2026-03-17 17:34:12 +02:00
6699213bdc
Merge branch 'dev' into test/storage-core-coverage
2026-03-17 17:31:31 +02:00
440da11259
Merge branch 'main' into dev
2026-03-17 17:28:38 +02:00
7bfaf204b7
docs: CJK Latin spacing fixes for the JP README
2026-03-17 17:12:41 +02:00
74abd54a0b
docs: Add translations to the new sponsor
2026-03-17 17:00:32 +02:00
3d60792cbc
docs: update Zensical
2026-03-17 16:53:55 +02:00
3b543a6bdd
docs: add a new sponsor
2026-03-17 16:45:04 +02:00
5bf921b308
fix: preserve HTTP method across retries in spider session
...
SessionManager.fetch() pops `method` from `_session_kwargs`,
which mutates the original request dict. When the engine retries
a blocked request via request.copy(), the copy no longer has
`method`, so it defaults to GET.
Steps to reproduce:
1. Yield Request(url, method="POST", data=...)
2. Target returns a response that triggers is_blocked()
3. Engine retries via request.copy() → second fetch uses GET
Fix: copy the kwargs dict before popping, so the original
request stays intact.
2026-03-17 00:53:52 +09:00
012e9f9f5e
docs: images adjustments
2026-03-16 16:23:08 +02:00
f25bca75bf
docs: fixing broken links
...
Also, Fixes #199
2026-03-16 15:38:23 +02:00
ca6c1f3c1e
docs: CJK Latin spacing fixes for the Chinese README
2026-03-16 15:26:25 +02:00
cec5acd68a
test: add edge case tests for filter, iterancestors, and find_similar
...
- test_selectors_filter.py: covers chained filter(), empty result,
all-pass predicate, and calling filter() on empty Selectors
- test_ancestor_navigation.py: covers iterancestors() order, depth,
text-node safety, root-element edge case, and find_ancestor() with
no match
- test_find_similar_advanced.py: covers similarity_threshold levels,
match_text=True behavior, ignore_attributes combinations, and
text-node safety
2026-03-16 18:01:11 +08:00
72a2c8de81
Merge branch 'main' into dev
2026-03-15 22:44:16 +02:00
eb32285c49
docs: add the sponsor translated versions
2026-03-15 22:25:50 +02:00
f286575f31
docs: adding a new sponsor
2026-03-15 22:17:12 +02:00
1e7704864e
Merge branch 'dev' into test/custom-types-coverage
2026-03-15 16:38:48 +02:00
e63e57e6dc
test(ai): add _normalize_credentials edge case coverage ( #192 )
2026-03-15 16:25:57 +02:00
656dbc7d12
Merge branch 'dev' into test/normalize-credentials-coverage
2026-03-15 16:24:59 +02:00
d3c251c1ab
fix: add max retry limit to _get_page_content to prevent infinite loop
...
Both _get_page_content and _get_async_page_content use a while-True
loop that retries page.content() on PlaywrightError with no upper
bound. If the page is in a permanently broken state (crashed tab,
closed context), this loops forever and hangs the process.
Replace with a bounded for-loop (default 10 retries = 5s), returning
an empty string if all attempts fail. This preserves the existing
retry behavior for the transient Windows issue (playwright#16108)
while preventing hangs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-03-15 17:53:23 +08:00
a31763afde
fix: replace bare raise with return False in _restore_from_checkpoint
...
When _checkpoint_system_enabled is False, the method uses a bare
`raise` with no active exception, which causes RuntimeError at
runtime. The method's docstring says it returns False when restoration
is not possible, so return False is the correct behavior.
The caller in crawl() currently guards with `if
self._checkpoint_system_enabled`, but the method's own contract
should be self-consistent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-03-15 17:52:57 +08:00
Karim shoair