0f6dcccf5f
fix(mcp): remove unneeded code and fix type hint for mypy
2026-03-28 17:19:42 +02:00
c458ab65a2
feat(mcp): Add three new tools to control browser sessions
...
Now you can open a browser, keep using it for other requests as you want, and close it when you want.
2026-03-28 00:16:14 +02:00
a356dd2f2b
refactor(mcp)!: Cleaning and unifying functions to async
...
- `get()` now delegates to `bulk_get([url])[0]` (was a separate sync implementation)
- `fetch()` now delegates to `bulk_fetch([url])[0]` (eliminated duplicate fetcher call)
- `stealthy_fetch()` now delegates to `bulk_stealthy_fetch([url])[0]` (same)
- Replaced 6x repeated `_content_translator(Convertor._extract_content(...), page)` with a single `_translate_response()` helper
- Removed unused imports (`Fetcher`, `DynamicFetcher`, `StealthyFetcher`, `Generator`)
2026-03-27 19:30:58 +02:00
8e147db7f8
refactor(cli): Code cleaning for easier maintenance/adding new features
...
Shortened the code by 210 lines. Also, removed docstrings because they are not needed for CLI commands (more maintenance burden).
- `_common_http_options`: shared decorator for 10 Click options used by get/post/put/delete (was repeated 4x)
- `_common_browser_options`: shared decorator for 11 Click options used by fetch/stealthy_fetch (was repeated 2x)
- `_data_options`: shared decorator for `--data`/`--json` options used by post/put
- `__http_command()`: shared implementation body for all HTTP commands (was 4 separate `from scrapling.fetchers import Fetcher` + `__Request_and_Save` blocks)
- `__build_browser_kwargs()`: shared kwargs builder for fetch/stealthy_fetch (was duplicated)
2026-03-27 18:02:33 +02:00
d47599f204
Merge branch 'main' into dev
2026-03-26 17:54:49 +02:00
f36ddb4b16
docs: update a sponsor url
2026-03-26 00:04:47 +02:00
8e19f825a4
docs(skill): adjustment
2026-03-25 01:47:52 +02:00
c2c02dc78e
docs: updating the agent skill for Clawhub validation
2026-03-25 01:38:11 +02:00
422b4713ec
build: pump up version
2026-03-25 00:19:58 +02:00
9fee49c928
test: fixes to storage tests
2026-03-25 00:13:42 +02:00
3757cf6d77
test: add save/retrieve round-trip and core storage coverage ( #193 )
2026-03-25 00:11:42 +02:00
30ffe76cfa
Merge branch 'dev' into test/storage-core-coverage
2026-03-24 22:42:58 +02:00
3b4cae6889
Merge branch 'main' into dev
2026-03-24 22:21:07 +02:00
6635a4ad72
docs: adding a new sponsor
2026-03-24 17:13:24 +02:00
718db42ff0
Merge branch 'main' into dev
2026-03-23 16:40:12 +02:00
32cec404f6
docs: adjustment to ad plans
2026-03-23 15:00:26 +02:00
e230403232
docs: removing an old sponsor
2026-03-23 14:56:35 +02:00
df4b1ac7e3
Merge branch 'dev' into test/storage-core-coverage
2026-03-22 18:05:42 +02:00
d7c08073d4
Merge branch 'main' into dev
2026-03-22 18:05:26 +02:00
a96809a340
docs: removing an old sponsor
2026-03-22 16:48:26 +02:00
bc509e9fcd
Merge branch 'main' into dev
2026-03-19 17:01:25 +02:00
3ed59c2a84
fix(agent): update skill zip file with the latest fix
2026-03-19 17:01:05 +02:00
2e78b3f8aa
fix(skill): Fix broken markdown links in skill references ( #204 )
...
Fixes #203
2026-03-19 16:58:06 +02:00
a4f31ae205
simple md fix
2026-03-19 10:37:02 -04:00
7ca02ebb80
test: add edge case tests for filter, iterancestors, and find_similar ( #200 )
2026-03-17 22:16:15 +02:00
bf9e2da329
Merge branch 'dev' into test/edge-cases-filter-ancestors-find-similar
2026-03-17 22:15:59 +02:00
1dc0b7a1bd
fix(fetchers/content): increase the default max number of retries and raise error on max retries
...
Ref.: https://github.com/D4Vinci/Scrapling/pull/197#issuecomment-4077705587
2026-03-17 22:14:02 +02:00
17ea982162
fix(fetchers): add max retry limit to _get_page_content to prevent infinite loop ( #197 )
2026-03-17 22:09:05 +02:00
8e4e59e13e
Merge branch 'dev' into fix/page-content-infinite-loop
2026-03-17 22:08:47 +02:00
d9932f2308
fix(spider/fetcher): preserve HTTP method across retries in spider session ( #201 )
2026-03-17 22:06:01 +02:00
4c07b294ae
Merge branch 'dev' into fix/preserve-http-method-on-retry
2026-03-17 22:04:57 +02:00
27f306259e
Merge branch 'dev' into fix/page-content-infinite-loop
2026-03-17 21:55:26 +02:00
136c389787
fix(Texthandler): Replace get_all with getall to match the Selector class
2026-03-17 21:53:17 +02:00
980be18a2a
test: add coverage for TextHandler regex paths and TextHandlers.re() ( #194 )
2026-03-17 21:50:05 +02:00
5123590a15
Merge branch 'dev' into test/custom-types-coverage
2026-03-17 21:42:17 +02:00
cc6c0dbfb9
fix: adjust test to the _restore_from_checkpoint fix
2026-03-17 17:46:47 +02:00
1fb7961d42
fix: replace bare raise with return False in _restore_from_checkpoint ( #196 )
2026-03-17 17:45:52 +02:00
bcd45e47bb
Merge branch 'dev' into fix/checkpoint-restore-bare-raise
2026-03-17 17:36:11 +02:00
5eefbc6c05
Merge branch 'dev' into test/custom-types-coverage
2026-03-17 17:34:12 +02:00
6699213bdc
Merge branch 'dev' into test/storage-core-coverage
2026-03-17 17:31:31 +02:00
440da11259
Merge branch 'main' into dev
2026-03-17 17:28:38 +02:00
7bfaf204b7
docs: CJK Latin spacing fixes for the JP README
2026-03-17 17:12:41 +02:00
74abd54a0b
docs: Add translations to the new sponsor
2026-03-17 17:00:32 +02:00
3d60792cbc
docs: update Zensical
2026-03-17 16:53:55 +02:00
3b543a6bdd
docs: add a new sponsor
2026-03-17 16:45:04 +02:00
5bf921b308
fix: preserve HTTP method across retries in spider session
...
SessionManager.fetch() pops `method` from `_session_kwargs`,
which mutates the original request dict. When the engine retries
a blocked request via request.copy(), the copy no longer has
`method`, so it defaults to GET.
Steps to reproduce:
1. Yield Request(url, method="POST", data=...)
2. Target returns a response that triggers is_blocked()
3. Engine retries via request.copy() → second fetch uses GET
Fix: copy the kwargs dict before popping, so the original
request stays intact.
2026-03-17 00:53:52 +09:00
012e9f9f5e
docs: images adjustments
2026-03-16 16:23:08 +02:00
f25bca75bf
docs: fixing broken links
...
Also, Fixes #199
2026-03-16 15:38:23 +02:00
ca6c1f3c1e
docs: CJK Latin spacing fixes for the Chinese README
2026-03-16 15:26:25 +02:00
cec5acd68a
test: add edge case tests for filter, iterancestors, and find_similar
...
- test_selectors_filter.py: covers chained filter(), empty result,
all-pass predicate, and calling filter() on empty Selectors
- test_ancestor_navigation.py: covers iterancestors() order, depth,
text-node safety, root-element edge case, and find_ancestor() with
no match
- test_find_similar_advanced.py: covers similarity_threshold levels,
match_text=True behavior, ignore_attributes combinations, and
text-node safety
2026-03-16 18:01:11 +08:00
Karim shoair