freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update the agent skill with the new features

Browse Source 
Before I forget lol
This commit is contained in:
Karim shoair
2026-03-29 23:46:25 +02:00
parent 7f552bed73
commit bcd39d57c0
6 changed files with 76 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent-skill/Scrapling-Skill.zip
BIN
View File
Binary file not shown.
agent-skill/Scrapling-Skill/SKILL.md
+9 -3
View File
@@ -346,19 +346,25 @@ async with FetcherSession(http3=True) as session: # `FetcherSession` is context
async with AsyncStealthySession(max_pages=2) as session:
tasks = []
urls = ['https://example.com/page1', 'https://example.com/page2']
for url in urls:
task = session.fetch(url)
tasks.append(task)
print(session.get_pool_stats()) # Optional - The status of the browser tabs pool (busy/free/error)
results = await asyncio.gather(*tasks)
print(session.get_pool_stats())
# Capture XHR/fetch API calls during page load
async with AsyncDynamicSession(capture_xhr=r"https://api\.example\.com/.*") as session:
page = await session.fetch('https://example.com')
for xhr in page.captured_xhr: # Each is a full Response object
print(xhr.url, xhr.status, xhr.body)
```
## References
You already had a good glimpse of what the library can do. Use the references below to dig deeper when needed
- `references/mcp-server.md` — MCP server tools and capabilities
- `references/mcp-server.md` — MCP server tools, persistent session management, and capabilities
- `references/parsing` — Everything you need for parsing HTML
- `references/fetching` — Everything you need to fetch websites and session persistence
- `references/spiders` — Everything you need to write spiders, proxy rotation, and advanced features. It follows a Scrapy-like format
agent-skill/Scrapling-Skill/references/fetching/choosing.md
+1
View File
@@ -71,6 +71,7 @@ The `Response` object is the same as the [Selector](parsing/main_classes.md#sele
>>> page.body # Raw response body as bytes
>>> page.encoding # Response encoding
>>> page.meta # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
>>> page.captured_xhr # List of captured XHR/fetch responses (when capture_xhr is enabled on a browser session)
```
All fetchers return the `Response` object.
agent-skill/Scrapling-Skill/references/fetching/dynamic.md
+19
View File
@@ -79,6 +79,7 @@ All arguments for `DynamicFetcher` and its session classes:
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `blocked_domains`, `proxy`, and `selector_config`.
@@ -201,6 +202,24 @@ The states the fetcher can wait for can be any of the following ([source](https:
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
### Capturing XHR/Fetch Requests
Many SPAs load data through background API calls (XHR/fetch). You can capture these requests by passing a regex URL pattern to `capture_xhr` at the session level:
```python
from scrapling.fetchers import DynamicSession
with DynamicSession(capture_xhr=r"https://api\.example\.com/.*", headless=True) as session:
page = session.fetch('https://example.com')
# Access captured XHR responses
for xhr in page.captured_xhr:
print(xhr.url, xhr.status)
print(xhr.body) # Raw response body as bytes
```
Each item in `captured_xhr` is a full `Response` object with the same properties (`.url`, `.status`, `.headers`, `.body`, etc.). When `capture_xhr` is not set or is `None`, `captured_xhr` is an empty list.
### Some Stealth Features
```python
agent-skill/Scrapling-Skill/references/fetching/stealthy.md
+1
View File
@@ -61,6 +61,7 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.
agent-skill/Scrapling-Skill/references/mcp-server.md
+46 -5
View File
@@ -1,8 +1,8 @@
# Scrapling MCP Server
The Scrapling MCP server exposes six web scraping tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results) and three levels of scraping capability: plain HTTP, browser-rendered, and stealth (anti-bot bypass).
The Scrapling MCP server exposes nine web scraping tools over the MCP protocol. It supports CSS-selector-based content narrowing (reducing tokens by extracting only relevant elements before returning results), three levels of scraping capability (plain HTTP, browser-rendered, and stealth/anti-bot bypass), and persistent browser session management.
All tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str).
All scraping tools return a `ResponseModel` with fields: `status` (int), `content` (list of strings), `url` (str).
## Tools
@@ -66,10 +66,11 @@ Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynam
| `cookies` | list or null | null | Playwright-format cookies |
| `timezone_id` | str or null | null | Browser timezone, e.g. `"America/New_York"` |
| `locale` | str or null | null | Browser locale, e.g. `"en-GB"` |
| `session_id` | str or null | null | Reuse a persistent session from `open_session` instead of creating a new browser |
### `bulk_fetch` -- Browser fetch (multiple URLs)
Concurrent browser version of `fetch`. Same parameters except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.
Concurrent browser version of `fetch`. Same parameters (including `session_id`) except `url` is replaced by `urls` (list of strings). Each URL opens in a separate browser tab. Returns a list of `ResponseModel`.
### `stealthy_fetch` -- Stealth browser fetch (single URL)
@@ -84,12 +85,51 @@ Anti-bot bypass fetcher with fingerprint spoofing. Use this for sites with Cloud
| `block_webrtc` | bool | false | Force WebRTC to respect proxy settings (prevents IP leak) |
| `allow_webgl` | bool | true | Keep WebGL enabled (disabling is detectable by WAFs) |
| `additional_args` | dict or null | null | Extra Playwright context args (overrides Scrapling defaults) |
| `session_id` | str or null | null | Reuse a persistent stealthy session from `open_session` |
All parameters from `fetch` are also accepted.
### `bulk_stealthy_fetch` -- Stealth browser fetch (multiple URLs)
Concurrent stealth version. Same parameters as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.
Concurrent stealth version. Same parameters (including `session_id`) as `stealthy_fetch` except `url` is replaced by `urls` (list of strings). Returns a list of `ResponseModel`.
### `open_session` -- Create a persistent browser session
Opens a browser session that stays alive across multiple fetch calls, avoiding the overhead of launching a new browser each time. Returns a `SessionCreatedModel` with `session_id`, `session_type`, `created_at`, `is_alive`, and `message`.
**Key parameters:**
| Parameter | Type | Default | Description |
|--------------------|-----------------------------|--------------|---------------------------------------------------------------------|
| `session_type` | `"dynamic"` / `"stealthy"` | required | Type of browser session to create |
| `headless` | bool | true | Run browser hidden or visible |
| `max_pages` | int | 5 | Max concurrent browser tabs (1-50) |
| `proxy` | str or dict or null | null | Proxy for all requests in this session |
| `timeout` | number | 30000 | Default timeout in ms |
| `solve_cloudflare` | bool | false | (Stealthy only) Auto-solve Cloudflare challenges |
| `hide_canvas` | bool | false | (Stealthy only) Canvas fingerprint noise |
| `block_webrtc` | bool | false | (Stealthy only) Block WebRTC IP leak |
| `allow_webgl` | bool | true | (Stealthy only) Keep WebGL enabled |
Plus all other browser session parameters (`google_search`, `real_chrome`, `cdp_url`, `locale`, `timezone_id`, `useragent`, `extra_headers`, `cookies`, `disable_resources`, `network_idle`, `wait_selector`, `wait_selector_state`).
A dynamic session can only be used with `fetch`/`bulk_fetch`. A stealthy session can only be used with `stealthy_fetch`/`bulk_stealthy_fetch`.
### `close_session` -- Close a persistent browser session
Closes a session and frees its browser resources. Always close sessions when done.
| Parameter | Type | Default | Description |
|--------------|------|----------|----------------------------------|
| `session_id` | str | required | Session ID from `open_session` |
Returns a `SessionClosedModel` with `session_id` and `message`.
### `list_sessions` -- List active sessions
Returns a list of `SessionInfo` objects, each with `session_id`, `session_type`, `created_at`, and `is_alive`.
No parameters.
## Tool selection guide
@@ -101,8 +141,9 @@ Concurrent stealth version. Same parameters as `stealthy_fetch` except `url` is
| Multiple JS-rendered pages | `bulk_fetch` |
| Cloudflare or strong anti-bot protection | `stealthy_fetch` (with `solve_cloudflare=true` for Turnstile) |
| Multiple protected pages | `bulk_stealthy_fetch` |
| Multiple pages from the same site | `open_session` + `fetch`/`stealthy_fetch` with `session_id` |
Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked.
Start with `get` (fastest, lowest resource cost). Escalate to `fetch` if content requires JS rendering. Escalate to `stealthy_fetch` only if blocked. For multiple pages from the same site, use a persistent session to avoid browser launch overhead.
## Content extraction tips