freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(agent): update the agent skills docs and version

Browse Source
This commit is contained in:
Karim shoair
2026-03-09 00:27:10 +02:00
parent a929de6ca4
commit 8e3bf89e6c
6 changed files with 12 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent-skill/Scrapling-Skill/SKILL.md
+2 -2
View File
@@ -1,7 +1,7 @@
--- ---
name: scrapling-official name: scrapling-official
description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders. description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
version: 0.4.1 version: 0.4.2
license: Complete terms in LICENSE.txt license: Complete terms in LICENSE.txt
--- ---
@@ -22,7 +22,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo
Create a virtual Python environment through any way available, like `venv`, then inside the environment do: Create a virtual Python environment through any way available, like `venv`, then inside the environment do:
`pip install "scrapling[all]>=0.4.1"` `pip install "scrapling[all]>=0.4.2"`
Then do this to download all the browsers' dependencies: Then do this to download all the browsers' dependencies:
agent-skill/Scrapling-Skill/examples/README.md
+1 -1
View File
@@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.
Make sure Scrapling is installed: Make sure Scrapling is installed:
```bash ```bash
pip install "scrapling[all]>=0.4.1" pip install "scrapling[all]>=0.4.2"
scrapling install --force scrapling install --force
``` ```
agent-skill/Scrapling-Skill/references/fetching/dynamic.md
+3 -3
View File
@@ -64,8 +64,8 @@ All arguments for `DynamicFetcher` and its session classes:
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ | | wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ | | init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ | | wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ | | google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ | | extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ | | proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ | | real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ | | locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
@@ -84,7 +84,7 @@ In session classes, all these arguments can be set globally for the session. Sti
**Notes:** **Notes:**
1. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading. 1. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading.
2. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument. 2. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`. 3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`.
4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions. 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
agent-skill/Scrapling-Skill/references/fetching/static.md
+1 -1
View File
@@ -14,7 +14,7 @@ Check out how to configure the parsing options [here](choosing.md#parser-configu
All methods for making requests here share some arguments, so let's discuss them first. All methods for making requests here share some arguments, so let's discuss them first.
- **url**: The targeted URL - **url**: The targeted URL
- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets the referer header as if this request came from a Google search of the URL's domain. - **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
- **follow_redirects**: As the name implies, tell the fetcher to follow redirections. **Enabled by default** - **follow_redirects**: As the name implies, tell the fetcher to follow redirections. **Enabled by default**
- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**. - **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.
- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**. - **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.
agent-skill/Scrapling-Skill/references/fetching/stealthy.md
+3 -3
View File
@@ -43,8 +43,8 @@ Scrapling provides many options with this fetcher and its session classes. Befor
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ | | wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ | | init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ | | wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ | | google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ | | extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ | | proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ | | real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ | | locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
@@ -69,7 +69,7 @@ In session classes, all these arguments can be set globally for the session. Sti
1. It's basically the same arguments as [DynamicFetcher](dynamic.md) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`. 1. It's basically the same arguments as [DynamicFetcher](dynamic.md) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
2. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading. 2. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading.
3. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument. 3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions. 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
## Examples ## Examples
agent-skill/Scrapling-Skill/references/mcp-server.md
+2 -2
View File
@@ -25,7 +25,7 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo
| `timeout` | number | 30 | Seconds before timeout | | `timeout` | number | 30 | Seconds before timeout |
| `retries` | int | 3 | Retry attempts on failure | | `retries` | int | 3 | Retry attempts on failure |
| `retry_delay` | int | 1 | Seconds between retries | | `retry_delay` | int | 1 | Seconds between retries |
| `stealthy_headers` | bool | true | Generate realistic browser headers and Google-search referer | | `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer |
| `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) | | `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) |
| `follow_redirects` | bool | true | Follow HTTP redirects | | `follow_redirects` | bool | true | Follow HTTP redirects |
| `max_redirects` | int | 30 | Max redirects (-1 for unlimited) | | `max_redirects` | int | 30 | Max redirects (-1 for unlimited) |
@@ -58,7 +58,7 @@ Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynam
| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` | | `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
| `network_idle` | bool | false | Wait until no network activity for 500ms | | `network_idle` | bool | false | Wait until no network activity for 500ms |
| `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed | | `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed |
| `google_search` | bool | true | Set referer as if from Google search | | `google_search` | bool | true | Set a Google referer header |
| `real_chrome` | bool | false | Use locally installed Chrome instead of bundled Chromium | | `real_chrome` | bool | false | Use locally installed Chrome instead of bundled Chromium |
| `cdp_url` | str or null | null | Connect to existing browser via CDP URL | | `cdp_url` | str or null | null | Connect to existing browser via CDP URL |
| `extra_headers` | dict or null | null | Additional request headers | | `extra_headers` | dict or null | null | Additional request headers |