fix(agent): update the agent skills docs and version
This commit is contained in:
Karim shoair
@@ -1,7 +1,7 @@
|
||||
---
|
||||
name: scrapling-official
|
||||
description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
|
||||
version: 0.4.1
|
||||
version: 0.4.2
|
||||
license: Complete terms in LICENSE.txt
|
||||
---
|
||||
|
||||
@@ -22,7 +22,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo
|
||||
|
||||
Create a virtual Python environment through any way available, like `venv`, then inside the environment do:
|
||||
|
||||
`pip install "scrapling[all]>=0.4.1"`
|
||||
`pip install "scrapling[all]>=0.4.2"`
|
||||
|
||||
Then do this to download all the browsers' dependencies:
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**.
|
||||
Make sure Scrapling is installed:
|
||||
|
||||
```bash
|
||||
pip install "scrapling[all]>=0.4.1"
|
||||
pip install "scrapling[all]>=0.4.2"
|
||||
scrapling install --force
|
||||
```
|
||||
|
||||
|
||||
@@ -64,8 +64,8 @@ All arguments for `DynamicFetcher` and its session classes:
|
||||
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
|
||||
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
|
||||
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
|
||||
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
|
||||
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
|
||||
| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
|
||||
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
|
||||
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
|
||||
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
|
||||
| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
|
||||
@@ -84,7 +84,7 @@ In session classes, all these arguments can be set globally for the session. Sti
|
||||
|
||||
**Notes:**
|
||||
1. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading.
|
||||
2. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
|
||||
2. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
|
||||
3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`.
|
||||
4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ Check out how to configure the parsing options [here](choosing.md#parser-configu
|
||||
All methods for making requests here share some arguments, so let's discuss them first.
|
||||
|
||||
- **url**: The targeted URL
|
||||
- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets the referer header as if this request came from a Google search of the URL's domain.
|
||||
- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header.
|
||||
- **follow_redirects**: As the name implies, tell the fetcher to follow redirections. **Enabled by default**
|
||||
- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.
|
||||
- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.
|
||||
|
||||
@@ -43,8 +43,8 @@ Scrapling provides many options with this fetcher and its session classes. Befor
|
||||
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
|
||||
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
|
||||
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
|
||||
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
|
||||
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
|
||||
| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ |
|
||||
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ |
|
||||
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
|
||||
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
|
||||
| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
|
||||
@@ -69,7 +69,7 @@ In session classes, all these arguments can be set globally for the session. Sti
|
||||
|
||||
1. It's basically the same arguments as [DynamicFetcher](dynamic.md) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
|
||||
2. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading.
|
||||
3. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
|
||||
3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
|
||||
4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -25,7 +25,7 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo
|
||||
| `timeout` | number | 30 | Seconds before timeout |
|
||||
| `retries` | int | 3 | Retry attempts on failure |
|
||||
| `retry_delay` | int | 1 | Seconds between retries |
|
||||
| `stealthy_headers` | bool | true | Generate realistic browser headers and Google-search referer |
|
||||
| `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer |
|
||||
| `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) |
|
||||
| `follow_redirects` | bool | true | Follow HTTP redirects |
|
||||
| `max_redirects` | int | 30 | Max redirects (-1 for unlimited) |
|
||||
@@ -58,7 +58,7 @@ Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynam
|
||||
| `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` |
|
||||
| `network_idle` | bool | false | Wait until no network activity for 500ms |
|
||||
| `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed |
|
||||
| `google_search` | bool | true | Set referer as if from Google search |
|
||||
| `google_search` | bool | true | Set a Google referer header |
|
||||
| `real_chrome` | bool | false | Use locally installed Chrome instead of bundled Chromium |
|
||||
| `cdp_url` | str or null | null | Connect to existing browser via CDP URL |
|
||||
| `extra_headers` | dict or null | null | Additional request headers |
|
||||
|
||||
Reference in New Issue
Block a user