diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md index 4578893..724e704 100644 --- a/agent-skill/Scrapling-Skill/SKILL.md +++ b/agent-skill/Scrapling-Skill/SKILL.md @@ -1,7 +1,7 @@ --- name: scrapling-official description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders. -version: 0.4.1 +version: 0.4.2 license: Complete terms in LICENSE.txt --- @@ -22,7 +22,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo Create a virtual Python environment through any way available, like `venv`, then inside the environment do: -`pip install "scrapling[all]>=0.4.1"` +`pip install "scrapling[all]>=0.4.2"` Then do this to download all the browsers' dependencies: diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md index 4d5741a..dfc65e4 100644 --- a/agent-skill/Scrapling-Skill/examples/README.md +++ b/agent-skill/Scrapling-Skill/examples/README.md @@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**. Make sure Scrapling is installed: ```bash -pip install "scrapling[all]>=0.4.1" +pip install "scrapling[all]>=0.4.2" scrapling install --force ``` diff --git a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md index ea3369e..a87fdf6 100644 --- a/agent-skill/Scrapling-Skill/references/fetching/dynamic.md +++ b/agent-skill/Scrapling-Skill/references/fetching/dynamic.md @@ -64,8 +64,8 @@ All arguments for `DynamicFetcher` and its session classes: | wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ | | init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ | | wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ | -| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ | -| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ | +| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ | +| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ | | proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ | | real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ | | locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ | @@ -84,7 +84,7 @@ In session classes, all these arguments can be set globally for the session. Sti **Notes:** 1. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading. -2. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument. +2. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there. 3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`. 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions. diff --git a/agent-skill/Scrapling-Skill/references/fetching/static.md b/agent-skill/Scrapling-Skill/references/fetching/static.md index 72ff74a..fa4d2a5 100644 --- a/agent-skill/Scrapling-Skill/references/fetching/static.md +++ b/agent-skill/Scrapling-Skill/references/fetching/static.md @@ -14,7 +14,7 @@ Check out how to configure the parsing options [here](choosing.md#parser-configu All methods for making requests here share some arguments, so let's discuss them first. - **url**: The targeted URL -- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets the referer header as if this request came from a Google search of the URL's domain. +- **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header. - **follow_redirects**: As the name implies, tell the fetcher to follow redirections. **Enabled by default** - **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**. - **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**. diff --git a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md index 1bde6f4..98aecd5 100644 --- a/agent-skill/Scrapling-Skill/references/fetching/stealthy.md +++ b/agent-skill/Scrapling-Skill/references/fetching/stealthy.md @@ -43,8 +43,8 @@ Scrapling provides many options with this fetcher and its session classes. Befor | wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ | | init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ | | wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ | -| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ | -| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ | +| google_search | Enabled by default, Scrapling will set a Google referer header. | ✔️ | +| extra_headers | A dictionary of extra headers to add to the request. _The referer set by `google_search` takes priority over the referer set here if used together._ | ✔️ | | proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ | | real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ | | locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ | @@ -69,7 +69,7 @@ In session classes, all these arguments can be set globally for the session. Sti 1. It's basically the same arguments as [DynamicFetcher](dynamic.md) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`. 2. The `disable_resources` option made requests ~25% faster in tests for some websites and can help save proxy usage, but be careful with it, as it can cause some websites to never finish loading. -3. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument. +3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there. 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions. ## Examples diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md index fae1f14..fbe1b7d 100644 --- a/agent-skill/Scrapling-Skill/references/mcp-server.md +++ b/agent-skill/Scrapling-Skill/references/mcp-server.md @@ -25,7 +25,7 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo | `timeout` | number | 30 | Seconds before timeout | | `retries` | int | 3 | Retry attempts on failure | | `retry_delay` | int | 1 | Seconds between retries | -| `stealthy_headers` | bool | true | Generate realistic browser headers and Google-search referer | +| `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer | | `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) | | `follow_redirects` | bool | true | Follow HTTP redirects | | `max_redirects` | int | 30 | Max redirects (-1 for unlimited) | @@ -58,7 +58,7 @@ Opens a Chromium browser via Playwright to render JavaScript. Suitable for dynam | `wait_selector_state` | str | `"attached"` | State for wait_selector: `"attached"` / `"visible"` / `"hidden"` / `"detached"` | | `network_idle` | bool | false | Wait until no network activity for 500ms | | `disable_resources` | bool | false | Block fonts, images, media, stylesheets, etc. for speed | -| `google_search` | bool | true | Set referer as if from Google search | +| `google_search` | bool | true | Set a Google referer header | | `real_chrome` | bool | false | Use locally installed Chrome instead of bundled Chromium | | `cdp_url` | str or null | null | Connect to existing browser via CDP URL | | `extra_headers` | dict or null | null | Additional request headers |