diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip index b03d956..8daa704 100644 Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md index 6b3cbf7..d4519bb 100644 --- a/agent-skill/Scrapling-Skill/SKILL.md +++ b/agent-skill/Scrapling-Skill/SKILL.md @@ -1,7 +1,7 @@ --- name: scrapling-official description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders. -version: "0.4.4" +version: "0.4.5" license: Complete terms in LICENSE.txt metadata: homepage: "https://scrapling.readthedocs.io/en/latest/index.html" @@ -40,7 +40,7 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo Create a virtual Python environment through any way available, like `venv`, then inside the environment do: -`pip install "scrapling[all]>=0.4.4"` +`pip install "scrapling[all]>=0.4.5"` Then do this to download all the browsers' dependencies: @@ -104,7 +104,7 @@ Those options are shared between the 4 HTTP request commands: | --proxy | TEXT | Proxy URL in format "http://username:password@host:port" | | -s, --css-selector | TEXT | CSS selector to extract specific content from the page. It returns all matches. | | -p, --params | TEXT | Query parameters in format "key=value" (can be used multiple times) | -| --follow-redirects / --no-follow-redirects | None | Whether to follow redirects (default: True) | +| --follow-redirects / --no-follow-redirects | None | Whether to follow redirects (default: "safe", rejects redirects to internal/private IPs) | | --verify / --no-verify | None | Whether to verify SSL certificates (default: True) | | --impersonate | TEXT | Browser to impersonate. Can be a single browser (e.g., Chrome) or a comma-separated list for random selection (e.g., Chrome, Firefox, Safari). | | --stealthy-headers / --no-stealthy-headers | None | Use stealthy browser headers (default: True) | diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md index 0c2f031..344dc0f 100644 --- a/agent-skill/Scrapling-Skill/examples/README.md +++ b/agent-skill/Scrapling-Skill/examples/README.md @@ -9,7 +9,7 @@ All examples collect **all 100 quotes across 10 pages**. Make sure Scrapling is installed: ```bash -pip install "scrapling[all]>=0.4.4" +pip install "scrapling[all]>=0.4.5" scrapling install --force ``` diff --git a/agent-skill/Scrapling-Skill/references/fetching/static.md b/agent-skill/Scrapling-Skill/references/fetching/static.md index fa4d2a5..4115483 100644 --- a/agent-skill/Scrapling-Skill/references/fetching/static.md +++ b/agent-skill/Scrapling-Skill/references/fetching/static.md @@ -15,7 +15,7 @@ All methods for making requests here share some arguments, so let's discuss them - **url**: The targeted URL - **stealthy_headers**: If enabled (default), it creates and adds real browser headers. It also sets a Google referer header. -- **follow_redirects**: As the name implies, tell the fetcher to follow redirections. **Enabled by default** +- **follow_redirects**: Controls redirect behavior. **Defaults to `"safe"`**, which follows redirects but rejects those targeting internal/private IPs (SSRF protection). Pass `True` to follow all redirects without restriction, or `False` to disable redirects entirely. - **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**. - **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**. - **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**. @@ -50,7 +50,7 @@ Examples are the best way to explain this: >>> from scrapling.fetchers import Fetcher >>> # Basic GET >>> page = Fetcher.get('https://example.com') ->>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True) +>>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True) >>> page = Fetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030') >>> # With parameters >>> page = Fetcher.get('https://example.com/search', params={'q': 'query'}) @@ -69,7 +69,7 @@ And for asynchronous requests, it's a small adjustment >>> from scrapling.fetchers import AsyncFetcher >>> # Basic GET >>> page = await AsyncFetcher.get('https://example.com') ->>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True, follow_redirects=True) +>>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True) >>> page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', proxy='http://username:password@localhost:8030') >>> # With parameters >>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'}) @@ -105,7 +105,7 @@ The `page` object in all cases is a [Response](choosing.md#response-object) obje >>> from scrapling.fetchers import Fetcher >>> # Basic POST >>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, params={'q': 'query'}) ->>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True) +>>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True) >>> page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome") >>> # Another example of form-encoded data >>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True) @@ -117,7 +117,7 @@ And for asynchronous requests, it's a small adjustment >>> from scrapling.fetchers import AsyncFetcher >>> # Basic POST >>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}) ->>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True) +>>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, stealthy_headers=True) >>> page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030', impersonate="chrome") >>> # Another example of form-encoded data >>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'}, http3=True) @@ -129,7 +129,7 @@ And for asynchronous requests, it's a small adjustment >>> from scrapling.fetchers import Fetcher >>> # Basic PUT >>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}) ->>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome") +>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome") >>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030') >>> # Another example of form-encoded data >>> page = Fetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']}) @@ -139,7 +139,7 @@ And for asynchronous requests, it's a small adjustment >>> from scrapling.fetchers import AsyncFetcher >>> # Basic PUT >>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}) ->>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True, impersonate="chrome") +>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, impersonate="chrome") >>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030') >>> # Another example of form-encoded data >>> page = await AsyncFetcher.put("https://scrapling.requestcatcher.com/put", data={'key': ['value1', 'value2']}) @@ -149,14 +149,14 @@ And for asynchronous requests, it's a small adjustment ```python >>> from scrapling.fetchers import Fetcher >>> page = Fetcher.delete('https://example.com/resource/123') ->>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True, impersonate="chrome") +>>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome") >>> page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030') ``` And for asynchronous requests, it's a small adjustment ```python >>> from scrapling.fetchers import AsyncFetcher >>> page = await AsyncFetcher.delete('https://example.com/resource/123') ->>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True, impersonate="chrome") +>>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, impersonate="chrome") >>> page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030') ``` diff --git a/agent-skill/Scrapling-Skill/references/mcp-server.md b/agent-skill/Scrapling-Skill/references/mcp-server.md index 0a63642..7bbc86b 100644 --- a/agent-skill/Scrapling-Skill/references/mcp-server.md +++ b/agent-skill/Scrapling-Skill/references/mcp-server.md @@ -27,7 +27,7 @@ Fast HTTP GET with browser fingerprint impersonation (TLS, headers). Suitable fo | `retry_delay` | int | 1 | Seconds between retries | | `stealthy_headers` | bool | true | Generate realistic browser headers and Google referer | | `http3` | bool | false | Use HTTP/3 (may conflict with `impersonate`) | -| `follow_redirects` | bool | true | Follow HTTP redirects | +| `follow_redirects` | bool or "safe" | "safe" | Follow redirects. "safe" rejects redirects to internal/private IPs | | `max_redirects` | int | 30 | Max redirects (-1 for unlimited) | | `headers` | dict or null | null | Custom request headers | | `cookies` | dict or null | null | Request cookies |