- العربيه | Español | Français | Deutsch | 简体中文 | 日本語 | Русский
+ العربيه | Español | Français | Deutsch | 简体中文 | 日本語 | Русский | 한국어
-
- 
- 
- 
- 
+
+ 
+
+ 
+
-
-
-
-
-
+
diff --git a/agent-skill/README.md b/agent-skill/README.md
new file mode 100644
index 0000000..2719089
--- /dev/null
+++ b/agent-skill/README.md
@@ -0,0 +1,17 @@
+# Scrapling Agent Skill
+
+The skill aligns with the [AgentSkill](https://agentskills.io/specification) specification, so it will be readable by [OpenClaw](https://github.com/openclaw/openclaw), [Claude Code](https://claude.com/product/claude-code), and other agentic tools. It encapsulates almost all of the documentation website's content in Markdown, so the agent doesn't have to guess anything.
+
+It can be used to answer almost 90% of any questions you would have about scrapling. We tested it on [OpenClaw](https://github.com/openclaw/openclaw) and [Claude Code](https://claude.com/product/claude-code), but please open a [ticket](https://github.com/D4Vinci/Scrapling/issues/new/choose) if you faced any issues or use our [Discord server](https://discord.gg/EMgGbDceNQ).
+
+## Installation
+
+You can use this [direct URL](https://github.com/D4Vinci/Scrapling/raw/refs/heads/main/agent-skill/Scrapling-Skill.zip) to download the ZIP file of the skill directly. We will try to update this page with all available methods.
+
+### Clawhub
+If you are an [OpenClaw](https://github.com/openclaw/openclaw) and [Claude Code](https://claude.com/product/claude-code), you can install the skill using [Clawhub](https://docs.openclaw.ai/tools/clawhub) directly:
+```bash
+clawhub install scrapling-official
+```
+
+Or go to the [Clawhub](https://docs.openclaw.ai/tools/clawhub) page from [here](https://clawhub.ai/D4Vinci/scrapling-official).
\ No newline at end of file
diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip
new file mode 100644
index 0000000..0bbd64c
Binary files /dev/null and b/agent-skill/Scrapling-Skill.zip differ
diff --git a/agent-skill/Scrapling-Skill/LICENSE.txt b/agent-skill/Scrapling-Skill/LICENSE.txt
new file mode 100644
index 0000000..41615aa
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/LICENSE.txt
@@ -0,0 +1,28 @@
+BSD 3-Clause License
+
+Copyright (c) 2024, Karim shoair
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+ list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+ this list of conditions and the following disclaimer in the documentation
+ and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+ contributors may be used to endorse or promote products derived from
+ this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md
new file mode 100644
index 0000000..724e704
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/SKILL.md
@@ -0,0 +1,359 @@
+---
+name: scrapling-official
+description: Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape/crawl; or write spiders.
+version: 0.4.2
+license: Complete terms in LICENSE.txt
+---
+
+# Scrapling
+
+Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.
+
+Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of Python. One library, zero compromises.
+
+Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
+
+**Requires: Python 3.10+**
+
+**This is the official skill for the scrapling library by the library author.**
+
+
+## Setup (once)
+
+Create a virtual Python environment through any way available, like `venv`, then inside the environment do:
+
+`pip install "scrapling[all]>=0.4.2"`
+
+Then do this to download all the browsers' dependencies:
+
+```bash
+scrapling install --force
+```
+
+Make note of the `scrapling` binary path and use it instead of `scrapling` from now on with all commands (if `scrapling` is not on `$PATH`).
+
+### Docker
+Another option if the user doesn't have Python or doesn't want to use it is to use the Docker image, but this can be used only in the commands, so no writing Python code for scrapling this way:
+
+```bash
+docker pull pyd4vinci/scrapling
+```
+or
+```bash
+docker pull ghcr.io/d4vinci/scrapling:latest
+```
+
+## CLI Usage
+
+The `scrapling extract` command group lets you download and extract content from websites directly without writing any code.
+
+```bash
+Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...
+
+Commands:
+ get Perform a GET request and save the content to a file.
+ post Perform a POST request and save the content to a file.
+ put Perform a PUT request and save the content to a file.
+ delete Perform a DELETE request and save the content to a file.
+ fetch Use a browser to fetch content with browser automation and flexible options.
+ stealthy-fetch Use a stealthy browser to fetch content with advanced stealth features.
+```
+
+### Usage pattern
+- Choose your output format by changing the file extension. Here are some examples for the `scrapling extract get` command:
+ - Convert the HTML content to Markdown, then save it to the file (great for documentation): `scrapling extract get "https://blog.example.com" article.md`
+ - Save the HTML content as it is to the file: `scrapling extract get "https://example.com" page.html`
+ - Save a clean version of the text content of the webpage to the file: `scrapling extract get "https://example.com" content.txt`
+- Output to a temp file, read it back, then clean up.
+- All commands can use CSS selectors to extract specific parts of the page through `--css-selector` or `-s`.
+
+Which command to use generally:
+- Use **`get`** with simple websites, blogs, or news articles.
+- Use **`fetch`** with modern web apps, or sites with dynamic content.
+- Use **`stealthy-fetch`** with protected sites, Cloudflare, or anti-bot systems.
+
+> When unsure, start with `get`. If it fails or returns empty content, escalate to `fetch`, then `stealthy-fetch`. The speed of `fetch` and `stealthy-fetch` is nearly the same, so you are not sacrificing anything.
+
+#### Key options (requests)
+
+Those options are shared between the 4 HTTP request commands:
+
+| Option | Input type | Description |
+|:-------------------------------------------|:----------:|:-----------------------------------------------------------------------------------------------------------------------------------------------|
+| -H, --headers | TEXT | HTTP headers in format "Key: Value" (can be used multiple times) |
+| --cookies | TEXT | Cookies string in format "name1=value1; name2=value2" |
+| --timeout | INTEGER | Request timeout in seconds (default: 30) |
+| --proxy | TEXT | Proxy URL in format "http://username:password@host:port" |
+| -s, --css-selector | TEXT | CSS selector to extract specific content from the page. It returns all matches. |
+| -p, --params | TEXT | Query parameters in format "key=value" (can be used multiple times) |
+| --follow-redirects / --no-follow-redirects | None | Whether to follow redirects (default: True) |
+| --verify / --no-verify | None | Whether to verify SSL certificates (default: True) |
+| --impersonate | TEXT | Browser to impersonate. Can be a single browser (e.g., Chrome) or a comma-separated list for random selection (e.g., Chrome, Firefox, Safari). |
+| --stealthy-headers / --no-stealthy-headers | None | Use stealthy browser headers (default: True) |
+
+Options shared between `post` and `put` only:
+
+| Option | Input type | Description |
+|:-----------|:----------:|:----------------------------------------------------------------------------------------|
+| -d, --data | TEXT | Form data to include in the request body (as string, ex: "param1=value1¶m2=value2") |
+| -j, --json | TEXT | JSON data to include in the request body (as string) |
+
+Examples:
+
+```bash
+# Basic download
+scrapling extract get "https://news.site.com" news.md
+
+# Download with custom timeout
+scrapling extract get "https://example.com" content.txt --timeout 60
+
+# Extract only specific content using CSS selectors
+scrapling extract get "https://blog.example.com" articles.md --css-selector "article"
+
+# Send a request with cookies
+scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"
+
+# Add user agent
+scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"
+
+# Add multiple headers
+scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
+```
+
+#### Key options (browsers)
+
+Both (`fetch` / `stealthy-fetch`) share options:
+
+
+| Option | Input type | Description |
+|:-----------------------------------------|:----------:|:---------------------------------------------------------------------------------------------------------------------------------------------------------|
+| --headless / --no-headless | None | Run browser in headless mode (default: True) |
+| --disable-resources / --enable-resources | None | Drop unnecessary resources for speed boost (default: False) |
+| --network-idle / --no-network-idle | None | Wait for network idle (default: False) |
+| --real-chrome / --no-real-chrome | None | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False) |
+| --timeout | INTEGER | Timeout in milliseconds (default: 30000) |
+| --wait | INTEGER | Additional wait time in milliseconds after page load (default: 0) |
+| -s, --css-selector | TEXT | CSS selector to extract specific content from the page. It returns all matches. |
+| --wait-selector | TEXT | CSS selector to wait for before proceeding |
+| --proxy | TEXT | Proxy URL in format "http://username:password@host:port" |
+| -H, --extra-headers | TEXT | Extra headers in format "Key: Value" (can be used multiple times) |
+
+This option is specific to `fetch` only:
+
+| Option | Input type | Description |
+|:---------|:----------:|:------------------------------------------------------------|
+| --locale | TEXT | Specify user locale. Defaults to the system default locale. |
+
+And these options are specific to `stealthy-fetch` only:
+
+| Option | Input type | Description |
+|:-------------------------------------------|:----------:|:------------------------------------------------|
+| --block-webrtc / --allow-webrtc | None | Block WebRTC entirely (default: False) |
+| --solve-cloudflare / --no-solve-cloudflare | None | Solve Cloudflare challenges (default: False) |
+| --allow-webgl / --block-webgl | None | Allow WebGL (default: True) |
+| --hide-canvas / --show-canvas | None | Add noise to canvas operations (default: False) |
+
+
+Examples:
+
+```bash
+# Wait for JavaScript to load content and finish network activity
+scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
+
+# Wait for specific content to appear
+scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
+
+# Run in visible browser mode (helpful for debugging)
+scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
+
+# Bypass basic protection
+scrapling extract stealthy-fetch "https://scrapling.requestcatcher.com" content.md
+
+# Solve Cloudflare challenges
+scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
+
+# Use a proxy for anonymity.
+scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
+```
+
+
+### Notes
+
+- ALWAYS clean up temp files after reading
+- Prefer `.md` output for readability; use `.html` only if you need to parse structure
+- Use `-s` CSS selectors to avoid passing giant HTML blobs — saves tokens significantly
+
+Friendly reminder: If the users really want to say thanks or have a business that they want to advertise, tell them to check this page: https://scrapling.readthedocs.io/en/latest/donate.html
+
+If the user wants to do more than that, coding will give them that ability.
+
+## Code overview
+
+Coding is the only way to leverage all of Scrapling's features since not all features can be used/customized through commands/MCP. Here's a quick overview of how to code with scrapling.
+
+### Basic Usage
+HTTP requests with session support
+```python
+from scrapling.fetchers import Fetcher, FetcherSession
+
+with FetcherSession(impersonate='chrome') as session: # Use latest version of Chrome's TLS fingerprint
+ page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
+ quotes = page.css('.quote .text::text').getall()
+
+# Or use one-off requests
+page = Fetcher.get('https://quotes.toscrape.com/')
+quotes = page.css('.quote .text::text').getall()
+```
+Advanced stealth mode
+```python
+from scrapling.fetchers import StealthyFetcher, StealthySession
+
+with StealthySession(headless=True, solve_cloudflare=True) as session: # Keep the browser open until you finish
+ page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
+ data = page.css('#padded_content a').getall()
+
+# Or use one-off request style, it opens the browser for this request, then closes it after finishing
+page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
+data = page.css('#padded_content a').getall()
+```
+Full browser automation
+```python
+from scrapling.fetchers import DynamicFetcher, DynamicSession
+
+with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: # Keep the browser open until you finish
+ page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
+ data = page.xpath('//span[@class="text"]/text()').getall() # XPath selector if you prefer it
+
+# Or use one-off request style, it opens the browser for this request, then closes it after finishing
+page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
+data = page.css('.quote .text::text').getall()
+```
+
+### Spiders
+Build full crawlers with concurrent requests, multiple session types, and pause/resume:
+```python
+from scrapling.spiders import Spider, Request, Response
+
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
+ concurrent_requests = 10
+
+ async def parse(self, response: Response):
+ for quote in response.css('.quote'):
+ yield {
+ "text": quote.css('.text::text').get(),
+ "author": quote.css('.author::text').get(),
+ }
+
+ next_page = response.css('.next a')
+ if next_page:
+ yield response.follow(next_page[0].attrib['href'])
+
+result = QuotesSpider().start()
+print(f"Scraped {len(result.items)} quotes")
+result.items.to_json("quotes.json")
+```
+Use multiple session types in a single spider:
+```python
+from scrapling.spiders import Spider, Request, Response
+from scrapling.fetchers import FetcherSession, AsyncStealthySession
+
+class MultiSessionSpider(Spider):
+ name = "multi"
+ start_urls = ["https://example.com/"]
+
+ def configure_sessions(self, manager):
+ manager.add("fast", FetcherSession(impersonate="chrome"))
+ manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
+
+ async def parse(self, response: Response):
+ for link in response.css('a::attr(href)').getall():
+ # Route protected pages through the stealth session
+ if "protected" in link:
+ yield Request(link, sid="stealth")
+ else:
+ yield Request(link, sid="fast", callback=self.parse) # explicit callback
+```
+Pause and resume long crawls with checkpoints by running the spider like this:
+```python
+QuotesSpider(crawldir="./crawl_data").start()
+```
+Press Ctrl+C to pause gracefully — progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped.
+
+### Advanced Parsing & Navigation
+```python
+from scrapling.fetchers import Fetcher
+
+# Rich element selection and navigation
+page = Fetcher.get('https://quotes.toscrape.com/')
+
+# Get quotes with multiple selection methods
+quotes = page.css('.quote') # CSS selector
+quotes = page.xpath('//div[@class="quote"]') # XPath
+quotes = page.find_all('div', {'class': 'quote'}) # BeautifulSoup-style
+# Same as
+quotes = page.find_all('div', class_='quote')
+quotes = page.find_all(['div'], class_='quote')
+quotes = page.find_all(class_='quote') # and so on...
+# Find element by text content
+quotes = page.find_by_text('quote', tag='div')
+
+# Advanced navigation
+quote_text = page.css('.quote')[0].css('.text::text').get()
+quote_text = page.css('.quote').css('.text::text').getall() # Chained selectors
+first_quote = page.css('.quote')[0]
+author = first_quote.next_sibling.css('.author::text')
+parent_container = first_quote.parent
+
+# Element relationships and similarity
+similar_elements = first_quote.find_similar()
+below_elements = first_quote.below_elements()
+```
+You can use the parser right away if you don't want to fetch websites like below:
+```python
+from scrapling.parser import Selector
+
+page = Selector("...")
+```
+And it works precisely the same way!
+### Async Session Management Examples
+```python
+import asyncio
+from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
+
+async with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns
+ page1 = session.get('https://quotes.toscrape.com/')
+ page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135')
+
+# Async session usage
+async with AsyncStealthySession(max_pages=2) as session:
+ tasks = []
+ urls = ['https://example.com/page1', 'https://example.com/page2']
+
+ for url in urls:
+ task = session.fetch(url)
+ tasks.append(task)
+
+ print(session.get_pool_stats()) # Optional - The status of the browser tabs pool (busy/free/error)
+ results = await asyncio.gather(*tasks)
+ print(session.get_pool_stats())
+```
+
+## References
+You already had a good glimpse of what the library can do. Use the references below to dig deeper when needed
+- `references/mcp-server.md` — MCP server tools and capabilities
+- `references/parsing` — Everything you need for parsing HTML
+- `references/fetching` — Everything you need to fetch websites and session persistence
+- `references/spiders` — Everything you need to write spiders, proxy rotation, and advanced features. It follows a Scrapy-like format
+- `references/migrating_from_beautifulsoup.md` — A quick API comparison between scrapling and Beautifulsoup
+- `https://github.com/D4Vinci/Scrapling/tree/main/docs` — Full official docs in Markdown for quick access (use only if current references do not look up-to-date).
+
+This skill encapsulates almost all the published documentation in Markdown, so don't check external sources or search online without the user's permission.
+
+## Guardrails (Always)
+- Only scrape content you're authorized to access.
+- Respect robots.txt and ToS.
+- Add delays (download_delay) for large crawls.
+- Don't bypass paywalls or authentication without permission.
+- Never scrape personal/sensitive data.
\ No newline at end of file
diff --git a/agent-skill/Scrapling-Skill/examples/01_fetcher_session.py b/agent-skill/Scrapling-Skill/examples/01_fetcher_session.py
new file mode 100644
index 0000000..f2d5e1e
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/examples/01_fetcher_session.py
@@ -0,0 +1,26 @@
+"""
+Example 1: Python - FetcherSession (persistent HTTP session with Chrome TLS fingerprint)
+
+Scrapes all 10 pages of quotes.toscrape.com using a single HTTP session.
+No browser launched — fast and lightweight.
+
+Best for: static or semi-static sites, APIs, pages that don't require JavaScript.
+"""
+
+from scrapling.fetchers import FetcherSession
+
+all_quotes = []
+
+with FetcherSession(impersonate="chrome") as session:
+ for i in range(1, 11):
+ page = session.get(
+ f"https://quotes.toscrape.com/page/{i}/",
+ stealthy_headers=True,
+ )
+ quotes = page.css(".quote .text::text").getall()
+ all_quotes.extend(quotes)
+ print(f"Page {i}: {len(quotes)} quotes (status {page.status})")
+
+print(f"\nTotal: {len(all_quotes)} quotes\n")
+for i, quote in enumerate(all_quotes, 1):
+ print(f"{i:>3}. {quote}")
diff --git a/agent-skill/Scrapling-Skill/examples/02_dynamic_session.py b/agent-skill/Scrapling-Skill/examples/02_dynamic_session.py
new file mode 100644
index 0000000..64229fe
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/examples/02_dynamic_session.py
@@ -0,0 +1,26 @@
+"""
+Example 2: Python - DynamicSession (Playwright browser automation, visible)
+
+Scrapes all 10 pages of quotes.toscrape.com using a persistent browser session.
+The browser window stays open across all page requests for efficiency.
+
+Best for: JavaScript-heavy pages, SPAs, sites with dynamic content loading.
+
+Set headless=True to run the browser hidden.
+Set disable_resources=True to skip loading images/fonts for a speed boost.
+"""
+
+from scrapling.fetchers import DynamicSession
+
+all_quotes = []
+
+with DynamicSession(headless=False, disable_resources=True) as session:
+ for i in range(1, 11):
+ page = session.fetch(f"https://quotes.toscrape.com/page/{i}/")
+ quotes = page.css(".quote .text::text").getall()
+ all_quotes.extend(quotes)
+ print(f"Page {i}: {len(quotes)} quotes (status {page.status})")
+
+print(f"\nTotal: {len(all_quotes)} quotes\n")
+for i, quote in enumerate(all_quotes, 1):
+ print(f"{i:>3}. {quote}")
diff --git a/agent-skill/Scrapling-Skill/examples/03_stealthy_session.py b/agent-skill/Scrapling-Skill/examples/03_stealthy_session.py
new file mode 100644
index 0000000..c3c68eb
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/examples/03_stealthy_session.py
@@ -0,0 +1,26 @@
+"""
+Example 3: Python - StealthySession (Patchright stealth browser, visible)
+
+Scrapes all 10 pages of quotes.toscrape.com using a persistent stealth browser session.
+Bypasses anti-bot protections automatically (Cloudflare Turnstile, fingerprinting, etc.).
+
+Best for: well-protected sites, Cloudflare-gated pages, sites that detect Playwright.
+
+Set headless=True to run the browser hidden.
+Add solve_cloudflare=True to auto-solve Cloudflare challenges.
+"""
+
+from scrapling.fetchers import StealthySession
+
+all_quotes = []
+
+with StealthySession(headless=False) as session:
+ for i in range(1, 11):
+ page = session.fetch(f"https://quotes.toscrape.com/page/{i}/")
+ quotes = page.css(".quote .text::text").getall()
+ all_quotes.extend(quotes)
+ print(f"Page {i}: {len(quotes)} quotes (status {page.status})")
+
+print(f"\nTotal: {len(all_quotes)} quotes\n")
+for i, quote in enumerate(all_quotes, 1):
+ print(f"{i:>3}. {quote}")
diff --git a/agent-skill/Scrapling-Skill/examples/04_spider.py b/agent-skill/Scrapling-Skill/examples/04_spider.py
new file mode 100644
index 0000000..c464007
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/examples/04_spider.py
@@ -0,0 +1,58 @@
+"""
+Example 4: Python - Spider (auto-crawling framework)
+
+Scrapes ALL pages of quotes.toscrape.com by following "Next" pagination links
+automatically. No manual page looping needed.
+
+The spider yields structured items (text + author + tags) and exports them to JSON.
+
+Best for: multi-page crawls, full-site scraping, anything needing pagination or
+link following across many pages.
+
+Outputs:
+ - Live stats to terminal during crawl
+ - Final crawl stats at the end
+ - quotes.json in the current directory
+"""
+
+from scrapling.spiders import Spider, Response
+
+
+class QuotesSpider(Spider):
+ name = "quotes"
+ start_urls = ["https://quotes.toscrape.com/"]
+ concurrent_requests = 5 # Fetch up to 5 pages at once
+
+ async def parse(self, response: Response):
+ # Extract all quotes on the current page
+ for quote in response.css(".quote"):
+ yield {
+ "text": quote.css(".text::text").get(),
+ "author": quote.css(".author::text").get(),
+ "tags": quote.css(".tags .tag::text").getall(),
+ }
+
+ # Follow the "Next" button to the next page (if it exists)
+ next_page = response.css(".next a")
+ if next_page:
+ yield response.follow(next_page[0].attrib["href"])
+
+
+if __name__ == "__main__":
+ result = QuotesSpider().start()
+
+ print(f"\n{'=' * 50}")
+ print(f"Scraped : {result.stats.items_scraped} quotes")
+ print(f"Requests: {result.stats.requests_count}")
+ print(f"Time : {result.stats.elapsed_seconds:.2f}s")
+ print(f"Speed : {result.stats.requests_per_second:.2f} req/s")
+ print(f"{'=' * 50}\n")
+
+ for i, item in enumerate(result.items, 1):
+ print(f"{i:>3}. [{item['author']}] {item['text']}")
+ if item["tags"]:
+ print(f" Tags: {', '.join(item['tags'])}")
+
+ # Export to JSON
+ result.items.to_json("quotes.json", indent=True)
+ print("\nExported to quotes.json")
diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md
new file mode 100644
index 0000000..dfc65e4
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/examples/README.md
@@ -0,0 +1,45 @@
+# Scrapling Examples
+
+These examples scrape [quotes.toscrape.com](https://quotes.toscrape.com) — a safe, purpose-built scraping sandbox — and demonstrate every tool available in Scrapling, from plain HTTP to full browser automation and spiders.
+
+All examples collect **all 100 quotes across 10 pages**.
+
+## Quick Start
+
+Make sure Scrapling is installed:
+
+```bash
+pip install "scrapling[all]>=0.4.2"
+scrapling install --force
+```
+
+## Examples
+
+| File | Tool | Type | Best For |
+|--------------------------|-------------------|-----------------------------|---------------------------------------|
+| `01_fetcher_session.py` | `FetcherSession` | Python — persistent HTTP | APIs, fast multi-page scraping |
+| `02_dynamic_session.py` | `DynamicSession` | Python — browser automation | Dynamic/SPA pages |
+| `03_stealthy_session.py` | `StealthySession` | Python — stealth browser | Cloudflare, fingerprint bypass |
+| `04_spider.py` | `Spider` | Python — auto-crawling | Multi-page crawls, full-site scraping |
+
+## Running
+
+**Python scripts:**
+
+```bash
+python examples/01_fetcher_session.py
+python examples/02_dynamic_session.py # Opens a visible browser
+python examples/03_stealthy_session.py # Opens a visible stealth browser
+python examples/04_spider.py # Auto-crawls all pages, exports quotes.json
+```
+
+## Escalation Guide
+
+Start with the fastest, lightest option and escalate only if needed:
+
+```
+get / FetcherSession
+ └─ If JS required → fetch / DynamicSession
+ └─ If blocked → stealthy-fetch / StealthySession
+ └─ If multi-page → Spider
+```
diff --git a/agent-skill/Scrapling-Skill/references/fetching/choosing.md b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
new file mode 100644
index 0000000..974b566
--- /dev/null
+++ b/agent-skill/Scrapling-Skill/references/fetching/choosing.md
@@ -0,0 +1,77 @@
+# Fetchers basics
+
+## Introduction
+Fetchers are classes that do requests or fetch pages in a single-line fashion with many features and return a [Response](#response-object) object. All fetchers have separate session classes to keep the session running (e.g., a browser fetcher keeps the browser open until you finish all requests).
+
+Fetchers are not wrappers built on top of other libraries. They use these libraries as an engine to request/fetch pages but add features the underlying engines don't have, while still fully leveraging and optimizing them for web scraping.
+
+## Fetchers Overview
+
+Scrapling provides three different fetcher classes with their session classes; each fetcher is designed for a specific use case.
+
+The following table compares them and can be quickly used for guidance.
+
+
+| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
+|--------------------|---------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
+| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇🐇 |
+| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
+| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
+| JavaScript loading | ❌ | ✅ | ✅ |
+| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
+| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites Description 1
+Description 2
+Description 1
+Description 2
+This is product 1
+ $10.99 +This is product 2
+ $20.99 +This is product 3
+ $15.99 +This is product 1
\n $10.99\nThis is product 1
+ $10.99 +{"some_key": "some_value"}
' + ``` + The `json` method can be used directly: + ```python + >>> page.json() + {'some_key': 'some_value'} + ``` + For JSON responses, the [Selector](#selector) class keeps a raw copy of the content it receives. When `.json()` is called, it checks for that raw copy first and converts it to JSON. If the raw copy is unavailable (as with sub-elements), it checks the current element's text content, then falls back to `get_all_text`. + +- The `.clean()` method removes all whitespace and consecutive spaces, returning a new `TextHandler` instance: +```python +>>> TextHandler('\n wonderful idea, \reh?').clean() +'wonderful idea, eh?' +``` +The `remove_entities` argument causes `clean` to replace HTML entities with their corresponding characters. + +- The `.sort()` method sorts the string characters: +```python +>>> TextHandler('acb').sort() +'abc' +``` +Or do it in reverse: +```python +>>> TextHandler('acb').sort(reverse=True) +'cba' +``` + +This class is returned in place of strings nearly everywhere in the library. + +## TextHandlers +This class inherits from standard lists, adding `re` and `re_first` as new methods. + +The `re_first` method runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. + +## AttributesHandler +This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element/[Selector](#selector) instance. +```python +>>> print(page.find('script').attrib) +{'id': 'page-data', 'type': 'application/json'} +>>> type(page.find('script').attrib).__name__ +'AttributesHandler' +``` +Because it's read-only, it will use fewer resources than the standard dictionary. Still, it has the same dictionary method and properties, except those that allow you to modify/override the data. + +It currently adds two extra simple methods: + +- The `search_values` method + + Searches the current attributes by values (rather than keys) and returns a dictionary of each matching item. + + A simple example would be + ```python + >>> for i in page.find('script').attrib.search_values('page-data'): + print(i) + {'id': 'page-data'} + ``` + But this method provides the `partial` argument as well, which allows you to search by part of the value: + ```python + >>> for i in page.find('script').attrib.search_values('page', partial=True): + print(i) + {'id': 'page-data'} + ``` + A more practical example is using it with `find_all` to find all elements that have a specific value in their attributes: + ```python + >>> page.find_all(lambda element: list(element.attrib.search_values('product'))) + [£51.77
' parent='£51.77
' parent='£53.74
' parent='£50.10
' parent='£47.82
' parent=',
+ ,
+ ,
+...]
+```
+Find all div elements with a class that equals `quote`.
+```python
+>>> page.find_all('div', class_='quote')
+[ ]
+```
+Find all elements that have children.
+```python
+>>> page.find_all(lambda element: len(element.children) > 0)
+[