diff --git a/agent-skill/Scrapling-Skill.zip b/agent-skill/Scrapling-Skill.zip index c89b5d7..b03d956 100644 Binary files a/agent-skill/Scrapling-Skill.zip and b/agent-skill/Scrapling-Skill.zip differ diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md index 60168a1..6b3cbf7 100644 --- a/agent-skill/Scrapling-Skill/SKILL.md +++ b/agent-skill/Scrapling-Skill/SKILL.md @@ -258,6 +258,7 @@ class QuotesSpider(Spider): name = "quotes" start_urls = ["https://quotes.toscrape.com/"] concurrent_requests = 10 + robots_txt_obey = True # Respect robots.txt rules async def parse(self, response: Response): for quote in response.css('.quote'): @@ -379,7 +380,7 @@ This skill encapsulates almost all the published documentation in Markdown, so d ## Guardrails (Always) - Only scrape content you're authorized to access. -- Respect robots.txt and ToS. -- Add delays (download_delay) for large crawls. +- Respect robots.txt and ToS. Use `robots_txt_obey = True` on spiders to enforce this automatically. +- Add delays (`download_delay`) for large crawls. - Don't bypass paywalls or authentication without permission. - Never scrape personal/sensitive data. \ No newline at end of file diff --git a/agent-skill/Scrapling-Skill/references/spiders/advanced.md b/agent-skill/Scrapling-Skill/references/spiders/advanced.md index 955706b..1244c9e 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/advanced.md +++ b/agent-skill/Scrapling-Skill/references/spiders/advanced.md @@ -9,6 +9,7 @@ The spider system uses three class attributes to control how aggressively it cra | `concurrent_requests` | `4` | Maximum number of requests being processed at the same time | | `concurrent_requests_per_domain` | `0` | Maximum concurrent requests per domain (0 = no per-domain limit) | | `download_delay` | `0.0` | Seconds to wait before each request | +| `robots_txt_obey` | `False` | Respect robots.txt rules (Disallow, Crawl-delay, Request-rate) | ```python class PoliteSpider(Spider): @@ -218,6 +219,7 @@ print(f"Requests: {stats.requests_count}") print(f"Failed: {stats.failed_requests_count}") print(f"Blocked: {stats.blocked_requests_count}") print(f"Offsite filtered: {stats.offsite_requests_count}") +print(f"Robots.txt disallowed: {stats.robots_disallowed_count}") print(f"Items scraped: {stats.items_scraped}") print(f"Items dropped: {stats.items_dropped}") print(f"Response bytes: {stats.response_bytes}") diff --git a/agent-skill/Scrapling-Skill/references/spiders/architecture.md b/agent-skill/Scrapling-Skill/references/spiders/architecture.md index 93b059b..9976f31 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/architecture.md +++ b/agent-skill/Scrapling-Skill/references/spiders/architecture.md @@ -10,7 +10,7 @@ Here's what happens step by step when you run a spider: 1. The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic. 2. The **Scheduler** receives requests and places them in a priority queue, and creates fingerprints for them. Higher-priority requests are dequeued first. -3. The **Crawler Engine** asks the **Scheduler** to dequeue the next request, respecting concurrency limits (global and per-domain) and download delays. Once the **Crawler Engine** receives the request, it passes it to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID). +3. The **Crawler Engine** asks the **Scheduler** to dequeue the next request, respecting concurrency limits (global and per-domain) and download delays. If `robots_txt_obey` is enabled, the engine checks the domain's robots.txt rules before proceeding -- disallowed requests are dropped silently. Once the **Crawler Engine** receives the request, it passes it to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID). 4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized. 5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing. 6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused. @@ -82,6 +82,7 @@ If you're coming from Scrapy, here's how Scrapling's spider system maps: | Blocked detection | Through custom middlewares | Built-in `is_blocked()` + `retry_blocked_request()` hooks | | Concurrency | `CONCURRENT_REQUESTS` setting | `concurrent_requests` class attribute | | Domain filtering | `allowed_domains` | `allowed_domains` | +| Robots.txt | `ROBOTSTXT_OBEY` setting | `robots_txt_obey` class attribute | | Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument | | Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks | | Running | `scrapy crawl spider_name` | `MySpider().start()` | diff --git a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md index 7420079..665fd7f 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md +++ b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md @@ -137,3 +137,28 @@ Subdomains are matched automatically, so setting `allowed_domains = {"example.co When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped. +## Robots.txt Compliance + +Set `robots_txt_obey = True` to make the spider respect robots.txt rules before crawling any domain: + +```python +class PoliteSpider(Spider): + name = "polite" + start_urls = ["https://example.com"] + robots_txt_obey = True + + async def parse(self, response: Response): + for link in response.css("a::attr(href)").getall(): + yield response.follow(link, callback=self.parse) +``` + +When enabled, the spider will: + +1. **Pre-fetch robots.txt** for all domains in `start_urls` before the crawl begins (concurrently). +2. **Check every request** against the domain's robots.txt `Disallow` rules. Disallowed requests are silently dropped and counted in `stats.robots_disallowed_count`. +3. **Respect `Crawl-delay` and `Request-rate` directives** by taking the maximum of the directive and your configured `download_delay`. This means robots.txt delays never reduce your configured delay, only increase it when needed. + +Robots.txt files are fetched using the spider's default session and cached per domain for the entire crawl. Domains discovered mid-crawl (not in `start_urls`) have their robots.txt fetched on the first request to that domain. + +**Note:** `robots_txt_obey` is turned off by default. It does not affect your concurrency settings -- only the delay between requests is adjusted. +