diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md index 412243b..a4d4bb1 100644 --- a/docs/spiders/architecture.md +++ b/docs/spiders/architecture.md @@ -9,7 +9,7 @@ Scrapling's spider system is a Scrapy-inspired async crawling framework designed If you're familiar with Scrapy, you'll feel right at home. If not, don't worry — the system is designed to be straightforward. -## Architecture Overview +## Data Flow The diagram below shows how data flows through the spider system when a crawl is running: @@ -47,47 +47,15 @@ graph TB class Checkpoint checkpointClass ``` -## Data Flow +Here's what happens step by step when you run a spider without many details: -Here's what happens step by step when you run a spider: - -### 1. Spider generates initial Requests - -The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic (e.g., POST requests, authentication, different callbacks). - -### 2. Requests are scheduled - -The **Scheduler** receives requests and places them in a priority queue. Before queuing, each request gets a fingerprint (SHA1 hash of URL + method + body + session ID). Duplicate requests are silently dropped unless `dont_filter=True` is set on the request. Higher-priority requests are dequeued first. - -### 3. Engine fetches via Session Manager - -The **Crawler Engine** dequeues the next request, respecting concurrency limits (global and per-domain) and download delays. It then hands the request to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID). - -A spider can have multiple sessions running simultaneously — for example, a fast `FetcherSession` for simple pages and a `AsyncStealthySession` for protected pages. The Session Manager handles starting, routing, and closing all of them. - -### 4. Response comes back - -The session fetches the page and returns a `Response` object. The engine records statistics (response bytes, status codes, per-session request counts) and checks for blocked responses. - -If the response is blocked (determined by `is_blocked()`), the engine retries the request up to `max_blocked_retries` times, calling `retry_blocked_request()` so you can modify the request before retrying (e.g., switch proxies, add headers). - -### 5. Spider processes the Response - -The engine passes the `Response` to the spider's callback (default: `parse()`). The callback is an async generator that yields results. - -### 6. Results are routed - -The callback can yield three types of values: - -- **`dict`** — Treated as a scraped item. Passed through `on_scraped_item()` for processing/filtering (return `None` to drop it), then added to the `ItemList` or streamed to the consumer. -- **`Request`** — A follow-up request. Checked against `allowed_domains`, normalized, and sent to the Scheduler for queuing. -- **`None`** — Silently ignored. - -The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused. - -### Checkpointing - -If `crawldir` is set, the engine periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state. +1. The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic. +2. The **Scheduler** receives requests and places them in a priority queue, and creates fingerprints for them. Higher-priority requests are dequeued first. +3. The **Crawler Engine** asks the **Scheduler** to dequeue the next request, respecting concurrency limits (global and per-domain) and download delays. Once the **Crawler Engine** receives the request, it passes it to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID). +4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized. +5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing. +6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused. +7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state. ## Components @@ -113,7 +81,7 @@ class MySpider(Spider): ### Crawler Engine -The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits using `anyio` task groups and `CapacityLimiter`, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you. +The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you. ### Scheduler @@ -123,40 +91,40 @@ A priority queue with built-in URL deduplication. Requests are fingerprinted bas Manages one or more named session instances. Each session is one of: -- **`FetcherSession`** — Async HTTP requests via `curl_cffi` (fastest, no JavaScript) -- **`AsyncDynamicSession`** — Playwright browser automation (JavaScript support) -- **`AsyncStealthySession`** — Stealthy browser with anti-bot bypass (Cloudflare, etc.) +- [FetcherSession](../fetching/static.md) +- [AsyncDynamicSession](../fetching/dynamic.md) +- [AsyncStealthySession](../fetching/stealthy.md) -When a request comes in, the Session Manager routes it to the correct session based on the request's `sid` field. Sessions can be started eagerly (default) or lazily (started on first use). +When a request comes in, the Session Manager routes it to the correct session based on the request's `sid` field. Sessions can be started with the spider start (default) or lazily (started on the first use). ### Checkpoint System -Saves the crawler's state (pending requests + seen URL fingerprints) to a pickle file on disk. Writes are atomic (temp file + rename) to prevent corruption. Checkpoints are saved periodically at a configurable interval and on graceful shutdown. On successful completion (not paused), checkpoint files are cleaned up automatically. +An optional system that, if enabled, saves the crawler's state (pending requests + seen URL fingerprints) to a pickle file on disk. Writes are atomic (temp file + rename) to prevent corruption. Checkpoints are saved periodically at a configurable interval and on graceful shutdown. Upon successful completion (not paused), checkpoint files are automatically cleaned up. ### Output -Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass with per-domain bytes, per-session request counts, status code distribution, and timing information. +Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info. ## Comparison with Scrapy If you're coming from Scrapy, here's how Scrapling's spider system maps: -| Concept | Scrapy | Scrapling | -|---------|--------|-----------| -| Spider definition | `scrapy.Spider` subclass | `scrapling.spiders.Spider` subclass | -| Initial requests | `start_requests()` | `start_requests()` | -| Callbacks | `def parse(self, response)` | `async def parse(self, response)` | -| Following links | `response.follow(url)` | `response.follow(url)` | -| Item output | `yield dict` or `yield Item` | `yield dict` | -| Request scheduling | Scheduler + Dupefilter | Scheduler with built-in deduplication | -| Downloading | Downloader + Middlewares | Session Manager with multi-session support | -| Item processing | Item Pipelines | `on_scraped_item()` hook | -| Blocked detection | Custom middlewares | Built-in `is_blocked()` + `retry_blocked_request()` | -| Concurrency | `CONCURRENT_REQUESTS` setting | `concurrent_requests` class attribute | -| Domain filtering | `allowed_domains` | `allowed_domains` | -| Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument | -| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` | -| Running | `scrapy crawl spider_name` | `MySpider().start()` | -| Streaming | N/A | `async for item in spider.stream()` | -| Multi-session | N/A | Multiple fetcher sessions per spider | +| Concept | Scrapy | Scrapling | +|--------------------|-------------------------------|-----------------------------------------------------------------| +| Spider definition | `scrapy.Spider` subclass | `scrapling.spiders.Spider` subclass | +| Initial requests | `start_requests()` | `async start_requests()` | +| Callbacks | `def parse(self, response)` | `async def parse(self, response)` | +| Following links | `response.follow(url)` | `response.follow(url)` | +| Item output | `yield dict` or `yield Item` | `yield dict` | +| Request scheduling | Scheduler + Dupefilter | Scheduler with built-in deduplication | +| Downloading | Downloader + Middlewares | Session Manager with multi-session support | +| Item processing | Item Pipelines | `on_scraped_item()` hook | +| Blocked detection | Through custom middlewares | Built-in `is_blocked()` + `retry_blocked_request()` hooks | +| Concurrency | `CONCURRENT_REQUESTS` setting | `concurrent_requests` class attribute | +| Domain filtering | `allowed_domains` | `allowed_domains` | +| Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument | +| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks | +| Running | `scrapy crawl spider_name` | `MySpider().start()` | +| Streaming | N/A | `async for item in spider.stream()` | +| Multi-session | N/A | Multiple sessions with different types per spider | \ No newline at end of file