diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.md new file mode 100644 index 0000000..1a363b9 --- /dev/null +++ b/docs/spiders/advanced.md @@ -0,0 +1,313 @@ +# Advanced usages + +## Introduction + +!!! success "Prerequisites" + + 1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider. + +This page covers the spider system's advanced features: concurrency control, pause/resume, streaming, lifecycle hooks, statistics, and logging. + +## Concurrency Control + +The spider system uses three class attributes to control how aggressively it crawls: + +| Attribute | Default | Description | +|----------------------------------|---------|------------------------------------------------------------------| +| `concurrent_requests` | `4` | Maximum number of requests being processed at the same time | +| `concurrent_requests_per_domain` | `0` | Maximum concurrent requests per domain (0 = no per-domain limit) | +| `download_delay` | `0.0` | Seconds to wait before each request | + +```python +class PoliteSpider(Spider): + name = "polite" + start_urls = ["https://example.com"] + + # Be gentle with the server + concurrent_requests = 4 + concurrent_requests_per_domain = 2 + download_delay = 1.0 # Wait 1 second between requests + + async def parse(self, response: Response): + yield {"title": response.css("title::text").get("")} +``` + +When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain. + +!!! tip + + The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting. + +### Using uvloop + +The `start()` method accepts a `use_uvloop` parameter to use the faster [uvloop](https://github.com/MagicStack/uvloop)/[winloop](https://github.com/nicktimko/winloop) event loop implementation, if available: + +```python +result = MySpider().start(use_uvloop=True) +``` + +This can improve throughput for I/O-heavy crawls. You'll need to install `uvloop` (Linux/macOS) or `winloop` (Windows) separately. + +## Pause & Resume + +The spider supports graceful pause-and-resume via checkpointing. To enable it, pass a `crawldir` directory to the spider constructor: + +```python +spider = MySpider(crawldir="crawl_data/my_spider") +result = spider.start() + +if result.paused: + print("Crawl was paused. Run again to resume.") +else: + print("Crawl completed!") +``` + +### How It Works + +1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits. +2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks. +3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`. +4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically. + +**Checkpoints are also saved periodically during the crawl (every 5 minutes by default).** + +You can change the interval as follows: + +```python +# Save checkpoint every 2 minutes +spider = MySpider(crawldir="crawl_data/my_spider", interval=120.0) +``` + +The writing to the disk is atomic, so it's totally safe. + +!!! tip + + Pressing `Ctrl+C` during a crawl always causes the spider to close gracefully, even if the checkpoint system is not enabled. Doing it again without waiting forces the spider to close immediately. + +### Knowing If You're Resuming + +The `on_start()` hook receives a `resuming` flag: + +```python +async def on_start(self, resuming: bool = False): + if resuming: + self.logger.info("Resuming from checkpoint!") + else: + self.logger.info("Starting fresh crawl") +``` + +## Streaming + +For long-running spiders or applications that need real-time access to scraped items, use the `stream()` method instead of `start()`: + +```python +import anyio + +async def main(): + spider = MySpider() + async for item in spider.stream(): + print(f"Got item: {item}") + # Access real-time stats + print(f"Items so far: {spider.stats.items_scraped}") + print(f"Requests made: {spider.stats.requests_count}") + +anyio.run(main) +``` + +Key differences from `start()`: + +- `stream()` must be called from an async context +- Items are yielded one by one as they're scraped, not collected into a list +- You can access `spider.stats` during iteration for real-time statistics + +!!! abstract + + The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results--statistics) + +You can use it with the checkpoint system too, so it's easy to build UI on top of spiders. UIs that have real-time data and can be paused/resumed. + +```python +import anyio + +async def main(): + spider = MySpider(crawldir="crawl_data/my_spider") + async for item in spider.stream(): + print(f"Got item: {item}") + # Access real-time stats + print(f"Items so far: {spider.stats.items_scraped}") + print(f"Requests made: {spider.stats.requests_count}") + +anyio.run(main) +``` +You can also use `spider.pause()` to shut down the spider in the code above. If you used it without enabling the checkpoint system, it will just close the crawl. + +## Lifecycle Hooks + +The spider provides several hooks you can override to add custom behavior at different stages of the crawl: + +### on_start + +Called before crawling begins. Use it for setup tasks like loading data or initializing resources: + +```python +async def on_start(self, resuming: bool = False): + self.logger.info("Spider starting up") + # Load seed URLs from a database, initialize counters, etc. +``` + +### on_close + +Called after crawling finishes (whether completed or paused). Use it for cleanup: + +```python +async def on_close(self): + self.logger.info("Spider shutting down") + # Close database connections, flush buffers, etc. +``` + +### on_error + +Called when a request fails with an exception. Use it for error tracking or custom recovery logic: + +```python +async def on_error(self, request: Request, error: Exception): + self.logger.error(f"Failed: {request.url} - {error}") + # Log to error tracker, save failed URL for later, etc. +``` + +### on_scraped_item + +Called for every scraped item before it's added to the results. Return the item (modified or not) to keep it, or return `None` to drop it: + +```python +async def on_scraped_item(self, item: dict) -> dict | None: + # Drop items without a title + if not item.get("title"): + return None + + # Modify items (e.g., add timestamps) + item["scraped_at"] = "2026-01-01" + return item +``` + +!!! tip + + This hook can also be used to direct items through your own pipelines and drop them from the spider. + +### start_requests + +Override `start_requests()` for custom initial request generation instead of using `start_urls`: + +```python +async def start_requests(self): + # POST request to log in first + yield Request( + "https://example.com/login", + method="POST", + data={"user": "admin", "pass": "secret"}, + callback=self.after_login, + ) + +async def after_login(self, response: Response): + # Now crawl the authenticated pages + yield response.follow("/dashboard", callback=self.parse) +``` + +## Results & Statistics + +The `CrawlResult` returned by `start()` contains both the scraped items and detailed statistics: + +```python +result = MySpider().start() + +# Items +print(f"Total items: {len(result.items)}") +result.items.to_json("output.json", indent=True) + +# Did the crawl complete? +print(f"Completed: {result.completed}") +print(f"Paused: {result.paused}") + +# Statistics +stats = result.stats +print(f"Requests: {stats.requests_count}") +print(f"Failed: {stats.failed_requests_count}") +print(f"Blocked: {stats.blocked_requests_count}") +print(f"Offsite filtered: {stats.offsite_requests_count}") +print(f"Items scraped: {stats.items_scraped}") +print(f"Items dropped: {stats.items_dropped}") +print(f"Response bytes: {stats.response_bytes}") +print(f"Duration: {stats.elapsed_seconds:.1f}s") +print(f"Speed: {stats.requests_per_second:.1f} req/s") +``` + +### Detailed Stats + +The `CrawlStats` object tracks granular information: + +```python +stats = result.stats + +# Status code distribution +print(stats.response_status_count) +# {'status_200': 150, 'status_404': 3, 'status_403': 1} + +# Bytes downloaded per domain +print(stats.domains_response_bytes) +# {'example.com': 1234567, 'api.example.com': 45678} + +# Requests per session +print(stats.sessions_requests_count) +# {'http': 120, 'stealth': 34} + +# Proxies used during the crawl +print(stats.proxies) +# ['http://proxy1:8080', 'http://proxy2:8080'] + +# Log level counts +print(stats.log_levels_counter) +# {'debug': 200, 'info': 50, 'warning': 3, 'error': 1, 'critical': 0} + +# Timing information +print(stats.start_time) # Unix timestamp when crawl started +print(stats.end_time) # Unix timestamp when crawl finished +print(stats.download_delay) # The download delay used (seconds) + +# Concurrency settings used +print(stats.concurrent_requests) # Global concurrency limit +print(stats.concurrent_requests_per_domain) # Per-domain concurrency limit + +# Custom stats (set by your spider code) +print(stats.custom_stats) +# {'login_attempts': 3, 'pages_with_errors': 5} + +# Export everything as a dict +print(stats.to_dict()) +``` + +## Logging + +The spider has a built-in logger accessible via `self.logger`. It's pre-configured with the spider's name and supports several customization options: + +| Attribute | Default | Description | +|-----------------------|--------------------------------------------------------------|----------------------------------------------------| +| `logging_level` | `logging.DEBUG` | Minimum log level | +| `logging_format` | `"[%(asctime)s]:({spider_name}) %(levelname)s: %(message)s"` | Log message format | +| `logging_date_format` | `"%Y-%m-%d %H:%M:%S"` | Date format in log messages | +| `log_file` | `None` | Path to a log file (in addition to console output) | + +```python +import logging + +class MySpider(Spider): + name = "my_spider" + start_urls = ["https://example.com"] + logging_level = logging.INFO + log_file = "logs/my_spider.log" + + async def parse(self, response: Response): + self.logger.info(f"Processing {response.url}") + yield {"title": response.css("title::text").get("")} +``` + +The log file directory is created automatically if it doesn't exist. Both console and file output use the same format. \ No newline at end of file diff --git a/docs/spiders/requests-responses.md b/docs/spiders/requests-responses.md index b61e727..c587af8 100644 --- a/docs/spiders/requests-responses.md +++ b/docs/spiders/requests-responses.md @@ -157,11 +157,11 @@ yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashbo You can fine-tune what goes into the fingerprint using class attributes on your spider: -| Attribute | Default | Effect | -|----------------------|---------|----------------------------------------------------------------------------------------------------------------| +| Attribute | Default | Effect | +|----------------------|---------|-----------------------------------------------------------------------------------------------------------------| | `fp_include_kwargs` | `False` | Include extra request kwargs (arguments you passed to the session fetch, like headers, etc.) in the fingerprint | -| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints | -| `fp_include_headers` | `False` | Include request headers in the fingerprint | +| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints | +| `fp_include_headers` | `False` | Include request headers in the fingerprint | For example, if you need to treat `https://example.com/page#section1` and `https://example.com/page#section2` as different URLs: diff --git a/zensical.toml b/zensical.toml index 084c9d4..fc94fa5 100644 --- a/zensical.toml +++ b/zensical.toml @@ -36,6 +36,7 @@ nav = [ {"Requests & Responses" = "spiders/requests-responses.md"}, {"Sessions" = "spiders/sessions.md"}, {"Proxy management & Blocking" = "spiders/proxy-blocking.md"}, + {"Advanced features" = "spiders/advanced.md"} ]}, {"Command Line Interface" = [ {Overview = "cli/overview.md"},