diff --git a/docs/spiders/requests-responses.md b/docs/spiders/requests-responses.md new file mode 100644 index 0000000..6ec78e5 --- /dev/null +++ b/docs/spiders/requests-responses.md @@ -0,0 +1,202 @@ +## Introduction + +!!! success "Prerequisites" + + 1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider. + +This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following. + +## The Request Object + +A `Request` represents a URL to be fetched. You create requests either directly or via `response.follow()`: + +```python +from scrapling.spiders import Request + +# Direct construction +request = Request( + "https://example.com/page", + callback=self.parse_page, + priority=5, +) + +# Via response.follow (preferred in callbacks) +request = response.follow("/page", callback=self.parse_page) +``` + +Here are all the arguments you can pass to `Request`: + +| Argument | Type | Default | Description | +|---------------|------------|------------|-------------------------------------------------------------------------------------------------------| +| `url` | `str` | *required* | The URL to fetch | +| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) | +| `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` | +| `priority` | `int` | `0` | Higher values are processed first | +| `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) | +| `meta` | `dict` | `{}` | Arbitrary metadata passed through to the response | +| `**kwargs` | | | Additional keyword arguments passed to the session's fetch method (e.g., `headers`, `method`, `data`) | + +Any extra keyword arguments are forwarded directly to the underlying session. For example, to make a POST request: + +```python +yield Request( + "https://example.com/api", + method="POST", + data={"key": "value"}, + callback=self.parse_result, +) +``` + +## Response.follow() + +`response.follow()` is the recommended way to create follow-up requests inside callbacks. It offers several advantages over constructing `Request` objects directly: + +- **Relative URLs** are resolved automatically against the current page URL +- **Referer header** is set to the current page URL by default +- **Session kwargs** from the original request are inherited (headers, proxy settings, etc.) +- **Callback, session ID, and priority** are inherited from the original request if not specified + +```python +async def parse(self, response: Response): + # Minimal — inherits callback, sid, priority from current request + yield response.follow("/next-page") + + # Override specific fields + yield response.follow( + "/product/123", + callback=self.parse_product, + priority=10, + ) + + # Pass additional metadata to + yield response.follow( + "/details", + callback=self.parse_details, + meta={"category": "electronics"}, + ) +``` + +| Argument | Type | Default | Description | +|--------------------|------------|------------|------------------------------------------------------------| +| `url` | `str` | *required* | URL to follow (absolute or relative) | +| `sid` | `str` | `""` | Session ID (inherits from original request if empty) | +| `callback` | `callable` | `None` | Callback method (inherits from original request if `None`) | +| `priority` | `int` | `None` | Priority (inherits from original request if `None`) | +| `dont_filter` | `bool` | `False` | Skip deduplication | +| `meta` | `dict` | `None` | Metadata (merged with existing response meta) | +| **`referer_flow`** | `bool` | `True` | Set current URL as Referer header | +| `**kwargs` | | | Merged with original request's session kwargs | + +### Disabling Referer Flow + +By default, `response.follow()` sets the `Referer` header to the current page URL. To disable this: + +```python +yield response.follow("/page", referer_flow=False) +``` + +## Callbacks + +Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types: + +- **`dict`** — A scraped item, added to the results +- **`Request`** — A follow-up request, added to the queue +- **`None`** — Silently ignored + +```python +class MySpider(Spider): + name = "my_spider" + start_urls = ["https://example.com"] + + async def parse(self, response: Response): + # Yield items (dicts) + yield {"url": response.url, "title": response.css("title::text").get("")} + + # Yield follow-up requests + for link in response.css("a::attr(href)").getall(): + yield response.follow(link, callback=self.parse_page) + + async def parse_page(self, response: Response): + yield {"content": response.css("article::text").get("")} +``` + +!!! tip "Note:" + + All callback methods must be `async def` and use `yield` (not `return`). Even if a callback only yields items with no follow-up requests, it must still be an async generator. + +## Request Priority + +Requests with higher priority values are processed first. This is useful when some pages are more important to be processed first before others: + +```python +async def parse(self, response: Response): + # High priority — process product pages first + for link in response.css("a.product::attr(href)").getall(): + yield response.follow(link, callback=self.parse_product, priority=10) + + # Low priority — pagination links processed after products + next_page = response.css("a.next::attr(href)").get() + if next_page: + yield response.follow(next_page, callback=self.parse, priority=0) +``` + +When using `response.follow()`, the priority is inherited from the original request unless you specify a new one. + +## Deduplication + +The spider automatically deduplicates requests based on a fingerprint computed from the URL, HTTP method, request body, and session ID. If two requests produce the same fingerprint, the second one is silently dropped. + +To allow duplicate requests (e.g., re-visiting a page after login), set `dont_filter=True`: + +```python +yield Request("https://example.com/dashboard", dont_filter=True, callback=self.parse_dashboard) + +# Or with response.follow +yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashboard) +``` + +You can fine-tune what goes into the fingerprint using class attributes on your spider: + +| Attribute | Default | Effect | +|----------------------|---------|----------------------------------------------------------------------------------------------------------------| +| `fp_include_kwargs` | `False` | Include extra request kwargs (arguments you passed to the session fetch, like headers, etc.) in the fingerprint | +| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints | +| `fp_include_headers` | `False` | Include request headers in the fingerprint | + +For example, if you need to treat `https://example.com/page#section1` and `https://example.com/page#section2` as different URLs: + +```python +class MySpider(Spider): + name = "my_spider" + fp_keep_fragments = True + # ... +``` + +## Request Meta + +The `meta` dictionary lets you pass arbitrary data between callbacks. This is useful when you need context from one page to process another: + +```python +async def parse(self, response: Response): + for product in response.css("div.product"): + category = product.css("span.category::text").get("") + link = product.css("a::attr(href)").get() + if link: + yield response.follow( + link, + callback=self.parse_product, + meta={"category": category}, + ) + +async def parse_product(self, response: Response): + yield { + "name": response.css("h1::text").get(""), + "price": response.css(".price::text").get(""), + # Access meta from the request + "category": response.meta.get("category", ""), + } +``` + +When using `response.follow()`, the meta from the current response is merged with the new meta you provide (new values take precedence). + +The spider system also automatically stores some metadata. For example, the proxy used for a request is available as `response.meta["proxy"]` when proxy rotation is enabled. \ No newline at end of file diff --git a/zensical.toml b/zensical.toml index a0dc9dc..b19a6ab 100644 --- a/zensical.toml +++ b/zensical.toml @@ -33,6 +33,7 @@ nav = [ {Spiders = [ {"Architecture" = "spiders/architecture.md"}, {"Getting started" = "spiders/getting-started.md"}, + {"Requests & Responses" = "spiders/requests-responses.md"}, ]}, {"Command Line Interface" = [ {Overview = "cli/overview.md"},