docs(spiders): add a new page for requests & responses
This commit is contained in:
Karim shoair
@@ -0,0 +1,202 @@
|
||||
## Introduction
|
||||
|
||||
!!! success "Prerequisites"
|
||||
|
||||
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
|
||||
|
||||
This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
|
||||
|
||||
## The Request Object
|
||||
|
||||
A `Request` represents a URL to be fetched. You create requests either directly or via `response.follow()`:
|
||||
|
||||
```python
|
||||
from scrapling.spiders import Request
|
||||
|
||||
# Direct construction
|
||||
request = Request(
|
||||
"https://example.com/page",
|
||||
callback=self.parse_page,
|
||||
priority=5,
|
||||
)
|
||||
|
||||
# Via response.follow (preferred in callbacks)
|
||||
request = response.follow("/page", callback=self.parse_page)
|
||||
```
|
||||
|
||||
Here are all the arguments you can pass to `Request`:
|
||||
|
||||
| Argument | Type | Default | Description |
|
||||
|---------------|------------|------------|-------------------------------------------------------------------------------------------------------|
|
||||
| `url` | `str` | *required* | The URL to fetch |
|
||||
| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) |
|
||||
| `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` |
|
||||
| `priority` | `int` | `0` | Higher values are processed first |
|
||||
| `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) |
|
||||
| `meta` | `dict` | `{}` | Arbitrary metadata passed through to the response |
|
||||
| `**kwargs` | | | Additional keyword arguments passed to the session's fetch method (e.g., `headers`, `method`, `data`) |
|
||||
|
||||
Any extra keyword arguments are forwarded directly to the underlying session. For example, to make a POST request:
|
||||
|
||||
```python
|
||||
yield Request(
|
||||
"https://example.com/api",
|
||||
method="POST",
|
||||
data={"key": "value"},
|
||||
callback=self.parse_result,
|
||||
)
|
||||
```
|
||||
|
||||
## Response.follow()
|
||||
|
||||
`response.follow()` is the recommended way to create follow-up requests inside callbacks. It offers several advantages over constructing `Request` objects directly:
|
||||
|
||||
- **Relative URLs** are resolved automatically against the current page URL
|
||||
- **Referer header** is set to the current page URL by default
|
||||
- **Session kwargs** from the original request are inherited (headers, proxy settings, etc.)
|
||||
- **Callback, session ID, and priority** are inherited from the original request if not specified
|
||||
|
||||
```python
|
||||
async def parse(self, response: Response):
|
||||
# Minimal — inherits callback, sid, priority from current request
|
||||
yield response.follow("/next-page")
|
||||
|
||||
# Override specific fields
|
||||
yield response.follow(
|
||||
"/product/123",
|
||||
callback=self.parse_product,
|
||||
priority=10,
|
||||
)
|
||||
|
||||
# Pass additional metadata to
|
||||
yield response.follow(
|
||||
"/details",
|
||||
callback=self.parse_details,
|
||||
meta={"category": "electronics"},
|
||||
)
|
||||
```
|
||||
|
||||
| Argument | Type | Default | Description |
|
||||
|--------------------|------------|------------|------------------------------------------------------------|
|
||||
| `url` | `str` | *required* | URL to follow (absolute or relative) |
|
||||
| `sid` | `str` | `""` | Session ID (inherits from original request if empty) |
|
||||
| `callback` | `callable` | `None` | Callback method (inherits from original request if `None`) |
|
||||
| `priority` | `int` | `None` | Priority (inherits from original request if `None`) |
|
||||
| `dont_filter` | `bool` | `False` | Skip deduplication |
|
||||
| `meta` | `dict` | `None` | Metadata (merged with existing response meta) |
|
||||
| **`referer_flow`** | `bool` | `True` | Set current URL as Referer header |
|
||||
| `**kwargs` | | | Merged with original request's session kwargs |
|
||||
|
||||
### Disabling Referer Flow
|
||||
|
||||
By default, `response.follow()` sets the `Referer` header to the current page URL. To disable this:
|
||||
|
||||
```python
|
||||
yield response.follow("/page", referer_flow=False)
|
||||
```
|
||||
|
||||
## Callbacks
|
||||
|
||||
Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:
|
||||
|
||||
- **`dict`** — A scraped item, added to the results
|
||||
- **`Request`** — A follow-up request, added to the queue
|
||||
- **`None`** — Silently ignored
|
||||
|
||||
```python
|
||||
class MySpider(Spider):
|
||||
name = "my_spider"
|
||||
start_urls = ["https://example.com"]
|
||||
|
||||
async def parse(self, response: Response):
|
||||
# Yield items (dicts)
|
||||
yield {"url": response.url, "title": response.css("title::text").get("")}
|
||||
|
||||
# Yield follow-up requests
|
||||
for link in response.css("a::attr(href)").getall():
|
||||
yield response.follow(link, callback=self.parse_page)
|
||||
|
||||
async def parse_page(self, response: Response):
|
||||
yield {"content": response.css("article::text").get("")}
|
||||
```
|
||||
|
||||
!!! tip "Note:"
|
||||
|
||||
All callback methods must be `async def` and use `yield` (not `return`). Even if a callback only yields items with no follow-up requests, it must still be an async generator.
|
||||
|
||||
## Request Priority
|
||||
|
||||
Requests with higher priority values are processed first. This is useful when some pages are more important to be processed first before others:
|
||||
|
||||
```python
|
||||
async def parse(self, response: Response):
|
||||
# High priority — process product pages first
|
||||
for link in response.css("a.product::attr(href)").getall():
|
||||
yield response.follow(link, callback=self.parse_product, priority=10)
|
||||
|
||||
# Low priority — pagination links processed after products
|
||||
next_page = response.css("a.next::attr(href)").get()
|
||||
if next_page:
|
||||
yield response.follow(next_page, callback=self.parse, priority=0)
|
||||
```
|
||||
|
||||
When using `response.follow()`, the priority is inherited from the original request unless you specify a new one.
|
||||
|
||||
## Deduplication
|
||||
|
||||
The spider automatically deduplicates requests based on a fingerprint computed from the URL, HTTP method, request body, and session ID. If two requests produce the same fingerprint, the second one is silently dropped.
|
||||
|
||||
To allow duplicate requests (e.g., re-visiting a page after login), set `dont_filter=True`:
|
||||
|
||||
```python
|
||||
yield Request("https://example.com/dashboard", dont_filter=True, callback=self.parse_dashboard)
|
||||
|
||||
# Or with response.follow
|
||||
yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashboard)
|
||||
```
|
||||
|
||||
You can fine-tune what goes into the fingerprint using class attributes on your spider:
|
||||
|
||||
| Attribute | Default | Effect |
|
||||
|----------------------|---------|----------------------------------------------------------------------------------------------------------------|
|
||||
| `fp_include_kwargs` | `False` | Include extra request kwargs (arguments you passed to the session fetch, like headers, etc.) in the fingerprint |
|
||||
| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints |
|
||||
| `fp_include_headers` | `False` | Include request headers in the fingerprint |
|
||||
|
||||
For example, if you need to treat `https://example.com/page#section1` and `https://example.com/page#section2` as different URLs:
|
||||
|
||||
```python
|
||||
class MySpider(Spider):
|
||||
name = "my_spider"
|
||||
fp_keep_fragments = True
|
||||
# ...
|
||||
```
|
||||
|
||||
## Request Meta
|
||||
|
||||
The `meta` dictionary lets you pass arbitrary data between callbacks. This is useful when you need context from one page to process another:
|
||||
|
||||
```python
|
||||
async def parse(self, response: Response):
|
||||
for product in response.css("div.product"):
|
||||
category = product.css("span.category::text").get("")
|
||||
link = product.css("a::attr(href)").get()
|
||||
if link:
|
||||
yield response.follow(
|
||||
link,
|
||||
callback=self.parse_product,
|
||||
meta={"category": category},
|
||||
)
|
||||
|
||||
async def parse_product(self, response: Response):
|
||||
yield {
|
||||
"name": response.css("h1::text").get(""),
|
||||
"price": response.css(".price::text").get(""),
|
||||
# Access meta from the request
|
||||
"category": response.meta.get("category", ""),
|
||||
}
|
||||
```
|
||||
|
||||
When using `response.follow()`, the meta from the current response is merged with the new meta you provide (new values take precedence).
|
||||
|
||||
The spider system also automatically stores some metadata. For example, the proxy used for a request is available as `response.meta["proxy"]` when proxy rotation is enabled.
|
||||
@@ -33,6 +33,7 @@ nav = [
|
||||
{Spiders = [
|
||||
{"Architecture" = "spiders/architecture.md"},
|
||||
{"Getting started" = "spiders/getting-started.md"},
|
||||
{"Requests & Responses" = "spiders/requests-responses.md"},
|
||||
]},
|
||||
{"Command Line Interface" = [
|
||||
{Overview = "cli/overview.md"},
|
||||
|
||||
Reference in New Issue
Block a user