docs: style adjustment
This commit is contained in:
Karim shoair
@@ -32,7 +32,7 @@ class PoliteSpider(Spider):
|
||||
yield {"title": response.css("title::text").get("")}
|
||||
```
|
||||
|
||||
When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain.
|
||||
When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously, as you can allow high global concurrency while being polite to each individual domain.
|
||||
|
||||
!!! tip
|
||||
|
||||
@@ -66,7 +66,7 @@ else:
|
||||
|
||||
1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits.
|
||||
2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks.
|
||||
3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`.
|
||||
3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off, skipping `start_requests()`.
|
||||
4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically.
|
||||
|
||||
**Checkpoints are also saved periodically during the crawl (every 5 minutes by default).**
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
|
||||
Scrapling's spider system is a Scrapy-inspired async crawling framework designed for concurrent, multi-session crawls with built-in pause/resume support. It brings together Scrapling's parsing engine and fetchers into a unified crawling API while adding scheduling, concurrency control, and checkpointing.
|
||||
|
||||
If you're familiar with Scrapy, you'll feel right at home. If not, don't worry — the system is designed to be straightforward.
|
||||
If you're familiar with Scrapy, you'll feel right at home. If not, don't worry - the system is designed to be straightforward.
|
||||
|
||||
## Data Flow
|
||||
|
||||
@@ -23,7 +23,7 @@ Here's what happens step by step when you run a spider without many details:
|
||||
4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized.
|
||||
5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing.
|
||||
6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused.
|
||||
7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state.
|
||||
7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off, skipping `start_requests()` and restoring the scheduler state.
|
||||
|
||||
|
||||
## Components
|
||||
@@ -49,7 +49,7 @@ class MySpider(Spider):
|
||||
|
||||
### Crawler Engine
|
||||
|
||||
The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you.
|
||||
The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly - the `Spider.start()` and `Spider.stream()` methods handle it for you.
|
||||
|
||||
### Scheduler
|
||||
|
||||
|
||||
@@ -31,9 +31,9 @@ class QuotesSpider(Spider):
|
||||
|
||||
Every spider needs three things:
|
||||
|
||||
1. **`name`** — A unique identifier for the spider.
|
||||
2. **`start_urls`** — A list of URLs to start crawling from.
|
||||
3. **`parse()`** — An async generator method that processes each response and yields results.
|
||||
1. **`name`** - A unique identifier for the spider.
|
||||
2. **`start_urls`** - A list of URLs to start crawling from.
|
||||
3. **`parse()`** - An async generator method that processes each response and yields results.
|
||||
|
||||
The `parse()` method is where the magic happens. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items.
|
||||
|
||||
@@ -45,7 +45,7 @@ To run your spider, create an instance and call `start()`:
|
||||
result = QuotesSpider().start()
|
||||
```
|
||||
|
||||
The `start()` method handles all the async machinery internally — no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
|
||||
The `start()` method handles all the async machinery internally, so no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
|
||||
|
||||
Those stats are in the returned `CrawlResult` object, which gives you everything you need:
|
||||
|
||||
@@ -90,7 +90,7 @@ class QuotesSpider(Spider):
|
||||
yield response.follow(next_page, callback=self.parse)
|
||||
```
|
||||
|
||||
`response.follow()` handles relative URLs automatically — it joins them with the current page's URL. It also sets the current page as the `Referer` header by default.
|
||||
`response.follow()` handles relative URLs automatically by joining them with the current page's URL. It also sets the current page as the `Referer` header by default.
|
||||
|
||||
You can point follow-up requests at different callback methods for different page types:
|
||||
|
||||
@@ -145,7 +145,7 @@ class MySpider(Spider):
|
||||
yield response.follow(link, callback=self.parse)
|
||||
```
|
||||
|
||||
Subdomains are matched automatically — setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.
|
||||
Subdomains are matched automatically, so setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.
|
||||
|
||||
When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped.
|
||||
|
||||
@@ -153,7 +153,7 @@ When a request is filtered out, it's counted in `stats.offsite_requests_count` s
|
||||
|
||||
Now that you have the basics, you can explore:
|
||||
|
||||
- [Requests & Responses](requests-responses.md) — learn about request priority, deduplication, metadata, and more.
|
||||
- [Sessions](sessions.md) — use multiple fetcher types (HTTP, browser, stealth) in a single spider.
|
||||
- [Proxy management & blocking](proxy-blocking.md) — rotate proxies across requests and how to handle blocking in the spider.
|
||||
- [Advanced features](advanced.md) — concurrency control, pause/resume, streaming, lifecycle hooks, and logging.
|
||||
- [Requests & Responses](requests-responses.md) - learn about request priority, deduplication, metadata, and more.
|
||||
- [Sessions](sessions.md) - use multiple fetcher types (HTTP, browser, stealth) in a single spider.
|
||||
- [Proxy management & blocking](proxy-blocking.md) - rotate proxies across requests and how to handle blocking in the spider.
|
||||
- [Advanced features](advanced.md) - concurrency control, pause/resume, streaming, lifecycle hooks, and logging.
|
||||
@@ -7,7 +7,7 @@
|
||||
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
|
||||
2. You've read the [Sessions](sessions.md) page and understand how to configure sessions.
|
||||
|
||||
When scraping at scale, you'll often need to rotate through multiple proxies to avoid rate limits and blocks. Scrapling's `ProxyRotator` makes this straightforward — it works with all session types and integrates with the spider's blocked request retry system.
|
||||
When scraping at scale, you'll often need to rotate through multiple proxies to avoid rate limits and blocks. Scrapling's `ProxyRotator` makes this straightforward. It works with all session types and integrates with the spider's blocked request retry system.
|
||||
|
||||
If you don't know what a proxy is or how to choose a good one, [this guide can help](https://substack.thewebscraping.club/p/everything-about-proxies).
|
||||
|
||||
@@ -70,7 +70,7 @@ def configure_sessions(self, manager):
|
||||
|
||||
## Custom Rotation Strategies
|
||||
|
||||
By default, `ProxyRotator` uses cyclic rotation — it iterates through proxies sequentially, wrapping around at the end.
|
||||
By default, `ProxyRotator` uses cyclic rotation, iterating through proxies sequentially and wrapping around at the end.
|
||||
|
||||
You can provide a custom strategy function to change this behavior, but it has to match the below signature:
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
|
||||
|
||||
This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
|
||||
This page covers the `Request` object in detail: how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
|
||||
|
||||
## The Request Object
|
||||
|
||||
@@ -29,7 +29,7 @@ Here are all the arguments you can pass to `Request`:
|
||||
| Argument | Type | Default | Description |
|
||||
|---------------|------------|------------|-------------------------------------------------------------------------------------------------------|
|
||||
| `url` | `str` | *required* | The URL to fetch |
|
||||
| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) |
|
||||
| `sid` | `str` | `""` | Session ID - routes the request to a specific session (see [Sessions](sessions.md)) |
|
||||
| `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` |
|
||||
| `priority` | `int` | `0` | Higher values are processed first |
|
||||
| `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) |
|
||||
@@ -58,7 +58,7 @@ yield Request(
|
||||
|
||||
```python
|
||||
async def parse(self, response: Response):
|
||||
# Minimal — inherits callback, sid, priority from current request
|
||||
# Minimal - inherits callback, sid, priority from current request
|
||||
yield response.follow("/next-page")
|
||||
|
||||
# Override specific fields
|
||||
@@ -99,9 +99,9 @@ yield response.follow("/page", referer_flow=False)
|
||||
|
||||
Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:
|
||||
|
||||
- **`dict`** — A scraped item, added to the results
|
||||
- **`Request`** — A follow-up request, added to the queue
|
||||
- **`None`** — Silently ignored
|
||||
- **`dict`** - A scraped item, added to the results
|
||||
- **`Request`** - A follow-up request, added to the queue
|
||||
- **`None`** - Silently ignored
|
||||
|
||||
```python
|
||||
class MySpider(Spider):
|
||||
@@ -130,11 +130,11 @@ Requests with higher priority values are processed first. This is useful when so
|
||||
|
||||
```python
|
||||
async def parse(self, response: Response):
|
||||
# High priority — process product pages first
|
||||
# High priority - process product pages first
|
||||
for link in response.css("a.product::attr(href)").getall():
|
||||
yield response.follow(link, callback=self.parse_product, priority=10)
|
||||
|
||||
# Low priority — pagination links processed after products
|
||||
# Low priority - pagination links processed after products
|
||||
next_page = response.css("a.next::attr(href)").get()
|
||||
if next_page:
|
||||
yield response.follow(next_page, callback=self.parse, priority=0)
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
|
||||
2. You're familiar with [Fetchers basics](../fetching/choosing.md) and the differences between HTTP, Dynamic, and Stealthy sessions.
|
||||
|
||||
A spider can use multiple fetcher sessions simultaneously — for example, a fast HTTP session for simple pages and a stealth browser session for protected pages. This page shows you how to configure and use sessions.
|
||||
A spider can use multiple fetcher sessions simultaneously. For example, a fast HTTP session for simple pages and a stealth browser session for protected pages. This page shows you how to configure and use sessions.
|
||||
|
||||
## What are Sessions?
|
||||
|
||||
@@ -23,7 +23,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md
|
||||
|
||||
## Configuring Sessions
|
||||
|
||||
Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance — use `manager.add()` to register sessions:
|
||||
Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance. Use `manager.add()` to register sessions:
|
||||
|
||||
```python
|
||||
from scrapling.spiders import Spider, Response
|
||||
@@ -101,7 +101,7 @@ class ProductSpider(Spider):
|
||||
}
|
||||
```
|
||||
|
||||
The key is the `sid` parameter — it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
|
||||
The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
|
||||
|
||||
Note that the sessions don't have to be from different classes only, but can be the same session, but different instances with different configurations, for example, like below:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user