docs: style adjustment

2026-03-30 03:53:56 +02:00
parent 1f1e475772
commit a403156b2e
36 changed files with 159 additions and 159 deletions
@@ -24,7 +24,7 @@ class PoliteSpider(Spider):
        yield {"title": response.css("title::text").get("")}
 ```

-When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain.
+When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously - you can allow high global concurrency while being polite to each individual domain.

 **Tip:** The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting.

@@ -56,7 +56,7 @@ else:

 1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits.
 2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks.
-3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`.
+3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off, skipping `start_requests()`.
 4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically.

 **Checkpoints are also saved periodically during the crawl (every 5 minutes by default).** 
@@ -14,7 +14,7 @@ Here's what happens step by step when you run a spider:
 4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized.
 5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing.
 6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused.
-7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state.
+7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off, skipping `start_requests()` and restoring the scheduler state.


 ## Components
@@ -40,7 +40,7 @@ class MySpider(Spider):

 ### Crawler Engine

-The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you.
+The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly - the `Spider.start()` and `Spider.stream()` methods handle it for you.

 ### Scheduler

@@ -21,9 +21,9 @@ class QuotesSpider(Spider):

 Every spider needs three things:

-1. **`name`** — A unique identifier for the spider.
-2. **`start_urls`** — A list of URLs to start crawling from.
-3. **`parse()`** — An async generator method that processes each response and yields results.
+1. **`name`**: A unique identifier for the spider.
+2. **`start_urls`**: A list of URLs to start crawling from.
+3. **`parse()`**: An async generator method that processes each response and yields results.

 The `parse()` method processes each response. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items.

@@ -35,7 +35,7 @@ To run your spider, create an instance and call `start()`:
 result = QuotesSpider().start()
 ```

-The `start()` method handles all the async machinery internally — no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
+The `start()` method handles all the async machinery internally, so there is no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.

 Those stats are in the returned `CrawlResult` object, which gives you everything you need:

@@ -80,7 +80,7 @@ class QuotesSpider(Spider):
            yield response.follow(next_page, callback=self.parse)
 ```

-`response.follow()` handles relative URLs automatically — it joins them with the current page's URL. It also sets the current page as the `Referer` header by default.
+`response.follow()` handles relative URLs automatically by joining them with the current page's URL. It also sets the current page as the `Referer` header by default.

 You can point follow-up requests at different callback methods for different page types:

@@ -133,7 +133,7 @@ class MySpider(Spider):
            yield response.follow(link, callback=self.parse)
 ```

-Subdomains are matched automatically — setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.
+Subdomains are matched automatically, so setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.

 When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped.

@@ -61,7 +61,7 @@ def configure_sessions(self, manager):

 ## Custom Rotation Strategies

-By default, `ProxyRotator` uses cyclic rotation — it iterates through proxies sequentially, wrapping around at the end.
+By default, `ProxyRotator` uses cyclic rotation - it iterates through proxies sequentially, wrapping around at the end.

 You can provide a custom strategy function to change this behavior, but it has to match the below signature:

@@ -1,6 +1,6 @@
 # Requests & Responses

-This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
+This page covers the `Request` object in detail: how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.

 ## The Request Object

@@ -25,7 +25,7 @@ Here are all the arguments you can pass to `Request`:
 | Argument      | Type       | Default    | Description                                                                                           |
 |---------------|------------|------------|-------------------------------------------------------------------------------------------------------|
 | `url`         | `str`      | *required* | The URL to fetch                                                                                      |
-| `sid`         | `str`      | `""`       | Session ID — routes the request to a specific session (see [Sessions](sessions.md))                   |
+| `sid`         | `str`      | `""`       | Session ID - routes the request to a specific session (see [Sessions](sessions.md))                   |
 | `callback`    | `callable` | `None`     | Async generator method to process the response. Defaults to `parse()`                                 |
 | `priority`    | `int`      | `0`        | Higher values are processed first                                                                     |
 | `dont_filter` | `bool`     | `False`    | If `True`, skip deduplication (allow duplicate requests)                                              |
@@ -54,7 +54,7 @@ yield Request(

 ```python
 async def parse(self, response: Response):
-    # Minimal — inherits callback, sid, priority from current request
+    # Minimal - inherits callback, sid, priority from current request
    yield response.follow("/next-page")

    # Override specific fields
@@ -95,9 +95,9 @@ yield response.follow("/page", referer_flow=False)

 Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:

- **`dict`** — A scraped item, added to the results
- **`Request`** — A follow-up request, added to the queue
- **`None`** — Silently ignored
+- **`dict`**: A scraped item, added to the results
+- **`Request`**: A follow-up request, added to the queue
+- **`None`**: Silently ignored

 ```python
 class MySpider(Spider):
@@ -124,11 +124,11 @@ Requests with higher priority values are processed first. This is useful when so

 ```python
 async def parse(self, response: Response):
-    # High priority — process product pages first
+    # High priority - process product pages first
    for link in response.css("a.product::attr(href)").getall():
        yield response.follow(link, callback=self.parse_product, priority=10)

-    # Low priority — pagination links processed after products
+    # Low priority - pagination links processed after products
    next_page = response.css("a.next::attr(href)").get()
    if next_page:
        yield response.follow(next_page, callback=self.parse, priority=0)
@@ -1,6 +1,6 @@
 # Spiders sessions

-A spider can use multiple fetcher sessions simultaneously — for example, a fast HTTP session for simple pages and a stealth browser session for protected pages.
+A spider can use multiple fetcher sessions simultaneously. For example, a fast HTTP session for simple pages and a stealth browser session for protected pages.

 ## What are Sessions?

@@ -18,7 +18,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md

 ## Configuring Sessions

-Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance — use `manager.add()` to register sessions:
+Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance - use `manager.add()` to register sessions:

 ```python
 from scrapling.spiders import Spider, Response
@@ -90,7 +90,7 @@ class ProductSpider(Spider):
        }
 ```

-The key is the `sid` parameter — it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
+The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.

 Sessions can also be different instances of the same class with different configurations: