freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Upload the library agent skill

Browse Source
This commit is contained in:
Karim shoair
2026-03-08 03:06:01 +02:00
parent aa3f4b4b0e
commit f469fb9b8b
24 changed files with 4312 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent-skill/Scrapling-Skill/references/spiders/advanced.md
+297
View File
@@ -0,0 +1,297 @@
# Advanced usages
## Concurrency Control
The spider system uses three class attributes to control how aggressively it crawls:
| Attribute | Default | Description |
|----------------------------------|---------|------------------------------------------------------------------|
| `concurrent_requests` | `4` | Maximum number of requests being processed at the same time |
| `concurrent_requests_per_domain` | `0` | Maximum concurrent requests per domain (0 = no per-domain limit) |
| `download_delay` | `0.0` | Seconds to wait before each request |
```python
class PoliteSpider(Spider):
name = "polite"
start_urls = ["https://example.com"]
# Be gentle with the server
concurrent_requests = 4
concurrent_requests_per_domain = 2
download_delay = 1.0 # Wait 1 second between requests
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
```
When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain.
**Tip:** The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting.
### Using uvloop
The `start()` method accepts a `use_uvloop` parameter to use the faster [uvloop](https://github.com/MagicStack/uvloop)/[winloop](https://github.com/nicktimko/winloop) event loop implementation, if available:
```python
result = MySpider().start(use_uvloop=True)
```
This can improve throughput for I/O-heavy crawls. You'll need to install `uvloop` (Linux/macOS) or `winloop` (Windows) separately.
## Pause & Resume
The spider supports graceful pause-and-resume via checkpointing. To enable it, pass a `crawldir` directory to the spider constructor:
```python
spider = MySpider(crawldir="crawl_data/my_spider")
result = spider.start()
if result.paused:
print("Crawl was paused. Run again to resume.")
else:
print("Crawl completed!")
```
### How It Works
1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits.
2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks.
3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`.
4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically.
**Checkpoints are also saved periodically during the crawl (every 5 minutes by default).**
You can change the interval as follows:
```python
# Save checkpoint every 2 minutes
spider = MySpider(crawldir="crawl_data/my_spider", interval=120.0)
```
The writing to the disk is atomic, so it's totally safe.
**Tip:** Pressing `Ctrl+C` during a crawl always causes the spider to close gracefully, even if the checkpoint system is not enabled. Doing it again without waiting forces the spider to close immediately.
### Knowing If You're Resuming
The `on_start()` hook receives a `resuming` flag:
```python
async def on_start(self, resuming: bool = False):
if resuming:
self.logger.info("Resuming from checkpoint!")
else:
self.logger.info("Starting fresh crawl")
```
## Streaming
For long-running spiders or applications that need real-time access to scraped items, use the `stream()` method instead of `start()`:
```python
import anyio
async def main():
spider = MySpider()
async for item in spider.stream():
print(f"Got item: {item}")
# Access real-time stats
print(f"Items so far: {spider.stats.items_scraped}")
print(f"Requests made: {spider.stats.requests_count}")
anyio.run(main)
```
Key differences from `start()`:
- `stream()` must be called from an async context
- Items are yielded one by one as they're scraped, not collected into a list
- You can access `spider.stats` during iteration for real-time statistics
**Note:** The full list of all stats that can be accessed by `spider.stats` is explained below [here](#results--statistics).
You can use it with the checkpoint system too, so it's easy to build UI on top of spiders. UIs that have real-time data and can be paused/resumed.
```python
import anyio
async def main():
spider = MySpider(crawldir="crawl_data/my_spider")
async for item in spider.stream():
print(f"Got item: {item}")
# Access real-time stats
print(f"Items so far: {spider.stats.items_scraped}")
print(f"Requests made: {spider.stats.requests_count}")
anyio.run(main)
```
You can also use `spider.pause()` to shut down the spider in the code above. If you used it without enabling the checkpoint system, it will just close the crawl.
## Lifecycle Hooks
The spider provides several hooks you can override to add custom behavior at different stages of the crawl:
### on_start
Called before crawling begins. Use it for setup tasks like loading data or initializing resources:
```python
async def on_start(self, resuming: bool = False):
self.logger.info("Spider starting up")
# Load seed URLs from a database, initialize counters, etc.
```
### on_close
Called after crawling finishes (whether completed or paused). Use it for cleanup:
```python
async def on_close(self):
self.logger.info("Spider shutting down")
# Close database connections, flush buffers, etc.
```
### on_error
Called when a request fails with an exception. Use it for error tracking or custom recovery logic:
```python
async def on_error(self, request: Request, error: Exception):
self.logger.error(f"Failed: {request.url} - {error}")
# Log to error tracker, save failed URL for later, etc.
```
### on_scraped_item
Called for every scraped item before it's added to the results. Return the item (modified or not) to keep it, or return `None` to drop it:
```python
async def on_scraped_item(self, item: dict) -> dict | None:
# Drop items without a title
if not item.get("title"):
return None
# Modify items (e.g., add timestamps)
item["scraped_at"] = "2026-01-01"
return item
```
**Tip:** This hook can also be used to direct items through your own pipelines and drop them from the spider.
### start_requests
Override `start_requests()` for custom initial request generation instead of using `start_urls`:
```python
async def start_requests(self):
# POST request to log in first
yield Request(
"https://example.com/login",
method="POST",
data={"user": "admin", "pass": "secret"},
callback=self.after_login,
)
async def after_login(self, response: Response):
# Now crawl the authenticated pages
yield response.follow("/dashboard", callback=self.parse)
```
## Results & Statistics
The `CrawlResult` returned by `start()` contains both the scraped items and detailed statistics:
```python
result = MySpider().start()
# Items
print(f"Total items: {len(result.items)}")
result.items.to_json("output.json", indent=True)
# Did the crawl complete?
print(f"Completed: {result.completed}")
print(f"Paused: {result.paused}")
# Statistics
stats = result.stats
print(f"Requests: {stats.requests_count}")
print(f"Failed: {stats.failed_requests_count}")
print(f"Blocked: {stats.blocked_requests_count}")
print(f"Offsite filtered: {stats.offsite_requests_count}")
print(f"Items scraped: {stats.items_scraped}")
print(f"Items dropped: {stats.items_dropped}")
print(f"Response bytes: {stats.response_bytes}")
print(f"Duration: {stats.elapsed_seconds:.1f}s")
print(f"Speed: {stats.requests_per_second:.1f} req/s")
```
### Detailed Stats
The `CrawlStats` object tracks granular information:
```python
stats = result.stats
# Status code distribution
print(stats.response_status_count)
# {'status_200': 150, 'status_404': 3, 'status_403': 1}
# Bytes downloaded per domain
print(stats.domains_response_bytes)
# {'example.com': 1234567, 'api.example.com': 45678}
# Requests per session
print(stats.sessions_requests_count)
# {'http': 120, 'stealth': 34}
# Proxies used during the crawl
print(stats.proxies)
# ['http://proxy1:8080', 'http://proxy2:8080']
# Log level counts
print(stats.log_levels_counter)
# {'debug': 200, 'info': 50, 'warning': 3, 'error': 1, 'critical': 0}
# Timing information
print(stats.start_time) # Unix timestamp when crawl started
print(stats.end_time) # Unix timestamp when crawl finished
print(stats.download_delay) # The download delay used (seconds)
# Concurrency settings used
print(stats.concurrent_requests) # Global concurrency limit
print(stats.concurrent_requests_per_domain) # Per-domain concurrency limit
# Custom stats (set by your spider code)
print(stats.custom_stats)
# {'login_attempts': 3, 'pages_with_errors': 5}
# Export everything as a dict
print(stats.to_dict())
```
## Logging
The spider has a built-in logger accessible via `self.logger`. It's pre-configured with the spider's name and supports several customization options:
| Attribute | Default | Description |
|-----------------------|--------------------------------------------------------------|----------------------------------------------------|
| `logging_level` | `logging.DEBUG` | Minimum log level |
| `logging_format` | `"[%(asctime)s]:({spider_name}) %(levelname)s: %(message)s"` | Log message format |
| `logging_date_format` | `"%Y-%m-%d %H:%M:%S"` | Date format in log messages |
| `log_file` | `None` | Path to a log file (in addition to console output) |
```python
import logging
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
logging_level = logging.INFO
log_file = "logs/my_spider.log"
async def parse(self, response: Response):
self.logger.info(f"Processing {response.url}")
yield {"title": response.css("title::text").get("")}
```
The log file directory is created automatically if it doesn't exist. Both console and file output use the same format.
agent-skill/Scrapling-Skill/references/spiders/architecture.md
+89
View File
@@ -0,0 +1,89 @@
# Spiders architecture
Scrapling's spider system is an async crawling framework designed for concurrent, multi-session crawls with built-in pause/resume support. It brings together Scrapling's parsing engine and fetchers into a unified crawling API while adding scheduling, concurrency control, and checkpointing.
## Data Flow
The diagram below shows how data flows through the spider system when a crawl is running:
Here's what happens step by step when you run a spider:
1. The **Spider** produces the first batch of `Request` objects. By default, it creates one request for each URL in `start_urls`, but you can override `start_requests()` for custom logic.
2. The **Scheduler** receives requests and places them in a priority queue, and creates fingerprints for them. Higher-priority requests are dequeued first.
3. The **Crawler Engine** asks the **Scheduler** to dequeue the next request, respecting concurrency limits (global and per-domain) and download delays. Once the **Crawler Engine** receives the request, it passes it to the **Session Manager**, which routes it to the correct session based on the request's `sid` (session ID).
4. The **session** fetches the page and returns a [Response](fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized.
5. The **Crawler Engine** passes the [Response](fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing.
6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused.
7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state.
## Components
### Spider
The central class you interact with. You subclass `Spider`, define your `start_urls` and `parse()` method, and optionally configure sessions and override lifecycle hooks.
```python
from scrapling.spiders import Spider, Response, Request
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
async def parse(self, response: Response):
for link in response.css("a::attr(href)").getall():
yield response.follow(link, callback=self.parse_page)
async def parse_page(self, response: Response):
yield {"title": response.css("h1::text").get("")}
```
### Crawler Engine
The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you.
### Scheduler
A priority queue with built-in URL deduplication. Requests are fingerprinted based on their URL, HTTP method, body, and session ID. The scheduler supports `snapshot()` and `restore()` for the checkpoint system, allowing the crawl state to be saved and resumed.
### Session Manager
Manages one or more named session instances. Each session is one of:
- [FetcherSession](fetching/static.md)
- [AsyncDynamicSession](fetching/dynamic.md)
- [AsyncStealthySession](fetching/stealthy.md)
When a request comes in, the Session Manager routes it to the correct session based on the request's `sid` field. Sessions can be started with the spider start (default) or lazily (started on the first use).
### Checkpoint System
An optional system that, if enabled, saves the crawler's state (pending requests + seen URL fingerprints) to a pickle file on disk. Writes are atomic (temp file + rename) to prevent corruption. Checkpoints are saved periodically at a configurable interval and on graceful shutdown. Upon successful completion (not paused), checkpoint files are automatically cleaned up.
### Output
Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.
## Comparison with Scrapy
If you're coming from Scrapy, here's how Scrapling's spider system maps:
| Concept | Scrapy | Scrapling |
|--------------------|-------------------------------|-----------------------------------------------------------------|
| Spider definition | `scrapy.Spider` subclass | `scrapling.spiders.Spider` subclass |
| Initial requests | `start_requests()` | `async start_requests()` |
| Callbacks | `def parse(self, response)` | `async def parse(self, response)` |
| Following links | `response.follow(url)` | `response.follow(url)` |
| Item output | `yield dict` or `yield Item` | `yield dict` |
| Request scheduling | Scheduler + Dupefilter | Scheduler with built-in deduplication |
| Downloading | Downloader + Middlewares | Session Manager with multi-session support |
| Item processing | Item Pipelines | `on_scraped_item()` hook |
| Blocked detection | Through custom middlewares | Built-in `is_blocked()` + `retry_blocked_request()` hooks |
| Concurrency | `CONCURRENT_REQUESTS` setting | `concurrent_requests` class attribute |
| Domain filtering | `allowed_domains` | `allowed_domains` |
| Pause/Resume | `JOBDIR` setting | `crawldir` constructor argument |
| Export | Feed exports | `result.items.to_json()` / `to_jsonl()` or custom through hooks |
| Running | `scrapy crawl spider_name` | `MySpider().start()` |
| Streaming | N/A | `async for item in spider.stream()` |
| Multi-session | N/A | Multiple sessions with different types per spider |
agent-skill/Scrapling-Skill/references/spiders/getting-started.md
+139
View File
@@ -0,0 +1,139 @@
# Getting started
## Your First Spider
A spider is a class that defines how to crawl and extract data from websites. Here's the simplest possible spider:
```python
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com"]
async def parse(self, response: Response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(""),
"author": quote.css("small.author::text").get(""),
}
```
Every spider needs three things:
1. **`name`** — A unique identifier for the spider.
2. **`start_urls`** — A list of URLs to start crawling from.
3. **`parse()`** — An async generator method that processes each response and yields results.
The `parse()` method processes each response. You use the same selection methods you'd use with Scrapling's [Selector](parsing/main_classes.md#selector)/[Response](fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items.
## Running the Spider
To run your spider, create an instance and call `start()`:
```python
result = QuotesSpider().start()
```
The `start()` method handles all the async machinery internally — no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
Those stats are in the returned `CrawlResult` object, which gives you everything you need:
```python
result = QuotesSpider().start()
# Access scraped items
for item in result.items:
print(item["text"], "-", item["author"])
# Check statistics
print(f"Scraped {result.stats.items_scraped} items")
print(f"Made {result.stats.requests_count} requests")
print(f"Took {result.stats.elapsed_seconds:.1f} seconds")
# Did the crawl finish or was it paused?
print(f"Completed: {result.completed}")
```
## Following Links
Most crawls need to follow links across multiple pages. Use `response.follow()` to create follow-up requests:
```python
from scrapling.spiders import Spider, Response
class QuotesSpider(Spider):
name = "quotes"
start_urls = ["https://quotes.toscrape.com"]
async def parse(self, response: Response):
# Extract items from the current page
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(""),
"author": quote.css("small.author::text").get(""),
}
# Follow the "next page" link
next_page = response.css("li.next a::attr(href)").get()
if next_page:
yield response.follow(next_page, callback=self.parse)
```
`response.follow()` handles relative URLs automatically — it joins them with the current page's URL. It also sets the current page as the `Referer` header by default.
You can point follow-up requests at different callback methods for different page types:
```python
async def parse(self, response: Response):
for link in response.css("a.product-link::attr(href)").getall():
yield response.follow(link, callback=self.parse_product)
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
```
**Note:** All callback methods must be async generators (using `async def` and `yield`).
## Exporting Data
The `ItemList` returned in `result.items` has built-in export methods:
```python
result = QuotesSpider().start()
# Export as JSON
result.items.to_json("quotes.json")
# Export as JSON with pretty-printing
result.items.to_json("quotes.json", indent=True)
# Export as JSON Lines (one JSON object per line)
result.items.to_jsonl("quotes.jsonl")
```
Both methods create parent directories automatically if they don't exist.
## Filtering Domains
Use `allowed_domains` to restrict the spider to specific domains. This prevents it from accidentally following links to external websites:
```python
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
allowed_domains = {"example.com"}
async def parse(self, response: Response):
for link in response.css("a::attr(href)").getall():
# Links to other domains are silently dropped
yield response.follow(link, callback=self.parse)
```
Subdomains are matched automatically — setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.
When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped.
agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md
+235
View File
@@ -0,0 +1,235 @@
# Proxy management and handling Blocks
Scrapling's `ProxyRotator` manages proxy rotation across requests. It works with all session types and integrates with the spider's blocked request retry system.
## ProxyRotator
The `ProxyRotator` class manages a list of proxies and rotates through them automatically. Pass it to any session type via the `proxy_rotator` parameter:
```python
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession, ProxyRotator
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
def configure_sessions(self, manager):
rotator = ProxyRotator([
"http://proxy1:8080",
"http://proxy2:8080",
"http://user:pass@proxy3:8080",
])
manager.add("default", FetcherSession(proxy_rotator=rotator))
async def parse(self, response: Response):
# Check which proxy was used
print(f"Proxy used: {response.meta.get('proxy')}")
yield {"title": response.css("title::text").get("")}
```
Each request automatically gets the next proxy in the rotation. The proxy used is stored in `response.meta["proxy"]` so you can track which proxy fetched which page.
Browser sessions support both string and dict proxy formats:
```python
from scrapling.fetchers import AsyncDynamicSession, AsyncStealthySession, ProxyRotator
# String proxies work for all session types
rotator = ProxyRotator([
"http://proxy1:8080",
"http://proxy2:8080",
])
# Dict proxies (Playwright format) work for browser sessions
rotator = ProxyRotator([
{"server": "http://proxy1:8080", "username": "user", "password": "pass"},
{"server": "http://proxy2:8080"},
])
# Then inside the spider
def configure_sessions(self, manager):
rotator = ProxyRotator(["http://proxy1:8080", "http://proxy2:8080"])
manager.add("browser", AsyncStealthySession(proxy_rotator=rotator))
```
**Important:**
1. You cannot use the `proxy_rotator` argument together with the static `proxy` or `proxies` parameters on the same session. Pick one approach when configuring the session, and override it per request later if needed.
2. By default, all browser-based sessions use a persistent browser context with a pool of tabs. However, since browsers can't set a proxy per tab, when you use a `ProxyRotator`, the fetcher will automatically open a separate context for each proxy, with one tab per context. Once the tab's job is done, both the tab and its context are closed.
## Custom Rotation Strategies
By default, `ProxyRotator` uses cyclic rotation — it iterates through proxies sequentially, wrapping around at the end.
You can provide a custom strategy function to change this behavior, but it has to match the below signature:
```python
from scrapling.core._types import ProxyType
def my_strategy(proxies: list, current_index: int) -> tuple[ProxyType, int]:
...
```
It receives the list of proxies and the current index, and must return the chosen proxy and the next index.
Below are some examples of custom rotation strategies you can use.
### Random Rotation
```python
import random
from scrapling.fetchers import ProxyRotator
def random_strategy(proxies, current_index):
idx = random.randint(0, len(proxies) - 1)
return proxies[idx], idx
rotator = ProxyRotator(
["http://proxy1:8080", "http://proxy2:8080", "http://proxy3:8080"],
strategy=random_strategy,
)
```
### Weighted Rotation
```python
import random
def weighted_strategy(proxies, current_index):
# First proxy gets 60% of traffic, others split the rest
weights = [60] + [40 // (len(proxies) - 1)] * (len(proxies) - 1)
proxy = random.choices(proxies, weights=weights, k=1)[0]
return proxy, current_index # Index doesn't matter for weighted
rotator = ProxyRotator(proxies, strategy=weighted_strategy)
```
## Per-Request Proxy Override
You can override the rotator for individual requests by passing `proxy=` as a keyword argument:
```python
async def parse(self, response: Response):
# This request uses the rotator's next proxy
yield response.follow("/page1", callback=self.parse_page)
# This request uses a specific proxy, bypassing the rotator
yield response.follow(
"/special-page",
callback=self.parse_page,
proxy="http://special-proxy:8080",
)
```
This is useful when certain pages require a specific proxy (e.g., a geo-located proxy for region-specific content).
## Blocked Request Handling
The spider has built-in blocked request detection and retry. By default, it considers the following HTTP status codes blocked: `401`, `403`, `407`, `429`, `444`, `500`, `502`, `503`, `504`.
The retry system works like this:
1. After a response comes back, the spider calls the `is_blocked(response)` method.
2. If blocked, it copies the request and calls the `retry_blocked_request()` method so you can modify it before retrying.
3. The retried request is re-queued with `dont_filter=True` (bypassing deduplication) and lower priority, so it's not retried right away.
4. This repeats up to `max_blocked_retries` times (default: 3).
**Tip:**
1. On retry, the previous `proxy`/`proxies` kwargs are cleared from the request automatically, so the rotator assigns a fresh proxy.
2. The `max_blocked_retries` attribute is different than the session retries and doesn't share the counter.
### Custom Block Detection
Override `is_blocked()` to add your own detection logic:
```python
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
async def is_blocked(self, response: Response) -> bool:
# Check status codes (default behavior)
if response.status in {403, 429, 503}:
return True
# Check response content
body = response.body.decode("utf-8", errors="ignore")
if "access denied" in body.lower() or "rate limit" in body.lower():
return True
return False
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
```
### Customizing Retries
Override `retry_blocked_request()` to modify the request before retrying. The `max_blocked_retries` attribute controls how many times a blocked request is retried (default: 3):
```python
from scrapling.spiders import Spider, SessionManager, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
max_blocked_retries = 5
def configure_sessions(self, manager: SessionManager) -> None:
manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari']))
manager.add('stealth', AsyncStealthySession(block_webrtc=True), lazy=True)
async def retry_blocked_request(self, request: Request, response: Response) -> Request:
request.sid = "stealth"
self.logger.info(f"Retrying blocked request: {request.url}")
return request
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
```
What happened above is that I left the blocking detection logic unchanged and had the spider mainly use requests until it got blocked, then switch to the stealthy browser.
Putting it all together:
```python
from scrapling.spiders import Spider, SessionManager, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession, ProxyRotator
cheap_proxies = ProxyRotator([ "http://proxy1:8080", "http://proxy2:8080"])
# A format acceptable by the browser
expensive_proxies = ProxyRotator([
{"server": "http://residential_proxy1:8080", "username": "user", "password": "pass"},
{"server": "http://residential_proxy2:8080", "username": "user", "password": "pass"},
{"server": "http://mobile_proxy1:8080", "username": "user", "password": "pass"},
{"server": "http://mobile_proxy2:8080", "username": "user", "password": "pass"},
])
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
max_blocked_retries = 5
def configure_sessions(self, manager: SessionManager) -> None:
manager.add('requests', FetcherSession(impersonate=['chrome', 'firefox', 'safari'], proxy_rotator=cheap_proxies))
manager.add('stealth', AsyncStealthySession(block_webrtc=True, proxy_rotator=expensive_proxies), lazy=True)
async def retry_blocked_request(self, request: Request, response: Response) -> Request:
request.sid = "stealth"
self.logger.info(f"Retrying blocked request: {request.url}")
return request
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
```
The above logic is: requests are made with cheap proxies, such as datacenter proxies, until they are blocked, then retried with higher-quality proxies, such as residential or mobile proxies.
agent-skill/Scrapling-Skill/references/spiders/requests-responses.md
+196
View File
@@ -0,0 +1,196 @@
# Requests & Responses
This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
## The Request Object
A `Request` represents a URL to be fetched. You create requests either directly or via `response.follow()`:
```python
from scrapling.spiders import Request
# Direct construction
request = Request(
"https://example.com/page",
callback=self.parse_page,
priority=5,
)
# Via response.follow (preferred in callbacks)
request = response.follow("/page", callback=self.parse_page)
```
Here are all the arguments you can pass to `Request`:
| Argument | Type | Default | Description |
|---------------|------------|------------|-------------------------------------------------------------------------------------------------------|
| `url` | `str` | *required* | The URL to fetch |
| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) |
| `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` |
| `priority` | `int` | `0` | Higher values are processed first |
| `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) |
| `meta` | `dict` | `{}` | Arbitrary metadata passed through to the response |
| `**kwargs` | | | Additional keyword arguments passed to the session's fetch method (e.g., `headers`, `method`, `data`) |
Any extra keyword arguments are forwarded directly to the underlying session. For example, to make a POST request:
```python
yield Request(
"https://example.com/api",
method="POST",
data={"key": "value"},
callback=self.parse_result,
)
```
## Response.follow()
`response.follow()` is the recommended way to create follow-up requests inside callbacks. It offers several advantages over constructing `Request` objects directly:
- **Relative URLs** are resolved automatically against the current page URL
- **Referer header** is set to the current page URL by default
- **Session kwargs** from the original request are inherited (headers, proxy settings, etc.)
- **Callback, session ID, and priority** are inherited from the original request if not specified
```python
async def parse(self, response: Response):
# Minimal — inherits callback, sid, priority from current request
yield response.follow("/next-page")
# Override specific fields
yield response.follow(
"/product/123",
callback=self.parse_product,
priority=10,
)
# Pass additional metadata to
yield response.follow(
"/details",
callback=self.parse_details,
meta={"category": "electronics"},
)
```
| Argument | Type | Default | Description |
|--------------------|------------|------------|------------------------------------------------------------|
| `url` | `str` | *required* | URL to follow (absolute or relative) |
| `sid` | `str` | `""` | Session ID (inherits from original request if empty) |
| `callback` | `callable` | `None` | Callback method (inherits from original request if `None`) |
| `priority` | `int` | `None` | Priority (inherits from original request if `None`) |
| `dont_filter` | `bool` | `False` | Skip deduplication |
| `meta` | `dict` | `None` | Metadata (merged with existing response meta) |
| **`referer_flow`** | `bool` | `True` | Set current URL as Referer header |
| `**kwargs` | | | Merged with original request's session kwargs |
### Disabling Referer Flow
By default, `response.follow()` sets the `Referer` header to the current page URL. To disable this:
```python
yield response.follow("/page", referer_flow=False)
```
## Callbacks
Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:
- **`dict`** — A scraped item, added to the results
- **`Request`** — A follow-up request, added to the queue
- **`None`** — Silently ignored
```python
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
async def parse(self, response: Response):
# Yield items (dicts)
yield {"url": response.url, "title": response.css("title::text").get("")}
# Yield follow-up requests
for link in response.css("a::attr(href)").getall():
yield response.follow(link, callback=self.parse_page)
async def parse_page(self, response: Response):
yield {"content": response.css("article::text").get("")}
```
**Note:** All callback methods must be `async def` and use `yield` (not `return`). Even if a callback only yields items with no follow-up requests, it must still be an async generator.
## Request Priority
Requests with higher priority values are processed first. This is useful when some pages are more important to be processed first before others:
```python
async def parse(self, response: Response):
# High priority — process product pages first
for link in response.css("a.product::attr(href)").getall():
yield response.follow(link, callback=self.parse_product, priority=10)
# Low priority — pagination links processed after products
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, callback=self.parse, priority=0)
```
When using `response.follow()`, the priority is inherited from the original request unless you specify a new one.
## Deduplication
The spider automatically deduplicates requests based on a fingerprint computed from the URL, HTTP method, request body, and session ID. If two requests produce the same fingerprint, the second one is silently dropped.
To allow duplicate requests (e.g., re-visiting a page after login), set `dont_filter=True`:
```python
yield Request("https://example.com/dashboard", dont_filter=True, callback=self.parse_dashboard)
# Or with response.follow
yield response.follow("/dashboard", dont_filter=True, callback=self.parse_dashboard)
```
You can fine-tune what goes into the fingerprint using class attributes on your spider:
| Attribute | Default | Effect |
|----------------------|---------|-----------------------------------------------------------------------------------------------------------------|
| `fp_include_kwargs` | `False` | Include extra request kwargs (arguments you passed to the session fetch, like headers, etc.) in the fingerprint |
| `fp_keep_fragments` | `False` | Keep URL fragments (`#section`) when computing fingerprints |
| `fp_include_headers` | `False` | Include request headers in the fingerprint |
For example, if you need to treat `https://example.com/page#section1` and `https://example.com/page#section2` as different URLs:
```python
class MySpider(Spider):
name = "my_spider"
fp_keep_fragments = True
# ...
```
## Request Meta
The `meta` dictionary lets you pass arbitrary data between callbacks. This is useful when you need context from one page to process another:
```python
async def parse(self, response: Response):
for product in response.css("div.product"):
category = product.css("span.category::text").get("")
link = product.css("a::attr(href)").get()
if link:
yield response.follow(
link,
callback=self.parse_product,
meta={"category": category},
)
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
# Access meta from the request
"category": response.meta.get("category", ""),
}
```
When using `response.follow()`, the meta from the current response is merged with the new meta you provide (new values take precedence).
The spider system also automatically stores some metadata. For example, the proxy used for a request is available as `response.meta["proxy"]` when proxy rotation is enabled.
agent-skill/Scrapling-Skill/references/spiders/sessions.md
+205
View File
@@ -0,0 +1,205 @@
# Spiders sessions
A spider can use multiple fetcher sessions simultaneously — for example, a fast HTTP session for simple pages and a stealth browser session for protected pages.
## What are Sessions?
A session is a pre-configured fetcher instance that stays alive for the duration of the crawl. Instead of creating a new connection or browser for every request, the spider reuses sessions, which is faster and more resource-efficient.
By default, every spider creates a single [FetcherSession](fetching/static.md). You can add more sessions or swap the default by overriding the `configure_sessions()` method, but you have to use the async version of each session only, as the table shows below:
| Session Type | Use Case |
|-------------------------------------------------|------------------------------------------|
| [FetcherSession](fetching/static.md) | Fast HTTP requests, no JavaScript |
| [AsyncDynamicSession](fetching/dynamic.md) | Browser automation, JavaScript rendering |
| [AsyncStealthySession](fetching/stealthy.md) | Anti-bot bypass, Cloudflare, etc. |
## Configuring Sessions
Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance — use `manager.add()` to register sessions:
```python
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession
class MySpider(Spider):
name = "my_spider"
start_urls = ["https://example.com"]
def configure_sessions(self, manager):
manager.add("default", FetcherSession())
async def parse(self, response: Response):
yield {"title": response.css("title::text").get("")}
```
The `manager.add()` method takes:
| Argument | Type | Default | Description |
|--------------|-----------|------------|----------------------------------------------|
| `session_id` | `str` | *required* | A name to reference this session in requests |
| `session` | `Session` | *required* | The session instance |
| `default` | `bool` | `False` | Make this the default session |
| `lazy` | `bool` | `False` | Start the session only when first used |
**Notes:**
1. In all requests, if you don't specify which session to use, the default session is used. The default session is determined in one of two ways:
1. The first session you add to the manager becomes the default automatically.
2. The session that gets `default=True` while added to the manager.
2. The instances you pass of each session don't have to be already started by you; the spider checks on all sessions if they are not already started and starts them.
3. If you want a specific session to start when used only, then use the `lazy` argument while adding that session to the manager. Example: start the browser only when you need it, not with the spider start.
## Multi-Session Spider
Here's a practical example: use a fast HTTP session for listing pages and a stealth browser for detail pages that have bot protection:
```python
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
class ProductSpider(Spider):
name = "products"
start_urls = ["https://shop.example.com/products"]
def configure_sessions(self, manager):
# Fast HTTP for listing pages (default)
manager.add("http", FetcherSession())
# Stealth browser for protected product pages
manager.add("stealth", AsyncStealthySession(
headless=True,
network_idle=True,
))
async def parse(self, response: Response):
for link in response.css("a.product::attr(href)").getall():
# Route product pages through the stealth session
yield response.follow(link, sid="stealth", callback=self.parse_product)
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page)
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
```
The key is the `sid` parameter — it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
Sessions can also be different instances of the same class with different configurations:
```python
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession
class ProductSpider(Spider):
name = "products"
start_urls = ["https://shop.example.com/products"]
def configure_sessions(self, manager):
chrome_requests = FetcherSession(impersonate="chrome")
firefox_requests = FetcherSession(impersonate="firefox")
manager.add("chrome", chrome_requests)
manager.add("firefox", firefox_requests)
async def parse(self, response: Response):
for link in response.css("a.product::attr(href)").getall():
yield response.follow(link, callback=self.parse_product)
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield response.follow(next_page, sid="firefox")
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
```
## Session Arguments
Extra keyword arguments passed to a `Request` (or through `response.follow(**kwargs)`) are forwarded to the session's fetch method. This lets you customize individual requests without changing the session configuration:
```python
async def parse(self, response: Response):
# Pass extra headers for this specific request
yield Request(
"https://api.example.com/data",
headers={"Authorization": "Bearer token123"},
callback=self.parse_api,
)
# Use a different HTTP method
yield Request(
"https://example.com/submit",
method="POST",
data={"field": "value"},
sid="firefox",
callback=self.parse_result,
)
```
**Warning:** When using `FetcherSession` in spiders, you cannot use `.get()` and `.post()` methods directly. By default, the request is an HTTP GET request; to use another HTTP method, pass it to the `method` argument as in the above example. This unifies the `Request` interface across all session types.
For browser sessions (`AsyncDynamicSession`, `AsyncStealthySession`), you can pass browser-specific arguments like `wait_selector`, `page_action`, or `extra_headers`:
```python
async def parse(self, response: Response):
# Use Cloudflare solver with the `AsyncStealthySession` we configured above
yield Request(
"https://nopecha.com/demo/cloudflare",
sid="stealth",
callback=self.parse_result,
solve_cloudflare=True,
block_webrtc=True,
hide_canvas=True,
google_search=True,
)
yield response.follow(
"/dynamic-page",
sid="browser",
callback=self.parse_dynamic,
wait_selector="div.loaded",
network_idle=True,
)
```
**Warning:** Session arguments (**kwargs) passed from the original request are inherited by `response.follow()`. New kwargs take precedence over inherited ones.
```python
from scrapling.spiders import Spider, Response
from scrapling.fetchers import FetcherSession
class ProductSpider(Spider):
name = "products"
start_urls = ["https://shop.example.com/products"]
def configure_sessions(self, manager):
manager.add("http", FetcherSession(impersonate='chrome'))
async def parse(self, response: Response):
# I don't want the follow request to impersonate a desktop Chrome like the previous request, but a mobile one
# so I override it like this
for link in response.css("a.product::attr(href)").getall():
yield response.follow(link, impersonate="chrome131_android", callback=self.parse_product)
next_page = response.css("a.next::attr(href)").get()
if next_page:
yield Request(next_page)
async def parse_product(self, response: Response):
yield {
"name": response.css("h1::text").get(""),
"price": response.css(".price::text").get(""),
}
```
**Note:** Upon spider closure, the manager automatically checks whether any sessions are still running and closes them before closing the spider.