docs: update pages with the XHR feature
This commit is contained in:
Karim shoair
@@ -77,6 +77,7 @@ The `Response` object is the same as the [Selector](../parsing/main_classes.md#s
|
||||
>>> page.body # Raw response body as bytes
|
||||
>>> page.encoding # Response encoding
|
||||
>>> page.meta # Response metadata dictionary (e.g., proxy used). Mainly helpful with the spiders system.
|
||||
>>> page.captured_xhr # List of captured XHR/fetch responses (when capture_xhr is enabled on a browser session)
|
||||
```
|
||||
All fetchers return the `Response` object.
|
||||
|
||||
|
||||
@@ -91,6 +91,7 @@ Scrapling provides many options with this fetcher and its session classes. To ma
|
||||
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
|
||||
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
|
||||
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
|
||||
| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
|
||||
|
||||
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `blocked_domains`, `proxy`, and `selector_config`.
|
||||
|
||||
@@ -217,6 +218,24 @@ The states the fetcher can wait for can be any of the following ([source](https:
|
||||
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
|
||||
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
|
||||
|
||||
### Capturing XHR/Fetch Requests
|
||||
|
||||
Many SPAs load data through background API calls (XHR/fetch). You can capture these requests by passing a regex URL pattern to `capture_xhr` at the session level:
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import DynamicSession
|
||||
|
||||
with DynamicSession(capture_xhr=r"https://api\.example\.com/.*", headless=True) as session:
|
||||
page = session.fetch('https://example.com')
|
||||
|
||||
# Access captured XHR responses
|
||||
for xhr in page.captured_xhr:
|
||||
print(xhr.url, xhr.status)
|
||||
print(xhr.body) # Raw response body as bytes
|
||||
```
|
||||
|
||||
Each item in `captured_xhr` is a full `Response` object with the same properties (`.url`, `.status`, `.headers`, `.body`, etc.). When `capture_xhr` is not set or is `None`, `captured_xhr` is an empty list.
|
||||
|
||||
### Some Stealth Features
|
||||
|
||||
```python
|
||||
|
||||
@@ -72,12 +72,13 @@ Scrapling provides many options with this fetcher and its session classes. Befor
|
||||
| proxy_rotator | A `ProxyRotator` instance for automatic proxy rotation. Cannot be combined with `proxy`. | ✔️ |
|
||||
| retries | Number of retry attempts for failed requests. Defaults to 3. | ✔️ |
|
||||
| retry_delay | Seconds to wait between retry attempts. Defaults to 1. | ✔️ |
|
||||
| capture_xhr | Pass a regex URL pattern string to capture XHR/fetch requests matching it during page load. Captured responses are available via `response.captured_xhr`. Defaults to `None` (disabled). | ✔️ |
|
||||
|
||||
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, `blocked_domains`, `proxy`, and `selector_config`.
|
||||
|
||||
!!! note "Notes:"
|
||||
|
||||
1. It's basically the same arguments as [DynamicFetcher](dynamic.md#introduction) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
|
||||
1. It's basically the same arguments as [DynamicFetcher](dynamic.md#introduction) class, but with these additional arguments: `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`. The `capture_xhr` argument is shared with `DynamicFetcher`.
|
||||
2. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
|
||||
3. The `google_search` argument is enabled by default for all requests, setting the referer to `https://www.google.com/`. If used together with `extra_headers`, it takes priority over the referer set there.
|
||||
4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
|
||||
|
||||
@@ -74,9 +74,11 @@ class ProductSpider(Spider):
|
||||
manager.add("http", FetcherSession())
|
||||
|
||||
# Stealth browser for protected product pages
|
||||
# capture_xhr captures background API calls matching the regex
|
||||
manager.add("stealth", AsyncStealthySession(
|
||||
headless=True,
|
||||
network_idle=True,
|
||||
capture_xhr=r"https://api\.shop\.example\.com/.*",
|
||||
))
|
||||
|
||||
async def parse(self, response: Response):
|
||||
@@ -89,6 +91,10 @@ class ProductSpider(Spider):
|
||||
yield response.follow(next_page)
|
||||
|
||||
async def parse_product(self, response: Response):
|
||||
# Access captured XHR/fetch API calls (if capture_xhr was set on the session)
|
||||
for xhr in response.captured_xhr:
|
||||
self.logger.info(f"Captured API call: {xhr.url} ({xhr.status})")
|
||||
|
||||
yield {
|
||||
"name": response.css("h1::text").get(""),
|
||||
"price": response.css(".price::text").get(""),
|
||||
|
||||
@@ -26,7 +26,18 @@ if TYPE_CHECKING:
|
||||
|
||||
|
||||
class Response(Selector):
|
||||
"""This class is returned by all engines as a way to unify the response type between different libraries."""
|
||||
"""This class is returned by all engines as a way to unify the response type between different libraries.
|
||||
|
||||
:param status: HTTP status code.
|
||||
:param reason: HTTP status message.
|
||||
:param cookies: Response cookies.
|
||||
:param headers: Response headers.
|
||||
:param request_headers: Request headers sent with the request.
|
||||
:param history: List of redirect responses, if any.
|
||||
:param meta: Metadata dictionary (e.g., proxy used).
|
||||
:param request: Associated spider Request object (set by crawler, in the spiders framework).
|
||||
:param captured_xhr: List of captured XHR/fetch ``Response`` objects. Populated when ``capture_xhr`` is set on a browser session.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
|
||||
Reference in New Issue
Block a user