docs: add new api references
This commit is contained in:
@@ -0,0 +1,18 @@
|
|||||||
|
---
|
||||||
|
search:
|
||||||
|
exclude: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Proxy Rotation
|
||||||
|
|
||||||
|
The `ProxyRotator` class provides thread-safe proxy rotation for any fetcher or session.
|
||||||
|
|
||||||
|
You can import it directly like below:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from scrapling.fetchers import ProxyRotator
|
||||||
|
```
|
||||||
|
|
||||||
|
## ::: scrapling.engines.toolbelt.proxy_rotation.ProxyRotator
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
---
|
||||||
|
search:
|
||||||
|
exclude: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Response Class
|
||||||
|
|
||||||
|
The `Response` class wraps HTTP responses returned by all fetchers, providing access to status, headers, body, cookies, and a `Selector` for parsing.
|
||||||
|
|
||||||
|
You can import the `Response` class like below:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from scrapling.engines.toolbelt.custom import Response
|
||||||
|
```
|
||||||
|
|
||||||
|
## ::: scrapling.engines.toolbelt.custom.Response
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
---
|
||||||
|
search:
|
||||||
|
exclude: true
|
||||||
|
---
|
||||||
|
|
||||||
|
# Spider Classes
|
||||||
|
|
||||||
|
Here's the reference information for the spider framework classes' parameters, attributes, and methods.
|
||||||
|
|
||||||
|
You can import them directly like below:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from scrapling.spiders import Spider, Request, CrawlResult, SessionManager, Response
|
||||||
|
```
|
||||||
|
|
||||||
|
## ::: scrapling.spiders.Spider
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
|
|
||||||
|
## ::: scrapling.spiders.Request
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
|
|
||||||
|
## Result Classes
|
||||||
|
|
||||||
|
## ::: scrapling.spiders.result.CrawlResult
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
|
|
||||||
|
## ::: scrapling.spiders.result.CrawlStats
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
|
|
||||||
|
## ::: scrapling.spiders.result.ItemList
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
|
|
||||||
|
## Session Management
|
||||||
|
|
||||||
|
## ::: scrapling.spiders.session.SessionManager
|
||||||
|
handler: python
|
||||||
|
:docstring:
|
||||||
+4
-1
@@ -54,7 +54,10 @@ nav = [
|
|||||||
{Selector = "api-reference/selector.md"},
|
{Selector = "api-reference/selector.md"},
|
||||||
{Fetchers = "api-reference/fetchers.md"},
|
{Fetchers = "api-reference/fetchers.md"},
|
||||||
{"MCP Server" = "api-reference/mcp-server.md"},
|
{"MCP Server" = "api-reference/mcp-server.md"},
|
||||||
{"Custom Types" = "api-reference/custom-types.md"}
|
{"Custom Types" = "api-reference/custom-types.md"},
|
||||||
|
{Response = "api-reference/response.md"},
|
||||||
|
{Spiders = "api-reference/spiders.md"},
|
||||||
|
{"Proxy Rotation" = "api-reference/proxy-rotation.md"}
|
||||||
]},
|
]},
|
||||||
{"Writing your retrieval system" = "development/adaptive_storage_system.md"},
|
{"Writing your retrieval system" = "development/adaptive_storage_system.md"},
|
||||||
{"Using Scrapling's custom types" = "development/scrapling_custom_types.md"}
|
{"Using Scrapling's custom types" = "development/scrapling_custom_types.md"}
|
||||||
|
|||||||
Reference in New Issue
Block a user