docs: Update all pages related to last changes
This commit is contained in:
Karim shoair
@@ -62,7 +62,7 @@ DynamicFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
|
||||
Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
|
||||
|
||||
## Full list of arguments
|
||||
Scrapling provides many options with this fetcher. To make it as simple as possible, we will list the options here and give examples of using most of them.
|
||||
Scrapling provides many options with this fetcher and its session classes. To make it as simple as possible, we will list the options here and give examples of using most of them.
|
||||
|
||||
| Argument | Description | Optional |
|
||||
|:-------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
|
||||
@@ -90,6 +90,9 @@ Scrapling provides many options with this fetcher. To make it as simple as possi
|
||||
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
|
||||
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
|
||||
|
||||
In the session classes, all these arguments can be set for the session globally. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, and `selector_config`.
|
||||
|
||||
|
||||
## Examples
|
||||
It's easier to understand with examples, so let's take a look.
|
||||
|
||||
|
||||
@@ -15,7 +15,7 @@ Check out how to configure the parsing options [here](choosing.md#parser-configu
|
||||
> Note: The async version of the `fetch` method is the `async_fetch` method, of course.
|
||||
|
||||
## Full list of arguments
|
||||
Before jumping to [examples](#examples), here's the full list of arguments
|
||||
Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments
|
||||
|
||||
|
||||
| Argument | Description | Optional |
|
||||
@@ -47,6 +47,7 @@ Before jumping to [examples](#examples), here's the full list of arguments
|
||||
| additional_args | Additional arguments to be passed to Camoufox as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
|
||||
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
|
||||
|
||||
In the session classes, all these arguments can be set for the session globally. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, and `selector_config`.
|
||||
|
||||
## Examples
|
||||
It's easier to understand with examples, so we will now review most of the arguments individually with examples.
|
||||
|
||||
+24
-20
@@ -114,29 +114,33 @@ Scrapling requires Python 3.10 or higher:
|
||||
pip install scrapling
|
||||
```
|
||||
|
||||
#### Fetchers Setup
|
||||
|
||||
If you are going to use any of the fetchers or their session classes, then install browser dependencies with
|
||||
```bash
|
||||
scrapling install
|
||||
```
|
||||
|
||||
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
|
||||
Starting with v0.3.2, this installation only includes the parser engine and its dependencies, without any fetchers.
|
||||
|
||||
### Optional Dependencies
|
||||
|
||||
- Install the MCP server feature:
|
||||
```bash
|
||||
pip install "scrapling[ai]"
|
||||
```
|
||||
- Install shell features (Web Scraping shell and the `extract` command):
|
||||
```bash
|
||||
pip install "scrapling[shell]"
|
||||
```
|
||||
- Install everything:
|
||||
```bash
|
||||
pip install "scrapling[all]"
|
||||
```
|
||||
1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies, and then install their browser dependencies with
|
||||
```bash
|
||||
pip install "scrapling[fetchers]"
|
||||
|
||||
scrapling install
|
||||
```
|
||||
|
||||
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
|
||||
|
||||
2. Extra features:
|
||||
- Install the MCP server feature:
|
||||
```bash
|
||||
pip install "scrapling[ai]"
|
||||
```
|
||||
- Install shell features (Web Scraping shell and the `extract` command):
|
||||
```bash
|
||||
pip install "scrapling[shell]"
|
||||
```
|
||||
- Install everything:
|
||||
```bash
|
||||
pip install "scrapling[all]"
|
||||
```
|
||||
Don't forget that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)
|
||||
|
||||
## How the documentation is organized
|
||||
Scrapling has a lot of documentation, so we try to follow a guideline called the [Diátaxis documentation framework](https://diataxis.fr/).
|
||||
|
||||
Reference in New Issue
Block a user