docs: Style updates and a lot of clarifications
This commit is contained in:
@@ -4,6 +4,12 @@ Here, we will discuss the `DynamicFetcher` class (previously known as `PlayWrigh
|
||||
|
||||
As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
|
||||
|
||||
> 💡 **Prerequisites:**
|
||||
>
|
||||
> 1. You’ve completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
|
||||
> 2. You’ve completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
|
||||
> 3. You’ve completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
|
||||
|
||||
## Basic Usage
|
||||
You have one primary way to import this Fetcher, which is the same for all fetchers.
|
||||
|
||||
@@ -275,7 +281,7 @@ async def scrape_multiple_sites():
|
||||
return pages
|
||||
```
|
||||
|
||||
You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
|
||||
You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
|
||||
|
||||
1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
|
||||
2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
|
||||
@@ -301,4 +307,4 @@ Use DynamicFetcher when:
|
||||
- Need custom browser config
|
||||
- Want flexible stealth options
|
||||
|
||||
If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
|
||||
If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
|
||||
@@ -2,6 +2,12 @@
|
||||
|
||||
The `Fetcher` class provides rapid and lightweight HTTP requests using the high-performance `curl_cffi` library with a lot of stealth capabilities.
|
||||
|
||||
> 💡 **Prerequisites:**
|
||||
>
|
||||
> 1. You’ve completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
|
||||
> 2. You’ve completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
|
||||
> 3. You’ve completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
|
||||
|
||||
## Basic Usage
|
||||
You have one primary way to import this Fetcher, which is the same for all fetchers.
|
||||
|
||||
|
||||
@@ -4,6 +4,12 @@ Here, we will discuss the `StealthyFetcher` class. This class is similar to [Dyn
|
||||
|
||||
As with [DynamicFetcher](dynamic.md#introduction), you will need some knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) to automate the page, as we will explain later.
|
||||
|
||||
> 💡 **Prerequisites:**
|
||||
>
|
||||
> 1. You’ve completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
|
||||
> 2. You’ve completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
|
||||
> 3. You’ve completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
|
||||
|
||||
## Basic Usage
|
||||
You have one primary way to import this Fetcher, which is the same for all fetchers.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user