docs: updating the Fetcher Basics and the HTTP requests pages
This commit is contained in:
Karim shoair
@@ -3,7 +3,7 @@ Fetchers are classes that can do requests or fetch pages for you easily in a sin
|
|||||||
|
|
||||||
This feature was introduced because, before v0.2, Scrapling was only a parsing engine. The target here is to gradually become the one-stop shop for all Web Scraping needs.
|
This feature was introduced because, before v0.2, Scrapling was only a parsing engine. The target here is to gradually become the one-stop shop for all Web Scraping needs.
|
||||||
|
|
||||||
> Fetchers are not wrappers built on top of other libraries. However, they utilize these libraries as an engine to request/fetch pages easily for you, while fully leveraging that engine and adding features for you. Some fetchers don't even use the official library for requests; instead, they use their own custom version. For example, `StealthyFetcher` utilizes `Camoufox` browser directly, without relying on its Python library for anything except launch options. This last part might change soon as well.
|
> Fetchers are not wrappers built on top of other libraries. However, they only use these libraries as an engine to request/fetch pages. To further clarify this, all fetchers have features that the underlying engines don't, while still fully leveraging those engines and optimizing them for Web Scraping.
|
||||||
|
|
||||||
## Fetchers Overview
|
## Fetchers Overview
|
||||||
|
|
||||||
@@ -13,14 +13,14 @@ The following table compares them and can be quickly used for guidance.
|
|||||||
|
|
||||||
|
|
||||||
| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
|
| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
|
||||||
|--------------------|---------------------------------------------------|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
|
|--------------------|---------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
|
||||||
| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇 |
|
| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇🐇 |
|
||||||
| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
|
| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
|
||||||
| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
|
| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
|
||||||
| JavaScript loading | ❌ | ✅ | ✅ |
|
| JavaScript loading | ❌ | ✅ | ✅ |
|
||||||
| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
|
| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
|
||||||
| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Slight protections | - Dynamically loaded websites <br/>- Small automation <br/>- Complicated protections |
|
| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Small-Mid protections | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
|
||||||
| Browser(s) | ❌ | Chromium and Google Chrome | Modified Firefox |
|
| Browser(s) | ❌ | Chromium and Google Chrome | Chromium and Google Chrome |
|
||||||
| Browser API used | ❌ | PlayWright | PlayWright |
|
| Browser API used | ❌ | PlayWright | PlayWright |
|
||||||
| Setup Complexity | Simple | Simple | Simple |
|
| Setup Complexity | Simple | Simple | Simple |
|
||||||
|
|
||||||
|
|||||||
@@ -25,11 +25,11 @@ Scrapling provides many options with this fetcher and its session classes. Befor
|
|||||||
|
|
||||||
|
|
||||||
| Argument | Description | Optional |
|
| Argument | Description | Optional |
|
||||||
|:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
|
|:-------------------:|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
|
||||||
| url | Target url | ❌ |
|
| url | Target url | ❌ |
|
||||||
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
|
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
|
||||||
| block_images | Prevent the loading of images through Firefox preferences. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ |
|
| block_images | Prevent the loading of images through Firefox preferences. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ |
|
||||||
| disable_resources | Drop requests for unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ |
|
| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
|
||||||
| cookies | Set cookies for the next request. | ✔️ |
|
| cookies | Set cookies for the next request. | ✔️ |
|
||||||
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
|
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
|
||||||
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
|
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
|
||||||
|
|||||||
Reference in New Issue
Block a user