First version of Scrapling full documentation
This commit is contained in:
Karim shoair
@@ -0,0 +1,77 @@
|
||||
## Introduction
|
||||
Fetchers are classes that can do requests or fetch pages for you easily in a single-line fashion with many features and then return a [Response](#response-object) object.
|
||||
|
||||
This feature was introduced because the only option before v0.2 was to fetch the page as you wanted, then pass it manually to the `Adaptor` class and start playing with it.
|
||||
|
||||
> Fetchers are not wrappers built on top of other libraries, but they use these libraries as an engine to make requests/fetch pages easily for you while fully utilizing that engine and adding features for you that aren't included in those engines
|
||||
|
||||
## Fetchers Overview
|
||||
|
||||
Scrapling provides three different fetcher classes, each designed for specific use cases.
|
||||
|
||||
The following table compares them and can be quickly used for guidance.
|
||||
|
||||
|
||||
| Feature | Fetcher | PlayWrightFetcher | StealthyFetcher |
|
||||
|--------------------|----------------|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
|
||||
| Relative speed | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ |
|
||||
| Stealth | ⭐ | ⭐⭐ | ⭐⭐⭐⭐ |
|
||||
| Anti-Bot options | ⭐ | ⭐⭐ | ⭐⭐⭐⭐ |
|
||||
| JavaScript loading | ❌ | ✅ | ✅ |
|
||||
| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
|
||||
| Best used for | Basic scraping | - Dynamically loaded websites <br/>- Small automation<br/>- Slight protections | - Dynamically loaded websites <br/>- Small automation <br/>- Complicated protections |
|
||||
| Browser(s) | ❌ | Chromium and Google Chrome | Modified Firefox |
|
||||
| Browser API used | ❌ | PlayWright | PlayWright |
|
||||
| Setup Complexity | Simple | Simple | Simple |
|
||||
|
||||
In the following pages, we will talk about each one in detail.
|
||||
|
||||
## Parser configuration in all fetchers
|
||||
All fetchers classes share the same import, as you will see in the upcoming pages
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
|
||||
```
|
||||
Then you use it right away without initializing like this, and it will use the default parser settings:
|
||||
```python
|
||||
>>> page = StealthyFetcher.fetch('https://example.com')
|
||||
```
|
||||
If you want to configure the parser ([Adaptor class](../parsing/main_classes.md#adaptor)) that will be used on the response before returning it for you, then do this first:
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
>>> Fetcher.configure(auto_match=True, encoding="utf8", keep_comments=False, keep_cdata=False) # and the rest
|
||||
```
|
||||
or
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
>>> Fetcher.auto_match=True
|
||||
>>> Fetcher.encoding="utf8"
|
||||
>>> Fetcher.keep_comments=False
|
||||
>>> Fetcher.keep_cdata=False # and the rest
|
||||
```
|
||||
Then, continue your code as usual.
|
||||
|
||||
The available configuration arguments are: `auto_match`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the `Adaptor` class. You can display the current configuration anytime by running `<fetcher_class>.display_config()`.
|
||||
|
||||
> Note: The `auto_match` argument is disabled by default; you must enable it to use that feature.
|
||||
|
||||
### Set parser config per request
|
||||
As you probably understood, the logic above for setting the parser config will work globally for all requests/fetches done through that class, and it's intended for simplicity.
|
||||
|
||||
If your use case requires a different configuration for each request/fetch, you can pass a dictionary to the request method (`fetch`/`get`/`post`/...) to an argument named `custom_config`.
|
||||
|
||||
## Response Object
|
||||
The `Response` object is the same as the [Adaptor](../parsing/main_classes.md#adaptor) class, but it has added details about the response like response headers, status, cookies, etc... as shown below:
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
>>> page = Fetcher.get('https://example.com')
|
||||
|
||||
>>> page.status # HTTP status code
|
||||
>>> page.reason # Status message
|
||||
>>> page.cookies # Response cookies as a dictionary
|
||||
>>> page.headers # Response headers
|
||||
>>> page.request_headers # Request headers
|
||||
>>> page.history # Response history of redirections, if any
|
||||
>>> page.body # Raw response body
|
||||
>>> page.encoding # Response encoding
|
||||
```
|
||||
All fetchers return the `Response` object.
|
||||
@@ -0,0 +1,248 @@
|
||||
# Introduction
|
||||
|
||||
Here, we will discuss the `PlayWrightFetcher` class. This class provides flexible browser automation with multiple configuration options and some stealth capabilities. It uses [PlayWright](https://playwright.dev/python/docs/intro) as an engine for fetching websites.
|
||||
|
||||
As we will explain later, to automate the page, you need some knowledge of [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page).
|
||||
|
||||
## Basic Usage
|
||||
You have one primary way to import this Fetcher, which is the same for all fetchers.
|
||||
|
||||
```python
|
||||
>>> from scrapling.fetchers import PlayWrightFetcher
|
||||
```
|
||||
Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
|
||||
|
||||
Now we will go over most of the arguments one by one with examples if you want to jump to a table of all arguments for quick reference [click here](#full-list-of-arguments)
|
||||
|
||||
> Notes:
|
||||
>
|
||||
> 1. Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the `domcontentloaded` state).
|
||||
> 2. Of course, the async version of the `fetch` method is the `async_fetch` method.
|
||||
|
||||
|
||||
This fetcher currently provides 4 main run options, but they can be mixed as you want.
|
||||
|
||||
Which are:
|
||||
|
||||
### 1. Vanilla Playwright
|
||||
```python
|
||||
PlayWrightFetcher.fetch('https://example.com')
|
||||
```
|
||||
Using it like that will open a Chromium browser and fetch the page. There are no tricks or extra features; it's just a plain PlayWright API.
|
||||
|
||||
### 2. Stealth Mode
|
||||
```python
|
||||
PlayWrightFetcher.fetch('https://example.com', stealth=True)
|
||||
```
|
||||
It's the same as the vanilla PlayWright option, but it provides a simple stealth mode suitable for websites with a small-to-medium protection layer(s).
|
||||
|
||||
Some of the things this fetcher's stealth mode does include:
|
||||
|
||||
* Patching the CDP runtime fingerprint.
|
||||
* Mimics some of the real browsers' properties by injecting several JS files and using custom options.
|
||||
* Custom flags are used on launch to hide Playwright even more and make it faster.
|
||||
* Generates real browser headers of the same type and user OS, then append them to the request's headers.
|
||||
|
||||
### 3. Real Chrome
|
||||
```python
|
||||
PlayWrightFetcher.fetch('https://example.com', real_chrome=True)
|
||||
```
|
||||
If you have a Google Chrome browser installed, use this option. It's the same as the first option but will use the Google Chrome browser you installed on your device instead of Chromium.
|
||||
|
||||
This will make your requests look more like requests coming from an actual human, so it's less detectable, and you can even use the `stealth=True` mode with it for better results like below:
|
||||
```python
|
||||
PlayWrightFetcher.fetch('https://example.com', real_chrome=True, stealth=True)
|
||||
```
|
||||
If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
|
||||
```commandline
|
||||
playwright install chrome
|
||||
```
|
||||
|
||||
### 4. CDP Connection
|
||||
```python
|
||||
PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
|
||||
```
|
||||
Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
|
||||
|
||||
This fetcher takes it even a step further. You can use [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5)'s [docker browserless](https://hub.docker.com/r/nstbrowser/browserless) option by passing the CDP URL and enabling `nstbrowser_mode` option like below
|
||||
```python
|
||||
PlayWrightFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222', nstbrowser_mode=True)
|
||||
```
|
||||
There's also a `nstbrowser_config` argument to send the config you want to send with the requests to the NSTBrowser. If you leave it empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config.
|
||||
|
||||
## Full list of arguments
|
||||
Scrapling provides many options with this fetcher, which works in all modes except the [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5) mode. To make it as simple as possible, we will list the options here and give examples of using most of them.
|
||||
|
||||
| Argument | Description | Optional |
|
||||
|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
|
||||
| url | Target url | ❌ |
|
||||
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
|
||||
| disable_resources | Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading._ | ✔️ |
|
||||
| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser.** | ✔️ |
|
||||
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
|
||||
| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30000. | ✔️ |
|
||||
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
|
||||
| page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation, then returns `page` again. | ✔️ |
|
||||
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
|
||||
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
|
||||
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name. | ✔️ |
|
||||
| extra_headers | A dictionary of extra headers to add to the request. The referer set by the `google_search` argument takes priority over the referer set here if used together. | ✔️ |
|
||||
| proxy | The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only. | ✔️ |
|
||||
| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
|
||||
| disable_webgl | Disables WebGL and WebGL 2.0 support entirely. | ✔️ |
|
||||
| stealth | Enables stealth mode; you should always check the documentation to see what stealth mode does currently. | ✔️ |
|
||||
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser and use it. | ✔️ |
|
||||
| locale | Set the locale for the browser if wanted. The default value is `en-US`. | ✔️ |
|
||||
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP. | ✔️ |
|
||||
| nstbrowser_mode | Enables NSTBrowser mode, **it have to be used with `cdp_url` argument or it will get completely ignored.** | ✔️ |
|
||||
| nstbrowser_config | The config you want to send with requests to the NSTBrowser. _Scrapling defaults to an optimized NSTBrowser's docker browserless config if you leave this argument empty._ | ✔️ |
|
||||
|
||||
|
||||
## Examples
|
||||
It's easier to understand with examples, so let's look at it.
|
||||
|
||||
### Resource Control
|
||||
|
||||
```python
|
||||
# Disable unnecessary resources
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com',
|
||||
disable_resources=True # Blocks fonts, images, media, etc...
|
||||
)
|
||||
```
|
||||
|
||||
### Network Control
|
||||
|
||||
```python
|
||||
# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
|
||||
page = PlayWrightFetcher.fetch('https://example.com', network_idle=True)
|
||||
|
||||
# Custom timeout (in milliseconds)
|
||||
page = PlayWrightFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
|
||||
|
||||
# Proxy support
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com',
|
||||
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
|
||||
)
|
||||
```
|
||||
|
||||
### Browser Automation
|
||||
This is where your knowledge about [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.
|
||||
|
||||
This function is executed right after waiting for network_idle (if enabled) and before waiting for the `wait_selector` argument, so it can be used for many things, not just automation. You can alter the page as you want.
|
||||
|
||||
In the example below, I used page [mouse events](https://playwright.dev/python/docs/api/class-mouse) to move the mouse wheel to scroll the page and then move the mouse.
|
||||
```python
|
||||
from playwright.sync_api import Page
|
||||
|
||||
def scroll_page(page: Page):
|
||||
page.mouse.wheel(10, 0)
|
||||
page.mouse.move(100, 400)
|
||||
page.mouse.up()
|
||||
return page
|
||||
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com',
|
||||
page_action=scroll_page
|
||||
)
|
||||
```
|
||||
Of course, if you use the async fetch version, the function must also be async.
|
||||
```python
|
||||
from playwright.async_api import Page
|
||||
|
||||
async def scroll_page(page: Page):
|
||||
await page.mouse.wheel(10, 0)
|
||||
await page.mouse.move(100, 400)
|
||||
await page.mouse.up()
|
||||
return page
|
||||
|
||||
page = await PlayWrightFetcher.async_fetch(
|
||||
'https://example.com',
|
||||
page_action=scroll_page
|
||||
)
|
||||
```
|
||||
|
||||
### Wait Conditions
|
||||
|
||||
```python
|
||||
# Wait for the selector
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com',
|
||||
wait_selector='h1',
|
||||
wait_selector_state='visible'
|
||||
)
|
||||
```
|
||||
This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the `wait_selector` argument, and the fetcher will wait for the state you passed in the `wait_selector_state` argument to be fulfilled. If you didn't pass a state, the default would be `attached`, which means it will wait for the element to be present in the DOM.
|
||||
|
||||
After that, the fetcher will check again to see if all JS files are loaded and executed (the `domcontentloaded` state) and wait for them to be. If you have enabled `network_idle` with this, the fetcher will wait for `network_idle` to be fulfilled again, as explained above.
|
||||
|
||||
The states the fetcher can wait for can be either ([source](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)):
|
||||
|
||||
- `attached`: Wait for an element to be present in DOM.
|
||||
- `detached`: Wait for an element to not be present in DOM.
|
||||
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
|
||||
- `hidden`: wait for an element to be either detached from DOM, or have an empty bounding box or `visibility:hidden`. This is opposite to the `'visible'` option.
|
||||
|
||||
### Some Stealth Features
|
||||
|
||||
```python
|
||||
# Full stealth mode
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com',
|
||||
stealth=True,
|
||||
hide_canvas=True,
|
||||
disable_webgl=True,
|
||||
google_search=True
|
||||
)
|
||||
|
||||
# Custom user agent
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com',
|
||||
useragent='Mozilla/5.0...'
|
||||
)
|
||||
|
||||
# Set browser locale
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com',
|
||||
locale='en-US'
|
||||
)
|
||||
```
|
||||
Hence, the `hide_canvas` argument doesn't disable canvas but hides it by adding random noise to canvas operations to prevent fingerprinting. Also, if you didn't set a useragent (preferred), the fetcher will generate a real Useragent of the same browser and use it.
|
||||
|
||||
The `google_search` argument is enabled by default, making the request look like it came from Google. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
|
||||
|
||||
### General example
|
||||
```python
|
||||
from scrapling.fetchers import PlayWrightFetcher
|
||||
|
||||
def scrape_dynamic_content():
|
||||
# Use PlayWright for JavaScript content
|
||||
page = PlayWrightFetcher.fetch(
|
||||
'https://example.com/dynamic',
|
||||
network_idle=True,
|
||||
wait_selector='.content'
|
||||
)
|
||||
|
||||
# Extract dynamic content
|
||||
content = page.css('.content')
|
||||
|
||||
return {
|
||||
'title': content.css_first('h1::text'),
|
||||
'items': [
|
||||
item.text for item in content.css('.item')
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## When to Use
|
||||
|
||||
Use PlayWrightFetcher when:
|
||||
|
||||
- Need browser automation
|
||||
- Want multiple browser options
|
||||
- Using a real Chrome browser
|
||||
- Need custom browser config
|
||||
- Want flexible stealth options
|
||||
|
||||
If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
|
||||
@@ -0,0 +1,300 @@
|
||||
# Introduction
|
||||
|
||||
The `Fetcher` class provides fast and lightweight HTTP requests with some stealth capabilities. This class uses [httpx](https://www.python-httpx.org/) as an engine for making requests. For advanced usages, you will need some knowledge about [httpx](https://www.python-httpx.org/), but it becomes simpler and simpler with user feedback and updates.
|
||||
|
||||
## Basic Usage
|
||||
You have one primary way to import this Fetcher, which is the same for all fetchers.
|
||||
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
```
|
||||
Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
|
||||
|
||||
### Shared arguments
|
||||
All methods for making requests here share some arguments, so let's discuss them first.
|
||||
|
||||
- **url**: The URL you want to request, of course :)
|
||||
- **proxy**: As the name implies, the proxy for this request is used to route all traffic (HTTP and HTTPS). The format accepted here is `http://username:password@localhost:8030`.
|
||||
- **stealthy_headers**: Generate and use real browser's headers, then create a referer header as if this request came from a Google search page of this URL's domain. Enabled by default, all headers generated can be overwritten by you through the `headers` argument.
|
||||
- **follow_redirects**: As the name implies, tell the fetcher to follow redirections. Enabled by default
|
||||
- **timeout**: The timeout to wait for each request to be finished in milliseconds. The default is 30000ms (30 seconds).
|
||||
- **retries**: The number of retries that [httpx](https://www.python-httpx.org/) will do for failed requests. The default number of retries is 3.
|
||||
|
||||
Other than this, you can pass any arguments that `httpx.<method_name>` takes, and that's why I said, in the beginning, you need a bit of knowledge about [httpx](https://www.python-httpx.org/), but in the following examples, we will try to cover most cases.
|
||||
|
||||
### HTTP Methods
|
||||
Examples are the best way to explain this
|
||||
|
||||
> Hence: `OPTIONS` and `HEAD` methods are not supported.
|
||||
#### GET
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
>>> # Basic GET
|
||||
>>> page = Fetcher.get('https://example.com')
|
||||
>>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = Fetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')
|
||||
>>> # With parameters
|
||||
>>> page = Fetcher.get('https://example.com/search', params={'q': 'query'})
|
||||
>>>
|
||||
>>> # With headers
|
||||
>>> page = Fetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
|
||||
>>> # Basic HTTP authentication
|
||||
>>> page = Fetcher.get("https://example.com", auth=("my_user", "password123"))
|
||||
```
|
||||
And for asynchronous requests, it's a small adjustment
|
||||
```python
|
||||
>>> from scrapling.fetchers import AsyncFetcher
|
||||
>>> # Basic GET
|
||||
>>> page = await AsyncFetcher.get('https://example.com')
|
||||
>>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = await AsyncFetcher.get('https://httpbin.org/get', proxy='http://username:password@localhost:8030')
|
||||
>>> # With parameters
|
||||
>>> page = await AsyncFetcher.get('https://example.com/search', params={'q': 'query'})
|
||||
>>>
|
||||
>>> # With headers
|
||||
>>> page = await AsyncFetcher.get('https://example.com', headers={'User-Agent': 'Custom/1.0'})
|
||||
>>> # Basic HTTP authentication
|
||||
>>> page = await AsyncFetcher.get("https://example.com", auth=("my_user", "password123"))
|
||||
```
|
||||
Needless to say, the `page` object in all cases is [Response](choosing.md#response-object) object, which is an `Adaptor` as we said, so you will use it directly
|
||||
```python
|
||||
>>> page.css('.something.something')
|
||||
|
||||
>>> page = Fetcher.get('https://api.github.com/events')
|
||||
>>> page.json()
|
||||
[{'id': '<redacted>',
|
||||
'type': 'PushEvent',
|
||||
'actor': {'id': '<redacted>',
|
||||
'login': '<redacted>',
|
||||
'display_login': '<redacted>',
|
||||
'gravatar_id': '',
|
||||
'url': 'https://api.github.com/users/<redacted>',
|
||||
'avatar_url': 'https://avatars.githubusercontent.com/u/<redacted>'},
|
||||
'repo': {'id': '<redacted>',
|
||||
...
|
||||
```
|
||||
#### POST
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
>>> # Basic POST
|
||||
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'})
|
||||
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
|
||||
>>> # Another example of form-encoded data
|
||||
>>> page = Fetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})
|
||||
>>> # JSON data
|
||||
>>> page = Fetcher.post('https://example.com/api', json={'key': 'value'})
|
||||
>>> # Uploading file
|
||||
>>> r = Fetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')})
|
||||
```
|
||||
And for asynchronous requests, it's a small adjustment
|
||||
```python
|
||||
>>> from scrapling.fetchers import AsyncFetcher
|
||||
>>> # Basic POST
|
||||
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'})
|
||||
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
|
||||
>>> # Another example of form-encoded data
|
||||
>>> page = await AsyncFetcher.post('https://example.com/submit', data={'username': 'user', 'password': 'pass'})
|
||||
>>> # JSON data
|
||||
>>> page = await AsyncFetcher.post('https://example.com/api', json={'key': 'value'})
|
||||
>>> # Uploading file
|
||||
>>> r = await AsyncFetcher.post("https://httpbin.org/post", files={'upload-file': open('something.xlsx', 'rb')})
|
||||
```
|
||||
#### PUT
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
>>> # Basic PUT
|
||||
>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'})
|
||||
>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = Fetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
|
||||
>>> # Another example of form-encoded data
|
||||
>>> page = Fetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})
|
||||
```
|
||||
And for asynchronous requests, it's a small adjustment
|
||||
```python
|
||||
>>> from scrapling.fetchers import AsyncFetcher
|
||||
>>> # Basic PUT
|
||||
>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'})
|
||||
>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = await AsyncFetcher.put('https://example.com/update', data={'status': 'updated'}, proxy='http://username:password@localhost:8030')
|
||||
>>> # Another example of form-encoded data
|
||||
>>> page = await AsyncFetcher.put("https://httpbin.org/put", data={'key': ['value1', 'value2']})
|
||||
```
|
||||
|
||||
#### DELETE
|
||||
```python
|
||||
>>> from scrapling.fetchers import Fetcher
|
||||
>>> page = Fetcher.delete('https://example.com/resource/123')
|
||||
>>> page = Fetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = Fetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
|
||||
```
|
||||
And for asynchronous requests, it's a small adjustment
|
||||
```python
|
||||
>>> from scrapling.fetchers import AsyncFetcher
|
||||
>>> page = await AsyncFetcher.delete('https://example.com/resource/123')
|
||||
>>> page = await AsyncFetcher.delete('https://example.com/resource/123', stealthy_headers=True, follow_redirects=True)
|
||||
>>> page = await AsyncFetcher.delete('https://example.com/resource/123', proxy='http://username:password@localhost:8030')
|
||||
```
|
||||
|
||||
## Examples
|
||||
Some well-rounded examples to aid newcomers to Web Scraping
|
||||
|
||||
### Basic HTTP Request
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher
|
||||
|
||||
# Make a request
|
||||
page = Fetcher.get('https://example.com')
|
||||
|
||||
# Check the status
|
||||
if page.status == 200:
|
||||
# Extract title
|
||||
title = page.css_first('title::text')
|
||||
print(f"Page title: {title}")
|
||||
|
||||
# Extract all links
|
||||
links = page.css('a::attr(href)')
|
||||
print(f"Found {len(links)} links")
|
||||
```
|
||||
|
||||
### Product Scraping
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher
|
||||
|
||||
def scrape_products():
|
||||
page = Fetcher.get('https://example.com/products')
|
||||
|
||||
# Find all product elements
|
||||
products = page.css('.product')
|
||||
|
||||
results = []
|
||||
for product in products:
|
||||
results.append({
|
||||
'title': product.css_first('.title::text'),
|
||||
'price': product.css_first('.price::text').re_first(r'\d+\.\d{2}'),
|
||||
'description': product.css_first('.description::text'),
|
||||
'in_stock': product.has_class('in-stock')
|
||||
})
|
||||
|
||||
return results
|
||||
```
|
||||
|
||||
### Pagination Handling
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher
|
||||
|
||||
def scrape_all_pages():
|
||||
base_url = 'https://example.com/products?page={}'
|
||||
page_num = 1
|
||||
all_products = []
|
||||
|
||||
while True:
|
||||
# Get current page
|
||||
page = Fetcher.get(base_url.format(page_num))
|
||||
|
||||
# Find products
|
||||
products = page.css('.product')
|
||||
if not products:
|
||||
break
|
||||
|
||||
# Process products
|
||||
for product in products:
|
||||
all_products.append({
|
||||
'name': product.css_first('.name::text'),
|
||||
'price': product.css_first('.price::text')
|
||||
})
|
||||
|
||||
# Next page
|
||||
page_num += 1
|
||||
|
||||
return all_products
|
||||
```
|
||||
|
||||
### Form Submission
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher
|
||||
|
||||
# Submit login form
|
||||
response = Fetcher.post(
|
||||
'https://example.com/login',
|
||||
data={
|
||||
'username': 'user@example.com',
|
||||
'password': 'password123'
|
||||
}
|
||||
)
|
||||
|
||||
# Check login success
|
||||
if response.status == 200:
|
||||
# Extract user info
|
||||
user_name = response.css_first('.user-name::text')
|
||||
print(f"Logged in as: {user_name}")
|
||||
```
|
||||
|
||||
### Table Extraction
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher
|
||||
|
||||
def extract_table():
|
||||
page = Fetcher.get('https://example.com/data')
|
||||
|
||||
# Find table
|
||||
table = page.css_first('table')
|
||||
|
||||
# Extract headers
|
||||
headers = [
|
||||
th.text for th in table.css('thead th')
|
||||
]
|
||||
|
||||
# Extract rows
|
||||
rows = []
|
||||
for row in table.css('tbody tr'):
|
||||
cells = [td.text for td in row.css('td')]
|
||||
rows.append(dict(zip(headers, cells)))
|
||||
|
||||
return rows
|
||||
```
|
||||
|
||||
### Navigation Menu
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher
|
||||
|
||||
def extract_menu():
|
||||
page = Fetcher.get('https://example.com')
|
||||
|
||||
# Find navigation
|
||||
nav = page.css_first('nav')
|
||||
|
||||
menu = {}
|
||||
for item in nav.css('li'):
|
||||
link = item.css_first('a')
|
||||
if link:
|
||||
menu[link.text] = {
|
||||
'url': link.attrib['href'],
|
||||
'has_submenu': bool(item.css('.submenu'))
|
||||
}
|
||||
|
||||
return menu
|
||||
```
|
||||
|
||||
## When to Use
|
||||
|
||||
Use `Fetcher` when:
|
||||
|
||||
- Need fast HTTP requests
|
||||
- Want minimal overhead
|
||||
- Don't need JavaScript
|
||||
- Want simple configuration
|
||||
- Need basic stealth features
|
||||
|
||||
Use other fetchers when:
|
||||
|
||||
- Need browser automation.
|
||||
- Need advanced anti-bot/stealth.
|
||||
- Need JavaScript support.
|
||||
@@ -0,0 +1,218 @@
|
||||
# Introduction
|
||||
|
||||
Here, we will discuss the `StealthyFetcher` class. This class is similar to [PlayWrightFetcher](dynamic.md#introduction) in many ways, like browser automation and using [PlayWright](https://playwright.dev/python/docs/intro) as an engine for fetching websites. The main difference is that this class provides advanced anti-bot protection bypass capabilities and a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), from which most stealth comes.
|
||||
|
||||
As with [PlayWrightFetcher](dynamic.md#introduction), you will need some knowledge about [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page) to automate the page, as we will explain later.
|
||||
|
||||
## Basic Usage
|
||||
You have one primary way to import this Fetcher, which is the same for all fetchers.
|
||||
|
||||
```python
|
||||
>>> from scrapling.fetchers import StealthyFetcher
|
||||
```
|
||||
Check out how to configure the parsing options [here](choosing.md#parser-configuration-in-all-fetchers)
|
||||
|
||||
> Notes:
|
||||
>
|
||||
> 1. Every time you fetch a website with this fetcher, it waits by default for all JavaScript to fully load and execute, so you don't have to (waits for the `domcontentloaded` state).
|
||||
> 2. Of course, the async version of the `fetch` method is the `async_fetch` method.
|
||||
|
||||
## Full list of arguments
|
||||
Before jumping to [examples](#examples), here's the full list of arguments
|
||||
|
||||
|
||||
| Argument | Description | Optional |
|
||||
|:--------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
|
||||
| url | Target url | ❌ |
|
||||
| headless | Pass `True` to run the browser in headless/hidden (**default**), `virtual` to run it in virtual screen mode, or `False` for headful/visible mode. The `virtual` mode requires having `xvfb` installed. | ✔️ |
|
||||
| block_images | Prevent the loading of images through Firefox preferences. _This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading._ | ✔️ |
|
||||
| disable_resources | Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be careful with this option as it makes some websites never finish loading._ | ✔️ |
|
||||
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search for this website's domain name. | ✔️ |
|
||||
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
|
||||
| block_webrtc | Blocks WebRTC entirely. | ✔️ |
|
||||
| page_action | Added for automation. A function that takes the `page` object and does the automation you need, then returns `page` again. | ✔️ |
|
||||
| addons | List of Firefox addons to use. **Must be paths to extracted addons.** | ✔️ |
|
||||
| humanize | Humanize the cursor movement. The cursor movement takes either True or the MAX duration in seconds. The cursor typically takes up to 1.5 seconds to move across the window. | ✔️ |
|
||||
| allow_webgl | Enabled by default. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
|
||||
| geoip | Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. | ✔️ |
|
||||
| os_randomize | If enabled, Scrapling will randomize the OS fingerprints used. The default is matching the fingerprints with the current OS. | ✔️ |
|
||||
| disable_ads | Disabled by default; this installs the `uBlock Origin` addon on the browser if enabled. | ✔️ |
|
||||
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
|
||||
| timeout | The timeout used in all operations and waits through the page. It's in milliseconds, and the default is 30000. | ✔️ |
|
||||
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
|
||||
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
|
||||
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
|
||||
| proxy | The proxy to be used with requests. It can be a string or a dictionary with the keys 'server', 'username', and 'password' only. | ✔️ |
|
||||
| additional_arguments | Arguments passed to Camoufox as additional settings that take higher priority than Scrapling's. | ✔️ |
|
||||
|
||||
|
||||
## Examples
|
||||
It's easier to understand with examples, so now we will go over most of the arguments individually with examples.
|
||||
|
||||
### Browser Modes
|
||||
|
||||
```python
|
||||
# Headless/hidden mode (default)
|
||||
page = StealthyFetcher.fetch('https://example.com', headless=True)
|
||||
|
||||
# Virtual display mode (requires having `xvfb` installed)
|
||||
page = StealthyFetcher.fetch('https://example.com', headless='virtual')
|
||||
|
||||
# Visible browser mode
|
||||
page = StealthyFetcher.fetch('https://example.com', headless=False)
|
||||
```
|
||||
|
||||
### Resource Control
|
||||
|
||||
```python
|
||||
# Block images
|
||||
page = StealthyFetcher.fetch('https://example.com', block_images=True)
|
||||
|
||||
# Disable unnecessary resources
|
||||
page = StealthyFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
|
||||
```
|
||||
|
||||
### Additional stealth options
|
||||
|
||||
```python
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://example.com',
|
||||
block_webrtc=True, # Block WebRTC
|
||||
allow_webgl=False, # Disable WebGL
|
||||
humanize=True, # Make the mouse move as how a human would move it
|
||||
geoip=True, # Use IP's longitude, latitude, timezone, country, and locale, then spoof the WebRTC IP address...
|
||||
os_randomize=True, # Randomize the OS fingerprints used. The default is matching the fingerprints with the current OS.
|
||||
disable_ads=True, # Block ads with uBlock Origin addon (enabled by default)
|
||||
google_search=True
|
||||
)
|
||||
|
||||
# Custom user agent
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://example.com',
|
||||
useragent='Mozilla/5.0...'
|
||||
)
|
||||
|
||||
# Custom humanization duration
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://example.com',
|
||||
humanize=1.5 # Max 1.5 seconds for cursor movement
|
||||
)
|
||||
```
|
||||
|
||||
The `google_search` argument is enabled by default. It makes the request as if it came from Google, so for a request for `https://example.com`, it will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
|
||||
|
||||
### Network Control
|
||||
|
||||
```python
|
||||
# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
|
||||
page = StealthyFetcher.fetch('https://example.com', network_idle=True)
|
||||
|
||||
# Custom timeout (in milliseconds)
|
||||
page = StealthyFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
|
||||
|
||||
# Proxy support
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://example.com',
|
||||
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
|
||||
)
|
||||
```
|
||||
|
||||
### Browser Automation
|
||||
This is where your knowledge about [PlayWright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, does what you want, and then returns it again for the current fetcher to continue working on it.
|
||||
|
||||
This function is executed right after waiting for `network_idle` (if enabled) and before waiting for the `wait_selector` argument, so it can be used for many things, not just automation. You can alter the page as you want.
|
||||
|
||||
In the example below, I used page [mouse events](https://playwright.dev/python/docs/api/class-mouse) to move the mouse wheel to scroll the page and then move the mouse.
|
||||
```python
|
||||
from playwright.sync_api import Page
|
||||
|
||||
def scroll_page(page: Page):
|
||||
page.mouse.wheel(10, 0)
|
||||
page.mouse.move(100, 400)
|
||||
page.mouse.up()
|
||||
return page
|
||||
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://example.com',
|
||||
page_action=scroll_page
|
||||
)
|
||||
```
|
||||
Of course, if you use the async fetch version, the function must also be async.
|
||||
```python
|
||||
from playwright.async_api import Page
|
||||
|
||||
async def scroll_page(page: Page):
|
||||
await page.mouse.wheel(10, 0)
|
||||
await page.mouse.move(100, 400)
|
||||
await page.mouse.up()
|
||||
return page
|
||||
|
||||
page = await StealthyFetcher.async_fetch(
|
||||
'https://example.com',
|
||||
page_action=scroll_page
|
||||
)
|
||||
```
|
||||
|
||||
### Wait Conditions
|
||||
```python
|
||||
# Wait for the selector
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://example.com',
|
||||
wait_selector='h1',
|
||||
wait_selector_state='visible'
|
||||
)
|
||||
```
|
||||
This is the last wait the fetcher will do before returning the response (if enabled). You pass a CSS selector to the `wait_selector` argument, and the fetcher will wait for the state you passed in the `wait_selector_state` argument to be fulfilled. If you didn't pass a state, the default would be `attached`, which means it will wait for the element to be present in the DOM.
|
||||
|
||||
After that, the fetcher will check again to see if all JS files are loaded and executed (the `domcontentloaded` state) and wait for them to be. If you have enabled `network_idle` with this, the fetcher will wait for `network_idle` to be fulfilled again, as explained above.
|
||||
|
||||
The states the fetcher can wait for can be either ([source](https://playwright.dev/python/docs/api/class-page#page-wait-for-selector)):
|
||||
|
||||
- `attached`: wait for the element to be present in DOM.
|
||||
- `detached`: wait for the element to not be present in DOM.
|
||||
- `visible`: wait for the element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
|
||||
- `hidden`: Wait for the element to be detached from DOM, have an empty bounding box, or have `visibility:hidden`. This is opposite to the `'visible'` option.
|
||||
|
||||
### Firefox Addons
|
||||
|
||||
```python
|
||||
# Custom Firefox addons
|
||||
page = StealthyFetcher.fetch(
|
||||
'https://example.com',
|
||||
addons=['/path/to/addon1', '/path/to/addon2']
|
||||
)
|
||||
```
|
||||
The paths here must be paths of extracted addons, which will be installed automatically upon browser launch.
|
||||
|
||||
### Real-world example (Amazon)
|
||||
This is for educational purposes only; this example was generated by AI, which shows too how easy it is to work with Scrapling through AI
|
||||
```python
|
||||
def scrape_amazon_product(url):
|
||||
# Use StealthyFetcher to bypass protection
|
||||
page = StealthyFetcher.fetch(url)
|
||||
|
||||
# Extract product details
|
||||
return {
|
||||
'title': page.css_first('#productTitle::text').clean(),
|
||||
'price': page.css_first('.a-price .a-offscreen::text'),
|
||||
'rating': page.css_first('[data-feature-name="averageCustomerReviews"] .a-popover-trigger .a-color-base::text'),
|
||||
'reviews_count': page.css('#acrCustomerReviewText::text').re_first(r'[\d,]+'),
|
||||
'features': [
|
||||
li.clean() for li in page.css('#feature-bullets li span::text')
|
||||
],
|
||||
'availability': page.css_first('#availability').get_all_text(strip=True),
|
||||
'images': [
|
||||
img.attrib['src'] for img in page.css('#altImages img')
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## When to Use
|
||||
|
||||
Use StealthyFetcher when:
|
||||
|
||||
- Bypassing anti-bot protection
|
||||
- Need a reliable browser fingerprint
|
||||
- Full JavaScript support needed
|
||||
- Want automatic stealth features
|
||||
- Need browser automation
|
||||
Reference in New Issue
Block a user