freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: updating overview and benchmarks pages with the new changes

Browse Source
This commit is contained in:
Karim shoair
2025-12-27 20:51:28 +02:00
parent fb3a1b4c17
commit a35cc4bdd6
2 changed files with 23 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/overview.md
+22 -12
View File
@@ -1,4 +1,4 @@
We will start by quickly reviewing the parsing capabilities. Then, we will fetch websites with custom browsers, make requests, and parse the response.
We will start by quickly reviewing the parsing capabilities. Then we will fetch websites using custom browsers, make requests, and parse the responses.
Here's an HTML document generated by ChatGPT that we will be using as an example throughout this page:
```html
@@ -82,7 +82,7 @@ page.get_all_text(ignore_tags=('script', 'style'))
```
## Finding elements
If there's an element you want to find on the page, you will! Your creativity level is the only limitation!
If there's an element you want to find on the page, you will find it! Your creativity level is the only limitation!
Finding the first HTML `section` element
```python
@@ -94,14 +94,14 @@ Find all `section` elements
section_elements = page.find_all('section')
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
```
Find all `section` elements whose `id` attribute value is `products`
Find all `section` elements whose `id` attribute value is `products`.
```python
section_elements = page.find_all('section', {'id':"products"})
# Same as
section_elements = page.find_all('section', id="products")
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
```
Find all `section` elements whose `id` attribute value contains `product`
Find all `section` elements whose `id` attribute value contains `product`.
```python
section_elements = page.find_all('section', {'id*':"product"})
```
@@ -218,7 +218,7 @@ Using the elements we found above
<data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
>>> section_element.siblings
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`.
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
>>> section_element.children.css('h2::text')
['Products']
@@ -237,7 +237,7 @@ You can search for a specific ancestor of an element that satisfies a function;
```
## Fetching websites
Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers.
Instead of passing the raw HTML to Scrapling, you can retrieve a website's response directly via HTTP requests or by fetching it in a browser.
A fetcher is made for every use case.
@@ -267,7 +267,7 @@ For Async requests, you will replace the import like below:
> Notes:
>
> 1. You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from a Google search of this domain. It's enabled by default.
> 2. The `impersonate` argument allows you to fake the TLS fingerprint for a specific version of a browser.
> 2. The `impersonate` argument lets you fake the TLS fingerprint for a specific browser version.
> 3. There's also the `http3` argument, which, when enabled, makes the fetcher use HTTP/3 for requests, which makes your requests more authentic
This is just the tip of the iceberg with this fetcher; check out the rest from [here](fetching/static.md)
@@ -275,7 +275,7 @@ This is just the tip of the iceberg with this fetcher; check out the rest from [
### Dynamic loading
We have you covered if you deal with dynamic websites like most today!
The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides many options to fetch/load websites' pages through browsers.
The `DynamicFetcher` class (formerly `PlayWrightFetcher`) offers many options for fetching and loading web pages using Chromium-based browsers.
```python
>>> from scrapling.fetchers import DynamicFetcher
>>> page = DynamicFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
@@ -286,10 +286,9 @@ The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides ma
>>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
```
It's built on top of [Playwright](https://playwright.dev/python/) and it's currently providing three main run options that can be mixed as you want:
It's built on top of [Playwright](https://playwright.dev/python/), and it's currently providing two main run options that can be mixed as you want:
- Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser.
- Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode, but it bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/). Check out the `StealthyFetcher` class below for more advanced stealth mode. It uses the Chromium browser.
- Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
@@ -298,7 +297,18 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
### Dynamic anti-protection loading
We also have you covered if you deal with dynamic websites with annoying anti-protections!
The `StealthyFetcher` class uses a custom version of a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), bypassing most bot detections by default. Scrapling offers a faster custom version, includes extra tools, and features easy configurations to further increase undetectability.
The `StealthyFetcher` class uses a stealthy version of the `DynamicFetcher` explained above.
Some of the things it does:
1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
6. It makes requests look as if they came from Google's search page of the requested website.
7. and other anti-protection options...
```python
>>> from scrapling.fetchers import StealthyFetcher
>>> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default
@@ -318,4 +328,4 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
---
That's Scrapling at a glance. If you want to learn more about it, continue to the next section.
That's Scrapling at a glance. If you want to learn more, continue to the next section.