freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Update README reflect current version accurately

Browse Source
This commit is contained in:
Karim shoair
2025-03-26 19:25:49 +02:00
parent 811cd2d9b0
commit 8551d7eeea
1 changed files with 51 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+34 -33
View File
@@ -7,6 +7,7 @@ Scrapling is a high-performance, intelligent web scraping library for Python tha
```python ```python
>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher >> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
>> StealthyFetcher.auto_match = True
# Fetch websites' source under the radar! # Fetch websites' source under the radar!
>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) >> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
>> print(page.status) >> print(page.status)
@@ -50,30 +51,33 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL
--- ---
## Table of content ## Table of content
* [Key Features](#key-features) * [Key Features](#key-features)
* [Fetch websites as you prefer](#fetch-websites-as-you-prefer-with-async-support) * [Fetch websites as you prefer with async support](#fetch-websites-as-you-prefer-with-async-support)
* [Adaptive Scraping](#adaptive-scraping) * [Adaptive Scraping](#adaptive-scraping)
* [Performance](#performance) * [High Performance](#high-performance)
* [Developing Experience](#developing-experience) * [Developer Friendly](#developer-friendly)
* [Getting Started](#getting-started) * [Getting Started](#getting-started)
* [Parsing Performance](#parsing-performance) * [Parsing Performance](#parsing-performance)
* [Text Extraction Speed Test (5000 nested elements).](#text-extraction-speed-test-5000-nested-elements) * [Text Extraction Speed Test (5000 nested elements).](#text-extraction-speed-test-5000-nested-elements)
* [Extraction By Text Speed Test](#extraction-by-text-speed-test) * [Extraction By Text Speed Test](#extraction-by-text-speed-test)
* [Installation](#installation) * [Installation](#installation)
* [Fetching Websites](#fetching-websites) * [Fetching Websites](#fetching-websites)
* [Features](#features) * [Features](#features)
* [Fetcher class](#fetcher) * [Set parser config per request](#set-parser-config-per-request)
* [StealthyFetcher class](#stealthyfetcher) * [Fetcher](#fetcher)
* [PlayWrightFetcher class](#playwrightfetcher) * [StealthyFetcher](#stealthyfetcher)
* [Advanced Parsing Features](#advanced-parsing-features) * [The complete list of arguments](#the-complete-list-of-arguments)
* [PlayWrightFetcher](#playwrightfetcher)
* [The complete list of arguments](#the-complete-list-of-arguments-1)
* [Advanced Parsing Features](#advanced-parsing-features)
* [Smart Navigation](#smart-navigation) * [Smart Navigation](#smart-navigation)
* [Content-based Selection & Finding Similar Elements](#content-based-selection--finding-similar-elements) * [Content-based Selection & Finding Similar Elements](#content-based-selection--finding-similar-elements)
* [Handling Structural Changes](#handling-structural-changes) * [Handling Structural Changes](#handling-structural-changes)
* [Real World Scenario](#real-world-scenario) * [Real-World Scenario](#real-world-scenario)
* [Find elements by filters](#find-elements-by-filters) * [Find elements by filters](#find-elements-by-filters)
* [Is That All?](#is-that-all) * [Is That All?](#is-that-all)
* [More Advanced Usage](#more-advanced-usage) * [More Advanced Usage](#more-advanced-usage)
* [⚡ Enlightening Questions and FAQs](#-enlightening-questions-and-faqs) * [⚡ Enlightening Questions and FAQs](#-enlightening-questions-and-faqs)
* [How does auto-matching work?](#how-does-auto-matching-work) * [How does auto-matching work?](#how-does-auto-matching-work)
* [How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?](#how-does-the-auto-matching-work-if-i-didnt-pass-a-url-while-initializing-the-adaptor-object) * [How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?](#how-does-the-auto-matching-work-if-i-didnt-pass-a-url-while-initializing-the-adaptor-object)
* [If all things about an element can change or get removed, what are the unique properties to be saved?](#if-all-things-about-an-element-can-change-or-get-removed-what-are-the-unique-properties-to-be-saved) * [If all things about an element can change or get removed, what are the unique properties to be saved?](#if-all-things-about-an-element-can-change-or-get-removed-what-are-the-unique-properties-to-be-saved)
@@ -82,13 +86,13 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL
* [Can Scrapling replace code built on top of BeautifulSoup4?](#can-scrapling-replace-code-built-on-top-of-beautifulsoup4) * [Can Scrapling replace code built on top of BeautifulSoup4?](#can-scrapling-replace-code-built-on-top-of-beautifulsoup4)
* [Can Scrapling replace code built on top of AutoScraper?](#can-scrapling-replace-code-built-on-top-of-autoscraper) * [Can Scrapling replace code built on top of AutoScraper?](#can-scrapling-replace-code-built-on-top-of-autoscraper)
* [Is Scrapling thread-safe?](#is-scrapling-thread-safe) * [Is Scrapling thread-safe?](#is-scrapling-thread-safe)
* [More Sponsors!](#more-sponsors) * [More Sponsors!](#more-sponsors)
* [Contributing](#contributing) * [Contributing](#contributing)
* [Disclaimer for Scrapling Project](#disclaimer-for-scrapling-project) * [Disclaimer for Scrapling Project](#disclaimer-for-scrapling-project)
* [License](#license) * [License](#license)
* [Acknowledgments](#acknowledgments) * [Acknowledgments](#acknowledgments)
* [Thanks and References](#thanks-and-references) * [Thanks and References](#thanks-and-references)
* [Known Issues](#known-issues) * [Known Issues](#known-issues)
## Key Features ## Key Features
@@ -120,10 +124,8 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL
```python ```python
from scrapling.fetchers import Fetcher from scrapling.fetchers import Fetcher
fetcher = Fetcher(auto_match=False)
# Do http GET request to a web page and create an Adaptor instance # Do http GET request to a web page and create an Adaptor instance
page = fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True) page = Fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True)
# Get all text content from all HTML tags in the page except `script` and `style` tags # Get all text content from all HTML tags in the page except `script` and `style` tags
page.get_all_text(ignore_tags=('script', 'style')) page.get_all_text(ignore_tags=('script', 'style'))
@@ -221,7 +223,7 @@ The available configuration arguments are: `auto_match`, `huge_tree`, `keep_comm
Also, the `Response` object returned from all fetchers is the same as the `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, `history`, and `request_headers`. All `cookies`, `headers`, and `request_headers` are always of type `dictionary`. Also, the `Response` object returned from all fetchers is the same as the `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, `history`, and `request_headers`. All `cookies`, `headers`, and `request_headers` are always of type `dictionary`.
> [!NOTE] > [!NOTE]
> The `auto_match` argument is enabled by default which is the one you should care about the most as you will see later. > The `auto_match` argument is disabled by default, you need to enable it to use that feature.
#### Set parser config per request #### Set parser config per request
As you probably understood, the logic above for setting the parser config will work globally for all requests/fetches done through that class and it's intended. As you probably understood, the logic above for setting the parser config will work globally for all requests/fetches done through that class and it's intended.
@@ -262,7 +264,7 @@ True
``` ```
> Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :) > Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :)
<details><summary><strong>For the sake of simplicity, expand this for the complete list of arguments</strong></summary> #### The complete list of arguments
| Argument | Description | Optional | | Argument | Description | Optional |
|:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| |:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
@@ -286,7 +288,6 @@ True
| os_randomize | If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS. | ✔️ | | os_randomize | If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS. | ✔️ |
| wait_selector_state | The state to wait for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ | | wait_selector_state | The state to wait for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
</details>
This list isn't final so expect a lot more additions and flexibility to be added in the next versions! This list isn't final so expect a lot more additions and flexibility to be added in the next versions!
@@ -316,7 +317,7 @@ Using this Fetcher class, you can make requests with:
Add that to a lot of controlling/hiding options as you will see in the arguments list below. Add that to a lot of controlling/hiding options as you will see in the arguments list below.
<details><summary><strong>Expand this for the complete list of arguments</strong></summary> #### The complete list of arguments
| Argument | Description | Optional | | Argument | Description | Optional |
|:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| |:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
@@ -341,7 +342,6 @@ Add that to a lot of controlling/hiding options as you will see in the arguments
| nstbrowser_mode | Enables NSTBrowser mode, **it have to be used with `cdp_url` argument or it will get completely ignored.** | ✔️ | | nstbrowser_mode | Enables NSTBrowser mode, **it have to be used with `cdp_url` argument or it will get completely ignored.** | ✔️ |
| nstbrowser_config | The config you want to send with requests to the NSTBrowser. _If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config._ | ✔️ | | nstbrowser_config | The config you want to send with requests to the NSTBrowser. _If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config._ | ✔️ |
</details>
This list isn't final so expect a lot more additions and flexibility to be added in the next versions! This list isn't final so expect a lot more additions and flexibility to be added in the next versions!
@@ -500,7 +500,7 @@ The selector will no longer function and your code needs maintenance. That's whe
```python ```python
from scrapling.parser import Adaptor from scrapling.parser import Adaptor
# Before the change # Before the change
page = Adaptor(page_source, url='example.com') page = Adaptor(page_source, auto_match=True, url='example.com')
element = page.css('#p1' auto_save=True) element = page.css('#p1' auto_save=True)
if not element: # One day website changes? if not element: # One day website changes?
element = page.css('#p1', auto_match=True) # Scrapling still finds it! element = page.css('#p1', auto_match=True) # Scrapling still finds it!
@@ -520,12 +520,13 @@ Now let's test the same selector in both versions
>> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a' >> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
>> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/" >> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
>> new_url = "https://stackoverflow.com/" >> new_url = "https://stackoverflow.com/"
>> Fetcher.configure(auto_match=True, automatch_domain='stackoverflow.com')
>> >>
>> page = Fetcher(automatch_domain='stackoverflow.com').get(old_url, timeout=30) >> page = Fetcher.get(old_url, timeout=30)
>> element1 = page.css_first(selector, auto_save=True) >> element1 = page.css_first(selector, auto_save=True)
>> >>
>> # Same selector but used in the updated website >> # Same selector but used in the updated website
>> page = Fetcher(automatch_domain="stackoverflow.com").get(new_url) >> page = Fetcher.get(new_url)
>> element2 = page.css_first(selector, auto_match=True) >> element2 = page.css_first(selector, auto_match=True)
>> >>
>> if element1.text == element2.text: >> if element1.text == element2.text: