Product 1
-Description 1
-diff --git a/README.md b/README.md index 9e3f379..c1ac3b6 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,36 @@ # 🕷️ Scrapling: Undetectable, Lightning-Fast, and Easy Web Scraping with Python -[](https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml) [](https://badge.fury.io/py/Scrapling) [](https://pypi.org/project/scrapling/) [](https://pepy.tech/project/scrapling) +
++ + Installation + + · + + Overview + + · + + Selection methods + + · + + Choosing a fetcher + + · + + Migrating from Beautifulsoup + +
Dealing with failing web scrapers due to anti-bot protections or website changes? Meet Scrapling. @@ -21,7 +52,7 @@ Scrapling is a high-performance, intelligent web scraping library for Python tha [Scrapeless Deep SerpApi](https://www.scrapeless.com/en/product/deep-serp-api?utm_source=website&utm_medium=ads&utm_campaign=scraping&utm_term=d4vinci) From $0.10 per 1,000 queries with a 1-2 second response time! -Deep SerpApi is a dedicated search engine designed for large language models (LLMs) and AI agents, aiming to provide real-time, accurate and unbiased information to help AI applications retrieve and process data efficiently. +Deep SerpApi is a dedicated search engine designed for large language models (LLMs) and AI agents. It aims to provide real-time, accurate, and unbiased information to help AI applications retrieve and process data efficiently. - covering 20+ Google SERP scenarios and mainstream search engines. - support real-time data updates to ensure real-time and accurate information. - It can integrate information from all available online channels and search engines. @@ -50,62 +81,18 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL [](https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling) --- -## Table of content -* [Key Features](#key-features) - * [Fetch websites as you prefer with async support](#fetch-websites-as-you-prefer-with-async-support) - * [Adaptive Scraping](#adaptive-scraping) - * [High Performance](#high-performance) - * [Developer Friendly](#developer-friendly) -* [Getting Started](#getting-started) -* [Parsing Performance](#parsing-performance) - * [Text Extraction Speed Test (5000 nested elements).](#text-extraction-speed-test-5000-nested-elements) - * [Extraction By Text Speed Test](#extraction-by-text-speed-test) -* [Installation](#installation) -* [Fetching Websites](#fetching-websites) - * [Features](#features) - * [Set parser config per request](#set-parser-config-per-request) - * [Fetcher](#fetcher) - * [StealthyFetcher](#stealthyfetcher) - * [The complete list of arguments](#the-complete-list-of-arguments) - * [PlayWrightFetcher](#playwrightfetcher) - * [The complete list of arguments](#the-complete-list-of-arguments-1) -* [Advanced Parsing Features](#advanced-parsing-features) - * [Smart Navigation](#smart-navigation) - * [Content-based Selection & Finding Similar Elements](#content-based-selection--finding-similar-elements) - * [Handling Structural Changes](#handling-structural-changes) - * [Real-World Scenario](#real-world-scenario) - * [Find elements by filters](#find-elements-by-filters) - * [Is That All?](#is-that-all) -* [More Advanced Usage](#more-advanced-usage) -* [⚡ Enlightening Questions and FAQs](#-enlightening-questions-and-faqs) - * [How does auto-matching work?](#how-does-auto-matching-work) - * [How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?](#how-does-the-auto-matching-work-if-i-didnt-pass-a-url-while-initializing-the-adaptor-object) - * [If all things about an element can change or get removed, what are the unique properties to be saved?](#if-all-things-about-an-element-can-change-or-get-removed-what-are-the-unique-properties-to-be-saved) - * [I have enabled the `auto_save`/`auto_match` parameter while selecting and it got completely ignored with a warning message](#i-have-enabled-the-auto_saveauto_match-parameter-while-selecting-and-it-got-completely-ignored-with-a-warning-message) - * [I have done everything as the docs but the auto-matching didn't return anything, what's wrong?](#i-have-done-everything-as-the-docs-but-the-auto-matching-didnt-return-anything-whats-wrong) - * [Can Scrapling replace code built on top of BeautifulSoup4?](#can-scrapling-replace-code-built-on-top-of-beautifulsoup4) - * [Can Scrapling replace code built on top of AutoScraper?](#can-scrapling-replace-code-built-on-top-of-autoscraper) - * [Is Scrapling thread-safe?](#is-scrapling-thread-safe) -* [More Sponsors!](#more-sponsors) -* [Contributing](#contributing) -* [Disclaimer for Scrapling Project](#disclaimer-for-scrapling-project) -* [License](#license) -* [Acknowledgments](#acknowledgments) -* [Thanks and References](#thanks-and-references) -* [Known Issues](#known-issues) - ## Key Features ### Fetch websites as you prefer with async support - **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. - **Dynamic Loading & Automation**: Fetch dynamic websites with the `PlayWrightFetcher` class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless! -- **Anti-bot Protections Bypass**: Easily bypass protections with `StealthyFetcher` and `PlayWrightFetcher` classes. +- **Anti-bot Protections Bypass**: Easily bypass protections with the `StealthyFetcher` and `PlayWrightFetcher` classes. ### Adaptive Scraping -- 🔄 **Smart Element Tracking**: Relocate elements after website changes, using an intelligent similarity system and integrated storage. -- 🎯 **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search and more. +- 🔄 **Smart Element Tracking**: Relocate elements after website changes using an intelligent similarity system and integrated storage. +- 🎯 **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search, and more. - 🔍 **Find Similar Elements**: Automatically locate elements similar to the element you found! -- 🧠 **Smart Content Scraping**: Extract data from multiple websites without specific selectors using Scrapling powerful features. +- 🧠 **Smart Content Scraping**: Extract data from multiple websites using Scrapling's powerful features without specific selectors. ### High Performance - 🚀 **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries. @@ -114,7 +101,7 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL ### Developer Friendly - 🛠️ **Powerful Navigation API**: Easy DOM traversal in all directions. -- 🧬 **Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that takes less memory than standard dictionaries with added methods. +- 🧬 **Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries with added methods that consume less memory than standard dictionaries. - 📝 **Auto Selectors Generation**: Generate robust short and full CSS/XPath selectors for any element. - 🔌 **Familiar API**: Similar to Scrapy/BeautifulSoup and the same pseudo-elements used in Scrapy. - 📘 **Type hints**: Complete type/doc-strings coverage for future-proofing and best autocompletion support. @@ -124,12 +111,12 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL ```python from scrapling.fetchers import Fetcher -# Do http GET request to a web page and create an Adaptor instance +# Do HTTP GET request to a web page and create an Adaptor instance page = Fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True) -# Get all text content from all HTML tags in the page except `script` and `style` tags +# Get all text content from all HTML tags in the page except the `script` and `style` tags page.get_all_text(ignore_tags=('script', 'style')) -# Get all quotes elements, any of these methods will return a list of strings directly (TextHandlers) +# Get all quotes elements; any of these methods will return a list of strings directly (TextHandlers) quotes = page.css('.quote .text::text') # CSS selector quotes = page.xpath('//span[@class="text"]/text()') # XPath quotes = page.css('.quote').css('.text::text') # Chained selectors @@ -147,13 +134,16 @@ quotes = page.find_all(['div'], class_='quote') quotes = page.find_all(class_='quote') # and so on... # Working with elements -quote.html_content # Get Inner HTML of this element +quote.html_content # Get the Inner HTML of this element quote.prettify() # Prettified version of Inner HTML above quote.attrib # Get that element's attributes quote.path # DOM path to element (List of all ancestors from tag till the element itself) ``` To keep it simple, all methods can be chained on top of each other! +> [!NOTE] +> Check out the full documentation from [here](https://scrapling.readthedocs.io/en/latest/) + ## Parsing Performance Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents. @@ -161,32 +151,45 @@ Here are benchmarks comparing Scrapling to popular Python libraries in two tests ### Text Extraction Speed Test (5000 nested elements). +This test consists of extracting the text content of 5000 nested div elements. + + | # | Library | Time (ms) | vs Scrapling | |---|:-----------------:|:---------:|:------------:| -| 1 | Scrapling | 5.55 | 1.0x | -| 2 | Parsel/Scrapy | 5.67 | 1.022x | -| 3 | Raw Lxml | 6.69 | 1.205x | -| 4 | PyQuery | 20.84 | 3.755x | -| 5 | Selectolax | 84.41 | 15.209x | -| 6 | BS4 with Lxml | 1313.45 | 236.658x | -| 7 | MechanicalSoup | 1313.66 | 236.695x | -| 8 | BS4 with html5lib | 3383.27 | 609.598x | +| 1 | Scrapling | 5.44 | 1.0x | +| 2 | Parsel/Scrapy | 5.53 | 1.017x | +| 3 | Raw Lxml | 6.76 | 1.243x | +| 4 | PyQuery | 21.96 | 4.037x | +| 5 | Selectolax | 67.12 | 12.338x | +| 6 | BS4 with Lxml | 1307.03 | 240.263x | +| 7 | MechanicalSoup | 1322.64 | 243.132x | +| 8 | BS4 with html5lib | 3373.75 | 620.175x | -As you see, Scrapling is on par with Scrapy and slightly faster than Lxml which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml but still, Scrapling is ~4 times faster. +As you see, Scrapling is on par with Scrapy and slightly faster than Lxml, which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml, but Scrapling is four times faster. ### Extraction By Text Speed Test -| Library | Time (ms) | vs Scrapling | -|:-----------:|:---------:|:------------:| -| Scrapling | 2.35 | 1.0x | -| AutoScraper | 11.44 | 4.868x | +Scrapling can find elements based on its text content and find elements similar to these elements. The only known library with these two features, too, is AutoScraper. -Scrapling can find elements with more methods and it returns full element `Adaptor` objects not only the text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. As you see, Scrapling is still 4.8 times faster at the same task. +So, we compared this to see how fast Scrapling can be in these two tasks compared to AutoScraper. + +Here are the results: + +| Library | Time (ms) | vs Scrapling | +|-------------|:---------:|:------------:| +| Scrapling | 2.51 | 1.0x | +| AutoScraper | 11.41 | 4.546x | + +Scrapling can find elements with more methods and returns the entire element's `Adaptor` object, not only text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. + +As you see, Scrapling is still 4.5 times faster at the same task. + +If we made Scrapling extract the elements only without stopping to extract each element's text, we would get speed twice as fast as this, but as I said, to make it fair comparison a bit :smile: > All benchmarks' results are an average of 100 runs. See our [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology and to run your comparisons. ## Installation -Scrapling is a breeze to get started with; Starting from version 0.2.9, we require at least Python 3.9 to work. +Scrapling is a breeze to get started with. Starting from version 0.2.9, we require at least Python 3.9 to work. ```bash pip3 install scrapling ``` @@ -196,600 +199,6 @@ scrapling install ``` If you have any installation issues, please open an issue. -## Fetching Websites -Fetchers are interfaces built on top of other libraries with added features that do requests or fetch pages for you in a single request fashion and then return an `Adaptor` object. This feature was introduced because the only option we had before was to fetch the page as you wanted it, then pass it manually to the `Adaptor` class to create an `Adaptor` instance and start playing around with the page. - -### Features -You might be slightly confused by now so let me clear things up. All fetcher-type classes are imported in the same way -```python ->>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher -``` -then use it right away without initializing like this and it will use the default parser settings: -```python ->>> page = StealthyFetcher.fetch('https://example.com') -``` -If you want to configure the parser (Adaptor class) that will be used on the response before returning it for you then do this first: -```python ->>> StealthyFetcher.configure(auto_match=True, keep_comments=False) -``` -or -```python ->>> StealthyFetcher.auto_match = True ->>> StealthyFetcher.keep_comments = False -``` -Then continue your code as normal. - -The available configuration arguments are: `auto_match`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the `Adaptor` class. You can display the current configuration anytime by running `£51.77
' parent='£53.74
' parent='£50.10
' parent='£47.82
' parent=',
- ,
- ,
-...]
-
-# You will notice that the number of elements is 19 not 20 because the current element is not included.
->>> len(page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title']))
-19
-
-# Get the `href` attribute from all similar elements
->>> [element.attrib['href'] for element in page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])]
-['catalogue/a-light-in-the-attic_1000/index.html',
- 'catalogue/soumission_998/index.html',
- 'catalogue/sharp-objects_997/index.html',
- ...]
-```
-To increase the complexity a little bit, let's say we want to get all books' data using that element as a starting point for some reason
-```python
->>> for product in page.find_by_text('Tipping the Velvet').parent.parent.find_similar():
- print({
- "name": product.css_first('h3 a::text'),
- "price": product.css_first('.price_color').re_first(r'[\d\.]+'),
- "stock": product.css('.availability::text')[-1].clean()
- })
-{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
-{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
-{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
-...
-```
-The [documentation](https://github.com/D4Vinci/Scrapling/tree/main/docs/Examples) will provide more advanced examples.
-
-### Handling Structural Changes
-Let's say you are scraping a page with a structure like this:
-```html
- Description 1 Description 2 Description 1 Description 2Product 1
- Product 2
- Product 1
- Product 2
-
It filters all elements in the current page/element in the following order:
-
-1. All elements with the passed tag name(s).
-2. All elements that match all passed attribute(s).
-3. All elements that its text content match all passed regex patterns.
-4. All elements that fulfill all passed function(s).
-
-Note: The filtering process always starts from the first filter it finds in the filtering order above so if no tag name(s) are passed but attributes are passed, the process starts from that layer and so on. **But the order in which you pass the arguments doesn't matter.**
-
-Examples to clear any confusion :)
-
-```python
->> from scrapling.fetchers import Fetcher
->> page = Fetcher.get('https://quotes.toscrape.com/')
-# Find all elements with tag name `div`.
->> page.find_all('div')
-[
-> I'm trying to rush creating the website, researching new ideas, and adding more features/tests/benchmarks but time is tight with too many spinning plates between work, personal life, and working on Scrapling. I have been working on Scrapling for months for free after all.
-> If you like `Scrapling` and want it to keep improving then this is a friendly reminder that you can help by supporting me through the [sponsor button](https://github.com/sponsors/D4Vinci).
-
-## ⚡ Enlightening Questions and FAQs
-This section addresses common questions about Scrapling, please read this section before opening an issue.
-
-### How does auto-matching work?
- 1. You need to get a working selector and run it at least once with methods `css` or `xpath` with the `auto_save` parameter set to `True` before structural changes happen.
- 2. Before returning results for you, Scrapling uses its configured database and saves unique properties about that element.
- 3. Now because everything about the element can be changed or removed, nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
- 1. The domain of the URL you gave while initializing the first Adaptor object
- 2. The `identifier` parameter you passed to the method while selecting. If you didn't pass one, then the selector string itself will be used as an identifier but remember you will have to use it as an identifier value later when the structure changes and you want to pass the new selector.
-
- Together both are used to retrieve the element's unique properties from the database later.
- 4. Now later when you enable the `auto_match` parameter for both the Adaptor instance and the method call. The element properties are retrieved and Scrapling loops over all elements in the page and compares each one's unique properties to the unique properties we already have for this element and a score is calculated for each one.
- 5. Comparing elements is not exact but more about finding how similar these values are, so everything is taken into consideration, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
- 6. The score for each element is stored in the table, and the element(s) with the highest combined similarity scores are returned.
-
-### How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?
-Not a big problem as it depends on your usage. The word `default` will be used in place of the URL field while saving the element's unique properties. So this will only be an issue if you used the same identifier later for a different website that you didn't pass the URL parameter while initializing it as well. The save process will overwrite the previous data and auto-matching uses the latest saved properties only.
-
-### If all things about an element can change or get removed, what are the unique properties to be saved?
-For each element, Scrapling will extract:
-- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
-- Element's parent tag name, attributes (names and values), and text.
-
-### I have enabled the `auto_save`/`auto_match` parameter while selecting and it got completely ignored with a warning message
-That's because passing the `auto_save`/`auto_match` argument without setting `auto_match` to `True` while initializing the Adaptor object will only result in ignoring the `auto_save`/`auto_match` argument value. This behavior is purely for performance reasons so the database gets created only when you are planning to use the auto-matching features.
-
-### I have done everything as the docs but the auto-matching didn't return anything, what's wrong?
-It could be one of these reasons:
-1. No data were saved/stored for this element before.
-2. The selector passed is not the one used while storing element data. The solution is simple
- - Pass the old selector again as an identifier to the method called.
- - Retrieve the element with the retrieve method using the old selector as identifier then save it again with the save method and the new selector as identifier.
- - Start using the identifier argument more often if you are planning to use every new selector from now on.
-3. The website had some extreme structural changes like a new full design. If this happens a lot with this website, the solution would be to make your code as selector-free as possible using Scrapling features.
-
-### Can Scrapling replace code built on top of BeautifulSoup4?
-Pretty much yeah, almost all features you get from BeautifulSoup can be found or achieved in Scrapling one way or another. In fact, if you see there's a feature in bs4 that is missing in Scrapling, please make a feature request from the issues tab to let me know.
-
-### Can Scrapling replace code built on top of AutoScraper?
-Of course, you can find elements by text/regex, find similar elements in a more reliable way than AutoScraper, and finally save/retrieve elements manually to use later as the model feature in AutoScraper. I have pulled all top articles about AutoScraper from Google and tested Scrapling against examples in them. In all examples, Scrapling got the same results as AutoScraper in much less time.
-
-### Is Scrapling thread-safe?
-Yes, Scrapling instances are thread-safe. Each Adaptor instance maintains its state.
## More Sponsors!
@@ -802,7 +211,7 @@ Please read the [contributing file](https://github.com/D4Vinci/Scrapling/blob/ma
## Disclaimer for Scrapling Project
> [!CAUTION]
-> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international laws regarding data scraping and privacy. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules like the `robots.txt` file, for example.
+> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules, such as the `robots.txt` file.
## License
This work is licensed under BSD-3
@@ -819,7 +228,7 @@ This project includes code adapted from:
- [rebrowser-patches](https://github.com/rebrowser/rebrowser-patches)
## Known Issues
-- In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page that are in different locations, auto-matching will probably return to you the first element only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector for example) as these selectors get separated and each selector gets executed alone.
+- In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. If the selector you are using selects different elements on the page in different locations, auto-matching will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
---
-
+
\ No newline at end of file