From b50c85fa60faa66c01dedba43039b14bbb9cf7f3 Mon Sep 17 00:00:00 2001 From: Karim shoair Date: Sat, 23 Aug 2025 19:22:35 +0300 Subject: [PATCH] docs: Update the README file to reflect the new version changes --- README.md | 285 ++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 192 insertions(+), 93 deletions(-) diff --git a/README.md b/README.md index 58cab59..3040e1e 100644 --- a/README.md +++ b/README.md @@ -46,9 +46,11 @@

-Dealing with failing web scrapers due to anti-bot protections or website changes? Meet Scrapling. +**Stop fighting anti-bot systems. Stop rewriting selectors after every website update.** -Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity. +Scrapling isn't just another Web Scraping library. It's the first **adaptive** scraping library that learns from website changes and evolves with them. While other libraries break when websites update their structure, Scrapling automatically relocates your elements and keeps your scrapers running. + +Built for the modern Web, Scrapling has its own rapid parsing engine and its fetchers to handle all Web Scraping challenges you are facing or will face. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone. ```python >> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher @@ -79,148 +81,245 @@ Scrapling is a high-performance, intelligent web scraping library for Python tha ## Key Features -### Fetch websites as you prefer with async support -- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. -- **Dynamic Loading & Automation**: Fetch dynamic websites with the `DynamicFetcher` class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless! -- **Anti-bot Protections Bypass**: Easily bypass protections with the `StealthyFetcher` and `DynamicFetcher` classes. +### Advanced Websites Fetching with Session Support +- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP3. +- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, real Chrome, and custom stealth mode. +- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` using a modified version of Firefox and fingerprint spoofing. Can bypass all levels of Cloudflare's Turnstile with automation easily. +- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests. +- **Async Support**: Complete async support across all fetchers and dedicated async session classes. -### Adaptive Scraping -- πŸ”„ **Smart Element Tracking**: Relocate elements after website changes using an intelligent similarity system and integrated storage. -- 🎯 **Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more. -- πŸ” **Find Similar Elements**: Automatically locate elements similar to the element you found! -- 🧠 **Smart Content Scraping**: Extract data from multiple websites using Scrapling's powerful features without specific selectors. +### Adaptive Scraping & AI Integration +- πŸ”„ **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms. +- 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more. +- πŸ” **Find Similar Elements**: Automatically locate elements similar to found elements. +- πŸ€– **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. -### High Performance -- πŸš€ **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries. -- πŸ”‹ **Memory Efficient**: Optimized data structures for minimal memory footprint. -- ⚑ **Fast JSON serialization**: 10x faster than standard library. +### High-Performance & battle-tested Architecture +- πŸš€ **Lightning Fast**: Optimized performance outperforming most Python scraping libraries. +- πŸ”‹ **Memory Efficient**: Optimized data structures and lazy loading for a minimal memory footprint. +- ⚑ **Fast JSON Serialization**: 10x faster than the standard library. +- πŸ—οΈ **Battle tested**: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year. -### Developer Friendly -- πŸ› οΈ **Powerful Navigation API**: Easy DOM traversal in all directions. -- 🧬 **Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries with added methods that consume less memory than standard dictionaries. -- πŸ“ **Auto Selectors Generation**: Generate robust short and full CSS/XPath selectors for any element. -- πŸ”Œ **Familiar API**: Similar to Scrapy/BeautifulSoup and the same pseudo-elements used in Scrapy. -- πŸ“˜ **Type hints**: Complete type/doc-strings coverage for future-proofing and best autocompletion support. +### Developer/Web Scraper Friendly Experience +- 🎯 **Interactive Web Scraping Shell**: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser. +- πŸš€ **Use it directly from the Terminal**: Optionally, you can use Scrapling to scrape a URL without writing a single code! +- πŸ› οΈ **Rich Navigation API**: Advanced DOM traversal with parent, sibling, and child navigation methods. +- 🧬 **Enhanced Text Processing**: Built-in regex, cleaning methods, and optimized string operations. +- πŸ“ **Auto Selector Generation**: Generate robust CSS/XPath selectors for any element. +- πŸ”Œ **Familiar API**: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel. +- πŸ“˜ **Complete Type Coverage**: Full type hints for excellent IDE support and code completion. + +### New Session Architecture +Scrapling 0.3 introduces a completely revamped session system: +- **Persistent Sessions**: Maintain cookies, headers, and authentication across multiple requests +- **Automatic Session Management**: Smart session lifecycle handling with proper cleanup +- **Session Inheritance**: All fetchers support both one-off requests and persistent session usage +- **Concurrent Session Support**: Run multiple isolated sessions simultaneously ## Getting Started +### Basic Usage +```python +from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher +from scrapling.fetchers import FetcherSession, StealthySession, DynamicSession + +# HTTP requests with session support +with FetcherSession(impersonate='chrome') as session: # Use latest version of Chrome's TLS fingerprint + page = session.get('https://quotes.toscrape.com/', stealthy_headers=True) + quotes = page.css('.quote .text::text') + +# Or use one-off requests +page = Fetcher.get('https://quotes.toscrape.com/') +quotes = page.css('.quote .text::text') + +# Advanced stealth mode (Keep the browser open until you finish) +with StealthySession(headless=True, solve_cloudflare=True) as session: + page = session.fetch('https://nopecha.com/demo/cloudflare') + data = page.css('#padded_content a') + +# Or use one-off request style, it opens the browser for this request, then closes it after finishing +page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare') +data = page.css('#padded_content a') + +# Full browser automation (Keep the browser open until you finish) +with DynamicSession(headless=True, disable_resources=False, network_idle=True) as session: + page = session.fetch('https://quotes.toscrape.com/') + data = page.xpath('//span[@class="text"]/text()') # XPath selector if you prefer it + +# Or use one-off request style, it opens the browser for this request, then closes it after finishing +page = DynamicFetcher.fetch('https://quotes.toscrape.com/') +data = page.css('.quote .text::text') +``` + +### Advanced Parsing & Navigation ```python from scrapling.fetchers import Fetcher -# Do HTTP GET request to a web page and create a Selector instance -page = Fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True) -# Get all text content from all HTML tags in the page except the `script` and `style` tags -page.get_all_text(ignore_tags=('script', 'style')) +# Rich element selection and navigation +page = Fetcher.get('https://quotes.toscrape.com/') -# Get all quotes elements; any of these methods will return a list of strings directly (TextHandlers) -quotes = page.css('.quote .text::text') # CSS selector -quotes = page.xpath('//span[@class="text"]/text()') # XPath -quotes = page.css('.quote').css('.text::text') # Chained selectors -quotes = [element.text for element in page.css('.quote .text')] # Slower than bulk query above - -# Get the first quote element -quote = page.css_first('.quote') # same as page.css('.quote').first or page.css('.quote')[0] - -# Tired of selectors? Use find_all/find -# Get all 'div' HTML tags that one of its 'class' values is 'quote' -quotes = page.find_all('div', {'class': 'quote'}) +# Get quotes with multiple selection methods +quotes = page.css('.quote') # CSS selector +quotes = page.xpath('//div[@class="quote"]') # XPath +quotes = page.find_all('div', {'class': 'quote'}) # BeautifulSoup-style # Same as quotes = page.find_all('div', class_='quote') quotes = page.find_all(['div'], class_='quote') quotes = page.find_all(class_='quote') # and so on... +# Find element by text content +quotes = page.find_by_text('quote', tag='div') -# Working with elements -quote.html_content # Get the Inner HTML of this element -quote.prettify() # Prettified version of Inner HTML above -quote.attrib # Get that element's attributes -quote.path # DOM path to element (List of all ancestors from tag till the element itself) +# Advanced navigation +first_quote = page.css_first('.quote') +quote_text = first_quote.css('.text::text') +quote_text = page.css('.quote').css_first('.text::text') # Chained selectors +quote_text = page.css_first('.quote .text').text # Using `css_first` is faster than `css` if you want the first element +author = first_quote.next_sibling.css('.author::text') +parent_container = first_quote.parent + +# Element relationships and similarity +similar_elements = first_quote.find_similar() +below_elements = first_quote.below_elements() +``` +You can use the parser right away if you don't want to fetch websites like below: +```python +from scrapling.parser import Selector + +page = Selector("...") +``` +And it works exactly the same! + +### Async Session Management Examples +```python +import asyncio +from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession + +async with FetcherSession(http3=True) as session: # `FetcherSession` is context-aware and can work in both sync/async patterns + page1 = session.get('https://quotes.toscrape.com/') + page2 = session.get('https://quotes.toscrape.com/', impersonate='firefox135') + +# Async session usage +async with AsyncStealthySession(max_pages=2) as session: + tasks = [] + urls = ['https://example.com/page1', 'https://example.com/page2'] + + for url in urls: + task = session.fetch(url) + tasks.append(task) + + print(session.get_pool_stats()) # Optional - The status of the browser tabs pool (busy/free/error) + results = await asyncio.gather(*tasks) + print(session.get_pool_stats()) +``` + +## CLI & Interactive Shell + +Scrapling v0.3 includes a powerful command-line interface: + +```bash +# Launch interactive Web Scraping shell +scrapling shell + +# Extract pages to a file directly without programming (Extracts the content inside `body` tag by default) +# If the output file ends with `.txt`, then the text content of the target will be extracted. +# If ended with `.md`, it will be a markdown representation of the HTML content, and `.html` will be the HTML content right away. +scrapling extract get 'https://example.com' content.md +scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts' +scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless +scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.html --css-selector '#padded_content a' --solve-cloudflare ``` -To keep it simple, all methods can be chained on top of each other! > [!NOTE] -> Check out the full documentation from [here](https://scrapling.readthedocs.io/en/latest/) +> There are many additional features, but we want to keep this page short, like the MCP server and the interactive Web Scraping Shell. Check out the full documentation [here](https://scrapling.readthedocs.io/en/latest/) -## Parsing Performance +## Performance Benchmarks -Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents. -Here are benchmarks comparing Scrapling to popular Python libraries in two tests. - -### Text Extraction Speed Test (5000 nested elements). - -This test consists of extracting the text content of 5000 nested div elements. +Scrapling isn't just powerfulβ€”it's also blazing fast, and version 0.3 delivers exceptional performance improvements across all operations! +### Text Extraction Speed Test (5000 nested elements) | # | Library | Time (ms) | vs Scrapling | |---|:-----------------:|:---------:|:------------:| -| 1 | Scrapling | 5.44 | 1.0x | -| 2 | Parsel/Scrapy | 5.53 | 1.017x | -| 3 | Raw Lxml | 6.76 | 1.243x | -| 4 | PyQuery | 21.96 | 4.037x | -| 5 | Selectolax | 67.12 | 12.338x | -| 6 | BS4 with Lxml | 1307.03 | 240.263x | -| 7 | MechanicalSoup | 1322.64 | 243.132x | -| 8 | BS4 with html5lib | 3373.75 | 620.175x | +| 1 | Scrapling | 1.88 | 1.0x | +| 2 | Parsel/Scrapy | 1.96 | 1.043x | +| 3 | Raw Lxml | 2.32 | 1.234x | +| 4 | PyQuery | 20.2 | ~11x | +| 5 | Selectolax | 85.2 | ~45x | +| 6 | MechanicalSoup | 1305.84 | ~695x | +| 7 | BS4 with Lxml | 1307.92 | ~696x | +| 8 | BS4 with html5lib | 3336.28 | ~1775x | -As you see, Scrapling is on par with Scrapy and slightly faster than Lxml, which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml, but Scrapling is four times faster. +### Element Similarity & Text Search Performance -### Extraction By Text Speed Test - -Scrapling can find elements based on its text content and find elements similar to these elements. The only known library with these two features, too, is AutoScraper. - -So, we compared this to see how fast Scrapling can be in these two tasks compared to AutoScraper. - -Here are the results: +Scrapling's adaptive element finding capabilities significantly outperform alternatives: | Library | Time (ms) | vs Scrapling | |-------------|:---------:|:------------:| -| Scrapling | 2.51 | 1.0x | -| AutoScraper | 11.41 | 4.546x | +| Scrapling | 2.02 | 1.0x | +| AutoScraper | 10.26 | 5.08x | -Scrapling can find elements with more methods and returns the entire element's `Selector` object, not only text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. -As you see, Scrapling is still 4.5 times faster at the same task. - -If we made Scrapling extract the elements only without stopping to extract each element's text, we would get speed twice as fast as this, but as I said, to make it fair comparison a bit :smile: - -> All benchmarks' results are an average of 100 runs. See our [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology and to run your comparisons. +> All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology. ## Installation -Scrapling is a breeze to get started with. Starting from version 0.2.9, we require at least Python 3.9 to work. + +Scrapling requires Python 3.10 or higher: + ```bash -pip3 install scrapling +pip install scrapling ``` -Then run this command to install browsers' dependencies needed to use Fetcher classes + +### Fetchers Setup + +If you are going to use any of the fetchers or their classes, then install browser dependencies with ```bash scrapling install ``` -If you have any installation issues, please open an issue. +This downloads all browsers with their system dependencies and fingerprint manipulation dependencies. + +### Optional Dependencies + +Install the MCP server feature: +```bash +pip install "scrapling[ai]" +``` + +Install with shell features (Web Scraping shell and the `extract` command): +```bash +pip install "scrapling[shell]" +``` + +Install everything: +```bash +pip install "scrapling[all]" +``` ## Contributing -Everybody is invited and welcome to contribute to Scrapling. There is a lot to do! -Please read the [contributing file](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md) before doing anything. +We welcome contributions! Please read our [contributing guidelines](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md) before getting started. + +## Disclaimer -## Disclaimer for Scrapling Project > [!CAUTION] -> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules, such as the `robots.txt` file. +> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect website terms of service and robots.txt files. ## License -This work is licensed under BSD-3 + +This work is licensed under the BSD-3-Clause License. ## Acknowledgments + This project includes code adapted from: -- Parsel (BSD License) - Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/translator.py) submodule +- Parsel (BSD License)β€”Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) submodule ## Thanks and References -- [Daijro](https://github.com/daijro)'s brilliant work on both [BrowserForge](https://github.com/daijro/browserforge) and [Camoufox](https://github.com/daijro/camoufox) -- [Vinyzu](https://github.com/Vinyzu)'s work on Playwright's mock on [Botright](https://github.com/Vinyzu/Botright) -- [brotector](https://github.com/kaliiiiiiiiii/brotector) -- [fakebrowser](https://github.com/kkoooqq/fakebrowser) -- [rebrowser-patches](https://github.com/rebrowser/rebrowser-patches) -## Known Issues -- In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. If the selector you are using selects different elements on the page in different locations, auto-matching will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone. +- [Daijro](https://github.com/daijro)'s brilliant work on [BrowserForge](https://github.com/daijro/browserforge) and [Camoufox](https://github.com/daijro/camoufox) +- [Vinyzu](https://github.com/Vinyzu)'s work on [Botright](https://github.com/Vinyzu/Botright) +- [brotector](https://github.com/kaliiiiiiiiii/brotector) for browser detection bypass techniques +- [fakebrowser](https://github.com/kkoooqq/fakebrowser) for fingerprinting research +- [rebrowser-patches](https://github.com/rebrowser/rebrowser-patches) for stealth improvements --- -
Designed & crafted with ❀️ by Karim Shoair.

+
Designed & crafted with ❀️ by Karim Shoair.

\ No newline at end of file