# πŸ•·οΈ Scrapling: Undetectable, Lightning-Fast, and Easy Web Scraping with Python [![Tests](https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml/badge.svg)](https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml) [![PyPI version](https://badge.fury.io/py/Scrapling.svg)](https://badge.fury.io/py/Scrapling) [![Supported Python versions](https://img.shields.io/pypi/pyversions/scrapling.svg)](https://pypi.org/project/scrapling/) [![PyPI Downloads](https://static.pepy.tech/badge/scrapling)](https://pepy.tech/project/scrapling) Dealing with failing web scrapers due to anti-bot protections or website changes? Meet Scrapling. Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity. ```python >> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher >> StealthyFetcher.auto_match = True # Fetch websites' source under the radar! >> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) >> print(page.status) 200 >> products = page.css('.product', auto_save=True) # Scrape data that survives website design changes! >> # Later, if the website structure changes, pass `auto_match=True` >> products = page.css('.product', auto_match=True) # and Scrapling still finds them! ``` # Sponsors [Scrapeless Deep SerpApi](https://www.scrapeless.com/en/product/deep-serp-api?utm_source=website&utm_medium=ads&utm_campaign=scraping&utm_term=d4vinci) From $0.10 per 1,000 queries with a 1-2 second response time! Deep SerpApi is a dedicated search engine designed for large language models (LLMs) and AI agents, aiming to provide real-time, accurate and unbiased information to help AI applications retrieve and process data efficiently. - covering 20+ Google SERP scenarios and mainstream search engines. - support real-time data updates to ensure real-time and accurate information. - It can integrate information from all available online channels and search engines. - Deep SerpApi will simplify the process of integrating dynamic web information into AI solutions, and ultimately achieve an ALL-in-One API for one-click search and extraction of web data. - **Developer Support Program**: Integrate Scrapeless Deep SerpApi into your AI tools, applications or projects. [We already support Dify, and will soon support frameworks such as Langchain, Langflow, FlowiseAI]. Then share your results on GitHub or social media, and you will get a 1-12 month free developer support opportunity, up to 500 free usage per month. - πŸš€ **Scraping API**: Effortless and highly customizable data extraction with a single API call, providing structured data from any website. - ⚑ **Scraping Browser**: AI-powered and LLM-driven, it simulates human-like behavior with genuine fingerprints and headless browser support, ensuring seamless, block-free scraping. - 🌐 **Proxies**: Use high-quality, rotating proxies to scrape top platforms like Amazon, Shopee, and more, with global coverage in 195+ countries. [![Scrapeless Banner](https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/scrapeless.jpg)](https://www.scrapeless.com/en/product/deep-serp-api?utm_source=website&utm_medium=ads&utm_campaign=scraping&utm_term=d4vinci) --- [Evomi](https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling) is your Swiss Quality Proxy Provider, starting at **$0.49/GB** - πŸ‘©β€πŸ’» **$0.49 per GB Residential Proxies**: Our price is unbeatable - πŸ‘©β€πŸ’» **24/7 Expert Support**: We will join your Slack Channel - 🌍 **Global Presence**: Available in 150+ Countries - ⚑ **Low Latency** - πŸ”’ **Swiss Quality and Privacy** - 🎁 **Free Trial** - πŸ›‘οΈ **99.9% Uptime** - 🀝 **Special IP Pool selection**: Optimize for fast, quality or quantity of ips - πŸ”§ **Easy Integration**: Compatible with most software and programming languages [![Evomi Banner](https://my.evomi.com/images/brand/cta.png)](https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling) --- ## Table of content * [Key Features](#key-features) * [Fetch websites as you prefer with async support](#fetch-websites-as-you-prefer-with-async-support) * [Adaptive Scraping](#adaptive-scraping) * [High Performance](#high-performance) * [Developer Friendly](#developer-friendly) * [Getting Started](#getting-started) * [Parsing Performance](#parsing-performance) * [Text Extraction Speed Test (5000 nested elements).](#text-extraction-speed-test-5000-nested-elements) * [Extraction By Text Speed Test](#extraction-by-text-speed-test) * [Installation](#installation) * [Fetching Websites](#fetching-websites) * [Features](#features) * [Set parser config per request](#set-parser-config-per-request) * [Fetcher](#fetcher) * [StealthyFetcher](#stealthyfetcher) * [The complete list of arguments](#the-complete-list-of-arguments) * [PlayWrightFetcher](#playwrightfetcher) * [The complete list of arguments](#the-complete-list-of-arguments-1) * [Advanced Parsing Features](#advanced-parsing-features) * [Smart Navigation](#smart-navigation) * [Content-based Selection & Finding Similar Elements](#content-based-selection--finding-similar-elements) * [Handling Structural Changes](#handling-structural-changes) * [Real-World Scenario](#real-world-scenario) * [Find elements by filters](#find-elements-by-filters) * [Is That All?](#is-that-all) * [More Advanced Usage](#more-advanced-usage) * [⚑ Enlightening Questions and FAQs](#-enlightening-questions-and-faqs) * [How does auto-matching work?](#how-does-auto-matching-work) * [How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?](#how-does-the-auto-matching-work-if-i-didnt-pass-a-url-while-initializing-the-adaptor-object) * [If all things about an element can change or get removed, what are the unique properties to be saved?](#if-all-things-about-an-element-can-change-or-get-removed-what-are-the-unique-properties-to-be-saved) * [I have enabled the `auto_save`/`auto_match` parameter while selecting and it got completely ignored with a warning message](#i-have-enabled-the-auto_saveauto_match-parameter-while-selecting-and-it-got-completely-ignored-with-a-warning-message) * [I have done everything as the docs but the auto-matching didn't return anything, what's wrong?](#i-have-done-everything-as-the-docs-but-the-auto-matching-didnt-return-anything-whats-wrong) * [Can Scrapling replace code built on top of BeautifulSoup4?](#can-scrapling-replace-code-built-on-top-of-beautifulsoup4) * [Can Scrapling replace code built on top of AutoScraper?](#can-scrapling-replace-code-built-on-top-of-autoscraper) * [Is Scrapling thread-safe?](#is-scrapling-thread-safe) * [More Sponsors!](#more-sponsors) * [Contributing](#contributing) * [Disclaimer for Scrapling Project](#disclaimer-for-scrapling-project) * [License](#license) * [Acknowledgments](#acknowledgments) * [Thanks and References](#thanks-and-references) * [Known Issues](#known-issues) ## Key Features ### Fetch websites as you prefer with async support - **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. - **Dynamic Loading & Automation**: Fetch dynamic websites with the `PlayWrightFetcher` class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless! - **Anti-bot Protections Bypass**: Easily bypass protections with `StealthyFetcher` and `PlayWrightFetcher` classes. ### Adaptive Scraping - πŸ”„ **Smart Element Tracking**: Relocate elements after website changes, using an intelligent similarity system and integrated storage. - 🎯 **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search and more. - πŸ” **Find Similar Elements**: Automatically locate elements similar to the element you found! - 🧠 **Smart Content Scraping**: Extract data from multiple websites without specific selectors using Scrapling powerful features. ### High Performance - πŸš€ **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries. - πŸ”‹ **Memory Efficient**: Optimized data structures for minimal memory footprint. - ⚑ **Fast JSON serialization**: 10x faster than standard library. ### Developer Friendly - πŸ› οΈ **Powerful Navigation API**: Easy DOM traversal in all directions. - 🧬 **Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that takes less memory than standard dictionaries with added methods. - πŸ“ **Auto Selectors Generation**: Generate robust short and full CSS/XPath selectors for any element. - πŸ”Œ **Familiar API**: Similar to Scrapy/BeautifulSoup and the same pseudo-elements used in Scrapy. - πŸ“˜ **Type hints**: Complete type/doc-strings coverage for future-proofing and best autocompletion support. ## Getting Started ```python from scrapling.fetchers import Fetcher # Do http GET request to a web page and create an Adaptor instance page = Fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True) # Get all text content from all HTML tags in the page except `script` and `style` tags page.get_all_text(ignore_tags=('script', 'style')) # Get all quotes elements, any of these methods will return a list of strings directly (TextHandlers) quotes = page.css('.quote .text::text') # CSS selector quotes = page.xpath('//span[@class="text"]/text()') # XPath quotes = page.css('.quote').css('.text::text') # Chained selectors quotes = [element.text for element in page.css('.quote .text')] # Slower than bulk query above # Get the first quote element quote = page.css_first('.quote') # same as page.css('.quote').first or page.css('.quote')[0] # Tired of selectors? Use find_all/find # Get all 'div' HTML tags that one of its 'class' values is 'quote' quotes = page.find_all('div', {'class': 'quote'}) # Same as quotes = page.find_all('div', class_='quote') quotes = page.find_all(['div'], class_='quote') quotes = page.find_all(class_='quote') # and so on... # Working with elements quote.html_content # Get Inner HTML of this element quote.prettify() # Prettified version of Inner HTML above quote.attrib # Get that element's attributes quote.path # DOM path to element (List of all ancestors from tag till the element itself) ``` To keep it simple, all methods can be chained on top of each other! ## Parsing Performance Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents. Here are benchmarks comparing Scrapling to popular Python libraries in two tests. ### Text Extraction Speed Test (5000 nested elements). | # | Library | Time (ms) | vs Scrapling | |---|:-----------------:|:---------:|:------------:| | 1 | Scrapling | 5.55 | 1.0x | | 2 | Parsel/Scrapy | 5.67 | 1.022x | | 3 | Raw Lxml | 6.69 | 1.205x | | 4 | PyQuery | 20.84 | 3.755x | | 5 | Selectolax | 84.41 | 15.209x | | 6 | BS4 with Lxml | 1313.45 | 236.658x | | 7 | MechanicalSoup | 1313.66 | 236.695x | | 8 | BS4 with html5lib | 3383.27 | 609.598x | As you see, Scrapling is on par with Scrapy and slightly faster than Lxml which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml but still, Scrapling is ~4 times faster. ### Extraction By Text Speed Test | Library | Time (ms) | vs Scrapling | |:-----------:|:---------:|:------------:| | Scrapling | 2.35 | 1.0x | | AutoScraper | 11.44 | 4.868x | Scrapling can find elements with more methods and it returns full element `Adaptor` objects not only the text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. As you see, Scrapling is still 4.8 times faster at the same task. > All benchmarks' results are an average of 100 runs. See our [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology and to run your comparisons. ## Installation Scrapling is a breeze to get started with; Starting from version 0.2.9, we require at least Python 3.9 to work. ```bash pip3 install scrapling ``` Then run this command to install browsers' dependencies needed to use Fetcher classes ```bash scrapling install ``` If you have any installation issues, please open an issue. ## Fetching Websites Fetchers are interfaces built on top of other libraries with added features that do requests or fetch pages for you in a single request fashion and then return an `Adaptor` object. This feature was introduced because the only option we had before was to fetch the page as you wanted it, then pass it manually to the `Adaptor` class to create an `Adaptor` instance and start playing around with the page. ### Features You might be slightly confused by now so let me clear things up. All fetcher-type classes are imported in the same way ```python >>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher ``` then use it right away without initializing like this and it will use the default parser settings: ```python >>> page = StealthyFetcher.fetch('https://example.com') ``` If you want to configure the parser (Adaptor class) that will be used on the response before returning it for you then do this first: ```python >>> StealthyFetcher.configure(auto_match=True, keep_comments=False) ``` or ```python >>> StealthyFetcher.auto_match = True >>> StealthyFetcher.keep_comments = False ``` Then continue your code as normal. The available configuration arguments are: `auto_match`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the `Adaptor` class. You can display the current configuration anytime by running `.display_config()` Also, the `Response` object returned from all fetchers is the same as the `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, `history`, and `request_headers`. All `cookies`, `headers`, and `request_headers` are always of type `dictionary`. > [!NOTE] > The `auto_match` argument is disabled by default, you need to enable it to use that feature. #### Set parser config per request As you probably understood, the logic above for setting the parser config will work globally for all requests/fetches done through that class and it's intended. If your use case requires you to use different config for each request/fetch, then you can pass a dictionary to the request method (`fetch`/`get`/`post`/...) to an argument named `custom_config`. ### Fetcher This class is built on top of [httpx](https://www.python-httpx.org/) with additional configuration options, here you can do `GET`, `POST`, `PUT`, and `DELETE` requests. For all methods, you have `stealthy_headers` which makes `Fetcher` create and use real browser's headers then create a referer header as if this request came from Google's search of this URL's domain. It's enabled by default. You can also set the number of retries with the argument `retries` for all methods and this will make httpx retry requests if it failed for any reason. The default number of retries for all `Fetcher` methods is 3. > Hence: All headers generated by `stealthy_headers` argument can be overwritten by you through the `headers` argument You can route all traffic (HTTP and HTTPS) to a proxy for any of these methods in this format `http://username:password@localhost:8030` ```python >> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True) >> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030') >> page = Fetcher.put('https://httpbin.org/put', data={'key': 'value'}) >> page = Fetcher.delete('https://httpbin.org/delete') ``` For Async requests, you will just replace the import like below: ```python >> from scrapling.fetchers import AsyncFetcher >> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True) >> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030') >> page = await AsyncFetcher.put('https://httpbin.org/put', data={'key': 'value'}) >> page = await AsyncFetcher.delete('https://httpbin.org/delete') ``` ### StealthyFetcher This class is built on top of [Camoufox](https://github.com/daijro/camoufox), bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to increase performance and undetectability even further. ```python >> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default >> page.status == 200 True >> page = await StealthyFetcher.async_fetch('https://www.browserscan.net/bot-detection') # the async version of fetch >> page.status == 200 True ``` > Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :) #### The complete list of arguments | Argument | Description | Optional | |:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| | url | Target url | ❌ | | headless | Pass `True` to run the browser in headless/hidden (**default**), `virtual` to run it in virtual screen mode, or `False` for headful/visible mode. The `virtual` mode requires having `xvfb` installed. | βœ”οΈ | | block_images | Prevent the loading of images through Firefox preferences. _This can help save your proxy usage but be careful with this option as it makes some websites never finish loading._ | βœ”οΈ | | disable_resources | Drop requests of unnecessary resources for a speed boost. It depends but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage but be careful with this option as it makes some websites never finish loading._ | βœ”οΈ | | google_search | Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name. | βœ”οΈ | | extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | βœ”οΈ | | block_webrtc | Blocks WebRTC entirely. | βœ”οΈ | | page_action | Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again. | βœ”οΈ | | addons | List of Firefox addons to use. **Must be paths to extracted addons.** | βœ”οΈ | | humanize | Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window. | βœ”οΈ | | allow_webgl | Enabled by default. Disabling it WebGL not recommended as many WAFs now checks if WebGL is enabled. | βœ”οΈ | | geoip | Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. | βœ”οΈ | | disable_ads | Disabled by default, this installs `uBlock Origin` addon on the browser if enabled. | βœ”οΈ | | network_idle | Wait for the page until there are no network connections for at least 500 ms. | βœ”οΈ | | timeout | The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000. | βœ”οΈ | | wait_selector | Wait for a specific css selector to be in a specific state. | βœ”οΈ | | proxy | The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. | βœ”οΈ | | os_randomize | If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS. | βœ”οΈ | | wait_selector_state | The state to wait for the selector given with `wait_selector`. _Default state is `attached`._ | βœ”οΈ | This list isn't final so expect a lot more additions and flexibility to be added in the next versions! ### PlayWrightFetcher This class is built on top of [Playwright](https://playwright.dev/python/) which currently provides 4 main run options but they can be mixed as you want. ```python >> page = PlayWrightFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option >> page.css_first("#search a::attr(href)") 'https://github.com/D4Vinci/Scrapling' >> page = await PlayWrightFetcher.async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # the async version of fetch >> page.css_first("#search a::attr(href)") 'https://github.com/D4Vinci/Scrapling' ``` > Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :) Using this Fetcher class, you can make requests with: 1) Vanilla Playwright without any modifications other than the ones you chose. 2) Stealthy Playwright with the stealth mode I wrote for it. It's still a WIP but it bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/).
Some of the things this fetcher's stealth mode does include: * Patching the CDP runtime fingerprint. * Mimics some of the real browsers' properties by injecting several JS files and using custom options. * Using custom flags on launch to hide Playwright even more and make it faster. * Generates real browser's headers of the same type and same user OS then append it to the request's headers. 3) Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher and most of the options can be enabled on it. 4) [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5)'s [docker browserless](https://hub.docker.com/r/nstbrowser/browserless) option by passing the CDP URL and enabling `nstbrowser_mode` option. > Hence using the `real_chrome` argument requires that you have Chrome browser installed on your device Add that to a lot of controlling/hiding options as you will see in the arguments list below. #### The complete list of arguments | Argument | Description | Optional | |:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| | url | Target url | ❌ | | headless | Pass `True` to run the browser in headless/hidden (**default**), or `False` for headful/visible mode. | βœ”οΈ | | disable_resources | Drop requests of unnecessary resources for a speed boost. It depends but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage but be careful with this option as it makes some websites never finish loading._ | βœ”οΈ | | useragent | Pass a useragent string to be used. **Otherwise the fetcher will generate a real Useragent of the same browser and use it.** | βœ”οΈ | | network_idle | Wait for the page until there are no network connections for at least 500 ms. | βœ”οΈ | | timeout | The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000. | βœ”οΈ | | page_action | Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again. | βœ”οΈ | | wait_selector | Wait for a specific css selector to be in a specific state. | βœ”οΈ | | wait_selector_state | The state to wait for the selector given with `wait_selector`. _Default state is `attached`._ | βœ”οΈ | | google_search | Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name. | βœ”οΈ | | extra_headers | A dictionary of extra headers to add to the request. The referer set by the `google_search` argument takes priority over the referer set here if used together. | βœ”οΈ | | proxy | The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. | βœ”οΈ | | hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | βœ”οΈ | | disable_webgl | Disables WebGL and WebGL 2.0 support entirely. | βœ”οΈ | | stealth | Enables stealth mode, always check the documentation to see what stealth mode does currently. | βœ”οΈ | | real_chrome | If you have Chrome browser installed on your device, enable this and the Fetcher will launch an instance of your browser and use it. | βœ”οΈ | | locale | Set the locale for the browser if wanted. The default value is `en-US`. | βœ”οΈ | | cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP. | βœ”οΈ | | nstbrowser_mode | Enables NSTBrowser mode, **it have to be used with `cdp_url` argument or it will get completely ignored.** | βœ”οΈ | | nstbrowser_config | The config you want to send with requests to the NSTBrowser. _If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config._ | βœ”οΈ | This list isn't final so expect a lot more additions and flexibility to be added in the next versions! ## Advanced Parsing Features ### Smart Navigation ```python >>> quote.tag 'div' >>> quote.parent
...'> >>> quote.parent.tag 'div' >>> quote.children [β€œThe...' parent='
, Tags: