Product 1
-Description 1
-diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 3690dc7..4e94941 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -2,7 +2,7 @@ name: Publish Python π distributions π¦ to PyPI on: release: - types: [created] + types: [created,published] jobs: build-n-publish: diff --git a/README.md b/README.md index 88a7fda..c1ac3b6 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,44 @@ # π·οΈ Scrapling: Undetectable, Lightning-Fast, and Easy Web Scraping with Python -[](https://github.com/D4Vinci/Scrapling/actions/workflows/tests.yml) [](https://badge.fury.io/py/Scrapling) [](https://pypi.org/project/scrapling/) [](https://pepy.tech/project/scrapling) +
++ + Installation + + Β· + + Overview + + Β· + + Selection methods + + Β· + + Choosing a fetcher + + Β· + + Migrating from Beautifulsoup + +
Dealing with failing web scrapers due to anti-bot protections or website changes? Meet Scrapling. Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. For both beginners and experts, Scrapling provides powerful features while maintaining simplicity. ```python ->> from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher +>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher +>> StealthyFetcher.auto_match = True # Fetch websites' source under the radar! >> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) >> print(page.status) @@ -20,7 +52,7 @@ Scrapling is a high-performance, intelligent web scraping library for Python tha [Scrapeless Deep SerpApi](https://www.scrapeless.com/en/product/deep-serp-api?utm_source=website&utm_medium=ads&utm_campaign=scraping&utm_term=d4vinci) From $0.10 per 1,000 queries with a 1-2 second response time! -Deep SerpApi is a dedicated search engine designed for large language models (LLMs) and AI agents, aiming to provide real-time, accurate and unbiased information to help AI applications retrieve and process data efficiently. +Deep SerpApi is a dedicated search engine designed for large language models (LLMs) and AI agents. It aims to provide real-time, accurate, and unbiased information to help AI applications retrieve and process data efficiently. - covering 20+ Google SERP scenarios and mainstream search engines. - support real-time data updates to ensure real-time and accurate information. - It can integrate information from all available online channels and search engines. @@ -49,59 +81,18 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL [](https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling) --- -## Table of content - * [Key Features](#key-features) - * [Fetch websites as you prefer](#fetch-websites-as-you-prefer-with-async-support) - * [Adaptive Scraping](#adaptive-scraping) - * [Performance](#performance) - * [Developing Experience](#developing-experience) - * [Getting Started](#getting-started) - * [Parsing Performance](#parsing-performance) - * [Text Extraction Speed Test (5000 nested elements).](#text-extraction-speed-test-5000-nested-elements) - * [Extraction By Text Speed Test](#extraction-by-text-speed-test) - * [Installation](#installation) - * [Fetching Websites](#fetching-websites) - * [Features](#features) - * [Fetcher class](#fetcher) - * [StealthyFetcher class](#stealthyfetcher) - * [PlayWrightFetcher class](#playwrightfetcher) - * [Advanced Parsing Features](#advanced-parsing-features) - * [Smart Navigation](#smart-navigation) - * [Content-based Selection & Finding Similar Elements](#content-based-selection--finding-similar-elements) - * [Handling Structural Changes](#handling-structural-changes) - * [Real World Scenario](#real-world-scenario) - * [Find elements by filters](#find-elements-by-filters) - * [Is That All?](#is-that-all) - * [More Advanced Usage](#more-advanced-usage) - * [β‘ Enlightening Questions and FAQs](#-enlightening-questions-and-faqs) - * [How does auto-matching work?](#how-does-auto-matching-work) - * [How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?](#how-does-the-auto-matching-work-if-i-didnt-pass-a-url-while-initializing-the-adaptor-object) - * [If all things about an element can change or get removed, what are the unique properties to be saved?](#if-all-things-about-an-element-can-change-or-get-removed-what-are-the-unique-properties-to-be-saved) - * [I have enabled the `auto_save`/`auto_match` parameter while selecting and it got completely ignored with a warning message](#i-have-enabled-the-auto_saveauto_match-parameter-while-selecting-and-it-got-completely-ignored-with-a-warning-message) - * [I have done everything as the docs but the auto-matching didn't return anything, what's wrong?](#i-have-done-everything-as-the-docs-but-the-auto-matching-didnt-return-anything-whats-wrong) - * [Can Scrapling replace code built on top of BeautifulSoup4?](#can-scrapling-replace-code-built-on-top-of-beautifulsoup4) - * [Can Scrapling replace code built on top of AutoScraper?](#can-scrapling-replace-code-built-on-top-of-autoscraper) - * [Is Scrapling thread-safe?](#is-scrapling-thread-safe) - * [More Sponsors!](#more-sponsors) - * [Contributing](#contributing) - * [Disclaimer for Scrapling Project](#disclaimer-for-scrapling-project) - * [License](#license) - * [Acknowledgments](#acknowledgments) - * [Thanks and References](#thanks-and-references) - * [Known Issues](#known-issues) - ## Key Features ### Fetch websites as you prefer with async support - **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. - **Dynamic Loading & Automation**: Fetch dynamic websites with the `PlayWrightFetcher` class through your real browser, Scrapling's stealth mode, Playwright's Chrome browser, or [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless! -- **Anti-bot Protections Bypass**: Easily bypass protections with `StealthyFetcher` and `PlayWrightFetcher` classes. +- **Anti-bot Protections Bypass**: Easily bypass protections with the `StealthyFetcher` and `PlayWrightFetcher` classes. ### Adaptive Scraping -- π **Smart Element Tracking**: Relocate elements after website changes, using an intelligent similarity system and integrated storage. -- π― **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search and more. +- π **Smart Element Tracking**: Relocate elements after website changes using an intelligent similarity system and integrated storage. +- π― **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search, and more. - π **Find Similar Elements**: Automatically locate elements similar to the element you found! -- π§ **Smart Content Scraping**: Extract data from multiple websites without specific selectors using Scrapling powerful features. +- π§ **Smart Content Scraping**: Extract data from multiple websites using Scrapling's powerful features without specific selectors. ### High Performance - π **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries. @@ -110,7 +101,7 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL ### Developer Friendly - π οΈ **Powerful Navigation API**: Easy DOM traversal in all directions. -- 𧬠**Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that takes less memory than standard dictionaries with added methods. +- 𧬠**Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries with added methods that consume less memory than standard dictionaries. - π **Auto Selectors Generation**: Generate robust short and full CSS/XPath selectors for any element. - π **Familiar API**: Similar to Scrapy/BeautifulSoup and the same pseudo-elements used in Scrapy. - π **Type hints**: Complete type/doc-strings coverage for future-proofing and best autocompletion support. @@ -120,14 +111,12 @@ Deep SerpApi is a dedicated search engine designed for large language models (LL ```python from scrapling.fetchers import Fetcher -fetcher = Fetcher(auto_match=False) - -# Do http GET request to a web page and create an Adaptor instance -page = fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True) -# Get all text content from all HTML tags in the page except `script` and `style` tags +# Do HTTP GET request to a web page and create an Adaptor instance +page = Fetcher.get('https://quotes.toscrape.com/', stealthy_headers=True) +# Get all text content from all HTML tags in the page except the `script` and `style` tags page.get_all_text(ignore_tags=('script', 'style')) -# Get all quotes elements, any of these methods will return a list of strings directly (TextHandlers) +# Get all quotes elements; any of these methods will return a list of strings directly (TextHandlers) quotes = page.css('.quote .text::text') # CSS selector quotes = page.xpath('//span[@class="text"]/text()') # XPath quotes = page.css('.quote').css('.text::text') # Chained selectors @@ -145,13 +134,16 @@ quotes = page.find_all(['div'], class_='quote') quotes = page.find_all(class_='quote') # and so on... # Working with elements -quote.html_content # Get Inner HTML of this element +quote.html_content # Get the Inner HTML of this element quote.prettify() # Prettified version of Inner HTML above quote.attrib # Get that element's attributes quote.path # DOM path to element (List of all ancestors from tag till the element itself) ``` To keep it simple, all methods can be chained on top of each other! +> [!NOTE] +> Check out the full documentation from [here](https://scrapling.readthedocs.io/en/latest/) + ## Parsing Performance Scrapling isn't just powerful - it's also blazing fast. Scrapling implements many best practices, design patterns, and numerous optimizations to save fractions of seconds. All of that while focusing exclusively on parsing HTML documents. @@ -159,6 +151,9 @@ Here are benchmarks comparing Scrapling to popular Python libraries in two tests ### Text Extraction Speed Test (5000 nested elements). +This test consists of extracting the text content of 5000 nested div elements. + + | # | Library | Time (ms) | vs Scrapling | |---|:-----------------:|:---------:|:------------:| | 1 | Scrapling | 5.44 | 1.0x | @@ -170,21 +165,31 @@ Here are benchmarks comparing Scrapling to popular Python libraries in two tests | 7 | MechanicalSoup | 1322.64 | 243.132x | | 8 | BS4 with html5lib | 3373.75 | 620.175x | -As you see, Scrapling is on par with Scrapy and slightly faster than Lxml which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml but still, Scrapling is 4 times faster. +As you see, Scrapling is on par with Scrapy and slightly faster than Lxml, which both libraries are built on top of. These are the closest results to Scrapling. PyQuery is also built on top of Lxml, but Scrapling is four times faster. ### Extraction By Text Speed Test +Scrapling can find elements based on its text content and find elements similar to these elements. The only known library with these two features, too, is AutoScraper. + +So, we compared this to see how fast Scrapling can be in these two tasks compared to AutoScraper. + +Here are the results: + | Library | Time (ms) | vs Scrapling | -|:-----------:|:---------:|:------------:| +|-------------|:---------:|:------------:| | Scrapling | 2.51 | 1.0x | | AutoScraper | 11.41 | 4.546x | -Scrapling can find elements with more methods and it returns full element `Adaptor` objects not only the text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. As you see, Scrapling is still 4.5 times faster at the same task. +Scrapling can find elements with more methods and returns the entire element's `Adaptor` object, not only text like AutoScraper. So, to make this test fair, both libraries will extract an element with text, find similar elements, and then extract the text content for all of them. + +As you see, Scrapling is still 4.5 times faster at the same task. + +If we made Scrapling extract the elements only without stopping to extract each element's text, we would get speed twice as fast as this, but as I said, to make it fair comparison a bit :smile: > All benchmarks' results are an average of 100 runs. See our [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology and to run your comparisons. ## Installation -Scrapling is a breeze to get started with; Starting from version 0.2.9, we require at least Python 3.9 to work. +Scrapling is a breeze to get started with. Starting from version 0.2.9, we require at least Python 3.9 to work. ```bash pip3 install scrapling ``` @@ -194,589 +199,6 @@ scrapling install ``` If you have any installation issues, please open an issue. -## Fetching Websites -Fetchers are interfaces built on top of other libraries with added features that do requests or fetch pages for you in a single request fashion and then return an `Adaptor` object. This feature was introduced because the only option we had before was to fetch the page as you wanted it, then pass it manually to the `Adaptor` class to create an `Adaptor` instance and start playing around with the page. - -### Features -You might be slightly confused by now so let me clear things up. All fetcher-type classes are imported in the same way -```python -from scrapling.fetchers import Fetcher, StealthyFetcher, PlayWrightFetcher -``` -All of them can take these initialization arguments: `auto_match`, `huge_tree`, `keep_comments`, `keep_cdata`, `storage`, and `storage_args`, which are the same ones you give to the `Adaptor` class. - -If you don't want to pass arguments to the generated `Adaptor` object and want to use the default values, you can use this import instead for cleaner code: -```python -from scrapling.defaults import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher -``` -then use it right away without initializing like: -```python -page = StealthyFetcher.fetch('https://example.com') -``` - -Also, the `Response` object returned from all fetchers is the same as the `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, `history`, and `request_headers`. All `cookies`, `headers`, and `request_headers` are always of type `dictionary`. -> [!NOTE] -> The `auto_match` argument is enabled by default which is the one you should care about the most as you will see later. -### Fetcher -This class is built on top of [httpx](https://www.python-httpx.org/) with additional configuration options, here you can do `GET`, `POST`, `PUT`, and `DELETE` requests. - -For all methods, you have `stealthy_headers` which makes `Fetcher` create and use real browser's headers then create a referer header as if this request came from Google's search of this URL's domain. It's enabled by default. You can also set the number of retries with the argument `retries` for all methods and this will make httpx retry requests if it failed for any reason. The default number of retries for all `Fetcher` methods is 3. - -> Hence: All headers generated by `stealthy_headers` argument can be overwritten by you through the `headers` argument - -You can route all traffic (HTTP and HTTPS) to a proxy for any of these methods in this format `http://username:password@localhost:8030` -```python ->> page = Fetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True) ->> page = Fetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030') ->> page = Fetcher().put('https://httpbin.org/put', data={'key': 'value'}) ->> page = Fetcher().delete('https://httpbin.org/delete') -``` -For Async requests, you will just replace the import like below: -```python ->> from scrapling.fetchers import AsyncFetcher ->> page = await AsyncFetcher().get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True) ->> page = await AsyncFetcher().post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030') ->> page = await AsyncFetcher().put('https://httpbin.org/put', data={'key': 'value'}) ->> page = await AsyncFetcher().delete('https://httpbin.org/delete') -``` -### StealthyFetcher -This class is built on top of [Camoufox](https://github.com/daijro/camoufox), bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to increase performance and undetectability even further. -```python ->> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection') # Running headless by default ->> page.status == 200 -True ->> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection') # the async version of fetch ->> page.status == 200 -True -``` -> Note: all requests done by this fetcher are waiting by default for all JS to be fully loaded and executed so you don't have to :) - -Β£51.77
' parent='Β£53.74
' parent='Β£50.10
' parent='Β£47.82
' parent=',
- ,
- ,
-...]
-
-# You will notice that the number of elements is 19 not 20 because the current element is not included.
->>> len(page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title']))
-19
-
-# Get the `href` attribute from all similar elements
->>> [element.attrib['href'] for element in page.find_by_text('Tipping the Velvet').find_similar(ignore_attributes=['title'])]
-['catalogue/a-light-in-the-attic_1000/index.html',
- 'catalogue/soumission_998/index.html',
- 'catalogue/sharp-objects_997/index.html',
- ...]
-```
-To increase the complexity a little bit, let's say we want to get all books' data using that element as a starting point for some reason
-```python
->>> for product in page.find_by_text('Tipping the Velvet').parent.parent.find_similar():
- print({
- "name": product.css_first('h3 a::text'),
- "price": product.css_first('.price_color').re_first(r'[\d\.]+'),
- "stock": product.css('.availability::text')[-1].clean()
- })
-{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
-{'name': 'Soumission', 'price': '50.10', 'stock': 'In stock'}
-{'name': 'Sharp Objects', 'price': '47.82', 'stock': 'In stock'}
-...
-```
-The [documentation](https://github.com/D4Vinci/Scrapling/tree/main/docs/Examples) will provide more advanced examples.
-
-### Handling Structural Changes
-Let's say you are scraping a page with a structure like this:
-```html
- Description 1 Description 2 Description 1 Description 2Product 1
- Product 2
- Product 1
- Product 2
-
It filters all elements in the current page/element in the following order:
-
-1. All elements with the passed tag name(s).
-2. All elements that match all passed attribute(s).
-3. All elements that its text content match all passed regex patterns.
-4. All elements that fulfill all passed function(s).
-
-Note: The filtering process always starts from the first filter it finds in the filtering order above so if no tag name(s) are passed but attributes are passed, the process starts from that layer and so on. **But the order in which you pass the arguments doesn't matter.**
-
-Examples to clear any confusion :)
-
-```python
->> from scrapling.fetchers import Fetcher
->> page = Fetcher().get('https://quotes.toscrape.com/')
-# Find all elements with tag name `div`.
->> page.find_all('div')
-[
-> I'm trying to rush creating the website, researching new ideas, and adding more features/tests/benchmarks but time is tight with too many spinning plates between work, personal life, and working on Scrapling. I have been working on Scrapling for months for free after all.
-> If you like `Scrapling` and want it to keep improving then this is a friendly reminder that you can help by supporting me through the [sponsor button](https://github.com/sponsors/D4Vinci).
-
-## β‘ Enlightening Questions and FAQs
-This section addresses common questions about Scrapling, please read this section before opening an issue.
-
-### How does auto-matching work?
- 1. You need to get a working selector and run it at least once with methods `css` or `xpath` with the `auto_save` parameter set to `True` before structural changes happen.
- 2. Before returning results for you, Scrapling uses its configured database and saves unique properties about that element.
- 3. Now because everything about the element can be changed or removed, nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
- 1. The domain of the URL you gave while initializing the first Adaptor object
- 2. The `identifier` parameter you passed to the method while selecting. If you didn't pass one, then the selector string itself will be used as an identifier but remember you will have to use it as an identifier value later when the structure changes and you want to pass the new selector.
-
- Together both are used to retrieve the element's unique properties from the database later.
- 4. Now later when you enable the `auto_match` parameter for both the Adaptor instance and the method call. The element properties are retrieved and Scrapling loops over all elements in the page and compares each one's unique properties to the unique properties we already have for this element and a score is calculated for each one.
- 5. Comparing elements is not exact but more about finding how similar these values are, so everything is taken into consideration, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
- 6. The score for each element is stored in the table, and the element(s) with the highest combined similarity scores are returned.
-
-### How does the auto-matching work if I didn't pass a URL while initializing the Adaptor object?
-Not a big problem as it depends on your usage. The word `default` will be used in place of the URL field while saving the element's unique properties. So this will only be an issue if you used the same identifier later for a different website that you didn't pass the URL parameter while initializing it as well. The save process will overwrite the previous data and auto-matching uses the latest saved properties only.
-
-### If all things about an element can change or get removed, what are the unique properties to be saved?
-For each element, Scrapling will extract:
-- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
-- Element's parent tag name, attributes (names and values), and text.
-
-### I have enabled the `auto_save`/`auto_match` parameter while selecting and it got completely ignored with a warning message
-That's because passing the `auto_save`/`auto_match` argument without setting `auto_match` to `True` while initializing the Adaptor object will only result in ignoring the `auto_save`/`auto_match` argument value. This behavior is purely for performance reasons so the database gets created only when you are planning to use the auto-matching features.
-
-### I have done everything as the docs but the auto-matching didn't return anything, what's wrong?
-It could be one of these reasons:
-1. No data were saved/stored for this element before.
-2. The selector passed is not the one used while storing element data. The solution is simple
- - Pass the old selector again as an identifier to the method called.
- - Retrieve the element with the retrieve method using the old selector as identifier then save it again with the save method and the new selector as identifier.
- - Start using the identifier argument more often if you are planning to use every new selector from now on.
-3. The website had some extreme structural changes like a new full design. If this happens a lot with this website, the solution would be to make your code as selector-free as possible using Scrapling features.
-
-### Can Scrapling replace code built on top of BeautifulSoup4?
-Pretty much yeah, almost all features you get from BeautifulSoup can be found or achieved in Scrapling one way or another. In fact, if you see there's a feature in bs4 that is missing in Scrapling, please make a feature request from the issues tab to let me know.
-
-### Can Scrapling replace code built on top of AutoScraper?
-Of course, you can find elements by text/regex, find similar elements in a more reliable way than AutoScraper, and finally save/retrieve elements manually to use later as the model feature in AutoScraper. I have pulled all top articles about AutoScraper from Google and tested Scrapling against examples in them. In all examples, Scrapling got the same results as AutoScraper in much less time.
-
-### Is Scrapling thread-safe?
-Yes, Scrapling instances are thread-safe. Each Adaptor instance maintains its state.
## More Sponsors!
@@ -789,7 +211,7 @@ Please read the [contributing file](https://github.com/D4Vinci/Scrapling/blob/ma
## Disclaimer for Scrapling Project
> [!CAUTION]
-> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international laws regarding data scraping and privacy. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules like the `robots.txt` file, for example.
+> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. This library should not be used to violate the rights of others, for unethical purposes, or to use data in an unauthorized or illegal manner. Do not use it on any website unless you have permission from the website owner or within their allowed rules, such as the `robots.txt` file.
## License
This work is licensed under BSD-3
@@ -806,7 +228,7 @@ This project includes code adapted from:
- [rebrowser-patches](https://github.com/rebrowser/rebrowser-patches)
## Known Issues
-- In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page that are in different locations, auto-matching will probably return to you the first element only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector for example) as these selectors get separated and each selector gets executed alone.
+- In the auto-matching save process, the unique properties of the first element from the selection results are the only ones that get saved. If the selector you are using selects different elements on the page in different locations, auto-matching will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
---
-
+
\ No newline at end of file
diff --git a/scrapling/__init__.py b/scrapling/__init__.py
index feda050..ee918f1 100644
--- a/scrapling/__init__.py
+++ b/scrapling/__init__.py
@@ -1,6 +1,6 @@
__author__ = "Karim Shoair (karim.shoair@pm.me)"
-__version__ = "0.2.98"
+__version__ = "0.2.99"
__copyright__ = "Copyright (c) 2024 Karim Shoair"
diff --git a/scrapling/defaults.py b/scrapling/defaults.py
index 4098f76..903cf6f 100644
--- a/scrapling/defaults.py
+++ b/scrapling/defaults.py
@@ -1,19 +1,25 @@
-# If you are going to use Fetchers with the default settings, import them from this file instead for a cleaner looking code
+# Left this file for backward-compatibility before 0.2.99
+from scrapling.core.utils import log
+
# A lightweight approach to create lazy loader for each import for backward compatibility
# This will reduces initial memory footprint significantly (only loads what's used)
def __getattr__(name):
if name == 'Fetcher':
from scrapling.fetchers import Fetcher as cls
- return cls()
+ log.warning('This import is deprecated now and it will be removed with v0.3. Use `from scrapling.fetchers import Fetcher` instead')
+ return cls
elif name == 'AsyncFetcher':
from scrapling.fetchers import AsyncFetcher as cls
- return cls()
+ log.warning('This import is deprecated now and it will be removed with v0.3. Use `from scrapling.fetchers import AsyncFetcher` instead')
+ return cls
elif name == 'StealthyFetcher':
from scrapling.fetchers import StealthyFetcher as cls
- return cls()
+ log.warning('This import is deprecated now and it will be removed with v0.3. Use `from scrapling.fetchers import StealthyFetcher` instead')
+ return cls
elif name == 'PlayWrightFetcher':
from scrapling.fetchers import PlayWrightFetcher as cls
- return cls()
+ log.warning('This import is deprecated now and it will be removed with v0.3. Use `from scrapling.fetchers import PlayWrightFetcher` instead')
+ return cls
else:
raise AttributeError(f"module 'scrapling' has no attribute '{name}'")
diff --git a/scrapling/engines/camo.py b/scrapling/engines/camo.py
index 7893526..54d9555 100644
--- a/scrapling/engines/camo.py
+++ b/scrapling/engines/camo.py
@@ -16,12 +16,13 @@ from scrapling.engines.toolbelt import (Response, StatusText,
class CamoufoxEngine:
def __init__(
self, headless: Union[bool, Literal['virtual']] = True, block_images: bool = False, disable_resources: bool = False,
- block_webrtc: bool = False, allow_webgl: bool = True, network_idle: bool = False, humanize: Union[bool, float] = True,
+ block_webrtc: bool = False, allow_webgl: bool = True, network_idle: bool = False, humanize: Union[bool, float] = True, wait: Optional[int] = 0,
timeout: Optional[float] = 30000, page_action: Callable = None, wait_selector: Optional[str] = None, addons: Optional[List[str]] = None,
wait_selector_state: SelectorWaitStates = 'attached', google_search: bool = True, extra_headers: Optional[Dict[str, str]] = None,
proxy: Optional[Union[str, Dict[str, str]]] = None, os_randomize: bool = False, disable_ads: bool = False,
geoip: bool = False,
adaptor_arguments: Dict = None,
+ additional_arguments: Dict = None
):
"""An engine that utilizes Camoufox library, check the `StealthyFetcher` class for more documentation.
@@ -38,6 +39,7 @@ class CamoufoxEngine:
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param disable_ads: Disabled by default, this installs `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning `Response` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000
:param page_action: Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again.
:param wait_selector: Wait for a specific css selector to be in a specific state.
@@ -48,6 +50,7 @@ class CamoufoxEngine:
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param adaptor_arguments: The arguments that will be passed in the end while creating the final Adaptor's class.
+ :param additional_arguments: Additional arguments to be passed to Camoufox as additional settings and it takes higher priority than Scrapling's settings.
"""
self.headless = headless
self.block_images = bool(block_images)
@@ -60,10 +63,12 @@ class CamoufoxEngine:
self.disable_ads = bool(disable_ads)
self.geoip = bool(geoip)
self.extra_headers = extra_headers or {}
+ self.additional_arguments = additional_arguments or {}
self.proxy = construct_proxy_dict(proxy)
self.addons = addons or []
self.humanize = humanize
self.timeout = check_type_validity(timeout, [int, float], 30000)
+ self.wait = check_type_validity(wait, [int, float], 0)
# Page action callable validation
self.page_action = None
@@ -92,6 +97,7 @@ class CamoufoxEngine:
"block_webrtc": self.block_webrtc,
"block_images": self.block_images, # Careful! it makes some websites doesn't finish loading at all like stackoverflow even in headful
"os": None if self.os_randomize else get_os_name(),
+ **self.additional_arguments
}
def _process_response_history(self, first_response):
@@ -126,6 +132,38 @@ class CamoufoxEngine:
return history
+ async def _async_process_response_history(self, first_response):
+ """Process response history to build a list of Response objects"""
+ history = []
+ current_request = first_response.request.redirected_from
+
+ try:
+ while current_request:
+ try:
+ current_response = await current_request.response()
+ history.insert(0, Response(
+ url=current_request.url,
+ # using current_response.text() will trigger "Error: Response.text: Response body is unavailable for redirect responses"
+ text='',
+ body=b'',
+ status=current_response.status if current_response else 301,
+ reason=(current_response.status_text or StatusText.get(current_response.status)) if current_response else StatusText.get(301),
+ encoding=current_response.headers.get('content-type', '') or 'utf-8',
+ cookies={},
+ headers=await current_response.all_headers() if current_response else {},
+ request_headers=await current_request.all_headers(),
+ **self.adaptor_arguments
+ ))
+ except Exception as e:
+ log.error(f"Error processing redirect: {e}")
+ break
+
+ current_request = current_request.redirected_from
+ except Exception as e:
+ log.error(f"Error processing response history: {e}")
+
+ return history
+
def fetch(self, url: str) -> Response:
"""Opens up the browser and do your request based on your chosen options.
@@ -177,6 +215,7 @@ class CamoufoxEngine:
except Exception as e:
log.error(f"Error waiting for selector {self.wait_selector}: {e}")
+ page.wait_for_timeout(self.wait)
# In case we didn't catch a document type somehow
final_response = final_response if final_response else first_response
if not final_response:
@@ -263,6 +302,7 @@ class CamoufoxEngine:
except Exception as e:
log.error(f"Error waiting for selector {self.wait_selector}: {e}")
+ await page.wait_for_timeout(self.wait)
# In case we didn't catch a document type somehow
final_response = final_response if final_response else first_response
if not final_response:
@@ -273,7 +313,7 @@ class CamoufoxEngine:
# PlayWright API sometimes give empty status text for some reason!
status_text = final_response.status_text or StatusText.get(final_response.status)
- history = self._process_response_history(first_response)
+ history = await self._async_process_response_history(first_response)
try:
page_content = await page.content()
except Exception as e:
diff --git a/scrapling/engines/pw.py b/scrapling/engines/pw.py
index 0c71fec..b94043f 100644
--- a/scrapling/engines/pw.py
+++ b/scrapling/engines/pw.py
@@ -21,6 +21,7 @@ class PlaywrightEngine:
useragent: Optional[str] = None,
network_idle: bool = False,
timeout: Optional[float] = 30000,
+ wait: Optional[int] = 0,
page_action: Callable = None,
wait_selector: Optional[str] = None,
locale: Optional[str] = 'en-US',
@@ -46,6 +47,7 @@ class PlaywrightEngine:
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning `Response` object.
:param page_action: Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again.
:param wait_selector: Wait for a specific css selector to be in a specific state.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
@@ -76,6 +78,7 @@ class PlaywrightEngine:
self.cdp_url = cdp_url
self.useragent = useragent
self.timeout = check_type_validity(timeout, [int, float], 30000)
+ self.wait = check_type_validity(wait, [int, float], 0)
if page_action is not None:
if callable(page_action):
self.page_action = page_action
@@ -220,6 +223,38 @@ class PlaywrightEngine:
return history
+ async def _async_process_response_history(self, first_response):
+ """Process response history to build a list of Response objects"""
+ history = []
+ current_request = first_response.request.redirected_from
+
+ try:
+ while current_request:
+ try:
+ current_response = await current_request.response()
+ history.insert(0, Response(
+ url=current_request.url,
+ # using current_response.text() will trigger "Error: Response.text: Response body is unavailable for redirect responses"
+ text='',
+ body=b'',
+ status=current_response.status if current_response else 301,
+ reason=(current_response.status_text or StatusText.get(current_response.status)) if current_response else StatusText.get(301),
+ encoding=current_response.headers.get('content-type', '') or 'utf-8',
+ cookies={},
+ headers=await current_response.all_headers() if current_response else {},
+ request_headers=await current_request.all_headers(),
+ **self.adaptor_arguments
+ ))
+ except Exception as e:
+ log.error(f"Error processing redirect: {e}")
+ break
+
+ current_request = current_request.redirected_from
+ except Exception as e:
+ log.error(f"Error processing response history: {e}")
+
+ return history
+
def fetch(self, url: str) -> Response:
"""Opens up the browser and do your request based on your chosen options.
@@ -289,6 +324,7 @@ class PlaywrightEngine:
except Exception as e:
log.error(f"Error waiting for selector {self.wait_selector}: {e}")
+ page.wait_for_timeout(self.wait)
# In case we didn't catch a document type somehow
final_response = final_response if final_response else first_response
if not final_response:
@@ -392,6 +428,7 @@ class PlaywrightEngine:
except Exception as e:
log.error(f"Error waiting for selector {self.wait_selector}: {e}")
+ await page.wait_for_timeout(self.wait)
# In case we didn't catch a document type somehow
final_response = final_response if final_response else first_response
if not final_response:
@@ -402,7 +439,7 @@ class PlaywrightEngine:
# PlayWright API sometimes give empty status text for some reason!
status_text = final_response.status_text or StatusText.get(final_response.status)
- history = self._process_response_history(first_response)
+ history = await self._async_process_response_history(first_response)
try:
page_content = await page.content()
except Exception as e:
diff --git a/scrapling/engines/toolbelt/custom.py b/scrapling/engines/toolbelt/custom.py
index c91a3a8..eb01345 100644
--- a/scrapling/engines/toolbelt/custom.py
+++ b/scrapling/engines/toolbelt/custom.py
@@ -105,41 +105,77 @@ class Response(Adaptor):
class BaseFetcher:
- def __init__(
- self, huge_tree: bool = True, keep_comments: Optional[bool] = False, auto_match: Optional[bool] = True,
- storage: Any = SQLiteStorageSystem, storage_args: Optional[Dict] = None,
- automatch_domain: Optional[str] = None, keep_cdata: Optional[bool] = False,
- ):
- """Arguments below are the same from the Adaptor class so you can pass them directly, the rest of Adaptor's arguments
- are detected and passed automatically from the Fetcher based on the response for accessibility.
+ __slots__ = ()
+ huge_tree: bool = True
+ auto_match: Optional[bool] = False
+ storage: Any = SQLiteStorageSystem
+ keep_cdata: Optional[bool] = False
+ storage_args: Optional[Dict] = None
+ keep_comments: Optional[bool] = False
+ automatch_domain: Optional[str] = None
+ parser_keywords: Tuple = ('huge_tree', 'auto_match', 'storage', 'keep_cdata', 'storage_args', 'keep_comments', 'automatch_domain',) # Left open for the user
- :param huge_tree: Enabled by default, should always be enabled when parsing large HTML documents. This controls
- libxml2 feature that forbids parsing certain large documents to protect from possible memory exhaustion.
- :param keep_comments: While parsing the HTML body, drop comments or not. Disabled by default for obvious reasons
- :param keep_cdata: While parsing the HTML body, drop cdata or not. Disabled by default for cleaner HTML.
- :param auto_match: Globally turn-off the auto-match feature in all functions, this argument takes higher
- priority over all auto-match related arguments/functions in the class.
- :param storage: The storage class to be passed for auto-matching functionalities, see ``Docs`` for more info.
- :param storage_args: A dictionary of ``argument->value`` pairs to be passed for the storage class.
- If empty, default values will be used.
- :param automatch_domain: For cases where you want to automatch selectors across different websites as if they were on the same website, use this argument to unify them.
- Otherwise, the domain of the request is used by default.
+ def __init__(self, *args, **kwargs):
+ # For backward-compatibility before 0.2.99
+ args_str = ", ".join(args) or ''
+ kwargs_str = ", ".join(f'{k}={v}' for k, v in kwargs.items()) or ''
+ if args_str:
+ args_str += ', '
+
+ log.warning(f'This logic is deprecated now, and have no effect; It will be removed with v0.3. Use `{self.__class__.__name__}.configure({args_str}{kwargs_str})` instead before fetching')
+ pass
+
+ @classmethod
+ def display_config(cls):
+ return dict(
+ huge_tree=cls.huge_tree,
+ keep_comments=cls.keep_comments,
+ keep_cdata=cls.keep_cdata,
+ auto_match=cls.auto_match,
+ storage=cls.storage,
+ storage_args=cls.storage_args,
+ automatch_domain=cls.automatch_domain,
+ )
+
+ @classmethod
+ def configure(cls, **kwargs):
+ """Set multiple arguments for the parser at once globally
+
+ :param kwargs: The keywords can be any arguments of the following: huge_tree, keep_comments, keep_cdata, auto_match, storage, storage_args, automatch_domain
"""
+ for key, value in kwargs.items():
+ key = key.strip().lower()
+ if hasattr(cls, key):
+ if key in cls.parser_keywords:
+ setattr(cls, key, value)
+ else:
+ # Yup, no fun allowed LOL
+ raise AttributeError(f'Unknown parser argument: "{key}"; maybe you meant {cls.parser_keywords}?')
+ else:
+ raise ValueError(f'Unknown parser argument: "{key}"; maybe you meant {cls.parser_keywords}?')
+
+ if not kwargs:
+ raise AttributeError(f'You must pass a keyword to configure, current keywords: {cls.parser_keywords}?')
+
+ @classmethod
+ def _generate_parser_arguments(cls) -> Dict:
# Adaptor class parameters
# I won't validate Adaptor's class parameters here again, I will leave it to be validated later
- self.adaptor_arguments = dict(
- huge_tree=huge_tree,
- keep_comments=keep_comments,
- keep_cdata=keep_cdata,
- auto_match=auto_match,
- storage=storage,
- storage_args=storage_args
+ parser_arguments = dict(
+ huge_tree=cls.huge_tree,
+ keep_comments=cls.keep_comments,
+ keep_cdata=cls.keep_cdata,
+ auto_match=cls.auto_match,
+ storage=cls.storage,
+ storage_args=cls.storage_args
)
- if automatch_domain:
- if type(automatch_domain) is not str:
+ if cls.automatch_domain:
+ if type(cls.automatch_domain) is not str:
log.warning('[Ignored] The argument "automatch_domain" must be of string type')
else:
- self.adaptor_arguments.update({'automatch_domain': automatch_domain})
+ parser_arguments.update({'automatch_domain': cls.automatch_domain})
+
+ return parser_arguments
class StatusText:
diff --git a/scrapling/fetchers.py b/scrapling/fetchers.py
index 4dadfe0..ad0e94c 100644
--- a/scrapling/fetchers.py
+++ b/scrapling/fetchers.py
@@ -10,9 +10,10 @@ class Fetcher(BaseFetcher):
Any additional keyword arguments passed to the methods below are passed to the respective httpx's method directly.
"""
+ @classmethod
def get(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP GET request for you but with some added flavors.
:param url: Target url.
@@ -22,16 +23,23 @@ class Fetcher(BaseFetcher):
create a referer header as if this request had came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.get()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries, adaptor_arguments=adaptor_arguments).get(**kwargs)
return response_object
+ @classmethod
def post(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP POST request for you but with some added flavors.
:param url: Target url.
@@ -41,16 +49,23 @@ class Fetcher(BaseFetcher):
create a referer header as if this request came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.post()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries, adaptor_arguments=adaptor_arguments).post(**kwargs)
return response_object
+ @classmethod
def put(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP PUT request for you but with some added flavors.
:param url: Target url
@@ -60,17 +75,24 @@ class Fetcher(BaseFetcher):
create a referer header as if this request came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.put()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries, adaptor_arguments=adaptor_arguments).put(**kwargs)
return response_object
+ @classmethod
def delete(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP DELETE request for you but with some added flavors.
:param url: Target url
@@ -80,18 +102,25 @@ class Fetcher(BaseFetcher):
create a referer header as if this request came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.delete()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries, adaptor_arguments=adaptor_arguments).delete(**kwargs)
return response_object
class AsyncFetcher(Fetcher):
+ @classmethod
async def get(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP GET request for you but with some added flavors.
:param url: Target url.
@@ -101,16 +130,23 @@ class AsyncFetcher(Fetcher):
create a referer header as if this request had came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.get()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = await StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries=retries, adaptor_arguments=adaptor_arguments).async_get(**kwargs)
return response_object
+ @classmethod
async def post(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP POST request for you but with some added flavors.
:param url: Target url.
@@ -120,16 +156,23 @@ class AsyncFetcher(Fetcher):
create a referer header as if this request came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.post()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = await StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries=retries, adaptor_arguments=adaptor_arguments).async_post(**kwargs)
return response_object
+ @classmethod
async def put(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP PUT request for you but with some added flavors.
:param url: Target url
@@ -139,16 +182,23 @@ class AsyncFetcher(Fetcher):
create a referer header as if this request came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.put()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = await StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries=retries, adaptor_arguments=adaptor_arguments).async_put(**kwargs)
return response_object
+ @classmethod
async def delete(
- self, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
- proxy: Optional[str] = None, retries: Optional[int] = 3, **kwargs: Dict) -> Response:
+ cls, url: str, follow_redirects: bool = True, timeout: Optional[Union[int, float]] = 10, stealthy_headers: bool = True,
+ proxy: Optional[str] = None, retries: Optional[int] = 3, custom_config: Dict = None, **kwargs: Dict) -> Response:
"""Make basic HTTP DELETE request for you but with some added flavors.
:param url: Target url
@@ -158,10 +208,16 @@ class AsyncFetcher(Fetcher):
create a referer header as if this request came from Google's search of this URL's domain.
:param proxy: A string of a proxy to use for http and https requests, the format accepted is `http://username:password@localhost:8030`
:param retries: The number of retries to do through httpx if the request failed for any reason. The default is 3 retries.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:param kwargs: Any additional keyword arguments are passed directly to `httpx.delete()` function so check httpx documentation for details.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
- adaptor_arguments = tuple(self.adaptor_arguments.items())
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
+ adaptor_arguments = tuple({**cls._generate_parser_arguments(), **custom_config}.items())
response_object = await StaticEngine(url, proxy, stealthy_headers, follow_redirects, timeout, retries=retries, adaptor_arguments=adaptor_arguments).async_delete(**kwargs)
return response_object
@@ -172,12 +228,14 @@ class StealthyFetcher(BaseFetcher):
It works as real browsers passing almost all online tests/protections based on Camoufox.
Other added flavors include setting the faked OS fingerprints to match the user's OS and the referer of every request is set as if this request came from Google's search of this URL's domain.
"""
+ @classmethod
def fetch(
- self, url: str, headless: Union[bool, Literal['virtual']] = True, block_images: bool = False, disable_resources: bool = False,
- block_webrtc: bool = False, allow_webgl: bool = True, network_idle: bool = False, addons: Optional[List[str]] = None,
+ cls, url: str, headless: Union[bool, Literal['virtual']] = True, block_images: bool = False, disable_resources: bool = False,
+ block_webrtc: bool = False, allow_webgl: bool = True, network_idle: bool = False, addons: Optional[List[str]] = None, wait: Optional[int] = 0,
timeout: Optional[float] = 30000, page_action: Callable = None, wait_selector: Optional[str] = None, humanize: Optional[Union[bool, float]] = True,
wait_selector_state: SelectorWaitStates = 'attached', google_search: bool = True, extra_headers: Optional[Dict[str, str]] = None,
proxy: Optional[Union[str, Dict[str, str]]] = None, os_randomize: bool = False, disable_ads: bool = False, geoip: bool = False,
+ custom_config: Dict = None, additional_arguments: Dict = None
) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
@@ -198,16 +256,25 @@ class StealthyFetcher(BaseFetcher):
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000.
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning `Response` object.
:param page_action: Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again.
:param wait_selector: Wait for a specific css selector to be in a specific state.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. Default state is `attached`.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
+ :param additional_arguments: Additional arguments to be passed to Camoufox as additional settings and it takes higher priority than Scrapling's settings.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
engine = CamoufoxEngine(
+ wait=wait,
proxy=proxy,
geoip=geoip,
addons=addons,
@@ -226,16 +293,19 @@ class StealthyFetcher(BaseFetcher):
extra_headers=extra_headers,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- adaptor_arguments=self.adaptor_arguments,
+ adaptor_arguments={**cls._generate_parser_arguments(), **custom_config},
+ additional_arguments=additional_arguments or {}
)
return engine.fetch(url)
+ @classmethod
async def async_fetch(
- self, url: str, headless: Union[bool, Literal['virtual']] = True, block_images: bool = False, disable_resources: bool = False,
- block_webrtc: bool = False, allow_webgl: bool = True, network_idle: bool = False, addons: Optional[List[str]] = None,
+ cls, url: str, headless: Union[bool, Literal['virtual']] = True, block_images: bool = False, disable_resources: bool = False,
+ block_webrtc: bool = False, allow_webgl: bool = True, network_idle: bool = False, addons: Optional[List[str]] = None, wait: Optional[int] = 0,
timeout: Optional[float] = 30000, page_action: Callable = None, wait_selector: Optional[str] = None, humanize: Optional[Union[bool, float]] = True,
wait_selector_state: SelectorWaitStates = 'attached', google_search: bool = True, extra_headers: Optional[Dict[str, str]] = None,
proxy: Optional[Union[str, Dict[str, str]]] = None, os_randomize: bool = False, disable_ads: bool = False, geoip: bool = False,
+ custom_config: Dict = None, additional_arguments: Dict = None
) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
@@ -257,15 +327,24 @@ class StealthyFetcher(BaseFetcher):
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning `Response` object.
:param page_action: Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again.
:param wait_selector: Wait for a specific css selector to be in a specific state.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. Default state is `attached`.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search for this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
+ :param additional_arguments: Additional arguments to be passed to Camoufox as additional settings and it takes higher priority than Scrapling's settings.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
engine = CamoufoxEngine(
+ wait=wait,
proxy=proxy,
geoip=geoip,
addons=addons,
@@ -284,7 +363,8 @@ class StealthyFetcher(BaseFetcher):
extra_headers=extra_headers,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- adaptor_arguments=self.adaptor_arguments,
+ adaptor_arguments={**cls._generate_parser_arguments(), **custom_config},
+ additional_arguments=additional_arguments or {}
)
return await engine.async_fetch(url)
@@ -305,15 +385,17 @@ class PlayWrightFetcher(BaseFetcher):
> Note that these are the main options with PlayWright but it can be mixed together.
"""
+ @classmethod
def fetch(
- self, url: str, headless: Union[bool, str] = True, disable_resources: bool = None,
- useragent: Optional[str] = None, network_idle: bool = False, timeout: Optional[float] = 30000,
+ cls, url: str, headless: Union[bool, str] = True, disable_resources: bool = None,
+ useragent: Optional[str] = None, network_idle: bool = False, timeout: Optional[float] = 30000, wait: Optional[int] = 0,
page_action: Optional[Callable] = None, wait_selector: Optional[str] = None, wait_selector_state: SelectorWaitStates = 'attached',
hide_canvas: bool = False, disable_webgl: bool = False, extra_headers: Optional[Dict[str, str]] = None, google_search: bool = True,
proxy: Optional[Union[str, Dict[str, str]]] = None, locale: Optional[str] = 'en-US',
stealth: bool = False, real_chrome: bool = False,
cdp_url: Optional[str] = None,
nstbrowser_mode: bool = False, nstbrowser_config: Optional[Dict] = None,
+ custom_config: Dict = None
) -> Response:
"""Opens up a browser and do your request based on your chosen options below.
@@ -324,7 +406,8 @@ class PlayWrightFetcher(BaseFetcher):
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000.
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning `Response` object.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param page_action: Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again.
:param wait_selector: Wait for a specific css selector to be in a specific state.
@@ -339,9 +422,16 @@ class PlayWrightFetcher(BaseFetcher):
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP.
:param nstbrowser_mode: Enables NSTBrowser mode, it have to be used with `cdp_url` argument or it will get completely ignored.
:param nstbrowser_config: The config you want to send with requests to the NSTBrowser. If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
engine = PlaywrightEngine(
+ wait=wait,
proxy=proxy,
locale=locale,
timeout=timeout,
@@ -361,19 +451,21 @@ class PlayWrightFetcher(BaseFetcher):
nstbrowser_config=nstbrowser_config,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- adaptor_arguments=self.adaptor_arguments,
+ adaptor_arguments={**cls._generate_parser_arguments(), **custom_config},
)
return engine.fetch(url)
+ @classmethod
async def async_fetch(
- self, url: str, headless: Union[bool, str] = True, disable_resources: bool = None,
- useragent: Optional[str] = None, network_idle: bool = False, timeout: Optional[float] = 30000,
+ cls, url: str, headless: Union[bool, str] = True, disable_resources: bool = None,
+ useragent: Optional[str] = None, network_idle: bool = False, timeout: Optional[float] = 30000, wait: Optional[int] = 0,
page_action: Optional[Callable] = None, wait_selector: Optional[str] = None, wait_selector_state: SelectorWaitStates = 'attached',
hide_canvas: bool = False, disable_webgl: bool = False, extra_headers: Optional[Dict[str, str]] = None, google_search: bool = True,
proxy: Optional[Union[str, Dict[str, str]]] = None, locale: Optional[str] = 'en-US',
stealth: bool = False, real_chrome: bool = False,
cdp_url: Optional[str] = None,
nstbrowser_mode: bool = False, nstbrowser_config: Optional[Dict] = None,
+ custom_config: Dict = None
) -> Response:
"""Opens up a browser and do your request based on your chosen options below.
@@ -384,7 +476,8 @@ class PlayWrightFetcher(BaseFetcher):
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30000.
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning `Response` object.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param page_action: Added for automation. A function that takes the `page` object, does the automation you need, then returns `page` again.
:param wait_selector: Wait for a specific css selector to be in a specific state.
@@ -399,9 +492,16 @@ class PlayWrightFetcher(BaseFetcher):
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers/NSTBrowser through CDP.
:param nstbrowser_mode: Enables NSTBrowser mode, it have to be used with `cdp_url` argument or it will get completely ignored.
:param nstbrowser_config: The config you want to send with requests to the NSTBrowser. If left empty, Scrapling defaults to an optimized NSTBrowser's docker browserless config.
+ :param custom_config: A dictionary of custom parser arguments to use with this request. Any argument passed will override any class parameters values.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`
"""
+ if not custom_config:
+ custom_config = {}
+ elif not isinstance(custom_config, dict):
+ ValueError(f"The custom parser config must be of type dictionary, got {cls.__class__}")
+
engine = PlaywrightEngine(
+ wait=wait,
proxy=proxy,
locale=locale,
timeout=timeout,
@@ -421,12 +521,13 @@ class PlayWrightFetcher(BaseFetcher):
nstbrowser_config=nstbrowser_config,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- adaptor_arguments=self.adaptor_arguments,
+ adaptor_arguments={**cls._generate_parser_arguments(), **custom_config},
)
return await engine.async_fetch(url)
class CustomFetcher(BaseFetcher):
- def fetch(self, url: str, browser_engine, **kwargs) -> Response:
- engine = check_if_engine_usable(browser_engine)(adaptor_arguments=self.adaptor_arguments, **kwargs)
+ @classmethod
+ def fetch(cls, url: str, browser_engine, **kwargs) -> Response:
+ engine = check_if_engine_usable(browser_engine)(adaptor_arguments=cls._generate_parser_arguments(), **kwargs)
return engine.fetch(url)
diff --git a/scrapling/parser.py b/scrapling/parser.py
index 98e0864..b3967ec 100644
--- a/scrapling/parser.py
+++ b/scrapling/parser.py
@@ -39,7 +39,7 @@ class Adaptor(SelectorsGeneration):
root: Optional[html.HtmlElement] = None,
keep_comments: Optional[bool] = False,
keep_cdata: Optional[bool] = False,
- auto_match: Optional[bool] = True,
+ auto_match: Optional[bool] = False,
storage: Any = SQLiteStorageSystem,
storage_args: Optional[Dict] = None,
**kwargs
diff --git a/setup.cfg b/setup.cfg
index 35d934c..17c82bd 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
[metadata]
name = scrapling
-version = 0.2.98
+version = 0.2.99
author = Karim Shoair
author_email = karim.shoair@pm.me
description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy again!
diff --git a/setup.py b/setup.py
index b060b9b..bc6c41b 100644
--- a/setup.py
+++ b/setup.py
@@ -6,7 +6,7 @@ with open("README.md", "r", encoding="utf-8") as fh:
setup(
name="scrapling",
- version="0.2.98",
+ version="0.2.99",
description="""Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy again! In an internet filled with complications,
it simplifies web scraping, even when websites' design changes, while providing impressive speed that surpasses almost all alternatives.""",
long_description=long_description,
@@ -65,7 +65,7 @@ setup(
python_requires=">=3.9",
url="https://github.com/D4Vinci/Scrapling",
project_urls={
- "Documentation": "https://github.com/D4Vinci/Scrapling/tree/main/docs", # For now
+ "Documentation": "https://scrapling.readthedocs.io/en/latest/",
"Source": "https://github.com/D4Vinci/Scrapling",
"Tracker": "https://github.com/D4Vinci/Scrapling/issues",
}
diff --git a/tests/fetchers/async/test_camoufox.py b/tests/fetchers/async/test_camoufox.py
index f2f495d..937ec43 100644
--- a/tests/fetchers/async/test_camoufox.py
+++ b/tests/fetchers/async/test_camoufox.py
@@ -3,13 +3,15 @@ import pytest_httpbin
from scrapling import StealthyFetcher
+StealthyFetcher.auto_match = True
+
@pytest_httpbin.use_class_based_httpbin
@pytest.mark.asyncio
class TestStealthyFetcher:
@pytest.fixture(scope="class")
def fetcher(self):
- return StealthyFetcher(auto_match=False)
+ return StealthyFetcher
@pytest.fixture(scope="class")
def urls(self, httpbin):
diff --git a/tests/fetchers/async/test_httpx.py b/tests/fetchers/async/test_httpx.py
index 67cc037..64b7fae 100644
--- a/tests/fetchers/async/test_httpx.py
+++ b/tests/fetchers/async/test_httpx.py
@@ -3,13 +3,15 @@ import pytest_httpbin
from scrapling.fetchers import AsyncFetcher
+AsyncFetcher.auto_match = True
+
@pytest_httpbin.use_class_based_httpbin
@pytest.mark.asyncio
class TestAsyncFetcher:
@pytest.fixture(scope="class")
def fetcher(self):
- return AsyncFetcher(auto_match=True)
+ return AsyncFetcher
@pytest.fixture(scope="class")
def urls(self, httpbin):
diff --git a/tests/fetchers/async/test_playwright.py b/tests/fetchers/async/test_playwright.py
index cf29ebf..ff50d09 100644
--- a/tests/fetchers/async/test_playwright.py
+++ b/tests/fetchers/async/test_playwright.py
@@ -3,12 +3,14 @@ import pytest_httpbin
from scrapling import PlayWrightFetcher
+PlayWrightFetcher.auto_match = True
+
@pytest_httpbin.use_class_based_httpbin
class TestPlayWrightFetcherAsync:
@pytest.fixture
def fetcher(self):
- return PlayWrightFetcher(auto_match=False)
+ return PlayWrightFetcher
@pytest.fixture
def urls(self, httpbin):
diff --git a/tests/fetchers/sync/test_camoufox.py b/tests/fetchers/sync/test_camoufox.py
index 33800f4..c613c58 100644
--- a/tests/fetchers/sync/test_camoufox.py
+++ b/tests/fetchers/sync/test_camoufox.py
@@ -3,13 +3,15 @@ import pytest_httpbin
from scrapling import StealthyFetcher
+StealthyFetcher.auto_match = True
+
@pytest_httpbin.use_class_based_httpbin
class TestStealthyFetcher:
@pytest.fixture(scope="class")
def fetcher(self):
"""Fixture to create a StealthyFetcher instance for the entire test class"""
- return StealthyFetcher(auto_match=False)
+ return StealthyFetcher
@pytest.fixture(autouse=True)
def setup_urls(self, httpbin):
diff --git a/tests/fetchers/sync/test_httpx.py b/tests/fetchers/sync/test_httpx.py
index 9f5ca80..0a1abd8 100644
--- a/tests/fetchers/sync/test_httpx.py
+++ b/tests/fetchers/sync/test_httpx.py
@@ -3,13 +3,15 @@ import pytest_httpbin
from scrapling import Fetcher
+Fetcher.auto_match = True
+
@pytest_httpbin.use_class_based_httpbin
class TestFetcher:
@pytest.fixture(scope="class")
def fetcher(self):
"""Fixture to create a Fetcher instance for the entire test class"""
- return Fetcher(auto_match=False)
+ return Fetcher
@pytest.fixture(autouse=True)
def setup_urls(self, httpbin):
diff --git a/tests/fetchers/sync/test_playwright.py b/tests/fetchers/sync/test_playwright.py
index f683402..c256d27 100644
--- a/tests/fetchers/sync/test_playwright.py
+++ b/tests/fetchers/sync/test_playwright.py
@@ -3,6 +3,8 @@ import pytest_httpbin
from scrapling import PlayWrightFetcher
+PlayWrightFetcher.auto_match = True
+
@pytest_httpbin.use_class_based_httpbin
class TestPlayWrightFetcher:
@@ -10,7 +12,7 @@ class TestPlayWrightFetcher:
@pytest.fixture(scope="class")
def fetcher(self):
"""Fixture to create a StealthyFetcher instance for the entire test class"""
- return PlayWrightFetcher(auto_match=False)
+ return PlayWrightFetcher
@pytest.fixture(autouse=True)
def setup_urls(self, httpbin):