diff --git a/docs/cli/overview.md b/docs/cli/overview.md
index 5b5d498..dc4346f 100644
--- a/docs/cli/overview.md
+++ b/docs/cli/overview.md
@@ -27,4 +27,4 @@ and the installation of the fetchers' dependencies with the following command
```bash
scrapling install
```
-This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
\ No newline at end of file
+This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
\ No newline at end of file
diff --git a/docs/development/adaptive_storage_system.md b/docs/development/adaptive_storage_system.md
index 5958e54..e8bb07f 100644
--- a/docs/development/adaptive_storage_system.md
+++ b/docs/development/adaptive_storage_system.md
@@ -1,6 +1,6 @@
-Scrapling uses SQLite by default, but this tutorial covers writing your storage system to store element properties there for `adaptive` feature.
+Scrapling uses SQLite by default, but this tutorial shows how to write your own storage system to store element properties for the `adaptive` feature.
-You might want to use FireBase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because the spiders will share with each other.
+You might want to use Firebase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because spiders can share adaptive data with each other.
So first, to make your storage class work, it must do the big 3:
@@ -8,7 +8,7 @@ So first, to make your storage class work, it must do the big 3:
2. Use the decorator `functools.lru_cache` on top of the class to follow the Singleton design pattern as other classes.
3. Implement methods `save` and `retrieve`, as you see from the type hints:
- The method `save` returns nothing and will get two arguments from the library
- * The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the function `element_to_dict` in submodule `scrapling.core.utils._StorageTools` to keep the same format and save it to your database as you wish.
+ * The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the `element_to_dict` function in the submodule `scrapling.core.utils._StorageTools` to maintain the same format, and then saved to your database as you wish.
* The second one is a string, the identifier used for retrieval. The combination result of this identifier and the `url` argument from initialization must be unique for each row, or the `adaptive` data will be messed up.
- The method `retrieve` takes a string, which is the identifier; using it with the `url` passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns `None`.
diff --git a/docs/development/scrapling_custom_types.md b/docs/development/scrapling_custom_types.md
index aee8c1f..d1dd585 100644
--- a/docs/development/scrapling_custom_types.md
+++ b/docs/development/scrapling_custom_types.md
@@ -1,6 +1,6 @@
> You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)
-### All current types can be imported alone like below
+### All current types can be imported alone, like below
```python
>>> from scrapling.core.custom_types import TextHandler, AttributesHandler
@@ -11,11 +11,11 @@
>>> somedict_2 = AttributesHandler(a=1)
```
-Note that `TextHandler` is a subclass of Python's `str`, so all normal operations/methods that work with Python strings will work.
-If you want to check for the type in your code, it's better to depend on Python's built-in function `issubclass`.
+Note that `TextHandler` is a subclass of Python's `str`, so all standard operations/methods that work with Python strings will work.
+If you want to check the type in your code, it's better to use Python's built-in `issubclass` function.
The class `AttributesHandler` is a subclass of `collections.abc.Mapping`, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the `_data` property, but be careful; it's of type `types.MappingProxyType`, so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds).
-So, to make it simple for you if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with class `AttributesHandler` except the ones that try to modify the actual data.
+So, to make it simple for you, if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with the class `AttributesHandler` except for the ones that try to modify the actual data.
-If you want to modify the data inside `AttributesHandler,` you have to convert it to a dictionary first, like using the `dict` function, and then modify it outside.
\ No newline at end of file
+If you want to modify the data inside `AttributesHandler`, you have to convert it to a dictionary first, e.g., using the `dict` function, and then change it outside.
\ No newline at end of file
diff --git a/docs/donate.md b/docs/donate.md
index 64fc5f6..f5464a0 100644
--- a/docs/donate.md
+++ b/docs/donate.md
@@ -1,6 +1,6 @@
-I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to provide these projects for free to the community. By becoming a sponsor, you would directly fund my coffee reserves, helping me continuously update existing projects and create new ones.
+I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to make them available to the community for free. By becoming a sponsor, you would directly fund my coffee reserves, helping me continuously update existing projects and create new ones.
-You can sponsor me directly through [GitHub sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me A Coffe](https://buymeacoffee.com/d4vinci).
+You can sponsor me directly through the [GitHub Sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me a Coffee](https://buymeacoffee.com/d4vinci).
Thank you, stay curious, and hack the planet! ❤️
@@ -22,4 +22,3 @@ Perks:
- The same logo will be featured at [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling).
- Your logo will be featured as a top sponsor on [Scrapling's website](https://scrapling.readthedocs.io/en/latest/) main page.
- A Shoutout with each [Release note](https://github.com/D4Vinci/Scrapling/releases).
-
diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.md
index a56459e..cfb3114 100644
--- a/docs/fetching/choosing.md
+++ b/docs/fetching/choosing.md
@@ -3,7 +3,7 @@ Fetchers are classes that can do requests or fetch pages for you easily in a sin
This feature was introduced because, before v0.2, Scrapling was only a parsing engine. The target here is to gradually become the one-stop shop for all Web Scraping needs.
-> Fetchers are not wrappers built on top of other libraries. However, they utilize these libraries as an engine to request/fetch pages easily for you, while fully leveraging that engine and adding features for you. Some fetchers don't even use the official library for requests; instead, they use their own custom version. For example, `StealthyFetcher` utilizes `Camoufox` browser directly, without relying on its Python library for anything except launch options. This last part might change soon as well.
+> Fetchers are not wrappers built on top of other libraries. However, they only use these libraries as an engine to request/fetch pages. To further clarify this, all fetchers have features that the underlying engines don't, while still fully leveraging those engines and optimizing them for Web Scraping.
## Fetchers Overview
@@ -12,17 +12,17 @@ Scrapling provides three different fetcher classes with their session classes; e
The following table compares them and can be quickly used for guidance.
-| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
-|--------------------|---------------------------------------------------|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
-| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇 |
-| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
-| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
-| JavaScript loading | ❌ | ✅ | ✅ |
-| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
-| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites Customer Revie...' parent=']
->>> section_element.next # gets the next element, the same logic applies to `quote.previous`
+>>> section_element.next # gets the next element, the same logic applies to `quote.previous`.
Customer Revie...' parent='
>>> section_element.children.css('h2::text')
['Products']
@@ -237,7 +237,7 @@ You can search for a specific ancestor of an element that satisfies a function;
```
## Fetching websites
-Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers.
+Instead of passing the raw HTML to Scrapling, you can retrieve a website's response directly via HTTP requests or by fetching it in a browser.
A fetcher is made for every use case.
@@ -267,7 +267,7 @@ For Async requests, you will replace the import like below:
> Notes:
>
> 1. You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from a Google search of this domain. It's enabled by default.
-> 2. The `impersonate` argument allows you to fake the TLS fingerprint for a specific version of a browser.
+> 2. The `impersonate` argument lets you fake the TLS fingerprint for a specific browser version.
> 3. There's also the `http3` argument, which, when enabled, makes the fetcher use HTTP/3 for requests, which makes your requests more authentic
This is just the tip of the iceberg with this fetcher; check out the rest from [here](fetching/static.md)
@@ -275,7 +275,7 @@ This is just the tip of the iceberg with this fetcher; check out the rest from [
### Dynamic loading
We have you covered if you deal with dynamic websites like most today!
-The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides many options to fetch/load websites' pages through browsers.
+The `DynamicFetcher` class (formerly `PlayWrightFetcher`) offers many options for fetching and loading web pages using Chromium-based browsers.
```python
>>> from scrapling.fetchers import DynamicFetcher
>>> page = DynamicFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
@@ -286,10 +286,9 @@ The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides ma
>>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
```
-It's built on top of [Playwright](https://playwright.dev/python/) and it's currently providing three main run options that can be mixed as you want:
+It's built on top of [Playwright](https://playwright.dev/python/), and it's currently providing two main run options that can be mixed as you want:
- Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser.
-- Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode, but it bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/). Check out the `StealthyFetcher` class below for more advanced stealth mode. It uses the Chromium browser.
- Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
@@ -298,7 +297,18 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
### Dynamic anti-protection loading
We also have you covered if you deal with dynamic websites with annoying anti-protections!
-The `StealthyFetcher` class uses a custom version of a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), bypassing most bot detections by default. Scrapling offers a faster custom version, includes extra tools, and features easy configurations to further increase undetectability.
+The `StealthyFetcher` class uses a stealthy version of the `DynamicFetcher` explained above.
+
+Some of the things it does:
+
+1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
+2. It bypasses CDP runtime leaks and WebRTC leaks.
+3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
+4. It generates canvas noise to prevent fingerprinting through canvas.
+5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
+6. It makes requests look as if they came from Google's search page of the requested website.
+7. and other anti-protection options...
+
```python
>>> from scrapling.fetchers import StealthyFetcher
>>> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default
@@ -318,4 +328,4 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
---
-That's Scrapling at a glance. If you want to learn more about it, continue to the next section.
\ No newline at end of file
+That's Scrapling at a glance. If you want to learn more, continue to the next section.
\ No newline at end of file
diff --git a/docs/parsing/adaptive.md b/docs/parsing/adaptive.md
index 6ba9288..33396e9 100644
--- a/docs/parsing/adaptive.md
+++ b/docs/parsing/adaptive.md
@@ -50,7 +50,7 @@ When website owners implement structural changes like
```
The selector will no longer function, and your code needs maintenance. That's where Scrapling's `adaptive` feature comes into play.
-With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element and without AI :)
+With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element, and without AI :)
```python
from scrapling import Selector, Fetcher
@@ -65,7 +65,7 @@ if not element: # One day website changes?
element = page.css('#p1', adaptive=True) # Scrapling still finds it!
# the rest of your code...
```
-Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
+Below, I will show you an example of how to use this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
## Real-World Scenario
Let's use a real website as an example and use one of the fetchers to fetch its source. To achieve this, we need to identify a website that is about to update its design/structure, copy its source, and then wait for the website to change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
@@ -98,7 +98,9 @@ Note that I introduced a new argument called `adaptive_domain`. This is because,
The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `adaptive_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
-Hence, in the two examples above, I used both the `Selector` class and the `Fetcher` class to show you that the logic for adaptive is the same.
+Hence, in the two examples above, I used both the `Selector` and `Fetcher` classes to show that the adaptive logic is the same.
+
+> Note: the main reason for creating the `adaptive_domain` argument was to handle if the website changed its URL while changing the design/structure. In that case, you can use it to continue using the previously stored adaptive data for the new URL. Otherwise, scrapling will consider it a new website and discard the old data.
## How the adaptive scraping feature works
Adaptive scraping works in two phases:
@@ -106,19 +108,19 @@ Adaptive scraping works in two phases:
1. **Save Phase**: Store unique properties of elements
2. **Match Phase**: Find elements with similar properties later
-Let's say you've got an element through selection or any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
+Let's say you've selected an element through any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
With as few technical details as possible, the general logic goes as follows:
1. You tell Scrapling to save that element's unique properties in one of the ways we will show below.
2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
- 3. Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
- 1. The domain of the current website. If you are using the `Selector` class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically.
- 2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this.
+ 3. Now, because everything about the element can be changed or removed by the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
+ 1. The domain of the current website. If you are using the `Selector` class, pass it when initializing; if you are using a fetcher, the domain will be automatically taken from the URL.
+ 2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself; we'll discuss this later.
- Together, they will be used to retrieve the element's unique properties from the database later.
+ Together, they will later be used to retrieve the element's unique properties from the database.
- 4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity with the desired element. In that comparison, everything is taken into consideration, as you will see later
+ 4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated based on their similarity to the desired element. In that comparison, everything is taken into consideration, as you will see later
5. The element(s) with the highest similarity score to the wanted element are returned.
### The unique properties
@@ -129,7 +131,7 @@ For Scrapling, the unique elements we are relying on are:
- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
- Element's parent tag name, attributes (names and values), and text.
-But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
+But you need to understand that the comparison between elements isn't exact; it's more about how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
## How to use adaptive feature
The adaptive feature can be applied to any found element, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.
@@ -146,11 +148,11 @@ Examples:
```
If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain.
-If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and the `adaptive` feature only uses the latest saved properties.
+If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you use the same identifier later for a different website and don't pass the URL parameter when initializing it. The save process overwrites previous data, and the `adaptive` feature uses only the latest saved properties.
-Besides those arguments, we have `storage` and `storage_args`. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
+Besides those arguments, we have `storage` and `storage_args`. Both are for the class to connect to the database; by default, it uses the SQLite class provided by the library. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
-Now, after enabling the `adaptive` feature globally, you have two main ways to use it.
+Now that you've enabled the `adaptive` feature globally, you have two main ways to use it.
### The CSS/XPath Selection way
As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page, like below
@@ -163,7 +165,7 @@ element = page.css('#p1', adaptive=True)
```
Pretty simple, eh?
-Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
+Well, a lot happened under the hood here. Remember the identifier we mentioned before that you need to set to retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
Additionally, for all these methods, you can pass the `identifier` argument to set it yourself. This is useful in some instances, or you can use it to save properties with the `auto_save` argument.
@@ -174,7 +176,7 @@ First, let's say you got an element like this by text:
```python
>>> element = page.find_by_text('Tipping the Velvet', first_match=True)
```
-You can save its unique properties with the `save` method, like below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
+You can save its unique properties using the `save` method, as shown below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
```python
>>> page.save(element, 'my_special_element')
```
@@ -221,7 +223,7 @@ page.save(product, 'specific_product')
```
## Known Issues
-In the `adaptive` save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
+In the `adaptive` save process, only the unique properties of the first element in the selection results are saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors are separated and each is executed alone.
## Final thoughts
Explaining this feature in detail without complications turned out to be challenging. However, still, if there's something left unclear, you can head out to the [discussions section](https://github.com/D4Vinci/Scrapling/discussions), and I will reply to you ASAP, or the Discord server, or reach out to me privately and have a chat :)
\ No newline at end of file
diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md
index fb1ccf0..ac8af75 100644
--- a/docs/parsing/main_classes.md
+++ b/docs/parsing/main_classes.md
@@ -41,7 +41,7 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th
I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.
You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested.
-After that, for the main page and elements within, most properties are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
+After that, most properties on the main page and its elements are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
### Properties
You have already seen much of this on the [overview](../overview.md) page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the [traversal](#traversal) section.
@@ -111,7 +111,7 @@ But if you try to get the direct text content, it will be empty because it doesn
```
The `get_all_text` method has the following optional arguments:
-1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'
+1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'.
2. **strip**: If enabled, strings will be stripped before concatenation. Disabled by default.
3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results and ignore any elements nested within them. The default is `('script', 'style',)`.
4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
@@ -132,7 +132,7 @@ If you use it on the page directly, you will find that you are operating on the
>>> page.tag
'html'
```
-Now, I think I hammered the (`page`/`element`) idea, so I won't return to it again.
+Now, I think I've hammered the (`page`/`element`) idea, so I won't return to it.
Getting the attributes of the element
```python
@@ -196,7 +196,7 @@ Same case with XPath
### Traversal
Using the elements we found above, we will go over the properties/methods for moving on the page in detail.
-If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding.
+If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online to better understand them.
If you are too lazy to search about it, here's a quick explanation to give you a good idea.
In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.
@@ -310,7 +310,7 @@ In the [Selector](#selector) class, all methods/properties that should return a
Let's see what [Selectors](#selectors) class adds to the table with that out of the way.
### Properties
-Apart from the normal operations on Python lists, such as iteration and slicing, etc.
+Apart from the standard operations on Python lists, such as iteration and slicing.
You can do the following:
@@ -334,7 +334,7 @@ Execute CSS and XPath selectors directly on the [Selector](#selector) instances
,
...]
```
-Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will still leave these methods to be explained in the [TextHandler](#texthandler) section below.
+Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will leave the explanation of these methods to the [TextHandler](#texthandler) section below.
However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance.
```python
@@ -355,7 +355,7 @@ However, in this class, the `re_first` behaves differently as it runs `re` on ea
```
With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`.
```python
-# Find all the products with price '53.23'
+# Find all the products with price '53.23'.
>>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
>>> page.css('.product_pod').search(search_function)
@@ -374,7 +374,7 @@ If you are too lazy like me and want to know the number of [Selector](#selector)
```python
page.css('.product_pod').length
```
-instead of this
+which is equivalent to
```python
len(page.css('.product_pod'))
```
@@ -389,15 +389,15 @@ Of course, TextHandler provides extra methods and properties that standard Pytho
### Usage
First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well.
-First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they will accept the same arguments as well.
+First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they accept the same arguments.
-- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
+- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but, as you probably figured out from the name, it returns only the first result as a `TextHandler` instance.
Also, it takes other helpful arguments, which are:
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- - **clean_match**: It's disabled by default. This causes the method to ignore all whitespace and consecutive spaces while matching.
- - **case_sensitive**: It's enabled by default. As the name implies, disabling it will cause the regex to ignore the case of letters while compiling.
+ - **clean_match**: It's disabled by default. This causes the method to ignore all whitespace, including consecutive spaces, while matching.
+ - **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
```python
@@ -471,14 +471,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
```python
>>> page.css_first('div::text').json()
```
- You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have direct text content at all.
+ You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.
In this case, the `get_all_text` method comes to the rescue, so you can do something like that
```python
>>> page.css_first('div').get_all_text(ignore_tags=[]).json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.
- Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example:
+ Another related behavior to be aware of occurs when using any fetcher, which we will explain later. If you have a JSON response like this example:
```python
>>> page = Selector("""{"some_key": "some_value"}""")
```
@@ -493,14 +493,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
{'some_key': 'some_value'}
```
You might wonder how this happened, given that the `html` tag doesn't contain direct text.
- Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.
This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
+ Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it uses the `get_all_text` method directly.
- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance
```python
>>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?'
```
-Also, you can pass `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
+Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists
```python
@@ -518,10 +518,10 @@ Other methods and properties will be added over time, but remember that this cla
## TextHandlers
You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
-The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing new needs to be explained here, but new methods will be added over time.
+The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. Nothing new needs to be explained here, but new methods will be added over time.
## AttributesHandler
-This is a read-only version of Python's standard dictionary, or `dict`, that is used solely to store the attributes of each element or each [Selector](#selector) instance.
+This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element or [Selector](#selector) instance.
```python
>>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'}
@@ -534,7 +534,7 @@ It currently adds two extra simple methods:
- The `search_values` method
- In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will require some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
+ In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
A simple example would be
```python
@@ -555,13 +555,13 @@ It currently adds two extra simple methods:
,
]
```
- All these elements have 'product' as a value for the attribute `class`.
+ All these elements have 'product' as the value for the `class` attribute.
Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
- The `json_string` property
- This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error
+ This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error.
```python
>>>page.find('script').attrib.json_string
diff --git a/docs/parsing/selection.md b/docs/parsing/selection.md
index a262f82..0b3877d 100644
--- a/docs/parsing/selection.md
+++ b/docs/parsing/selection.md
@@ -22,7 +22,7 @@ Scrapling implements CSS3 selectors as described in the [W3C specification](http
Also, Scrapling implements some non-standard pseudo-elements like:
-* To select text nodes, use ``::text``
+* To select text nodes, use ``::text``.
* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
@@ -36,19 +36,19 @@ In short, it is the same situation as CSS Selectors; if you come from Scrapy/Par
To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above, and `xpath_first` is faster.
-> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
+> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here, as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
### Selectors examples
Let's see some shared examples of using CSS and XPath Selectors.
-Select all elements with the class `product`
+Select all elements with the class `product`.
```python
products = page.css('.product')
products = page.xpath('//*[@class="product"]')
```
Note: The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class**
-Select the first element with the class `product`
+Select the first element with the class `product`.
```python
product = page.css_first('.product')
product = page.xpath_first('//*[@class="product"]')
@@ -73,12 +73,12 @@ Get the `href` attribute of the first element with the `a` tag name
link = page.css_first('a::attr(href)')
link = page.xpath_first('//a/@href')
```
-Select the text of the first element with the `h1` tag name, which contains 'Phone', and under an element with class 'product'
+Select the text of the first element with the `h1` tag name, which contains `Phone`, and under an element with class `product`.
```python
title = page.css_first('.product h1:contains("Phone")::text')
title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
```
-You can nest and chain selectors as you want, given that it returns results
+You can nest and chain selectors as you want, given that they return results
```python
page.css_first('.product').css_first('h1:contains("Phone")::text')
page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
@@ -104,7 +104,7 @@ Scrapling provides the ability to select elements based on their direct text con
What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
-With `find_by_text`, you will pass the text as the first argument; with the `find_by_regex` method, the regex pattern is the first argument. Both methods share the following arguments:
+With `find_by_text`, you pass the text as the first argument; with `find_by_regex`, the regex pattern is the first argument. Both methods share the following arguments:
* **first_match**: If `True` (the default), the method used will return the first result it finds.
* **case_sensitive**: If `True`, the case of the letters will be considered.
@@ -117,7 +117,7 @@ By default, Scrapling searches for the exact matching of the text/pattern you pa
Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
### Finding Similar Elements
-One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
+One of the most remarkable new features Scrapling puts on the table is the ability to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
So, how does it work?
@@ -129,8 +129,8 @@ Imagine a scenario where you found a product by its title, for example, and you
That's a lot of talking, I know, but I had to go deep. I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
-* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the values of the attributes of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what the other arguments do first.
-* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change a lot between elements, making them unreliable.
+* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the tag attributes of both elements should be at least 20% similar. If you want to turn off this check (basically Step 3), you can set this attribute to 0, but I recommend you read what the other arguments do first.
+* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change significantly across elements, making them unreliable.
* **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends.
Now, let's check out the examples below.
@@ -148,7 +148,7 @@ Find the first element whose text fully matches this text
>>> page.find_by_text('Tipping the Velvet')
```
-Combining it with `page.urljoin` to return the full URL from the relative `href`
+Combining it with `page.urljoin` to return the full URL from the relative `href`.
```python
>>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html'
@@ -174,7 +174,7 @@ Get all elements that contain the word `the` (Partial matching)
'Mesaerion: The Best Science ...',
"It's Only the Himalayas"]
```
-The search is case-insensitive, so those results have `The`, not only the lowercase one `the`; let's limit the search to the elements with `the` only.
+The search is case-insensitive, so those results include `The`, not just the lowercase `the`; let's limit the search to elements with `the` only.
```python
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
@@ -332,14 +332,14 @@ def extract_reviews(page):
]
```
## Filters-based searching
-This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier to learn for newcomers to Web Scraping than writing selectors.
+This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier for newcomers to Web Scraping to learn than writing selectors.
-Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
+Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can accept multiple filters and return all elements on the pages where all these filters apply.
To be more specific:
* Any string passed is considered a tag name.
-* Any iterable passed, like List/Tuple/Set, is considered an iterable of tag names.
+* Any iterable passed, like List/Tuple/Set, will be considered as an iterable of tag names.
* Any dictionary is considered a mapping of HTML element(s), attribute names, and attribute values.
* Any regex patterns passed are used to filter elements by content, like the `find_by_regex` method
* Any functions passed are used to filter elements
@@ -357,7 +357,7 @@ It filters all elements in the current page/element in the following order:
Notes:
1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on.
-2. The order in which you pass the arguments doesn't matter. The only order taken into consideration is the order explained above.
+2. The order in which you pass the arguments doesn't matter. The only order considered is the one explained above.
Check examples to clear any confusion :)
@@ -509,4 +509,4 @@ We will have a deep look at it while explaining the [TextHandler](main_classes.m
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']
```
-And so on. You get the idea. We will explain this in more detail on the next page while explaining the [TextHandler](main_classes.md#texthandler) class.
\ No newline at end of file
+And so on. You get the idea. We will explain this in more detail on the next page, along with the [TextHandler](main_classes.md#texthandler) class.
\ No newline at end of file
diff --git a/docs/requirements.txt b/docs/requirements.txt
index e996a0b..c2c5d54 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,5 +1,8 @@
-mkdocs-material
-mkdocstrings
-mkdocstrings-python
-mkdocs-material[imaging]
-black
\ No newline at end of file
+mkdocstrings>=1.0.0
+mkdocstrings-python>=2.0.1
+griffe-inherited-docstrings
+griffe-runtime-objects
+griffe-sphinx
+mkdocs-material[imaging]>=9.7.1
+black>=25.12.0
+pngquant
\ No newline at end of file
diff --git a/docs/tutorials/migrating_from_beautifulsoup.md b/docs/tutorials/migrating_from_beautifulsoup.md
index d1b7019..e5474bb 100644
--- a/docs/tutorials/migrating_from_beautifulsoup.md
+++ b/docs/tutorials/migrating_from_beautifulsoup.md
@@ -1,10 +1,10 @@
# Migrating from BeautifulSoup to Scrapling
-If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is incredibly faster, provides the same parsing capabilities, adds more parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
+If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is much faster, provides the same parsing capabilities as BS, adds additional parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
-Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to accomplish a specific task using BeautifulSoup and the corresponding method in Scrapling.
+Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to achieve a specific task using BeautifulSoup and the corresponding method in Scrapling.
-You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, but that's one of the reasons why BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short oneliner, there is no need to sacrifice performance to shorten that short line :)
+You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, which is one of the reasons BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short one-liner, there is no need to sacrifice performance to shorten that short line :)
| Task | BeautifulSoup Code | Scrapling Code |
@@ -56,7 +56,7 @@ Here's a simple example of scraping a web page to extract all the links using Be
import requests
from bs4 import BeautifulSoup
-url = 'http://example.com'
+url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
@@ -70,7 +70,7 @@ for link in links:
```python
from scrapling import Fetcher
-url = 'http://example.com'
+url = 'https://example.com'
page = Fetcher.get(url)
links = page.css('a::attr(href)')
@@ -78,12 +78,12 @@ for link in links:
print(link)
```
-As you can see, Scrapling simplifies the process by handling the fetching and parsing in a single step, making your code cleaner and more efficient.
+As you can see, Scrapling simplifies the process by combining fetching and parsing into a single step, making your code cleaner and more efficient.
**Additional Notes:**
- **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons.
-- **Element Types**: In BeautifulSoup, elements are `Tag` objects, while in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
+- **Element Types**: In BeautifulSoup, elements are `Tag` objects; in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.css_first()`). To avoid errors, check for `None` before accessing properties.
- **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can help remove extra whitespace, consecutive spaces, or unwanted characters. Please check out the documentation for the complete list.
diff --git a/docs/tutorials/replacing_ai.md b/docs/tutorials/replacing_ai.md
index 733e88b..200073c 100644
--- a/docs/tutorials/replacing_ai.md
+++ b/docs/tutorials/replacing_ai.md
@@ -26,13 +26,13 @@ How will you solve that manually? I'm referring to generic web scraping of vario
## AI to the rescue, but at a high cost
-Of course, the AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
+Of course, AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
This approach is, of course, beautiful. I love AI and find it very fascinating, especially Generative AI. You will probably spend a lot of time on prompt engineering and tweaking the prompts, but if that's cool with you, you will soon hit the real issue with using AI here.
Most websites have vast amounts of content per page, which you will need to pass to the AI somehow so it can do its magic. This will burn through tokens like fire in a haystack, quickly accumulating high costs.
-Unless money is irrelevant to you, you will try to find less expensive approaches, and that's why I made Scrapling :smile:
+Unless money is irrelevant to you, you will try to find less expensive approaches, and that's where Scrapling comes into play :smile:
## Scrapling got you covered
@@ -41,11 +41,11 @@ Scrapling can handle almost all issues you will face during Web Scraping, and th
### Solving issue T1: Rapidly changing website structures
That's why the [adaptive](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/) feature was made. You knew I would talk about it, and here we are :)
-While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties to find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
+While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties so you can find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
That's how the adaptive feature works: it stores everything unique about an element. When the website structure changes, it returns the element with the highest similarity score of the previous element.
-I have already explained that in more detail and with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
+I have already explained this in more detail, with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
### Solving issue T2: Unstable selectors
If you have been doing Web scraping for a long enough time, you have likely experienced this once. I'm referring to a website that employs poor design patterns, built on raw HTML without any IDs/classes, or uses random class names with nothing else to rely on, etc...
@@ -59,16 +59,16 @@ In these cases, standard selection methods with CSS/XPath selectors won't be opt
There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this.
### Solving issue T3: Increasingly complex anti-bot measures
-It's known that making an undetectable spider takes more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
+It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
-1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides many flexible options, like stealth mode suitable for small to medium protections and using your real browser.
-2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher utilizes our version of a modified Firefox browser, called [Camoufox](https://camoufox.com/stealth/), which nearly passes all known tests and incorporates additional tricks. **With v0.3, this fetcher can bypass Cloudflare for you automatically as well!**
+1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
+2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
-These two will be improved a lot with the upcoming updates, so stay tuned :)
+We keep improving these two with each update, so stay tuned :)
### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data
-This one is tough to handle, but it's possible with Scrapling's flexibility.
+This one is tough to handle, but Scrapling's flexibility makes it possible.
I talked with someone who uses AI to extract prices from different websites. He is only interested in prices and titles, so he uses AI to find the price for him.
@@ -94,7 +94,7 @@ It will be a bit boring, but it's definitely less expensive than AI.
This example illustrates the point I aim to convey here. Not every challenge will need AI to be solved, but sometimes you need to be creative, and that might save you a lot of money.
### Solving issue B3: Pagination variations
-This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the following updates :)
+This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :)
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea.
@@ -112,6 +112,6 @@ For a quick comparison.
This table is based on pricing from [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing)
## Conclusion
-While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit designed to tackle the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
+While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit for tackling the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects!
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index ea2dd7a..c8b9709 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -29,6 +29,7 @@ theme:
- navigation.sections
- navigation.tracking
- navigation.instant
+ - navigation.instant.prefetch
- navigation.instant.progress
# - navigation.tabs
# - navigation.expand
@@ -41,6 +42,7 @@ theme:
- content.action.view
- content.action.edit
- content.code.copy
+ - content.code.select
- content.code.annotate
- content.code.annotation
@@ -103,10 +105,13 @@ markdown_extensions:
plugins:
- search
-# - social:
-# cards_layout_options:
-# background_color: "#1f1f1f"
-# font_family: Roboto
+ - privacy:
+ links: false
+ - optimize
+ - social:
+ cards_layout_options:
+ background_color: "#1f1f1f"
+ font_family: Roboto
- mkdocstrings:
handlers:
python:
@@ -122,11 +127,19 @@ plugins:
unwrap_annotated: true
filters:
- '!^_'
+ - "^__"
merge_init_into_class: true
docstring_section_style: spacy
signature_crossrefs: true
show_symbol_type_heading: true
show_symbol_type_toc: true
+ show_inheritance_diagram: true
+ modernize_annotations: true
+ extensions:
+ - griffe_runtime_objects
+ - griffe_sphinx
+ - griffe_inherited_docstrings:
+ merge: true
extra:
homepage: https://scrapling.readthedocs.io/en/latest/
diff --git a/pyproject.toml b/pyproject.toml
index 0dd663b..a5b6f5c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "scrapling"
# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
-version = "0.3.12"
+version = "0.3.13"
description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
readme = {file = "docs/README.md", content-type = "text/markdown"}
license = {file = "LICENSE"}
@@ -60,7 +60,7 @@ dependencies = [
"lxml>=6.0.2",
"cssselect>=1.3.0",
"orjson>=3.11.5",
- "tldextract>=5.3.0",
+ "tldextract>=5.3.1",
]
[project.optional-dependencies]
@@ -69,8 +69,7 @@ fetchers = [
"curl_cffi>=0.14.0",
"playwright==1.56.0",
"patchright==1.56.0",
- "camoufox>=0.4.11",
- "geoip2>=5.2.0",
+ "browserforge>=1.2.1",
"msgspec>=0.20.0",
]
ai = [
diff --git a/scrapling/__init__.py b/scrapling/__init__.py
index 23f917b..0eca2f2 100644
--- a/scrapling/__init__.py
+++ b/scrapling/__init__.py
@@ -1,5 +1,5 @@
__author__ = "Karim Shoair (karim.shoair@pm.me)"
-__version__ = "0.3.12"
+__version__ = "0.3.13"
__copyright__ = "Copyright (c) 2024 Karim Shoair"
from typing import Any, TYPE_CHECKING
diff --git a/scrapling/cli.py b/scrapling/cli.py
index 99a2c63..68470a5 100644
--- a/scrapling/cli.py
+++ b/scrapling/cli.py
@@ -125,14 +125,9 @@ def install(force): # pragma: no cover
"playwright",
"install-deps",
"chromium",
- "firefox",
],
"Playwright dependencies",
)
- __Execute(
- [python_executable, "-m", "camoufox", "fetch", "--browserforge"],
- "Camoufox browser and databases",
- )
# if no errors raised by the above commands, then we add the below file
__PACKAGE_DIR__.joinpath(".scrapling_dependencies_installed").touch()
else:
@@ -610,17 +605,11 @@ def delete(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
-@option("--locale", default="en-US", help="Browser locale (default: en-US)")
-@option("--stealth/--no-stealth", default=False, help="Enable stealth mode (default: False)")
+@option("--locale", default=None, help="Specify user locale. Defaults to the system default locale.")
@option(
- "--hide-canvas/--show-canvas",
+ "--real-chrome/--no-real-chrome",
default=False,
- help="Add noise to canvas operations (default: False)",
-)
-@option(
- "--disable-webgl/--enable-webgl",
- default=False,
- help="Disable WebGL support (default: False)",
+ help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
@@ -640,9 +629,7 @@ def fetch(
css_selector,
wait_selector,
locale,
- stealth,
- hide_canvas,
- disable_webgl,
+ real_chrome,
proxy,
extra_headers,
):
@@ -659,9 +646,7 @@ def fetch(
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser.
- :param stealth: Enables stealth mode.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -676,9 +661,7 @@ def fetch(
"network_idle": network_idle,
"timeout": timeout,
"locale": locale,
- "stealth": stealth,
- "hide_canvas": hide_canvas,
- "disable_webgl": disable_webgl,
+ "real_chrome": real_chrome,
}
if wait > 0:
@@ -703,11 +686,6 @@ def fetch(
default=True,
help="Run browser in headless mode (default: True)",
)
-@option(
- "--block-images/--allow-images",
- default=False,
- help="Block image loading (default: False)",
-)
@option(
"--disable-resources/--enable-resources",
default=False,
@@ -718,11 +696,6 @@ def fetch(
default=False,
help="Block WebRTC entirely (default: False)",
)
-@option(
- "--humanize/--no-humanize",
- default=False,
- help="Humanize cursor movement (default: False)",
-)
@option(
"--solve-cloudflare/--no-solve-cloudflare",
default=False,
@@ -735,9 +708,14 @@ def fetch(
help="Wait for network idle (default: False)",
)
@option(
- "--disable-ads/--allow-ads",
+ "--real-chrome/--no-real-chrome",
default=False,
- help="Install uBlock Origin addon (default: False)",
+ help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
+)
+@option(
+ "--hide-canvas/--show-canvas",
+ default=False,
+ help="Add noise to canvas operations (default: False)",
)
@option(
"--timeout",
@@ -757,11 +735,6 @@ def fetch(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
-@option(
- "--geoip/--no-geoip",
- default=False,
- help="Use IP geolocation for timezone/locale (default: False)",
-)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
"--extra-headers",
@@ -773,19 +746,17 @@ def stealthy_fetch(
url,
output_file,
headless,
- block_images,
disable_resources,
block_webrtc,
- humanize,
solve_cloudflare,
allow_webgl,
network_idle,
- disable_ads,
+ real_chrome,
+ hide_canvas,
timeout,
wait,
css_selector,
wait_selector,
- geoip,
proxy,
extra_headers,
):
@@ -795,19 +766,17 @@ def stealthy_fetch(
:param url: Target url.
:param output_file: Output file path (.md for Markdown, .html for HTML).
:param headless: Run the browser in headless/hidden, or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
:param disable_resources: Drop requests of unnecessary resources for a speed boost.
:param block_webrtc: Blocks WebRTC entirely.
- :param humanize: Humanize the cursor movement.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges.
:param allow_webgl: Allow WebGL (recommended to keep enabled).
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Install the uBlock Origin addon on the browser.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before returning.
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Automatically use IP's longitude, latitude, timezone, country, locale.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -818,16 +787,14 @@ def stealthy_fetch(
# Build request arguments
kwargs = {
"headless": headless,
- "block_images": block_images,
"disable_resources": disable_resources,
"block_webrtc": block_webrtc,
- "humanize": humanize,
"solve_cloudflare": solve_cloudflare,
"allow_webgl": allow_webgl,
"network_idle": network_idle,
- "disable_ads": disable_ads,
+ "real_chrome": real_chrome,
+ "hide_canvas": hide_canvas,
"timeout": timeout,
- "geoip": geoip,
}
if wait > 0:
diff --git a/scrapling/core/_shell_signatures.py b/scrapling/core/_shell_signatures.py
index 803e6d5..d9a42fc 100644
--- a/scrapling/core/_shell_signatures.py
+++ b/scrapling/core/_shell_signatures.py
@@ -1,9 +1,13 @@
from scrapling.core._types import (
- Dict,
Any,
+ Dict,
List,
Tuple,
+ Sequence,
+ Callable,
Optional,
+ SetCookieParam,
+ SelectorWaitStates,
)
# Parameter definitions for shell function signatures (defined once at module level)
@@ -31,57 +35,58 @@ _REQUESTS_PARAMS = {
_FETCH_PARAMS = {
"headless": bool,
- "google_search": bool,
- "hide_canvas": bool,
- "disable_webgl": bool,
- "real_chrome": bool,
- "stealth": bool,
- "wait": int | float,
- "page_action": Optional[Any],
- "proxy": Optional[str | Dict],
- "locale": str,
- "extra_headers": Optional[Dict[str, str]],
- "useragent": Optional[str],
- "cdp_url": Optional[str],
- "timeout": int | float,
"disable_resources": bool,
- "wait_selector": Optional[str],
- "init_script": Optional[str],
- "cookies": Optional[List[Dict]],
"network_idle": bool,
"load_dom": bool,
- "wait_selector_state": Any,
- "extra_flags": Optional[List[str]],
- "additional_args": Optional[Dict],
+ "wait_selector": Optional[str],
+ "wait_selector_state": SelectorWaitStates,
+ "cookies": Sequence[SetCookieParam],
+ "google_search": bool,
+ "wait": int | float,
+ "timezone_id": str | None,
+ "page_action": Optional[Callable],
+ "proxy": Optional[str | Dict[str, str] | Tuple],
+ "extra_headers": Optional[Dict[str, str]],
+ "timeout": int | float,
+ "init_script": Optional[str],
+ "user_data_dir": str,
"selector_config": Optional[Dict],
+ "additional_args": Optional[Dict],
+ "locale": Optional[str],
+ "real_chrome": bool,
+ "cdp_url": Optional[str],
+ "useragent": Optional[str],
+ "extra_flags": Optional[List[str]],
}
_STEALTHY_FETCH_PARAMS = {
"headless": bool,
- "block_images": bool,
"disable_resources": bool,
- "block_webrtc": bool,
- "allow_webgl": bool,
"network_idle": bool,
"load_dom": bool,
- "humanize": bool | float,
- "solve_cloudflare": bool,
- "wait": int | float,
- "timeout": int | float,
- "page_action": Optional[Any],
"wait_selector": Optional[str],
- "init_script": Optional[str],
- "addons": Optional[List[str]],
- "wait_selector_state": Any,
- "cookies": Optional[List[Dict]],
+ "wait_selector_state": SelectorWaitStates,
+ "cookies": Sequence[SetCookieParam],
"google_search": bool,
+ "wait": int | float,
+ "timezone_id": str | None,
+ "page_action": Optional[Callable],
+ "proxy": Optional[str | Dict[str, str] | Tuple],
"extra_headers": Optional[Dict[str, str]],
- "proxy": Optional[str | Dict],
- "os_randomize": bool,
- "disable_ads": bool,
- "geoip": bool,
+ "timeout": int | float,
+ "init_script": Optional[str],
+ "user_data_dir": str,
"selector_config": Optional[Dict],
"additional_args": Optional[Dict],
+ "locale": Optional[str],
+ "real_chrome": bool,
+ "cdp_url": Optional[str],
+ "useragent": Optional[str],
+ "extra_flags": Optional[List[str]],
+ "allow_webgl": bool,
+ "hide_canvas": bool,
+ "block_webrtc": bool,
+ "solve_cloudflare": bool,
}
# Mapping of function names to their parameter definitions
diff --git a/scrapling/core/_types.py b/scrapling/core/_types.py
index ac19a91..d7c1f8b 100644
--- a/scrapling/core/_types.py
+++ b/scrapling/core/_types.py
@@ -57,3 +57,17 @@ except ImportError: # pragma: no cover
from typing_extensions import Self # Backport
except ImportError:
Self = object
+
+
+# Copied from `playwright._impl._api_structures.SetCookieParam`
+class SetCookieParam(TypedDict, total=False):
+ name: str
+ value: str
+ url: Optional[str]
+ domain: Optional[str]
+ path: Optional[str]
+ expires: Optional[float]
+ httpOnly: Optional[bool]
+ secure: Optional[bool]
+ sameSite: Optional[Literal["Lax", "None", "Strict"]]
+ partitionKey: Optional[str]
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 4d5277a..4d5929b 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -17,13 +17,15 @@ from scrapling.fetchers import (
from scrapling.core._types import (
Optional,
Tuple,
- extraction_types,
Mapping,
Dict,
List,
Any,
- SelectorWaitStates,
Generator,
+ Sequence,
+ SetCookieParam,
+ extraction_types,
+ SelectorWaitStates,
)
@@ -213,20 +215,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
- hide_canvas: bool = False,
- disable_webgl: bool = False,
real_chrome: bool = False,
- stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
- locale: str = "en-US",
+ timezone_id: str | None = None,
+ locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
- cookies: Optional[List[Dict]] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> ResponseModel:
@@ -242,21 +242,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -269,15 +267,13 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
- stealth=stealth,
cdp_url=cdp_url,
headless=headless,
useragent=useragent,
- hide_canvas=hide_canvas,
+ timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
- disable_webgl=disable_webgl,
extra_headers=extra_headers,
google_search=google_search,
disable_resources=disable_resources,
@@ -301,20 +297,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
- hide_canvas: bool = False,
- disable_webgl: bool = False,
real_chrome: bool = False,
- stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
- locale: str = "en-US",
+ timezone_id: str | None = None,
+ locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
- cookies: Optional[List[Dict]] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> List[ResponseModel]:
@@ -330,21 +324,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -356,17 +348,15 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
- stealth=stealth,
cdp_url=cdp_url,
headless=headless,
max_pages=len(urls),
useragent=useragent,
- hide_canvas=hide_canvas,
+ timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
google_search=google_search,
- disable_webgl=disable_webgl,
extra_headers=extra_headers,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
@@ -393,29 +383,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
- block_images: bool = False,
+ google_search: bool = True,
+ real_chrome: bool = False,
+ wait: int | float = 0,
+ proxy: Optional[str | Dict[str, str]] = None,
+ timezone_id: str | None = None,
+ locale: str | None = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ useragent: Optional[str] = None,
+ hide_canvas: bool = False,
+ cdp_url: Optional[str] = None,
+ timeout: int | float = 30000,
disable_resources: bool = False,
+ wait_selector: Optional[str] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
+ network_idle: bool = False,
+ wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
- network_idle: bool = False,
- humanize: bool | float = True,
solve_cloudflare: bool = False,
- wait: int | float = 0,
- timeout: int | float = 30000,
- wait_selector: Optional[str] = None,
- addons: Optional[List[str]] = None,
- wait_selector_state: SelectorWaitStates = "attached",
- cookies: Optional[List[Dict]] = None,
- google_search: bool = True,
- extra_headers: Optional[Dict[str, str]] = None,
- proxy: Optional[str | Dict[str, str]] = None,
- os_randomize: bool = False,
- disable_ads: bool = False,
- geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> ResponseModel:
- """Use Scrapling's version of the Camoufox browser to fetch a URL and return a structured output of the result.
- Note: This is best suitable for high protection levels. It's slower than the other tools.
+ """Use the stealthy fetcher to fetch a URL and return a structured output of the result.
+ Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param url: The URL to request.
@@ -426,54 +416,52 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
page = await StealthyFetcher.async_fetch(
url,
wait=wait,
proxy=proxy,
- geoip=geoip,
- addons=addons,
+ locale=locale,
+ cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
- humanize=humanize,
+ useragent=useragent,
+ timezone_id=timezone_id,
+ real_chrome=real_chrome,
+ hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
- disable_ads=disable_ads,
network_idle=network_idle,
- block_images=block_images,
block_webrtc=block_webrtc,
- os_randomize=os_randomize,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
+ additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- additional_args=additional_args,
)
return _ContentTranslator(
Convertor._extract_content(
@@ -492,29 +480,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
- block_images: bool = False,
+ google_search: bool = True,
+ real_chrome: bool = False,
+ wait: int | float = 0,
+ proxy: Optional[str | Dict[str, str]] = None,
+ timezone_id: str | None = None,
+ locale: str | None = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ useragent: Optional[str] = None,
+ hide_canvas: bool = False,
+ cdp_url: Optional[str] = None,
+ timeout: int | float = 30000,
disable_resources: bool = False,
+ wait_selector: Optional[str] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
+ network_idle: bool = False,
+ wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
- network_idle: bool = False,
- humanize: bool | float = True,
solve_cloudflare: bool = False,
- wait: int | float = 0,
- timeout: int | float = 30000,
- wait_selector: Optional[str] = None,
- addons: Optional[List[str]] = None,
- wait_selector_state: SelectorWaitStates = "attached",
- cookies: Optional[List[Dict]] = None,
- google_search: bool = True,
- extra_headers: Optional[Dict[str, str]] = None,
- proxy: Optional[str | Dict[str, str]] = None,
- os_randomize: bool = False,
- disable_ads: bool = False,
- geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> List[ResponseModel]:
- """Use Scrapling's version of the Camoufox browser to fetch a group of URLs at the same time, and for each page return a structured output of the result.
- Note: This is best suitable for high protection levels. It's slower than the other tools.
+ """Use the stealthy fetcher to fetch a group of URLs at the same time, and for each page return a structured output of the result.
+ Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param urls: A tuple of the URLs to request.
@@ -525,54 +513,51 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
async with AsyncStealthySession(
wait=wait,
proxy=proxy,
- geoip=geoip,
- addons=addons,
+ locale=locale,
+ cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
- humanize=humanize,
- max_pages=len(urls),
+ useragent=useragent,
+ timezone_id=timezone_id,
+ real_chrome=real_chrome,
+ hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
- disable_ads=disable_ads,
- block_images=block_images,
- block_webrtc=block_webrtc,
network_idle=network_idle,
- os_randomize=os_randomize,
+ block_webrtc=block_webrtc,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
+ additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- additional_args=additional_args,
) as session:
tasks = [session.fetch(url) for url in urls]
responses = await gather(*tasks)
diff --git a/scrapling/core/translator.py b/scrapling/core/translator.py
index bb6d405..c34ed7d 100644
--- a/scrapling/core/translator.py
+++ b/scrapling/core/translator.py
@@ -130,5 +130,5 @@ translator = HTMLTranslator()
@lru_cache(maxsize=256)
def css_to_xpath(query: str) -> str:
- """Return translated XPath version of a given CSS query"""
+ """Return the translated XPath version of a given CSS query"""
return translator.css_to_xpath(query)
diff --git a/scrapling/engines/_browsers/_base.py b/scrapling/engines/_browsers/_base.py
index fdb298c..bb14b5f 100644
--- a/scrapling/engines/_browsers/_base.py
+++ b/scrapling/engines/_browsers/_base.py
@@ -1,7 +1,6 @@
from time import time
from asyncio import sleep as asyncio_sleep, Lock
-from camoufox import DefaultAddons
from playwright.sync_api._generated import Page
from playwright.sync_api import (
Frame,
@@ -17,18 +16,18 @@ from playwright.async_api import (
BrowserContext as AsyncBrowserContext,
)
from playwright._impl._errors import Error as PlaywrightError
-from camoufox.pkgman import installed_verstr as camoufox_version
-from camoufox.utils import launch_options as generate_launch_options
from ._page import PageInfo, PagePool
from scrapling.parser import Selector
-from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING
-from scrapling.engines.toolbelt.fingerprints import get_os_name
-from ._validators import validate, PlaywrightConfig, CamoufoxConfig
-from ._config_tools import _compiled_stealth_scripts, _launch_kwargs, _context_kwargs
+from ._validators import validate, PlaywrightConfig, StealthConfig
+from ._config_tools import __default_chrome_useragent__, __default_useragent__
from scrapling.engines.toolbelt.navigation import intercept_route, async_intercept_route
-
-__ff_version_str__ = camoufox_version().split(".", 1)[0]
+from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING, overload, Tuple
+from scrapling.engines.constants import (
+ DEFAULT_STEALTH_FLAGS,
+ HARMFUL_DEFAULT_ARGS,
+ DEFAULT_FLAGS,
+)
class SyncSession:
@@ -84,10 +83,6 @@ class SyncSession:
if disable_resources:
page.route("**/*", intercept_route)
- if getattr(self, "stealth", False):
- for script in _compiled_stealth_scripts():
- page.add_init_script(script=script)
-
page_info = self.page_pool.add_page(page)
page_info.mark_busy()
return page_info
@@ -202,10 +197,6 @@ class AsyncSession:
if disable_resources:
await page.route("**/*", async_intercept_route)
- if getattr(self, "stealth", False):
- for script in _compiled_stealth_scripts():
- await page.add_init_script(script=script)
-
return self.page_pool.add_page(page)
def get_pool_stats(self) -> Dict[str, int]:
@@ -251,151 +242,118 @@ class AsyncSession:
return handle_response
-class DynamicSessionMixin:
- def __validate__(self, **params):
+class BaseSessionMixin:
+ @overload
+ def __validate_routine__(self, params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
+
+ @overload
+ def __validate_routine__(self, params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
+
+ def __validate_routine__(
+ self, params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]
+ ) -> PlaywrightConfig | StealthConfig:
+ # Dark color scheme bypasses the 'prefersLightColor' check in creepjs
+ self._context_options: Dict[str, Any] = {"color_scheme": "dark", "device_scale_factor": 2}
+ self._launch_options: Dict[str, Any] = self._context_options | {
+ "args": DEFAULT_FLAGS,
+ "ignore_default_args": HARMFUL_DEFAULT_ARGS,
+ }
if "__max_pages" in params:
params["max_pages"] = params.pop("__max_pages")
- config = validate(params, model=PlaywrightConfig)
+ config = validate(params, model=model)
+ self._headers_keys = (
+ {header.lower() for header in config.extra_headers.keys()} if config.extra_headers else set()
+ )
- self._max_pages = config.max_pages
- self._headless = config.headless
- self._hide_canvas = config.hide_canvas
- self._disable_webgl = config.disable_webgl
- self._real_chrome = config.real_chrome
- self._stealth = config.stealth
- self._google_search = config.google_search
- self._wait = config.wait
- self._proxy = config.proxy
- self._locale = config.locale
- self._extra_headers = config.extra_headers
- self._useragent = config.useragent
- self._timeout = config.timeout
- self._cookies = config.cookies
- self._disable_resources = config.disable_resources
- self._cdp_url = config.cdp_url
- self._network_idle = config.network_idle
- self._load_dom = config.load_dom
- self._wait_selector = config.wait_selector
- self._init_script = config.init_script
- self._wait_selector_state = config.wait_selector_state
- self._extra_flags = config.extra_flags
- self._selector_config = config.selector_config
- self._timezone_id = config.timezone_id
- self._additional_args = config.additional_args
- self._page_action = config.page_action
- self._user_data_dir = config.user_data_dir
- self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
- self.__initiate_browser_options__()
+ return config
- def __initiate_browser_options__(self):
- if TYPE_CHECKING:
- assert isinstance(self._proxy, tuple)
-
- if not self._cdp_url:
- # `launch_options` is used with persistent context
- self.launch_options = dict(
- _launch_kwargs(
- self._headless,
- self._proxy,
- self._locale,
- tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
- self._useragent,
- self._real_chrome,
- self._stealth,
- self._hide_canvas,
- self._disable_webgl,
- self._timezone_id,
- tuple(self._extra_flags) if self._extra_flags else tuple(),
- )
- )
- self.launch_options["extra_http_headers"] = dict(self.launch_options["extra_http_headers"])
- self.launch_options["proxy"] = dict(self.launch_options["proxy"]) or None
- self.launch_options["user_data_dir"] = self._user_data_dir
- self.launch_options.update(cast(Dict, self._additional_args))
- self.context_options = dict()
- else:
- # while `context_options` is left to be used when cdp mode is enabled
- self.launch_options = dict()
- self.context_options = dict(
- _context_kwargs(
- self._proxy,
- self._locale,
- tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
- self._useragent,
- self._stealth,
- )
- )
- self.context_options["extra_http_headers"] = dict(self.context_options["extra_http_headers"])
- self.context_options["proxy"] = dict(self.context_options["proxy"]) or None
- self.context_options.update(cast(Dict, self._additional_args))
-
-
-class StealthySessionMixin:
- def __validate__(self, **params):
- if "__max_pages" in params:
- params["max_pages"] = params.pop("__max_pages")
-
- config: CamoufoxConfig = validate(params, model=CamoufoxConfig)
-
- self._max_pages = config.max_pages
- self._headless = config.headless
- self._block_images = config.block_images
- self._disable_resources = config.disable_resources
- self._block_webrtc = config.block_webrtc
- self._allow_webgl = config.allow_webgl
- self._network_idle = config.network_idle
- self._load_dom = config.load_dom
- self._humanize = config.humanize
- self._solve_cloudflare = config.solve_cloudflare
- self._wait = config.wait
- self._timeout = config.timeout
- self._page_action = config.page_action
- self._wait_selector = config.wait_selector
- self._init_script = config.init_script
- self._addons = config.addons
- self._wait_selector_state = config.wait_selector_state
- self._cookies = config.cookies
- self._google_search = config.google_search
- self._extra_headers = config.extra_headers
- self._proxy = config.proxy
- self._os_randomize = config.os_randomize
- self._disable_ads = config.disable_ads
- self._geoip = config.geoip
- self._selector_config = config.selector_config
- self._additional_args = config.additional_args
- self._user_data_dir = config.user_data_dir
- self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
- self.__initiate_browser_options__()
-
- def __initiate_browser_options__(self):
- """Initiate browser options."""
- self.launch_options: Dict[str, Any] = generate_launch_options(
- **{
- "geoip": self._geoip,
- "proxy": dict(self._proxy) if self._proxy and isinstance(self._proxy, tuple) else self._proxy,
- "addons": self._addons,
- "exclude_addons": [] if self._disable_ads else [DefaultAddons.UBO],
- "headless": self._headless,
- "humanize": True if self._solve_cloudflare else self._humanize,
- "i_know_what_im_doing": True, # To turn warnings off with the user configurations
- "allow_webgl": self._allow_webgl,
- "block_webrtc": self._block_webrtc,
- "block_images": self._block_images, # Careful! it makes some websites don't finish loading at all like stackoverflow even in headful mode.
- "os": None if self._os_randomize else get_os_name(),
- "user_data_dir": self._user_data_dir,
- "ff_version": __ff_version_str__,
- "firefox_user_prefs": {
- # This is what enabling `enable_cache` does internally, so we do it from here instead
- "browser.sessionhistory.max_entries": 10,
- "browser.sessionhistory.max_total_viewers": -1,
- "browser.cache.memory.enable": True,
- "browser.cache.disk_cache_ssl": True,
- "browser.cache.disk.smart_size.enabled": True,
- },
- **cast(Dict, self._additional_args),
+ def __generate_options__(self, extra_flags: Tuple | None = None) -> None:
+ config = cast(PlaywrightConfig, getattr(self, "_config", None))
+ self._context_options.update(
+ {
+ "proxy": config.proxy,
+ "locale": config.locale,
+ "timezone_id": config.timezone_id,
+ "extra_http_headers": config.extra_headers,
}
)
+ # The default useragent in the headful is always correct now in the current versions of Playwright
+ if config.useragent:
+ self._context_options["user_agent"] = config.useragent
+ elif not config.useragent and config.headless:
+ self._context_options["user_agent"] = (
+ __default_chrome_useragent__ if config.real_chrome else __default_useragent__
+ )
+
+ if not config.cdp_url:
+ self._launch_options |= self._context_options
+ self._context_options = {}
+ flags = self._launch_options["args"]
+ if config.extra_flags or extra_flags:
+ flags = list(set(flags + (config.extra_flags or extra_flags)))
+
+ self._launch_options.update(
+ {
+ "args": flags,
+ "headless": config.headless,
+ "user_data_dir": config.user_data_dir,
+ "channel": "chrome" if config.real_chrome else "chromium",
+ }
+ )
+
+ if config.additional_args:
+ self._launch_options.update(config.additional_args)
+ else:
+ # while `context_options` is left to be used when cdp mode is enabled
+ self._launch_options = dict()
+ if config.additional_args:
+ self._context_options.update(config.additional_args)
+
+
+class DynamicSessionMixin(BaseSessionMixin):
+ def __validate__(self, **params):
+ self._config = self.__validate_routine__(params, model=PlaywrightConfig)
+ self.__generate_options__()
+
+
+class StealthySessionMixin(BaseSessionMixin):
+ def __validate__(self, **params):
+ self._config: StealthConfig = self.__validate_routine__(params, model=StealthConfig)
+ self._context_options.update(
+ {
+ "is_mobile": False,
+ "has_touch": False,
+ # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
+ "service_workers": "allow",
+ "ignore_https_errors": True,
+ "screen": {"width": 1920, "height": 1080},
+ "viewport": {"width": 1920, "height": 1080},
+ "permissions": ["geolocation", "notifications"],
+ }
+ )
+ self.__generate_stealth_options()
+
+ def __generate_stealth_options(self) -> None:
+ flags = tuple()
+ if not self._config.cdp_url:
+ flags = DEFAULT_FLAGS + DEFAULT_STEALTH_FLAGS
+
+ if self._config.block_webrtc:
+ flags += (
+ "--webrtc-ip-handling-policy=disable_non_proxied_udp",
+ "--force-webrtc-ip-handling-policy", # Ensures the policy is enforced
+ )
+ if not self._config.allow_webgl:
+ flags += (
+ "--disable-webgl",
+ "--disable-webgl-image-chromium",
+ "--disable-webgl2",
+ )
+ if self._config.hide_canvas:
+ flags += ("--fingerprinting-canvas-image-data-noise",)
+
+ super(StealthySessionMixin, self).__generate_options__(flags)
@staticmethod
def _detect_cloudflare(page_content: str) -> str | None:
diff --git a/scrapling/engines/_browsers/_config_tools.py b/scrapling/engines/_browsers/_config_tools.py
index eaf28d6..2d7691b 100644
--- a/scrapling/engines/_browsers/_config_tools.py
+++ b/scrapling/engines/_browsers/_config_tools.py
@@ -1,15 +1,10 @@
from functools import lru_cache
-from scrapling.core._types import Tuple
-from scrapling.engines.constants import (
- DEFAULT_STEALTH_FLAGS,
- HARMFUL_DEFAULT_ARGS,
- DEFAULT_FLAGS,
-)
from scrapling.engines.toolbelt.navigation import js_bypass_path
from scrapling.engines.toolbelt.fingerprints import generate_headers
__default_useragent__ = generate_headers(browser_mode=True).get("User-Agent")
+__default_chrome_useragent__ = generate_headers(browser_mode="chrome").get("User-Agent")
@lru_cache(1)
@@ -41,99 +36,3 @@ def _compiled_stealth_scripts():
with open(script_path, "r") as f:
scripts.append(f.read())
return tuple(scripts)
-
-
-@lru_cache(2, typed=True)
-def _set_flags(hide_canvas, disable_webgl): # pragma: no cover
- """Returns the flags that will be used while launching the browser if stealth mode is enabled"""
- flags = DEFAULT_STEALTH_FLAGS
- if hide_canvas:
- flags += ("--fingerprinting-canvas-image-data-noise",)
- if disable_webgl:
- flags += (
- "--disable-webgl",
- "--disable-webgl-image-chromium",
- "--disable-webgl2",
- )
-
- return flags
-
-
-@lru_cache(2, typed=True)
-def _launch_kwargs(
- headless,
- proxy: Tuple,
- locale,
- extra_headers,
- useragent,
- real_chrome,
- stealth,
- hide_canvas,
- disable_webgl,
- timezone_id,
- extra_flags: Tuple,
-) -> Tuple:
- """Creates the arguments we will use while launching playwright's browser"""
- base_args = DEFAULT_FLAGS
- if extra_flags:
- base_args = base_args + extra_flags
-
- launch_kwargs = {
- "locale": locale,
- "timezone_id": timezone_id or None,
- "headless": headless,
- "args": base_args,
- "color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
- "proxy": proxy or tuple(),
- "device_scale_factor": 2,
- "ignore_default_args": HARMFUL_DEFAULT_ARGS,
- "channel": "chrome" if real_chrome else "chromium",
- "extra_http_headers": extra_headers or tuple(),
- "user_agent": useragent or __default_useragent__,
- }
- if stealth:
- stealth_args = base_args + _set_flags(hide_canvas, disable_webgl)
- launch_kwargs.update(
- {
- "args": stealth_args,
- "chromium_sandbox": True,
- "is_mobile": False,
- "has_touch": False,
- # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
- "service_workers": "allow",
- "ignore_https_errors": True,
- "screen": {"width": 1920, "height": 1080},
- "viewport": {"width": 1920, "height": 1080},
- "permissions": ["geolocation", "notifications"],
- }
- )
-
- return tuple(launch_kwargs.items())
-
-
-@lru_cache(2, typed=True)
-def _context_kwargs(proxy, locale, extra_headers, useragent, stealth) -> Tuple:
- """Creates the arguments for the browser context"""
- context_kwargs = {
- "proxy": proxy or tuple(),
- "locale": locale,
- "color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
- "device_scale_factor": 2,
- "extra_http_headers": extra_headers or tuple(),
- "user_agent": useragent or __default_useragent__,
- }
- if stealth:
- context_kwargs.update(
- {
- "is_mobile": False,
- "has_touch": False,
- # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
- "service_workers": "allow",
- "ignore_https_errors": True,
- "screen": {"width": 1920, "height": 1080},
- "viewport": {"width": 1920, "height": 1080},
- "permissions": ["geolocation", "notifications"],
- }
- )
-
- return tuple(context_kwargs.items())
diff --git a/scrapling/engines/_browsers/_controllers.py b/scrapling/engines/_browsers/_controllers.py
index 9e48675..ff21438 100644
--- a/scrapling/engines/_browsers/_controllers.py
+++ b/scrapling/engines/_browsers/_controllers.py
@@ -9,8 +9,6 @@ from playwright.async_api import (
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
-from patchright.sync_api import sync_playwright as sync_patchright
-from patchright.async_api import async_playwright as async_patchright
from scrapling.core.utils import log
from scrapling.core._types import Unpack, TYPE_CHECKING
@@ -21,53 +19,27 @@ from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
-class DynamicSession(DynamicSessionMixin, SyncSession):
+class DynamicSession(SyncSession, DynamicSessionMixin):
"""A Browser session manager with page pooling."""
__slots__ = (
- "_max_pages",
- "_headless",
- "_hide_canvas",
- "_disable_webgl",
- "_real_chrome",
- "_stealth",
- "_google_search",
- "_proxy",
- "_locale",
- "_extra_headers",
- "_useragent",
- "_timeout",
- "_cookies",
- "_disable_resources",
- "_network_idle",
- "_load_dom",
- "_wait_selector",
- "_init_script",
- "_wait_selector_state",
- "_wait",
- "playwright",
- "browser",
- "context",
+ "_config",
+ "_context_options",
+ "_launch_options",
+ "max_pages",
"page_pool",
+ "_max_wait_for_page",
+ "playwright",
+ "context",
"_closed",
- "_selector_config",
- "_page_action",
- "launch_options",
- "context_options",
- "_cdp_url",
- "_headers_keys",
- "_extra_flags",
- "_additional_args",
- "_user_data_dir",
)
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -76,13 +48,11 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- :param timezone_id: Set the timezone for the browser if wanted.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
@@ -94,27 +64,24 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- sync_context = sync_patchright if self._stealth else sync_playwright
+ self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- self.playwright: Playwright = sync_context().start() # pyright: ignore [reportAttributeAccessIssue]
-
- if self._cdp_url: # pragma: no cover
- self.context = self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url).new_context(
- **self.context_options
- )
+ if self._config.cdp_url: # pragma: no cover
+ browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context = browser.new_context(**self._context_options)
else:
- self.context = self.playwright.chromium.launch_persistent_context(**self.launch_options)
+ self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
- if self._init_script: # pragma: no cover
- self.context.add_init_script(path=self._init_script)
+ if self._config.init_script: # pragma: no cover
+ self.context.add_init_script(path=self._config.init_script)
- if self._cookies: # pragma: no cover
- self.context.add_cookies(self._cookies)
+ if self._config.cookies: # pragma: no cover
+ self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -122,24 +89,21 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
-
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -193,16 +157,15 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
raise e
-class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
+class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
"""An async Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory."""
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -212,13 +175,11 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- :param timezone_id: Set the timezone for the browser if wanted.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -230,28 +191,26 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- async_context = async_patchright if self._stealth else async_playwright
+ self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- self.playwright: AsyncPlaywright = await async_context().start() # pyright: ignore [reportAttributeAccessIssue]
-
- if self._cdp_url:
- browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url)
- self.context: AsyncBrowserContext = await browser.new_context(**self.context_options)
+ if self._config.cdp_url:
+ browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
else:
self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
- **self.launch_options
+ **self._launch_options
)
- if self._init_script: # pragma: no cover
- await self.context.add_init_script(path=self._init_script)
+ if self._config.init_script: # pragma: no cover
+ await self.context.add_init_script(path=self._config.init_script)
- if self._cookies:
- await self.context.add_cookies(self._cookies) # pyright: ignore
+ if self._config.cookies:
+ await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
@@ -259,20 +218,18 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
diff --git a/scrapling/engines/_browsers/_camoufox.py b/scrapling/engines/_browsers/_stealth.py
similarity index 62%
rename from scrapling/engines/_browsers/_camoufox.py
rename to scrapling/engines/_browsers/_stealth.py
index e3cbc67..912d38e 100644
--- a/scrapling/engines/_browsers/_camoufox.py
+++ b/scrapling/engines/_browsers/_stealth.py
@@ -2,117 +2,101 @@ from random import randint
from re import compile as re_compile
from playwright.sync_api import (
- Page,
Locator,
- sync_playwright,
+ Page,
+ Playwright,
)
from playwright.async_api import (
- async_playwright,
Page as async_Page,
Locator as AsyncLocator,
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
+from patchright.sync_api import sync_playwright
+from patchright.async_api import async_playwright
from scrapling.core.utils import log
-from ._types import CamoufoxSession, CamoufoxFetchParams
-from scrapling.core._types import Any, Unpack, TYPE_CHECKING
+from scrapling.core._types import Any, Unpack
+from ._config_tools import _compiled_stealth_scripts
+from ._types import StealthSession, StealthFetchParams
from ._base import SyncSession, AsyncSession, StealthySessionMixin
-from ._validators import validate_fetch as _validate, CamoufoxConfig
+from ._validators import validate_fetch as _validate, StealthConfig
from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
__CF_PATTERN__ = re_compile("challenges.cloudflare.com/cdn-cgi/challenge-platform/.*")
-class StealthySession(StealthySessionMixin, SyncSession):
- """A Stealthy session manager with page pooling."""
+class StealthySession(SyncSession, StealthySessionMixin):
+ """A Stealthy Browser session manager with page pooling."""
__slots__ = (
- "_max_pages",
- "_headless",
- "_block_images",
- "_disable_resources",
- "_block_webrtc",
- "_allow_webgl",
- "_network_idle",
- "_load_dom",
- "_humanize",
- "_solve_cloudflare",
- "_wait",
- "_timeout",
- "_page_action",
- "_wait_selector",
- "_init_script",
- "_addons",
- "_wait_selector_state",
- "_cookies",
- "_google_search",
- "_extra_headers",
- "_proxy",
- "_os_randomize",
- "_disable_ads",
- "_geoip",
- "_selector_config",
- "_additional_args",
- "playwright",
- "browser",
- "context",
+ "_config",
+ "_context_options",
+ "_launch_options",
+ "max_pages",
"page_pool",
+ "_max_wait_for_page",
+ "playwright",
+ "context",
"_closed",
- "launch_options",
- "_headers_keys",
- "_user_data_dir",
)
- def __init__(self, **kwargs: Unpack[CamoufoxSession]):
- """A Browser session manager with page pooling
+ def __init__(self, **kwargs: Unpack[StealthSession]):
+ """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
- :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- self.playwright = sync_playwright().start()
- self.context = self.playwright.firefox.launch_persistent_context(**self.launch_options)
+ self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- if self._init_script: # pragma: no cover
- self.context.add_init_script(path=self._init_script)
+ if self._config.cdp_url: # pragma: no cover
+ browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context = browser.new_context(**self._context_options)
+ else:
+ self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
- if self._cookies: # pragma: no cover
- self.context.add_cookies(self._cookies)
+ for script in _compiled_stealth_scripts():
+ self.context.add_init_script(script=script)
+
+ if self._config.init_script: # pragma: no cover
+ self.context.add_init_script(path=self._config.init_script)
+
+ if self._config.cookies: # pragma: no cover
+ self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -148,22 +132,27 @@ class StealthySession(StealthySessionMixin, SyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
- self._wait_for_page_stability(iframe, True, True)
+ self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not iframe.frame_element().is_visible():
# Double-checking that the iframe is loaded
page.wait_for_timeout(500)
+
outer_box: Any = iframe.frame_element().bounding_box()
if not iframe or not outer_box:
+ if "Just a moment... " not in (ResponseFactory._get_page_content(page)):
+ log.info("Cloudflare captcha is solved")
+ return
+
outer_box: Any = page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
- page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
+ page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -182,29 +171,26 @@ class StealthySession(StealthySessionMixin, SyncSession):
log.info("Cloudflare captcha is solved")
return
- def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
+ def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
- params = _validate(kwargs, self, CamoufoxConfig)
-
+ params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -233,7 +219,7 @@ class StealthySession(StealthySessionMixin, SyncSession):
if params.page_action:
try:
_ = params.page_action(page_info.page)
- except Exception as e:
+ except Exception as e: # pragma: no cover
log.error(f"Error executing page_action: {e}")
if params.wait_selector:
@@ -242,10 +228,12 @@ class StealthySession(StealthySessionMixin, SyncSession):
waiter.first.wait_for(state=params.wait_selector_state)
# Wait again after waiting for the selector, helpful with protections like Cloudflare
self._wait_for_page_stability(page_info.page, params.load_dom, params.network_idle)
- except Exception as e:
+ except Exception as e: # pragma: no cover
log.error(f"Error waiting for selector {params.wait_selector}: {e}")
page_info.page.wait_for_timeout(params.wait)
+
+ # Create response object
response = ResponseFactory.from_playwright_response(
page_info.page, first_response, final_response[0], params.selector_config
)
@@ -256,72 +244,78 @@ class StealthySession(StealthySessionMixin, SyncSession):
return response
- except Exception as e: # pragma: no cover
+ except Exception as e:
page_info.mark_error()
raise e
-class AsyncStealthySession(StealthySessionMixin, AsyncSession):
- """A Stealthy session manager with page pooling."""
+class AsyncStealthySession(AsyncSession, StealthySessionMixin):
+ """An async Stealthy Browser session manager with page pooling."""
- def __init__(self, **kwargs: Unpack[CamoufoxSession]):
- """A Browser session manager with page pooling
+ def __init__(self, **kwargs: Unpack[StealthSession]):
+ """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
- :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param max_pages: The maximum number of tabs to be opened at the same time. It will be used in rotation through a PagePool.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- self.playwright: AsyncPlaywright = await async_playwright().start()
- self.context: AsyncBrowserContext = await self.playwright.firefox.launch_persistent_context(
- **self.launch_options
- )
+ self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- if self._init_script: # pragma: no cover
- await self.context.add_init_script(path=self._init_script)
+ if self._config.cdp_url:
+ browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
+ else:
+ self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
+ **self._launch_options
+ )
- if self._cookies:
- await self.context.add_cookies(self._cookies) # pyright: ignore [reportArgumentType]
+ for script in _compiled_stealth_scripts():
+ await self.context.add_init_script(script=script)
+
+ if self._config.init_script: # pragma: no cover
+ await self.context.add_init_script(path=self._config.init_script)
+
+ if self._config.cookies:
+ await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
- async def _cloudflare_solver(self, page: async_Page): # pragma: no cover
- """Solve the cloudflare challenge displayed on the playwright page passed. The async version
+ async def _cloudflare_solver(self, page: async_Page) -> None: # pragma: no cover
+ """Solve the cloudflare challenge displayed on the playwright page passed
- :param page: The async targeted page
+ :param page: The targeted page
:return:
"""
await self._wait_for_networkidle(page, timeout=5000)
@@ -331,7 +325,7 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
return
else:
log.info(f'The turnstile version discovered is "{challenge_type}"')
- if challenge_type == "non-interactive": # pragma: no cover
+ if challenge_type == "non-interactive":
while "Just a moment... " in (await ResponseFactory._get_async_page_content(page)):
log.info("Waiting for Cloudflare wait page to disappear.")
await page.wait_for_timeout(1000)
@@ -350,22 +344,27 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
- await self._wait_for_page_stability(iframe, True, True)
+ await self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not await (await iframe.frame_element()).is_visible():
# Double-checking that the iframe is loaded
await page.wait_for_timeout(500)
+
outer_box: Any = await (await iframe.frame_element()).bounding_box()
if not iframe or not outer_box:
+ if "Just a moment... " not in (await ResponseFactory._get_async_page_content(page)):
+ log.info("Cloudflare captcha is solved")
+ return
+
outer_box: Any = await page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
- await page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
+ await page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
await self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -377,35 +376,33 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
await page.wait_for_timeout(100)
attempts += 1
if challenge_type != "embedded":
- await page.locator(box_selector).wait_for(state="detached")
+ await page.locator(box_selector).last.wait_for(state="detached")
await page.locator(".zone-name-title").wait_for(state="hidden")
await self._wait_for_page_stability(page, True, False)
log.info("Cloudflare captcha is solved")
return
- async def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
+ async def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
- params = _validate(kwargs, self, CamoufoxConfig)
+ params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -418,10 +415,6 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
final_response = [None]
handle_response = self._create_response_handler(page_info, final_response)
- if TYPE_CHECKING:
- if not isinstance(page_info.page, async_Page):
- raise TypeError
-
try:
# Navigate to URL and wait for a specified state
page_info.page.on("response", handle_response)
@@ -461,9 +454,8 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
# Close the page to free up resources
await page_info.page.close()
self.page_pool.pages.remove(page_info)
-
return response
- except Exception as e:
+ except Exception as e: # pragma: no cover
page_info.mark_error()
raise e
diff --git a/scrapling/engines/_browsers/_types.py b/scrapling/engines/_browsers/_types.py
index d1edca5..cb8d760 100644
--- a/scrapling/engines/_browsers/_types.py
+++ b/scrapling/engines/_browsers/_types.py
@@ -11,9 +11,10 @@ from scrapling.core._types import (
Mapping,
Optional,
Callable,
- Iterable,
+ Sequence,
TypedDict,
TypeAlias,
+ SetCookieParam,
SelectorWaitStates,
TYPE_CHECKING,
)
@@ -53,7 +54,7 @@ if TYPE_CHECKING: # pragma: no cover
json: Optional[Dict | List]
# Types for browser session
- class BrowserSession(TypedDict, total=False):
+ class PlaywrightSession(TypedDict, total=False):
max_pages: int
headless: bool
disable_resources: bool
@@ -61,9 +62,10 @@ if TYPE_CHECKING: # pragma: no cover
load_dom: bool
wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates
- cookies: Optional[Iterable[Dict]]
+ cookies: Sequence[SetCookieParam] | None
google_search: bool
wait: int | float
+ timezone_id: str | None
page_action: Optional[Callable]
proxy: Optional[str | Dict[str, str] | Tuple]
extra_headers: Optional[Dict[str, str]]
@@ -72,42 +74,32 @@ if TYPE_CHECKING: # pragma: no cover
user_data_dir: str
selector_config: Optional[Dict]
additional_args: Optional[Dict]
-
- class PlaywrightSession(BrowserSession, total=False):
- cdp_url: Optional[str]
- hide_canvas: bool
- disable_webgl: bool
+ locale: Optional[str]
real_chrome: bool
- stealth: bool
- locale: str
+ cdp_url: Optional[str]
useragent: Optional[str]
extra_flags: Optional[List[str]]
class PlaywrightFetchParams(TypedDict, total=False):
+ load_dom: bool
+ wait: int | float
+ network_idle: bool
google_search: bool
timeout: int | float
- wait: int | float
- page_action: Optional[Callable]
- extra_headers: Optional[Dict[str, str]]
disable_resources: bool
wait_selector: Optional[str]
- wait_selector_state: SelectorWaitStates
- network_idle: bool
- load_dom: bool
+ page_action: Optional[Callable]
selector_config: Optional[Dict]
+ extra_headers: Optional[Dict[str, str]]
+ wait_selector_state: SelectorWaitStates
- class CamoufoxSession(BrowserSession, total=False):
- block_images: bool
- block_webrtc: bool
+ class StealthSession(PlaywrightSession, total=False):
allow_webgl: bool
- humanize: bool | float
+ hide_canvas: bool
+ block_webrtc: bool
solve_cloudflare: bool
- addons: Optional[List[str]]
- os_randomize: bool
- disable_ads: bool
- geoip: bool
- class CamoufoxFetchParams(PlaywrightFetchParams, total=False):
+ class StealthFetchParams(PlaywrightFetchParams, total=False):
solve_cloudflare: bool
else: # pragma: no cover
@@ -116,5 +108,5 @@ else: # pragma: no cover
DataRequestParams = TypedDict
PlaywrightSession = TypedDict
PlaywrightFetchParams = TypedDict
- CamoufoxSession = TypedDict
- CamoufoxFetchParams = TypedDict
+ StealthSession = TypedDict
+ StealthFetchParams = TypedDict
diff --git a/scrapling/engines/_browsers/_validators.py b/scrapling/engines/_browsers/_validators.py
index f8b9875..82270ef 100644
--- a/scrapling/engines/_browsers/_validators.py
+++ b/scrapling/engines/_browsers/_validators.py
@@ -13,12 +13,13 @@ from scrapling.core._types import (
Tuple,
Optional,
Callable,
- Iterable,
- SelectorWaitStates,
+ Sequence,
overload,
+ SetCookieParam,
+ SelectorWaitStates,
)
from scrapling.engines.toolbelt.navigation import construct_proxy_dict
-from scrapling.engines._browsers._types import PlaywrightFetchParams, CamoufoxFetchParams
+from scrapling.engines._browsers._types import PlaywrightFetchParams, StealthFetchParams
# Custom validators for msgspec
@@ -35,15 +36,6 @@ def _is_invalid_file_path(value: str) -> bool | str: # pragma: no cover
return False
-def _validate_addon_path(value: str) -> None: # pragma: no cover
- """Fast addon path validation"""
- path = Path(value)
- if not path.exists():
- raise FileNotFoundError(f"Addon path not found: {value}")
- if not path.is_dir():
- raise ValueError(f"Addon path must be a directory of the extracted addon: {value}")
-
-
@lru_cache(2)
def _is_invalid_cdp_url(cdp_url: str) -> bool | str:
"""Fast CDP URL validation"""
@@ -65,39 +57,36 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
"""Configuration struct for validation"""
max_pages: PagesCount = 1
- cdp_url: Optional[str] = None
headless: bool = True
- google_search: bool = True
- hide_canvas: bool = False
- disable_webgl: bool = False
- real_chrome: bool = False
- stealth: bool = False
- wait: Seconds = 0
- page_action: Optional[Callable] = None
- proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
- locale: str = "en-US"
- extra_headers: Optional[Dict[str, str]] = None
- useragent: Optional[str] = None
- timeout: Seconds = 30000
- init_script: Optional[str] = None
disable_resources: bool = False
- wait_selector: Optional[str] = None
- cookies: Optional[Iterable[Dict]] = None
network_idle: bool = False
load_dom: bool = True
+ wait_selector: Optional[str] = None
wait_selector_state: SelectorWaitStates = "attached"
+ cookies: Sequence[SetCookieParam] | None = []
+ google_search: bool = True
+ wait: Seconds = 0
+ timezone_id: str | None = ""
+ page_action: Optional[Callable] = None
+ proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
+ extra_headers: Optional[Dict[str, str]] = None
+ timeout: Seconds = 30000
+ init_script: Optional[str] = None
user_data_dir: str = ""
- timezone_id: str = ""
- extra_flags: Optional[List[str]] = None
selector_config: Optional[Dict] = {}
additional_args: Optional[Dict] = {}
+ locale: str | None = None
+ real_chrome: bool = False
+ cdp_url: Optional[str] = None
+ useragent: Optional[str] = None
+ extra_flags: Optional[List[str]] = None
def __post_init__(self): # pragma: no cover
"""Custom validation after msgspec validation"""
if self.page_action and not callable(self.page_action):
raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
if self.proxy:
- self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
+ self.proxy = construct_proxy_dict(self.proxy)
if self.cdp_url:
cdp_msg = _is_invalid_cdp_url(self.cdp_url)
if cdp_msg:
@@ -118,64 +107,18 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
raise ValueError(validation_msg)
-class CamoufoxConfig(Struct, kw_only=True, frozen=False, weakref=True):
- """Configuration struct for validation"""
-
- max_pages: PagesCount = 1
- headless: bool = True # noqa: F821
- block_images: bool = False
- disable_resources: bool = False
- block_webrtc: bool = False
+class StealthConfig(PlaywrightConfig, kw_only=True, frozen=False, weakref=True):
allow_webgl: bool = True
- network_idle: bool = False
- load_dom: bool = True
- humanize: bool | float = True
+ hide_canvas: bool = False
+ block_webrtc: bool = False
solve_cloudflare: bool = False
- wait: Seconds = 0
- timeout: Seconds = 30000
- init_script: Optional[str] = None
- page_action: Optional[Callable] = None
- wait_selector: Optional[str] = None
- addons: Optional[List[str]] = None
- wait_selector_state: SelectorWaitStates = "attached"
- cookies: Optional[Iterable[Dict]] = None
- google_search: bool = True
- extra_headers: Optional[Dict[str, str]] = None
- proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
- os_randomize: bool = False
- disable_ads: bool = False
- geoip: bool = False
- user_data_dir: str = ""
- selector_config: Optional[Dict] = {}
- additional_args: Optional[Dict] = {}
def __post_init__(self):
"""Custom validation after msgspec validation"""
- if self.page_action and not callable(self.page_action):
- raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
- if self.proxy:
- self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
-
- if self.addons:
- for addon in self.addons:
- _validate_addon_path(addon)
- else:
- self.addons = []
-
- if self.init_script is not None:
- validation_msg = _is_invalid_file_path(self.init_script)
- if validation_msg:
- raise ValueError(validation_msg)
-
- if not self.cookies:
- self.cookies = []
+ super(StealthConfig, self).__post_init__()
# Cloudflare timeout adjustment
if self.solve_cloudflare and self.timeout < 60_000:
self.timeout = 60_000
- if not self.selector_config:
- self.selector_config = {}
- if not self.additional_args:
- self.additional_args = {}
@dataclass
@@ -197,9 +140,9 @@ class _fetch_params:
def validate_fetch(
- method_kwargs: Dict | PlaywrightFetchParams | CamoufoxFetchParams,
+ method_kwargs: Dict | PlaywrightFetchParams | StealthFetchParams,
session: Any,
- model: type[PlaywrightConfig] | type[CamoufoxConfig],
+ model: type[PlaywrightConfig] | type[StealthConfig],
) -> _fetch_params: # pragma: no cover
result = {}
overrides = {}
@@ -210,21 +153,20 @@ def validate_fetch(
for key in fetch_param_fields:
if key in method_kwargs:
overrides[key] = method_kwargs[key]
- else:
- # Check for underscore-prefixed attribute (private)
- attr_name = f"_{key}"
- if hasattr(session, attr_name):
- result[key] = getattr(session, attr_name)
+ elif hasattr(session, "_config") and hasattr(session._config, key):
+ result[key] = getattr(session._config, key)
if overrides:
validated_config = validate(overrides, model)
- # Extract only the fields that _fetch_params needs from validated_config
+ # Extract ONLY the fields that were actually overridden (not all fields)
+ # This prevents validated defaults from overwriting session config values
validated_dict = {
- f.name: getattr(validated_config, f.name)
- for f in fields(_fetch_params)
- if hasattr(validated_config, f.name)
+ field: getattr(validated_config, field) for field in overrides.keys() if hasattr(validated_config, field)
}
- validated_dict.setdefault("solve_cloudflare", False)
+
+ # Preserve solve_cloudflare if the user explicitly provided it, even if the model doesn't have it
+ if "solve_cloudflare" in overrides:
+ validated_dict["solve_cloudflare"] = overrides["solve_cloudflare"]
# Start with session defaults, then overwrite with validated overrides
result.update(validated_dict)
@@ -238,7 +180,7 @@ def validate_fetch(
# Cache default values for each model to reduce validation overhead
models_default_values = {}
-for _model in (CamoufoxConfig, PlaywrightConfig):
+for _model in (StealthConfig, PlaywrightConfig):
_defaults = {}
if hasattr(_model, "__struct_defaults__") and hasattr(_model, "__struct_fields__"):
for field_name, default_value in zip(_model.__struct_fields__, _model.__struct_defaults__): # type: ignore
@@ -256,14 +198,14 @@ def _filter_defaults(params: Dict, model: str) -> Dict:
@overload
-def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
+def validate(params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
@overload
-def validate(params: Dict, model: type[CamoufoxConfig]) -> CamoufoxConfig: ...
+def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
-def validate(params: Dict, model: type[PlaywrightConfig] | type[CamoufoxConfig]) -> PlaywrightConfig | CamoufoxConfig:
+def validate(params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]) -> PlaywrightConfig | StealthConfig:
try:
# Filter out params with the default values (no need to validate them) to speed up validation
filtered = _filter_defaults(params, model.__name__)
diff --git a/scrapling/engines/constants.py b/scrapling/engines/constants.py
index df12ee3..db8618c 100644
--- a/scrapling/engines/constants.py
+++ b/scrapling/engines/constants.py
@@ -74,7 +74,6 @@ DEFAULT_STEALTH_FLAGS = (
"--disable-domain-reliability",
"--disable-threaded-animation",
"--disable-threaded-scrolling",
- # '--disable-reading-from-canvas', # For Firefox
"--enable-simple-cache-backend",
"--disable-background-networking",
"--enable-surface-synchronization",
@@ -94,9 +93,7 @@ DEFAULT_STEALTH_FLAGS = (
"--autoplay-policy=no-user-gesture-required",
"--disable-offer-store-unmasked-wallet-cards",
"--disable-blink-features=AutomationControlled",
- "--webrtc-ip-handling-policy=disable_non_proxied_udp",
"--disable-component-extensions-with-background-pages",
- "--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
"--enable-features=NetworkService,NetworkServiceInProcess,TrustTokens,TrustTokensAlwaysAllowIssuance",
"--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4",
"--disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process,TranslateUI,BlinkGenPropertyTrees",
diff --git a/scrapling/engines/toolbelt/fingerprints.py b/scrapling/engines/toolbelt/fingerprints.py
index c006ec2..2ddbbe5 100644
--- a/scrapling/engines/toolbelt/fingerprints.py
+++ b/scrapling/engines/toolbelt/fingerprints.py
@@ -13,19 +13,32 @@ from scrapling.core._types import Dict, Literal, Tuple
__OS_NAME__ = platform_system()
OSName = Literal["linux", "macos", "windows"]
+# Current versions hardcoded for now (Playwright doesn't allow to know the version of a browser without launching it)
+chromium_version = 141
+chrome_version = 143
@lru_cache(10, typed=True)
-def generate_convincing_referer(url: str) -> str:
+def generate_convincing_referer(url: str) -> str | None:
"""Takes the domain from the URL without the subdomain/suffix and make it look like you were searching Google for this website
>>> generate_convincing_referer('https://www.somewebsite.com/blah')
'https://www.google.com/search?q=somewebsite'
:param url: The URL you are about to fetch.
- :return: Google's search URL of the domain name
+ :return: Google's search URL of the domain name, or None for localhost/IP addresses
"""
- website_name = extract(url).domain
+ extracted = extract(url)
+ website_name = extracted.domain
+
+ # Skip generating referer for localhost, IP addresses, or when there's no valid domain
+ if not website_name or not extracted.suffix or website_name in ("localhost", "127.0.0.1", "::1"):
+ return None
+
+ # Check if it's an IP address (simple check for IPv4)
+ if all(part.isdigit() for part in website_name.split(".") if part):
+ return None
+
return f"https://www.google.com/search?q={website_name}"
@@ -46,7 +59,7 @@ def get_os_name() -> OSName | Tuple:
return SUPPORTED_OPERATING_SYSTEMS
-def generate_headers(browser_mode: bool = False) -> Dict:
+def generate_headers(browser_mode: bool | str = False) -> Dict:
"""Generate real browser-like headers using browserforge's generator
:param browser_mode: If enabled, the headers created are used for playwright, so it has to match everything
@@ -55,13 +68,14 @@ def generate_headers(browser_mode: bool = False) -> Dict:
# In the browser mode, we don't care about anything other than matching the OS and the browser type with the browser we are using,
# So we don't raise any inconsistency red flags while websites fingerprinting us
os_name = get_os_name()
- browsers = [Browser(name="chrome", min_version=130)]
+ ver = chrome_version if browser_mode and browser_mode == "chrome" else chromium_version
+ browsers = [Browser(name="chrome", min_version=ver, max_version=ver)]
if not browser_mode:
os_name = ("windows", "macos", "linux")
browsers.extend(
[
- Browser(name="firefox", min_version=130),
- Browser(name="edge", min_version=130),
+ Browser(name="firefox", min_version=142),
+ Browser(name="edge", min_version=140),
]
)
return HeaderGenerator(browser=browsers, os=os_name, device="desktop").generate()
diff --git a/scrapling/engines/toolbelt/navigation.py b/scrapling/engines/toolbelt/navigation.py
index 959aa5f..1b44f5a 100644
--- a/scrapling/engines/toolbelt/navigation.py
+++ b/scrapling/engines/toolbelt/navigation.py
@@ -11,7 +11,7 @@ from msgspec import Struct, structs, convert, ValidationError
from playwright.sync_api import Route
from scrapling.core.utils import log
-from scrapling.core._types import Dict, Tuple, overload, Literal
+from scrapling.core._types import Dict, Tuple
from scrapling.engines.constants import DEFAULT_DISABLED_RESOURCES
__BYPASSES_DIR__ = Path(__file__).parent / "bypasses"
@@ -49,20 +49,11 @@ async def async_intercept_route(route: async_Route):
await route.continue_()
-@overload
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[True]) -> Tuple: ...
-
-
-@overload
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[False] = False) -> Dict: ...
-
-
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: bool = False) -> Dict | Tuple:
+def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple) -> Dict:
"""Validate a proxy and return it in the acceptable format for Playwright
Reference: https://playwright.dev/python/docs/network#http-proxy
:param proxy_string: A string or a dictionary representation of the proxy.
- :param as_tuple: Return the proxy dictionary as a tuple to be cachable
:return:
"""
if isinstance(proxy_string, str):
@@ -78,7 +69,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
}
if proxy.port:
result["server"] += f":{proxy.port}"
- return tuple(result.items()) if as_tuple else result
+ return result
except ValueError:
# Urllib will say that one of the parameters above can't be casted to the correct type like `int` for port etc...
raise ValueError("The proxy argument's string is in invalid format!")
@@ -87,7 +78,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
try:
validated = convert(proxy_string, ProxyDict)
result_dict = structs.asdict(validated)
- return tuple(result_dict.items()) if as_tuple else result_dict
+ return result_dict
except ValidationError as e:
raise TypeError(f"Invalid proxy dictionary: {e}")
diff --git a/scrapling/fetchers/__init__.py b/scrapling/fetchers/__init__.py
index c135273..e2e5866 100644
--- a/scrapling/fetchers/__init__.py
+++ b/scrapling/fetchers/__init__.py
@@ -3,7 +3,7 @@ from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from scrapling.fetchers.requests import Fetcher, AsyncFetcher, FetcherSession
from scrapling.fetchers.chrome import DynamicFetcher, DynamicSession, AsyncDynamicSession
- from scrapling.fetchers.firefox import StealthyFetcher, StealthySession, AsyncStealthySession
+ from scrapling.fetchers.stealth_chrome import StealthyFetcher, StealthySession, AsyncStealthySession
# Lazy import mapping
@@ -14,9 +14,9 @@ _LAZY_IMPORTS = {
"DynamicFetcher": ("scrapling.fetchers.chrome", "DynamicFetcher"),
"DynamicSession": ("scrapling.fetchers.chrome", "DynamicSession"),
"AsyncDynamicSession": ("scrapling.fetchers.chrome", "AsyncDynamicSession"),
- "StealthyFetcher": ("scrapling.fetchers.firefox", "StealthyFetcher"),
- "StealthySession": ("scrapling.fetchers.firefox", "StealthySession"),
- "AsyncStealthySession": ("scrapling.fetchers.firefox", "AsyncStealthySession"),
+ "StealthyFetcher": ("scrapling.fetchers.stealth_chrome", "StealthyFetcher"),
+ "StealthySession": ("scrapling.fetchers.stealth_chrome", "StealthySession"),
+ "AsyncStealthySession": ("scrapling.fetchers.stealth_chrome", "AsyncStealthySession"),
}
__all__ = [
diff --git a/scrapling/fetchers/chrome.py b/scrapling/fetchers/chrome.py
index a706826..0d50c19 100644
--- a/scrapling/fetchers/chrome.py
+++ b/scrapling/fetchers/chrome.py
@@ -5,51 +5,37 @@ from scrapling.engines._browsers._controllers import DynamicSession, AsyncDynami
class DynamicFetcher(BaseFetcher):
- """A `Fetcher` class type that provide many options, all of them are based on PlayWright.
-
- Using this Fetcher class, you can do requests with:
- - Vanilla Playwright without any modifications other than the ones you chose.
- - Stealthy Playwright with the stealth mode I wrote for it. It's still a work in progress, but it bypasses many online tests like bot.sannysoft.com
- Some of the things stealth mode does include:
- 1) Patches the CDP runtime fingerprint.
- 2) Mimics some of the real browsers' properties by injecting several JS files and using custom options.
- 3) Using custom flags on launch to hide Playwright even more and make it faster.
- 4) Generates real browser's headers of the same type and same user OS, then append it to the request.
- - Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
-
- > Note that these are the main options with PlayWright, but it can be mixed.
- """
+ """A `Fetcher` that provide many options to fetch/load websites' pages through chromium-based browsers."""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[PlaywrightSession]) -> Response:
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- - cookies: Set cookies for the next request.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - locale: Set the locale for the browser if wanted. The default value is `en-US`.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - extra_flags: A list of additional browser flags to pass to the browser on launch.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request.
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
@@ -68,31 +54,30 @@ class DynamicFetcher(BaseFetcher):
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- - cookies: Set cookies for the next request.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - locale: Set the locale for the browser if wanted. The default value is `en-US`.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - extra_flags: A list of additional browser flags to pass to the browser on launch.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request.
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
diff --git a/scrapling/fetchers/firefox.py b/scrapling/fetchers/firefox.py
deleted file mode 100644
index a9361d6..0000000
--- a/scrapling/fetchers/firefox.py
+++ /dev/null
@@ -1,102 +0,0 @@
-from scrapling.core._types import Unpack
-from scrapling.engines._browsers._types import CamoufoxSession
-from scrapling.engines.toolbelt.custom import BaseFetcher, Response
-from scrapling.engines._browsers._camoufox import StealthySession, AsyncStealthySession
-
-
-class StealthyFetcher(BaseFetcher):
- """A `Fetcher` class type that is a completely stealthy fetcher that uses a modified version of Firefox.
-
- It works as real browsers passing almost all online tests/protections based on Camoufox.
- Other added flavors include setting the faked OS fingerprints to match the user's OS, and the referer of every request is set as if this request came from Google's search of this URL's domain.
- """
-
- @classmethod
- def fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
- """
- Opens up a browser and do your request based on your chosen options below.
-
- :param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - block_images: Prevent the loading of images through Firefox preferences.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - block_webrtc: Blocks WebRTC entirely.
- - allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - addons: List of Firefox addons to use. Must be paths to extracted addons.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - cookies: Set cookies for the next request.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- - disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- - geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Camoufox as additional settings.
- :return: A `Response` object.
- """
- selector_config = kwargs.get("selector_config", {}) or kwargs.get(
- "custom_config", {}
- ) # Checking `custom_config` for backward compatibility
- if not isinstance(selector_config, dict):
- raise TypeError("Argument `selector_config` must be a dictionary.")
-
- kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
-
- with StealthySession(**kwargs) as engine:
- return engine.fetch(url)
-
- @classmethod
- async def async_fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
- """
- Opens up a browser and do your request based on your chosen options below.
-
- :param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - block_images: Prevent the loading of images through Firefox preferences.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - block_webrtc: Blocks WebRTC entirely.
- - allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - addons: List of Firefox addons to use. Must be paths to extracted addons.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - cookies: Set cookies for the next request.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- - disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- - geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Camoufox as additional settings.
- :return: A `Response` object.
- """
- selector_config = kwargs.get("selector_config", {}) or kwargs.get(
- "custom_config", {}
- ) # Checking `custom_config` for backward compatibility
- if not isinstance(selector_config, dict):
- raise TypeError("Argument `selector_config` must be a dictionary.")
-
- kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
-
- async with AsyncStealthySession(**kwargs) as engine:
- return await engine.fetch(url)
diff --git a/scrapling/fetchers/stealth_chrome.py b/scrapling/fetchers/stealth_chrome.py
new file mode 100644
index 0000000..58574d3
--- /dev/null
+++ b/scrapling/fetchers/stealth_chrome.py
@@ -0,0 +1,107 @@
+from scrapling.core._types import Unpack
+from scrapling.engines._browsers._types import StealthSession
+from scrapling.engines.toolbelt.custom import BaseFetcher, Response
+from scrapling.engines._browsers._stealth import StealthySession, AsyncStealthySession
+
+
+class StealthyFetcher(BaseFetcher):
+ """A `Fetcher` class type which is a completely stealthy built on top of Chromium.
+
+ It works as real browsers passing almost all online tests/protections with many customization options.
+ """
+
+ @classmethod
+ def fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
+ """
+ Opens up a browser and do your request based on your chosen options below.
+
+ :param url: Target url.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
+ :return: A `Response` object.
+ """
+ selector_config = kwargs.get("selector_config", {}) or kwargs.get(
+ "custom_config", {}
+ ) # Checking `custom_config` for backward compatibility
+ if not isinstance(selector_config, dict):
+ raise TypeError("Argument `selector_config` must be a dictionary.")
+
+ kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
+
+ with StealthySession(**kwargs) as engine:
+ return engine.fetch(url)
+
+ @classmethod
+ async def async_fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
+ """
+ Opens up a browser and do your request based on your chosen options below.
+
+ :param url: Target url.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
+ :return: A `Response` object.
+ """
+ selector_config = kwargs.get("selector_config", {}) or kwargs.get(
+ "custom_config", {}
+ ) # Checking `custom_config` for backward compatibility
+ if not isinstance(selector_config, dict):
+ raise TypeError("Argument `selector_config` must be a dictionary.")
+
+ kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
+
+ async with AsyncStealthySession(**kwargs) as engine:
+ return await engine.fetch(url)
diff --git a/setup.cfg b/setup.cfg
index beef70a..1a02999 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
[metadata]
name = scrapling
-version = 0.3.12
+version = 0.3.13
author = Karim Shoair
author_email = karim.shoair@pm.me
description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!
diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py
index 4a79c25..e2f86f3 100644
--- a/tests/cli/test_cli.py
+++ b/tests/cli/test_cli.py
@@ -156,7 +156,6 @@ class TestCLI:
html_url,
str(output_file),
'--headless',
- '--stealth',
'--timeout', '60000'
]
)
diff --git a/tests/fetchers/async/test_dynamic.py b/tests/fetchers/async/test_dynamic.py
index 0d46365..3fe38cc 100644
--- a/tests/fetchers/async/test_dynamic.py
+++ b/tests/fetchers/async/test_dynamic.py
@@ -53,10 +53,7 @@ class TestDynamicFetcherAsync:
@pytest.mark.parametrize(
"kwargs",
[
- {"disable_webgl": True, "hide_canvas": False},
- {"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
- {"stealth": True}, # causes issues with GitHub Actions
- {"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
+ {"real_chrome": True, "disable_resources": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
diff --git a/tests/fetchers/async/test_dynamic_session.py b/tests/fetchers/async/test_dynamic_session.py
index 234854c..c403017 100644
--- a/tests/fetchers/async/test_dynamic_session.py
+++ b/tests/fetchers/async/test_dynamic_session.py
@@ -71,7 +71,6 @@ class TestAsyncDynamicSession:
"""Test AsyncDynamicSession with various options"""
async with AsyncDynamicSession(
headless=False,
- stealth=True,
disable_resources=True,
extra_headers={"X-Test": "value"}
) as session:
diff --git a/tests/fetchers/async/test_requests_session.py b/tests/fetchers/async/test_requests_session.py
index c4e1355..3846e69 100644
--- a/tests/fetchers/async/test_requests_session.py
+++ b/tests/fetchers/async/test_requests_session.py
@@ -1,4 +1,3 @@
-import pytest
from scrapling.engines.static import AsyncFetcherClient
diff --git a/tests/fetchers/async/test_camoufox.py b/tests/fetchers/async/test_stealth.py
similarity index 81%
rename from tests/fetchers/async/test_camoufox.py
rename to tests/fetchers/async/test_stealth.py
index e8bbd3f..bba39d8 100644
--- a/tests/fetchers/async/test_camoufox.py
+++ b/tests/fetchers/async/test_stealth.py
@@ -1,4 +1,3 @@
-from playwright._impl._errors import TimeoutError
import pytest
import pytest_httpbin
@@ -30,8 +29,8 @@ class TestStealthyFetcher:
async def test_basic_fetch(self, fetcher, urls):
"""Test doing a basic fetch request with multiple statuses"""
assert (await fetcher.async_fetch(urls["status_200"])).status == 200
- assert (await fetcher.async_fetch(urls["status_404"])).status == 404
- assert (await fetcher.async_fetch(urls["status_501"])).status == 501
+ # assert (await fetcher.async_fetch(urls["status_404"])).status == 404
+ # assert (await fetcher.async_fetch(urls["status_501"])).status == 501
async def test_cookies_loading(self, fetcher, urls):
"""Test if cookies are set after the request"""
@@ -55,10 +54,9 @@ class TestStealthyFetcher:
@pytest.mark.parametrize(
"kwargs",
[
- {"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
- {"block_webrtc": False, "allow_webgl": True, "block_images": True},
+ {"block_webrtc": True, "allow_webgl": True},
+ {"block_webrtc": False, "allow_webgl": True},
{"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
- {"block_images": True, "disable_resources": True, },
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
@@ -67,11 +65,8 @@ class TestStealthyFetcher:
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"google_search": True,
"extra_headers": {"ayo": ""},
- "os_randomize": True,
- "disable_ads": True,
- # "geoip": True,
"selector_config": {"keep_comments": False, "keep_cdata": False},
- "additional_args": {"window": (1920, 1080)},
+ "additional_args": {},
},
],
)
diff --git a/tests/fetchers/async/test_camoufox_session.py b/tests/fetchers/async/test_stealth_session.py
similarity index 96%
rename from tests/fetchers/async/test_camoufox_session.py
rename to tests/fetchers/async/test_stealth_session.py
index 05f7953..140742d 100644
--- a/tests/fetchers/async/test_camoufox_session.py
+++ b/tests/fetchers/async/test_stealth_session.py
@@ -72,9 +72,8 @@ class TestAsyncStealthySession:
"""Test AsyncStealthySession with various options"""
async with AsyncStealthySession(
max_pages=1,
- block_images=True,
- disable_ads=True,
- humanize=True
+ block_webrtc=True,
+ allow_webgl=True
) as session:
response = await session.fetch(urls["html"])
assert response.status == 200
diff --git a/tests/fetchers/sync/test_camoufox.py b/tests/fetchers/sync/test_camoufox.py
deleted file mode 100644
index 269238f..0000000
--- a/tests/fetchers/sync/test_camoufox.py
+++ /dev/null
@@ -1,79 +0,0 @@
-import pytest
-import pytest_httpbin
-
-from scrapling import StealthyFetcher
-StealthyFetcher.adaptive = True
-
-
-@pytest_httpbin.use_class_based_httpbin
-class TestStealthyFetcher:
- @pytest.fixture(scope="class")
- def fetcher(self):
- """Fixture to create a StealthyFetcher instance for the entire test class"""
- return StealthyFetcher
-
- @pytest.fixture(autouse=True)
- def setup_urls(self, httpbin):
- """Fixture to set up URLs for testing"""
- self.status_200 = f"{httpbin.url}/status/200"
- self.status_404 = f"{httpbin.url}/status/404"
- self.status_501 = f"{httpbin.url}/status/501"
- self.basic_url = f"{httpbin.url}/get"
- self.html_url = f"{httpbin.url}/html"
- self.delayed_url = f"{httpbin.url}/delay/10" # 10 Seconds delay response
- self.cookies_url = f"{httpbin.url}/cookies/set/test/value"
-
- def test_basic_fetch(self, fetcher):
- """Test doing a basic fetch request with multiple statuses"""
- assert fetcher.fetch(self.status_200).status == 200
- assert fetcher.fetch(self.status_404).status == 404
- assert fetcher.fetch(self.status_501).status == 501
-
- def test_cookies_loading(self, fetcher):
- """Test if cookies are set after the request"""
- response = fetcher.fetch(self.cookies_url)
- cookies = {response.cookies[0]['name']: response.cookies[0]['value']}
- assert cookies == {"test": "value"}
-
- def test_automation(self, fetcher):
- """Test if automation breaks the code or not"""
-
- def scroll_page(page):
- page.mouse.wheel(10, 0)
- page.mouse.move(100, 400)
- page.mouse.up()
- return page
-
- assert fetcher.fetch(self.html_url, page_action=scroll_page).status == 200
-
- @pytest.mark.parametrize(
- "kwargs",
- [
- {"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
- {"block_webrtc": False, "allow_webgl": True, "block_images": True},
- {"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
- {"block_images": True, "disable_resources": True, },
- {"wait_selector": "h1", "wait_selector_state": "attached"},
- {"wait_selector": "h1", "wait_selector_state": "visible"},
- {
- "network_idle": True,
- "wait": 10,
- "timeout": 30_000,
- "cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
- "google_search": True,
- "extra_headers": {"ayo": ""},
- "os_randomize": True,
- "disable_ads": True,
- # "geoip": True,
- "selector_config": {"keep_comments": False, "keep_cdata": False},
- "additional_args": {"window": (1920, 1080)},
- },
- ],
- )
- def test_properties(self, fetcher, kwargs):
- """Test if different arguments break the code or not"""
- response = fetcher.fetch(
- self.html_url,
- **kwargs
- )
- assert response.status == 200
diff --git a/tests/fetchers/sync/test_dynamic.py b/tests/fetchers/sync/test_dynamic.py
index e044e33..cac8a73 100644
--- a/tests/fetchers/sync/test_dynamic.py
+++ b/tests/fetchers/sync/test_dynamic.py
@@ -51,10 +51,7 @@ class TestDynamicFetcher:
@pytest.mark.parametrize(
"kwargs",
[
- {"disable_webgl": True, "hide_canvas": False},
- {"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
- {"stealth": True}, # causes issues with GitHub Actions
- {"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
+ {"disable_resources": True, "real_chrome": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
diff --git a/tests/fetchers/sync/test_camoufox_session.py b/tests/fetchers/sync/test_stealth_session.py
similarity index 86%
rename from tests/fetchers/sync/test_camoufox_session.py
rename to tests/fetchers/sync/test_stealth_session.py
index e282d98..e740b01 100644
--- a/tests/fetchers/sync/test_camoufox_session.py
+++ b/tests/fetchers/sync/test_stealth_session.py
@@ -2,11 +2,11 @@ import re
import pytest
import pytest_httpbin
-from scrapling.engines._browsers._camoufox import StealthySession, __CF_PATTERN__
+from scrapling.engines._browsers._stealth import StealthySession, __CF_PATTERN__
-class TestCamoufoxConstants:
- """Test Camoufox constants and patterns"""
+class TestStealthConstants:
+ """Test Stealth constants and patterns"""
def test_cf_pattern_regex(self):
"""Test __CF_PATTERN__ regex compilation"""
@@ -54,7 +54,6 @@ class TestStealthySession:
with StealthySession(
headless=True,
- block_images=True,
disable_resources=True,
solve_cloudflare=True,
wait=1000,
@@ -63,12 +62,11 @@ class TestStealthySession:
) as session:
assert session.max_pages == 1
- assert session._headless is True
- assert session._block_images is True
- assert session._disable_resources is True
- assert session._solve_cloudflare is True
- assert session._wait == 1000
- assert session._timeout == 60000
+ assert session._config.headless is True
+ assert session._config.disable_resources is True
+ assert session._config.solve_cloudflare is True
+ assert session._config.wait == 1000
+ assert session._config.timeout == 60000
assert session.context is not None
# Test Cloudflare detection
diff --git a/tests/fetchers/test_utils.py b/tests/fetchers/test_utils.py
index 0028902..fea3f3f 100644
--- a/tests/fetchers/test_utils.py
+++ b/tests/fetchers/test_utils.py
@@ -190,14 +190,6 @@ class TestConstructProxyDict:
}
assert result == expected
- def test_proxy_as_tuple(self):
- """Test returning proxy as a tuple"""
- result = construct_proxy_dict("http://proxy.example.com:8080", as_tuple=True)
-
- assert isinstance(result, tuple)
- result_dict = dict(result)
- assert result_dict["server"] == "http://proxy.example.com:8080"
-
def test_invalid_proxy_string(self):
"""Test invalid proxy string"""
with pytest.raises(ValueError):
diff --git a/tests/fetchers/test_validator.py b/tests/fetchers/test_validator.py
index 118554c..209fae0 100644
--- a/tests/fetchers/test_validator.py
+++ b/tests/fetchers/test_validator.py
@@ -1,8 +1,8 @@
import pytest
from scrapling.engines._browsers._validators import (
validate,
+ StealthConfig,
PlaywrightConfig,
- CamoufoxConfig
)
@@ -23,7 +23,7 @@ class TestValidators:
assert config.max_pages == 2
assert config.headless is True
assert config.timeout == 30000
- assert isinstance(config.proxy, tuple) # Should be converted to tuple
+ assert isinstance(config.proxy, dict)
def test_playwright_config_invalid_max_pages(self):
"""Test PlaywrightConfig with invalid max_pages"""
@@ -51,8 +51,8 @@ class TestValidators:
with pytest.raises(TypeError):
validate(params, PlaywrightConfig)
- def test_camoufox_config_valid(self):
- """Test valid CamoufoxConfig"""
+ def test_stealth_config_valid(self):
+ """Test valid StealthConfig"""
params = {
"max_pages": 1,
"headless": True,
@@ -60,20 +60,20 @@ class TestValidators:
"timeout": 30000
}
- config = validate(params, CamoufoxConfig)
+ config = validate(params, StealthConfig)
assert config.max_pages == 1
assert config.headless is True
assert config.solve_cloudflare is False
assert config.timeout == 30000
- def test_camoufox_config_cloudflare_timeout(self):
- """Test CamoufoxConfig timeout adjustment for Cloudflare"""
+ def test_stealth_config_cloudflare_timeout(self):
+ """Test StealthConfig timeout adjustment for Cloudflare"""
params = {
"solve_cloudflare": True,
"timeout": 10000 # Less than the required 60,000
}
- config = validate(params, CamoufoxConfig)
+ config = validate(params, StealthConfig)
assert config.timeout == 60000 # Should be increased
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 52f672c..7e9a957 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,7 +1,6 @@
pytest>=2.8.0,<9
pytest-cov
playwright
-camoufox
werkzeug<3.0.0
pytest-httpbin==2.1.0
pytest-asyncio
diff --git a/tox.ini b/tox.ini
index 2a24c42..c2a30ae 100644
--- a/tox.ini
+++ b/tox.ini
@@ -10,9 +10,8 @@ envlist = pre-commit,py{310,311,312,313}
usedevelop = True
changedir = tests
deps =
- playwright>=1.56.0
- patchright>=1.56.0
- camoufox>=0.4.11
+ playwright==1.56.0
+ patchright==1.56.0
-r{toxinidir}/tests/requirements.txt
extras = ai,shell
commands =
Customer Revie...' parent='
>>> section_element.children.css('h2::text')
['Products']
@@ -237,7 +237,7 @@ You can search for a specific ancestor of an element that satisfies a function;
```
## Fetching websites
-Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers.
+Instead of passing the raw HTML to Scrapling, you can retrieve a website's response directly via HTTP requests or by fetching it in a browser.
A fetcher is made for every use case.
@@ -267,7 +267,7 @@ For Async requests, you will replace the import like below:
> Notes:
>
> 1. You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from a Google search of this domain. It's enabled by default.
-> 2. The `impersonate` argument allows you to fake the TLS fingerprint for a specific version of a browser.
+> 2. The `impersonate` argument lets you fake the TLS fingerprint for a specific browser version.
> 3. There's also the `http3` argument, which, when enabled, makes the fetcher use HTTP/3 for requests, which makes your requests more authentic
This is just the tip of the iceberg with this fetcher; check out the rest from [here](fetching/static.md)
@@ -275,7 +275,7 @@ This is just the tip of the iceberg with this fetcher; check out the rest from [
### Dynamic loading
We have you covered if you deal with dynamic websites like most today!
-The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides many options to fetch/load websites' pages through browsers.
+The `DynamicFetcher` class (formerly `PlayWrightFetcher`) offers many options for fetching and loading web pages using Chromium-based browsers.
```python
>>> from scrapling.fetchers import DynamicFetcher
>>> page = DynamicFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
@@ -286,10 +286,9 @@ The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides ma
>>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
```
-It's built on top of [Playwright](https://playwright.dev/python/) and it's currently providing three main run options that can be mixed as you want:
+It's built on top of [Playwright](https://playwright.dev/python/), and it's currently providing two main run options that can be mixed as you want:
- Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser.
-- Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode, but it bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/). Check out the `StealthyFetcher` class below for more advanced stealth mode. It uses the Chromium browser.
- Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
@@ -298,7 +297,18 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
### Dynamic anti-protection loading
We also have you covered if you deal with dynamic websites with annoying anti-protections!
-The `StealthyFetcher` class uses a custom version of a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), bypassing most bot detections by default. Scrapling offers a faster custom version, includes extra tools, and features easy configurations to further increase undetectability.
+The `StealthyFetcher` class uses a stealthy version of the `DynamicFetcher` explained above.
+
+Some of the things it does:
+
+1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
+2. It bypasses CDP runtime leaks and WebRTC leaks.
+3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
+4. It generates canvas noise to prevent fingerprinting through canvas.
+5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
+6. It makes requests look as if they came from Google's search page of the requested website.
+7. and other anti-protection options...
+
```python
>>> from scrapling.fetchers import StealthyFetcher
>>> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default
@@ -318,4 +328,4 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
---
-That's Scrapling at a glance. If you want to learn more about it, continue to the next section.
\ No newline at end of file
+That's Scrapling at a glance. If you want to learn more, continue to the next section.
\ No newline at end of file
diff --git a/docs/parsing/adaptive.md b/docs/parsing/adaptive.md
index 6ba9288..33396e9 100644
--- a/docs/parsing/adaptive.md
+++ b/docs/parsing/adaptive.md
@@ -50,7 +50,7 @@ When website owners implement structural changes like
```
The selector will no longer function, and your code needs maintenance. That's where Scrapling's `adaptive` feature comes into play.
-With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element and without AI :)
+With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element, and without AI :)
```python
from scrapling import Selector, Fetcher
@@ -65,7 +65,7 @@ if not element: # One day website changes?
element = page.css('#p1', adaptive=True) # Scrapling still finds it!
# the rest of your code...
```
-Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
+Below, I will show you an example of how to use this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
## Real-World Scenario
Let's use a real website as an example and use one of the fetchers to fetch its source. To achieve this, we need to identify a website that is about to update its design/structure, copy its source, and then wait for the website to change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
@@ -98,7 +98,9 @@ Note that I introduced a new argument called `adaptive_domain`. This is because,
The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `adaptive_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
-Hence, in the two examples above, I used both the `Selector` class and the `Fetcher` class to show you that the logic for adaptive is the same.
+Hence, in the two examples above, I used both the `Selector` and `Fetcher` classes to show that the adaptive logic is the same.
+
+> Note: the main reason for creating the `adaptive_domain` argument was to handle if the website changed its URL while changing the design/structure. In that case, you can use it to continue using the previously stored adaptive data for the new URL. Otherwise, scrapling will consider it a new website and discard the old data.
## How the adaptive scraping feature works
Adaptive scraping works in two phases:
@@ -106,19 +108,19 @@ Adaptive scraping works in two phases:
1. **Save Phase**: Store unique properties of elements
2. **Match Phase**: Find elements with similar properties later
-Let's say you've got an element through selection or any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
+Let's say you've selected an element through any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
With as few technical details as possible, the general logic goes as follows:
1. You tell Scrapling to save that element's unique properties in one of the ways we will show below.
2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
- 3. Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
- 1. The domain of the current website. If you are using the `Selector` class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically.
- 2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this.
+ 3. Now, because everything about the element can be changed or removed by the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
+ 1. The domain of the current website. If you are using the `Selector` class, pass it when initializing; if you are using a fetcher, the domain will be automatically taken from the URL.
+ 2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself; we'll discuss this later.
- Together, they will be used to retrieve the element's unique properties from the database later.
+ Together, they will later be used to retrieve the element's unique properties from the database.
- 4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity with the desired element. In that comparison, everything is taken into consideration, as you will see later
+ 4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated based on their similarity to the desired element. In that comparison, everything is taken into consideration, as you will see later
5. The element(s) with the highest similarity score to the wanted element are returned.
### The unique properties
@@ -129,7 +131,7 @@ For Scrapling, the unique elements we are relying on are:
- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
- Element's parent tag name, attributes (names and values), and text.
-But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
+But you need to understand that the comparison between elements isn't exact; it's more about how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
## How to use adaptive feature
The adaptive feature can be applied to any found element, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.
@@ -146,11 +148,11 @@ Examples:
```
If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain.
-If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and the `adaptive` feature only uses the latest saved properties.
+If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you use the same identifier later for a different website and don't pass the URL parameter when initializing it. The save process overwrites previous data, and the `adaptive` feature uses only the latest saved properties.
-Besides those arguments, we have `storage` and `storage_args`. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
+Besides those arguments, we have `storage` and `storage_args`. Both are for the class to connect to the database; by default, it uses the SQLite class provided by the library. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
-Now, after enabling the `adaptive` feature globally, you have two main ways to use it.
+Now that you've enabled the `adaptive` feature globally, you have two main ways to use it.
### The CSS/XPath Selection way
As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page, like below
@@ -163,7 +165,7 @@ element = page.css('#p1', adaptive=True)
```
Pretty simple, eh?
-Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
+Well, a lot happened under the hood here. Remember the identifier we mentioned before that you need to set to retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
Additionally, for all these methods, you can pass the `identifier` argument to set it yourself. This is useful in some instances, or you can use it to save properties with the `auto_save` argument.
@@ -174,7 +176,7 @@ First, let's say you got an element like this by text:
```python
>>> element = page.find_by_text('Tipping the Velvet', first_match=True)
```
-You can save its unique properties with the `save` method, like below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
+You can save its unique properties using the `save` method, as shown below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
```python
>>> page.save(element, 'my_special_element')
```
@@ -221,7 +223,7 @@ page.save(product, 'specific_product')
```
## Known Issues
-In the `adaptive` save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
+In the `adaptive` save process, only the unique properties of the first element in the selection results are saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors are separated and each is executed alone.
## Final thoughts
Explaining this feature in detail without complications turned out to be challenging. However, still, if there's something left unclear, you can head out to the [discussions section](https://github.com/D4Vinci/Scrapling/discussions), and I will reply to you ASAP, or the Discord server, or reach out to me privately and have a chat :)
\ No newline at end of file
diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md
index fb1ccf0..ac8af75 100644
--- a/docs/parsing/main_classes.md
+++ b/docs/parsing/main_classes.md
@@ -41,7 +41,7 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th
I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.
You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested.
-After that, for the main page and elements within, most properties are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
+After that, most properties on the main page and its elements are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
### Properties
You have already seen much of this on the [overview](../overview.md) page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the [traversal](#traversal) section.
@@ -111,7 +111,7 @@ But if you try to get the direct text content, it will be empty because it doesn
```
The `get_all_text` method has the following optional arguments:
-1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'
+1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'.
2. **strip**: If enabled, strings will be stripped before concatenation. Disabled by default.
3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results and ignore any elements nested within them. The default is `('script', 'style',)`.
4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
@@ -132,7 +132,7 @@ If you use it on the page directly, you will find that you are operating on the
>>> page.tag
'html'
```
-Now, I think I hammered the (`page`/`element`) idea, so I won't return to it again.
+Now, I think I've hammered the (`page`/`element`) idea, so I won't return to it.
Getting the attributes of the element
```python
@@ -196,7 +196,7 @@ Same case with XPath
### Traversal
Using the elements we found above, we will go over the properties/methods for moving on the page in detail.
-If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding.
+If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online to better understand them.
If you are too lazy to search about it, here's a quick explanation to give you a good idea.
In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.
@@ -310,7 +310,7 @@ In the [Selector](#selector) class, all methods/properties that should return a
Let's see what [Selectors](#selectors) class adds to the table with that out of the way.
### Properties
-Apart from the normal operations on Python lists, such as iteration and slicing, etc.
+Apart from the standard operations on Python lists, such as iteration and slicing.
You can do the following:
@@ -334,7 +334,7 @@ Execute CSS and XPath selectors directly on the [Selector](#selector) instances
,
...]
```
-Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will still leave these methods to be explained in the [TextHandler](#texthandler) section below.
+Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will leave the explanation of these methods to the [TextHandler](#texthandler) section below.
However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance.
```python
@@ -355,7 +355,7 @@ However, in this class, the `re_first` behaves differently as it runs `re` on ea
```
With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`.
```python
-# Find all the products with price '53.23'
+# Find all the products with price '53.23'.
>>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
>>> page.css('.product_pod').search(search_function)
@@ -374,7 +374,7 @@ If you are too lazy like me and want to know the number of [Selector](#selector)
```python
page.css('.product_pod').length
```
-instead of this
+which is equivalent to
```python
len(page.css('.product_pod'))
```
@@ -389,15 +389,15 @@ Of course, TextHandler provides extra methods and properties that standard Pytho
### Usage
First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well.
-First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they will accept the same arguments as well.
+First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they accept the same arguments.
-- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
+- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but, as you probably figured out from the name, it returns only the first result as a `TextHandler` instance.
Also, it takes other helpful arguments, which are:
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- - **clean_match**: It's disabled by default. This causes the method to ignore all whitespace and consecutive spaces while matching.
- - **case_sensitive**: It's enabled by default. As the name implies, disabling it will cause the regex to ignore the case of letters while compiling.
+ - **clean_match**: It's disabled by default. This causes the method to ignore all whitespace, including consecutive spaces, while matching.
+ - **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
```python
@@ -471,14 +471,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
```python
>>> page.css_first('div::text').json()
```
- You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have direct text content at all.
+ You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.
In this case, the `get_all_text` method comes to the rescue, so you can do something like that
```python
>>> page.css_first('div').get_all_text(ignore_tags=[]).json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.
- Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example:
+ Another related behavior to be aware of occurs when using any fetcher, which we will explain later. If you have a JSON response like this example:
```python
>>> page = Selector("""{"some_key": "some_value"}""")
```
@@ -493,14 +493,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
{'some_key': 'some_value'}
```
You might wonder how this happened, given that the `html` tag doesn't contain direct text.
- Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.
This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
+ Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it uses the `get_all_text` method directly.
- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance
```python
>>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?'
```
-Also, you can pass `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
+Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists
```python
@@ -518,10 +518,10 @@ Other methods and properties will be added over time, but remember that this cla
## TextHandlers
You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
-The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing new needs to be explained here, but new methods will be added over time.
+The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. Nothing new needs to be explained here, but new methods will be added over time.
## AttributesHandler
-This is a read-only version of Python's standard dictionary, or `dict`, that is used solely to store the attributes of each element or each [Selector](#selector) instance.
+This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element or [Selector](#selector) instance.
```python
>>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'}
@@ -534,7 +534,7 @@ It currently adds two extra simple methods:
- The `search_values` method
- In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will require some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
+ In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
A simple example would be
```python
@@ -555,13 +555,13 @@ It currently adds two extra simple methods:
,
]
```
- All these elements have 'product' as a value for the attribute `class`.
+ All these elements have 'product' as the value for the `class` attribute.
Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
- The `json_string` property
- This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error
+ This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error.
```python
>>>page.find('script').attrib.json_string
diff --git a/docs/parsing/selection.md b/docs/parsing/selection.md
index a262f82..0b3877d 100644
--- a/docs/parsing/selection.md
+++ b/docs/parsing/selection.md
@@ -22,7 +22,7 @@ Scrapling implements CSS3 selectors as described in the [W3C specification](http
Also, Scrapling implements some non-standard pseudo-elements like:
-* To select text nodes, use ``::text``
+* To select text nodes, use ``::text``.
* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
@@ -36,19 +36,19 @@ In short, it is the same situation as CSS Selectors; if you come from Scrapy/Par
To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above, and `xpath_first` is faster.
-> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
+> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here, as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
### Selectors examples
Let's see some shared examples of using CSS and XPath Selectors.
-Select all elements with the class `product`
+Select all elements with the class `product`.
```python
products = page.css('.product')
products = page.xpath('//*[@class="product"]')
```
Note: The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class**
-Select the first element with the class `product`
+Select the first element with the class `product`.
```python
product = page.css_first('.product')
product = page.xpath_first('//*[@class="product"]')
@@ -73,12 +73,12 @@ Get the `href` attribute of the first element with the `a` tag name
link = page.css_first('a::attr(href)')
link = page.xpath_first('//a/@href')
```
-Select the text of the first element with the `h1` tag name, which contains 'Phone', and under an element with class 'product'
+Select the text of the first element with the `h1` tag name, which contains `Phone`, and under an element with class `product`.
```python
title = page.css_first('.product h1:contains("Phone")::text')
title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
```
-You can nest and chain selectors as you want, given that it returns results
+You can nest and chain selectors as you want, given that they return results
```python
page.css_first('.product').css_first('h1:contains("Phone")::text')
page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
@@ -104,7 +104,7 @@ Scrapling provides the ability to select elements based on their direct text con
What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
-With `find_by_text`, you will pass the text as the first argument; with the `find_by_regex` method, the regex pattern is the first argument. Both methods share the following arguments:
+With `find_by_text`, you pass the text as the first argument; with `find_by_regex`, the regex pattern is the first argument. Both methods share the following arguments:
* **first_match**: If `True` (the default), the method used will return the first result it finds.
* **case_sensitive**: If `True`, the case of the letters will be considered.
@@ -117,7 +117,7 @@ By default, Scrapling searches for the exact matching of the text/pattern you pa
Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
### Finding Similar Elements
-One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
+One of the most remarkable new features Scrapling puts on the table is the ability to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
So, how does it work?
@@ -129,8 +129,8 @@ Imagine a scenario where you found a product by its title, for example, and you
That's a lot of talking, I know, but I had to go deep. I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
-* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the values of the attributes of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what the other arguments do first.
-* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change a lot between elements, making them unreliable.
+* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the tag attributes of both elements should be at least 20% similar. If you want to turn off this check (basically Step 3), you can set this attribute to 0, but I recommend you read what the other arguments do first.
+* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change significantly across elements, making them unreliable.
* **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends.
Now, let's check out the examples below.
@@ -148,7 +148,7 @@ Find the first element whose text fully matches this text
>>> page.find_by_text('Tipping the Velvet')
```
-Combining it with `page.urljoin` to return the full URL from the relative `href`
+Combining it with `page.urljoin` to return the full URL from the relative `href`.
```python
>>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html'
@@ -174,7 +174,7 @@ Get all elements that contain the word `the` (Partial matching)
'Mesaerion: The Best Science ...',
"It's Only the Himalayas"]
```
-The search is case-insensitive, so those results have `The`, not only the lowercase one `the`; let's limit the search to the elements with `the` only.
+The search is case-insensitive, so those results include `The`, not just the lowercase `the`; let's limit the search to elements with `the` only.
```python
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
@@ -332,14 +332,14 @@ def extract_reviews(page):
]
```
## Filters-based searching
-This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier to learn for newcomers to Web Scraping than writing selectors.
+This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier for newcomers to Web Scraping to learn than writing selectors.
-Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
+Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can accept multiple filters and return all elements on the pages where all these filters apply.
To be more specific:
* Any string passed is considered a tag name.
-* Any iterable passed, like List/Tuple/Set, is considered an iterable of tag names.
+* Any iterable passed, like List/Tuple/Set, will be considered as an iterable of tag names.
* Any dictionary is considered a mapping of HTML element(s), attribute names, and attribute values.
* Any regex patterns passed are used to filter elements by content, like the `find_by_regex` method
* Any functions passed are used to filter elements
@@ -357,7 +357,7 @@ It filters all elements in the current page/element in the following order:
Notes:
1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on.
-2. The order in which you pass the arguments doesn't matter. The only order taken into consideration is the order explained above.
+2. The order in which you pass the arguments doesn't matter. The only order considered is the one explained above.
Check examples to clear any confusion :)
@@ -509,4 +509,4 @@ We will have a deep look at it while explaining the [TextHandler](main_classes.m
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']
```
-And so on. You get the idea. We will explain this in more detail on the next page while explaining the [TextHandler](main_classes.md#texthandler) class.
\ No newline at end of file
+And so on. You get the idea. We will explain this in more detail on the next page, along with the [TextHandler](main_classes.md#texthandler) class.
\ No newline at end of file
diff --git a/docs/requirements.txt b/docs/requirements.txt
index e996a0b..c2c5d54 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,5 +1,8 @@
-mkdocs-material
-mkdocstrings
-mkdocstrings-python
-mkdocs-material[imaging]
-black
\ No newline at end of file
+mkdocstrings>=1.0.0
+mkdocstrings-python>=2.0.1
+griffe-inherited-docstrings
+griffe-runtime-objects
+griffe-sphinx
+mkdocs-material[imaging]>=9.7.1
+black>=25.12.0
+pngquant
\ No newline at end of file
diff --git a/docs/tutorials/migrating_from_beautifulsoup.md b/docs/tutorials/migrating_from_beautifulsoup.md
index d1b7019..e5474bb 100644
--- a/docs/tutorials/migrating_from_beautifulsoup.md
+++ b/docs/tutorials/migrating_from_beautifulsoup.md
@@ -1,10 +1,10 @@
# Migrating from BeautifulSoup to Scrapling
-If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is incredibly faster, provides the same parsing capabilities, adds more parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
+If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is much faster, provides the same parsing capabilities as BS, adds additional parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
-Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to accomplish a specific task using BeautifulSoup and the corresponding method in Scrapling.
+Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to achieve a specific task using BeautifulSoup and the corresponding method in Scrapling.
-You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, but that's one of the reasons why BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short oneliner, there is no need to sacrifice performance to shorten that short line :)
+You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, which is one of the reasons BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short one-liner, there is no need to sacrifice performance to shorten that short line :)
| Task | BeautifulSoup Code | Scrapling Code |
@@ -56,7 +56,7 @@ Here's a simple example of scraping a web page to extract all the links using Be
import requests
from bs4 import BeautifulSoup
-url = 'http://example.com'
+url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
@@ -70,7 +70,7 @@ for link in links:
```python
from scrapling import Fetcher
-url = 'http://example.com'
+url = 'https://example.com'
page = Fetcher.get(url)
links = page.css('a::attr(href)')
@@ -78,12 +78,12 @@ for link in links:
print(link)
```
-As you can see, Scrapling simplifies the process by handling the fetching and parsing in a single step, making your code cleaner and more efficient.
+As you can see, Scrapling simplifies the process by combining fetching and parsing into a single step, making your code cleaner and more efficient.
**Additional Notes:**
- **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons.
-- **Element Types**: In BeautifulSoup, elements are `Tag` objects, while in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
+- **Element Types**: In BeautifulSoup, elements are `Tag` objects; in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.css_first()`). To avoid errors, check for `None` before accessing properties.
- **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can help remove extra whitespace, consecutive spaces, or unwanted characters. Please check out the documentation for the complete list.
diff --git a/docs/tutorials/replacing_ai.md b/docs/tutorials/replacing_ai.md
index 733e88b..200073c 100644
--- a/docs/tutorials/replacing_ai.md
+++ b/docs/tutorials/replacing_ai.md
@@ -26,13 +26,13 @@ How will you solve that manually? I'm referring to generic web scraping of vario
## AI to the rescue, but at a high cost
-Of course, the AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
+Of course, AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
This approach is, of course, beautiful. I love AI and find it very fascinating, especially Generative AI. You will probably spend a lot of time on prompt engineering and tweaking the prompts, but if that's cool with you, you will soon hit the real issue with using AI here.
Most websites have vast amounts of content per page, which you will need to pass to the AI somehow so it can do its magic. This will burn through tokens like fire in a haystack, quickly accumulating high costs.
-Unless money is irrelevant to you, you will try to find less expensive approaches, and that's why I made Scrapling :smile:
+Unless money is irrelevant to you, you will try to find less expensive approaches, and that's where Scrapling comes into play :smile:
## Scrapling got you covered
@@ -41,11 +41,11 @@ Scrapling can handle almost all issues you will face during Web Scraping, and th
### Solving issue T1: Rapidly changing website structures
That's why the [adaptive](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/) feature was made. You knew I would talk about it, and here we are :)
-While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties to find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
+While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties so you can find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
That's how the adaptive feature works: it stores everything unique about an element. When the website structure changes, it returns the element with the highest similarity score of the previous element.
-I have already explained that in more detail and with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
+I have already explained this in more detail, with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
### Solving issue T2: Unstable selectors
If you have been doing Web scraping for a long enough time, you have likely experienced this once. I'm referring to a website that employs poor design patterns, built on raw HTML without any IDs/classes, or uses random class names with nothing else to rely on, etc...
@@ -59,16 +59,16 @@ In these cases, standard selection methods with CSS/XPath selectors won't be opt
There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this.
### Solving issue T3: Increasingly complex anti-bot measures
-It's known that making an undetectable spider takes more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
+It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
-1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides many flexible options, like stealth mode suitable for small to medium protections and using your real browser.
-2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher utilizes our version of a modified Firefox browser, called [Camoufox](https://camoufox.com/stealth/), which nearly passes all known tests and incorporates additional tricks. **With v0.3, this fetcher can bypass Cloudflare for you automatically as well!**
+1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
+2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
-These two will be improved a lot with the upcoming updates, so stay tuned :)
+We keep improving these two with each update, so stay tuned :)
### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data
-This one is tough to handle, but it's possible with Scrapling's flexibility.
+This one is tough to handle, but Scrapling's flexibility makes it possible.
I talked with someone who uses AI to extract prices from different websites. He is only interested in prices and titles, so he uses AI to find the price for him.
@@ -94,7 +94,7 @@ It will be a bit boring, but it's definitely less expensive than AI.
This example illustrates the point I aim to convey here. Not every challenge will need AI to be solved, but sometimes you need to be creative, and that might save you a lot of money.
### Solving issue B3: Pagination variations
-This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the following updates :)
+This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :)
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea.
@@ -112,6 +112,6 @@ For a quick comparison.
This table is based on pricing from [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing)
## Conclusion
-While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit designed to tackle the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
+While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit for tackling the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects!
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index ea2dd7a..c8b9709 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -29,6 +29,7 @@ theme:
- navigation.sections
- navigation.tracking
- navigation.instant
+ - navigation.instant.prefetch
- navigation.instant.progress
# - navigation.tabs
# - navigation.expand
@@ -41,6 +42,7 @@ theme:
- content.action.view
- content.action.edit
- content.code.copy
+ - content.code.select
- content.code.annotate
- content.code.annotation
@@ -103,10 +105,13 @@ markdown_extensions:
plugins:
- search
-# - social:
-# cards_layout_options:
-# background_color: "#1f1f1f"
-# font_family: Roboto
+ - privacy:
+ links: false
+ - optimize
+ - social:
+ cards_layout_options:
+ background_color: "#1f1f1f"
+ font_family: Roboto
- mkdocstrings:
handlers:
python:
@@ -122,11 +127,19 @@ plugins:
unwrap_annotated: true
filters:
- '!^_'
+ - "^__"
merge_init_into_class: true
docstring_section_style: spacy
signature_crossrefs: true
show_symbol_type_heading: true
show_symbol_type_toc: true
+ show_inheritance_diagram: true
+ modernize_annotations: true
+ extensions:
+ - griffe_runtime_objects
+ - griffe_sphinx
+ - griffe_inherited_docstrings:
+ merge: true
extra:
homepage: https://scrapling.readthedocs.io/en/latest/
diff --git a/pyproject.toml b/pyproject.toml
index 0dd663b..a5b6f5c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "scrapling"
# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
-version = "0.3.12"
+version = "0.3.13"
description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
readme = {file = "docs/README.md", content-type = "text/markdown"}
license = {file = "LICENSE"}
@@ -60,7 +60,7 @@ dependencies = [
"lxml>=6.0.2",
"cssselect>=1.3.0",
"orjson>=3.11.5",
- "tldextract>=5.3.0",
+ "tldextract>=5.3.1",
]
[project.optional-dependencies]
@@ -69,8 +69,7 @@ fetchers = [
"curl_cffi>=0.14.0",
"playwright==1.56.0",
"patchright==1.56.0",
- "camoufox>=0.4.11",
- "geoip2>=5.2.0",
+ "browserforge>=1.2.1",
"msgspec>=0.20.0",
]
ai = [
diff --git a/scrapling/__init__.py b/scrapling/__init__.py
index 23f917b..0eca2f2 100644
--- a/scrapling/__init__.py
+++ b/scrapling/__init__.py
@@ -1,5 +1,5 @@
__author__ = "Karim Shoair (karim.shoair@pm.me)"
-__version__ = "0.3.12"
+__version__ = "0.3.13"
__copyright__ = "Copyright (c) 2024 Karim Shoair"
from typing import Any, TYPE_CHECKING
diff --git a/scrapling/cli.py b/scrapling/cli.py
index 99a2c63..68470a5 100644
--- a/scrapling/cli.py
+++ b/scrapling/cli.py
@@ -125,14 +125,9 @@ def install(force): # pragma: no cover
"playwright",
"install-deps",
"chromium",
- "firefox",
],
"Playwright dependencies",
)
- __Execute(
- [python_executable, "-m", "camoufox", "fetch", "--browserforge"],
- "Camoufox browser and databases",
- )
# if no errors raised by the above commands, then we add the below file
__PACKAGE_DIR__.joinpath(".scrapling_dependencies_installed").touch()
else:
@@ -610,17 +605,11 @@ def delete(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
-@option("--locale", default="en-US", help="Browser locale (default: en-US)")
-@option("--stealth/--no-stealth", default=False, help="Enable stealth mode (default: False)")
+@option("--locale", default=None, help="Specify user locale. Defaults to the system default locale.")
@option(
- "--hide-canvas/--show-canvas",
+ "--real-chrome/--no-real-chrome",
default=False,
- help="Add noise to canvas operations (default: False)",
-)
-@option(
- "--disable-webgl/--enable-webgl",
- default=False,
- help="Disable WebGL support (default: False)",
+ help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
@@ -640,9 +629,7 @@ def fetch(
css_selector,
wait_selector,
locale,
- stealth,
- hide_canvas,
- disable_webgl,
+ real_chrome,
proxy,
extra_headers,
):
@@ -659,9 +646,7 @@ def fetch(
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser.
- :param stealth: Enables stealth mode.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -676,9 +661,7 @@ def fetch(
"network_idle": network_idle,
"timeout": timeout,
"locale": locale,
- "stealth": stealth,
- "hide_canvas": hide_canvas,
- "disable_webgl": disable_webgl,
+ "real_chrome": real_chrome,
}
if wait > 0:
@@ -703,11 +686,6 @@ def fetch(
default=True,
help="Run browser in headless mode (default: True)",
)
-@option(
- "--block-images/--allow-images",
- default=False,
- help="Block image loading (default: False)",
-)
@option(
"--disable-resources/--enable-resources",
default=False,
@@ -718,11 +696,6 @@ def fetch(
default=False,
help="Block WebRTC entirely (default: False)",
)
-@option(
- "--humanize/--no-humanize",
- default=False,
- help="Humanize cursor movement (default: False)",
-)
@option(
"--solve-cloudflare/--no-solve-cloudflare",
default=False,
@@ -735,9 +708,14 @@ def fetch(
help="Wait for network idle (default: False)",
)
@option(
- "--disable-ads/--allow-ads",
+ "--real-chrome/--no-real-chrome",
default=False,
- help="Install uBlock Origin addon (default: False)",
+ help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
+)
+@option(
+ "--hide-canvas/--show-canvas",
+ default=False,
+ help="Add noise to canvas operations (default: False)",
)
@option(
"--timeout",
@@ -757,11 +735,6 @@ def fetch(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
-@option(
- "--geoip/--no-geoip",
- default=False,
- help="Use IP geolocation for timezone/locale (default: False)",
-)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
"--extra-headers",
@@ -773,19 +746,17 @@ def stealthy_fetch(
url,
output_file,
headless,
- block_images,
disable_resources,
block_webrtc,
- humanize,
solve_cloudflare,
allow_webgl,
network_idle,
- disable_ads,
+ real_chrome,
+ hide_canvas,
timeout,
wait,
css_selector,
wait_selector,
- geoip,
proxy,
extra_headers,
):
@@ -795,19 +766,17 @@ def stealthy_fetch(
:param url: Target url.
:param output_file: Output file path (.md for Markdown, .html for HTML).
:param headless: Run the browser in headless/hidden, or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
:param disable_resources: Drop requests of unnecessary resources for a speed boost.
:param block_webrtc: Blocks WebRTC entirely.
- :param humanize: Humanize the cursor movement.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges.
:param allow_webgl: Allow WebGL (recommended to keep enabled).
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Install the uBlock Origin addon on the browser.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before returning.
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Automatically use IP's longitude, latitude, timezone, country, locale.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -818,16 +787,14 @@ def stealthy_fetch(
# Build request arguments
kwargs = {
"headless": headless,
- "block_images": block_images,
"disable_resources": disable_resources,
"block_webrtc": block_webrtc,
- "humanize": humanize,
"solve_cloudflare": solve_cloudflare,
"allow_webgl": allow_webgl,
"network_idle": network_idle,
- "disable_ads": disable_ads,
+ "real_chrome": real_chrome,
+ "hide_canvas": hide_canvas,
"timeout": timeout,
- "geoip": geoip,
}
if wait > 0:
diff --git a/scrapling/core/_shell_signatures.py b/scrapling/core/_shell_signatures.py
index 803e6d5..d9a42fc 100644
--- a/scrapling/core/_shell_signatures.py
+++ b/scrapling/core/_shell_signatures.py
@@ -1,9 +1,13 @@
from scrapling.core._types import (
- Dict,
Any,
+ Dict,
List,
Tuple,
+ Sequence,
+ Callable,
Optional,
+ SetCookieParam,
+ SelectorWaitStates,
)
# Parameter definitions for shell function signatures (defined once at module level)
@@ -31,57 +35,58 @@ _REQUESTS_PARAMS = {
_FETCH_PARAMS = {
"headless": bool,
- "google_search": bool,
- "hide_canvas": bool,
- "disable_webgl": bool,
- "real_chrome": bool,
- "stealth": bool,
- "wait": int | float,
- "page_action": Optional[Any],
- "proxy": Optional[str | Dict],
- "locale": str,
- "extra_headers": Optional[Dict[str, str]],
- "useragent": Optional[str],
- "cdp_url": Optional[str],
- "timeout": int | float,
"disable_resources": bool,
- "wait_selector": Optional[str],
- "init_script": Optional[str],
- "cookies": Optional[List[Dict]],
"network_idle": bool,
"load_dom": bool,
- "wait_selector_state": Any,
- "extra_flags": Optional[List[str]],
- "additional_args": Optional[Dict],
+ "wait_selector": Optional[str],
+ "wait_selector_state": SelectorWaitStates,
+ "cookies": Sequence[SetCookieParam],
+ "google_search": bool,
+ "wait": int | float,
+ "timezone_id": str | None,
+ "page_action": Optional[Callable],
+ "proxy": Optional[str | Dict[str, str] | Tuple],
+ "extra_headers": Optional[Dict[str, str]],
+ "timeout": int | float,
+ "init_script": Optional[str],
+ "user_data_dir": str,
"selector_config": Optional[Dict],
+ "additional_args": Optional[Dict],
+ "locale": Optional[str],
+ "real_chrome": bool,
+ "cdp_url": Optional[str],
+ "useragent": Optional[str],
+ "extra_flags": Optional[List[str]],
}
_STEALTHY_FETCH_PARAMS = {
"headless": bool,
- "block_images": bool,
"disable_resources": bool,
- "block_webrtc": bool,
- "allow_webgl": bool,
"network_idle": bool,
"load_dom": bool,
- "humanize": bool | float,
- "solve_cloudflare": bool,
- "wait": int | float,
- "timeout": int | float,
- "page_action": Optional[Any],
"wait_selector": Optional[str],
- "init_script": Optional[str],
- "addons": Optional[List[str]],
- "wait_selector_state": Any,
- "cookies": Optional[List[Dict]],
+ "wait_selector_state": SelectorWaitStates,
+ "cookies": Sequence[SetCookieParam],
"google_search": bool,
+ "wait": int | float,
+ "timezone_id": str | None,
+ "page_action": Optional[Callable],
+ "proxy": Optional[str | Dict[str, str] | Tuple],
"extra_headers": Optional[Dict[str, str]],
- "proxy": Optional[str | Dict],
- "os_randomize": bool,
- "disable_ads": bool,
- "geoip": bool,
+ "timeout": int | float,
+ "init_script": Optional[str],
+ "user_data_dir": str,
"selector_config": Optional[Dict],
"additional_args": Optional[Dict],
+ "locale": Optional[str],
+ "real_chrome": bool,
+ "cdp_url": Optional[str],
+ "useragent": Optional[str],
+ "extra_flags": Optional[List[str]],
+ "allow_webgl": bool,
+ "hide_canvas": bool,
+ "block_webrtc": bool,
+ "solve_cloudflare": bool,
}
# Mapping of function names to their parameter definitions
diff --git a/scrapling/core/_types.py b/scrapling/core/_types.py
index ac19a91..d7c1f8b 100644
--- a/scrapling/core/_types.py
+++ b/scrapling/core/_types.py
@@ -57,3 +57,17 @@ except ImportError: # pragma: no cover
from typing_extensions import Self # Backport
except ImportError:
Self = object
+
+
+# Copied from `playwright._impl._api_structures.SetCookieParam`
+class SetCookieParam(TypedDict, total=False):
+ name: str
+ value: str
+ url: Optional[str]
+ domain: Optional[str]
+ path: Optional[str]
+ expires: Optional[float]
+ httpOnly: Optional[bool]
+ secure: Optional[bool]
+ sameSite: Optional[Literal["Lax", "None", "Strict"]]
+ partitionKey: Optional[str]
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 4d5277a..4d5929b 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -17,13 +17,15 @@ from scrapling.fetchers import (
from scrapling.core._types import (
Optional,
Tuple,
- extraction_types,
Mapping,
Dict,
List,
Any,
- SelectorWaitStates,
Generator,
+ Sequence,
+ SetCookieParam,
+ extraction_types,
+ SelectorWaitStates,
)
@@ -213,20 +215,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
- hide_canvas: bool = False,
- disable_webgl: bool = False,
real_chrome: bool = False,
- stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
- locale: str = "en-US",
+ timezone_id: str | None = None,
+ locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
- cookies: Optional[List[Dict]] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> ResponseModel:
@@ -242,21 +242,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -269,15 +267,13 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
- stealth=stealth,
cdp_url=cdp_url,
headless=headless,
useragent=useragent,
- hide_canvas=hide_canvas,
+ timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
- disable_webgl=disable_webgl,
extra_headers=extra_headers,
google_search=google_search,
disable_resources=disable_resources,
@@ -301,20 +297,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
- hide_canvas: bool = False,
- disable_webgl: bool = False,
real_chrome: bool = False,
- stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
- locale: str = "en-US",
+ timezone_id: str | None = None,
+ locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
- cookies: Optional[List[Dict]] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> List[ResponseModel]:
@@ -330,21 +324,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -356,17 +348,15 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
- stealth=stealth,
cdp_url=cdp_url,
headless=headless,
max_pages=len(urls),
useragent=useragent,
- hide_canvas=hide_canvas,
+ timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
google_search=google_search,
- disable_webgl=disable_webgl,
extra_headers=extra_headers,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
@@ -393,29 +383,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
- block_images: bool = False,
+ google_search: bool = True,
+ real_chrome: bool = False,
+ wait: int | float = 0,
+ proxy: Optional[str | Dict[str, str]] = None,
+ timezone_id: str | None = None,
+ locale: str | None = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ useragent: Optional[str] = None,
+ hide_canvas: bool = False,
+ cdp_url: Optional[str] = None,
+ timeout: int | float = 30000,
disable_resources: bool = False,
+ wait_selector: Optional[str] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
+ network_idle: bool = False,
+ wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
- network_idle: bool = False,
- humanize: bool | float = True,
solve_cloudflare: bool = False,
- wait: int | float = 0,
- timeout: int | float = 30000,
- wait_selector: Optional[str] = None,
- addons: Optional[List[str]] = None,
- wait_selector_state: SelectorWaitStates = "attached",
- cookies: Optional[List[Dict]] = None,
- google_search: bool = True,
- extra_headers: Optional[Dict[str, str]] = None,
- proxy: Optional[str | Dict[str, str]] = None,
- os_randomize: bool = False,
- disable_ads: bool = False,
- geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> ResponseModel:
- """Use Scrapling's version of the Camoufox browser to fetch a URL and return a structured output of the result.
- Note: This is best suitable for high protection levels. It's slower than the other tools.
+ """Use the stealthy fetcher to fetch a URL and return a structured output of the result.
+ Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param url: The URL to request.
@@ -426,54 +416,52 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
page = await StealthyFetcher.async_fetch(
url,
wait=wait,
proxy=proxy,
- geoip=geoip,
- addons=addons,
+ locale=locale,
+ cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
- humanize=humanize,
+ useragent=useragent,
+ timezone_id=timezone_id,
+ real_chrome=real_chrome,
+ hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
- disable_ads=disable_ads,
network_idle=network_idle,
- block_images=block_images,
block_webrtc=block_webrtc,
- os_randomize=os_randomize,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
+ additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- additional_args=additional_args,
)
return _ContentTranslator(
Convertor._extract_content(
@@ -492,29 +480,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
- block_images: bool = False,
+ google_search: bool = True,
+ real_chrome: bool = False,
+ wait: int | float = 0,
+ proxy: Optional[str | Dict[str, str]] = None,
+ timezone_id: str | None = None,
+ locale: str | None = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ useragent: Optional[str] = None,
+ hide_canvas: bool = False,
+ cdp_url: Optional[str] = None,
+ timeout: int | float = 30000,
disable_resources: bool = False,
+ wait_selector: Optional[str] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
+ network_idle: bool = False,
+ wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
- network_idle: bool = False,
- humanize: bool | float = True,
solve_cloudflare: bool = False,
- wait: int | float = 0,
- timeout: int | float = 30000,
- wait_selector: Optional[str] = None,
- addons: Optional[List[str]] = None,
- wait_selector_state: SelectorWaitStates = "attached",
- cookies: Optional[List[Dict]] = None,
- google_search: bool = True,
- extra_headers: Optional[Dict[str, str]] = None,
- proxy: Optional[str | Dict[str, str]] = None,
- os_randomize: bool = False,
- disable_ads: bool = False,
- geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> List[ResponseModel]:
- """Use Scrapling's version of the Camoufox browser to fetch a group of URLs at the same time, and for each page return a structured output of the result.
- Note: This is best suitable for high protection levels. It's slower than the other tools.
+ """Use the stealthy fetcher to fetch a group of URLs at the same time, and for each page return a structured output of the result.
+ Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param urls: A tuple of the URLs to request.
@@ -525,54 +513,51 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
async with AsyncStealthySession(
wait=wait,
proxy=proxy,
- geoip=geoip,
- addons=addons,
+ locale=locale,
+ cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
- humanize=humanize,
- max_pages=len(urls),
+ useragent=useragent,
+ timezone_id=timezone_id,
+ real_chrome=real_chrome,
+ hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
- disable_ads=disable_ads,
- block_images=block_images,
- block_webrtc=block_webrtc,
network_idle=network_idle,
- os_randomize=os_randomize,
+ block_webrtc=block_webrtc,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
+ additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- additional_args=additional_args,
) as session:
tasks = [session.fetch(url) for url in urls]
responses = await gather(*tasks)
diff --git a/scrapling/core/translator.py b/scrapling/core/translator.py
index bb6d405..c34ed7d 100644
--- a/scrapling/core/translator.py
+++ b/scrapling/core/translator.py
@@ -130,5 +130,5 @@ translator = HTMLTranslator()
@lru_cache(maxsize=256)
def css_to_xpath(query: str) -> str:
- """Return translated XPath version of a given CSS query"""
+ """Return the translated XPath version of a given CSS query"""
return translator.css_to_xpath(query)
diff --git a/scrapling/engines/_browsers/_base.py b/scrapling/engines/_browsers/_base.py
index fdb298c..bb14b5f 100644
--- a/scrapling/engines/_browsers/_base.py
+++ b/scrapling/engines/_browsers/_base.py
@@ -1,7 +1,6 @@
from time import time
from asyncio import sleep as asyncio_sleep, Lock
-from camoufox import DefaultAddons
from playwright.sync_api._generated import Page
from playwright.sync_api import (
Frame,
@@ -17,18 +16,18 @@ from playwright.async_api import (
BrowserContext as AsyncBrowserContext,
)
from playwright._impl._errors import Error as PlaywrightError
-from camoufox.pkgman import installed_verstr as camoufox_version
-from camoufox.utils import launch_options as generate_launch_options
from ._page import PageInfo, PagePool
from scrapling.parser import Selector
-from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING
-from scrapling.engines.toolbelt.fingerprints import get_os_name
-from ._validators import validate, PlaywrightConfig, CamoufoxConfig
-from ._config_tools import _compiled_stealth_scripts, _launch_kwargs, _context_kwargs
+from ._validators import validate, PlaywrightConfig, StealthConfig
+from ._config_tools import __default_chrome_useragent__, __default_useragent__
from scrapling.engines.toolbelt.navigation import intercept_route, async_intercept_route
-
-__ff_version_str__ = camoufox_version().split(".", 1)[0]
+from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING, overload, Tuple
+from scrapling.engines.constants import (
+ DEFAULT_STEALTH_FLAGS,
+ HARMFUL_DEFAULT_ARGS,
+ DEFAULT_FLAGS,
+)
class SyncSession:
@@ -84,10 +83,6 @@ class SyncSession:
if disable_resources:
page.route("**/*", intercept_route)
- if getattr(self, "stealth", False):
- for script in _compiled_stealth_scripts():
- page.add_init_script(script=script)
-
page_info = self.page_pool.add_page(page)
page_info.mark_busy()
return page_info
@@ -202,10 +197,6 @@ class AsyncSession:
if disable_resources:
await page.route("**/*", async_intercept_route)
- if getattr(self, "stealth", False):
- for script in _compiled_stealth_scripts():
- await page.add_init_script(script=script)
-
return self.page_pool.add_page(page)
def get_pool_stats(self) -> Dict[str, int]:
@@ -251,151 +242,118 @@ class AsyncSession:
return handle_response
-class DynamicSessionMixin:
- def __validate__(self, **params):
+class BaseSessionMixin:
+ @overload
+ def __validate_routine__(self, params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
+
+ @overload
+ def __validate_routine__(self, params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
+
+ def __validate_routine__(
+ self, params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]
+ ) -> PlaywrightConfig | StealthConfig:
+ # Dark color scheme bypasses the 'prefersLightColor' check in creepjs
+ self._context_options: Dict[str, Any] = {"color_scheme": "dark", "device_scale_factor": 2}
+ self._launch_options: Dict[str, Any] = self._context_options | {
+ "args": DEFAULT_FLAGS,
+ "ignore_default_args": HARMFUL_DEFAULT_ARGS,
+ }
if "__max_pages" in params:
params["max_pages"] = params.pop("__max_pages")
- config = validate(params, model=PlaywrightConfig)
+ config = validate(params, model=model)
+ self._headers_keys = (
+ {header.lower() for header in config.extra_headers.keys()} if config.extra_headers else set()
+ )
- self._max_pages = config.max_pages
- self._headless = config.headless
- self._hide_canvas = config.hide_canvas
- self._disable_webgl = config.disable_webgl
- self._real_chrome = config.real_chrome
- self._stealth = config.stealth
- self._google_search = config.google_search
- self._wait = config.wait
- self._proxy = config.proxy
- self._locale = config.locale
- self._extra_headers = config.extra_headers
- self._useragent = config.useragent
- self._timeout = config.timeout
- self._cookies = config.cookies
- self._disable_resources = config.disable_resources
- self._cdp_url = config.cdp_url
- self._network_idle = config.network_idle
- self._load_dom = config.load_dom
- self._wait_selector = config.wait_selector
- self._init_script = config.init_script
- self._wait_selector_state = config.wait_selector_state
- self._extra_flags = config.extra_flags
- self._selector_config = config.selector_config
- self._timezone_id = config.timezone_id
- self._additional_args = config.additional_args
- self._page_action = config.page_action
- self._user_data_dir = config.user_data_dir
- self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
- self.__initiate_browser_options__()
+ return config
- def __initiate_browser_options__(self):
- if TYPE_CHECKING:
- assert isinstance(self._proxy, tuple)
-
- if not self._cdp_url:
- # `launch_options` is used with persistent context
- self.launch_options = dict(
- _launch_kwargs(
- self._headless,
- self._proxy,
- self._locale,
- tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
- self._useragent,
- self._real_chrome,
- self._stealth,
- self._hide_canvas,
- self._disable_webgl,
- self._timezone_id,
- tuple(self._extra_flags) if self._extra_flags else tuple(),
- )
- )
- self.launch_options["extra_http_headers"] = dict(self.launch_options["extra_http_headers"])
- self.launch_options["proxy"] = dict(self.launch_options["proxy"]) or None
- self.launch_options["user_data_dir"] = self._user_data_dir
- self.launch_options.update(cast(Dict, self._additional_args))
- self.context_options = dict()
- else:
- # while `context_options` is left to be used when cdp mode is enabled
- self.launch_options = dict()
- self.context_options = dict(
- _context_kwargs(
- self._proxy,
- self._locale,
- tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
- self._useragent,
- self._stealth,
- )
- )
- self.context_options["extra_http_headers"] = dict(self.context_options["extra_http_headers"])
- self.context_options["proxy"] = dict(self.context_options["proxy"]) or None
- self.context_options.update(cast(Dict, self._additional_args))
-
-
-class StealthySessionMixin:
- def __validate__(self, **params):
- if "__max_pages" in params:
- params["max_pages"] = params.pop("__max_pages")
-
- config: CamoufoxConfig = validate(params, model=CamoufoxConfig)
-
- self._max_pages = config.max_pages
- self._headless = config.headless
- self._block_images = config.block_images
- self._disable_resources = config.disable_resources
- self._block_webrtc = config.block_webrtc
- self._allow_webgl = config.allow_webgl
- self._network_idle = config.network_idle
- self._load_dom = config.load_dom
- self._humanize = config.humanize
- self._solve_cloudflare = config.solve_cloudflare
- self._wait = config.wait
- self._timeout = config.timeout
- self._page_action = config.page_action
- self._wait_selector = config.wait_selector
- self._init_script = config.init_script
- self._addons = config.addons
- self._wait_selector_state = config.wait_selector_state
- self._cookies = config.cookies
- self._google_search = config.google_search
- self._extra_headers = config.extra_headers
- self._proxy = config.proxy
- self._os_randomize = config.os_randomize
- self._disable_ads = config.disable_ads
- self._geoip = config.geoip
- self._selector_config = config.selector_config
- self._additional_args = config.additional_args
- self._user_data_dir = config.user_data_dir
- self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
- self.__initiate_browser_options__()
-
- def __initiate_browser_options__(self):
- """Initiate browser options."""
- self.launch_options: Dict[str, Any] = generate_launch_options(
- **{
- "geoip": self._geoip,
- "proxy": dict(self._proxy) if self._proxy and isinstance(self._proxy, tuple) else self._proxy,
- "addons": self._addons,
- "exclude_addons": [] if self._disable_ads else [DefaultAddons.UBO],
- "headless": self._headless,
- "humanize": True if self._solve_cloudflare else self._humanize,
- "i_know_what_im_doing": True, # To turn warnings off with the user configurations
- "allow_webgl": self._allow_webgl,
- "block_webrtc": self._block_webrtc,
- "block_images": self._block_images, # Careful! it makes some websites don't finish loading at all like stackoverflow even in headful mode.
- "os": None if self._os_randomize else get_os_name(),
- "user_data_dir": self._user_data_dir,
- "ff_version": __ff_version_str__,
- "firefox_user_prefs": {
- # This is what enabling `enable_cache` does internally, so we do it from here instead
- "browser.sessionhistory.max_entries": 10,
- "browser.sessionhistory.max_total_viewers": -1,
- "browser.cache.memory.enable": True,
- "browser.cache.disk_cache_ssl": True,
- "browser.cache.disk.smart_size.enabled": True,
- },
- **cast(Dict, self._additional_args),
+ def __generate_options__(self, extra_flags: Tuple | None = None) -> None:
+ config = cast(PlaywrightConfig, getattr(self, "_config", None))
+ self._context_options.update(
+ {
+ "proxy": config.proxy,
+ "locale": config.locale,
+ "timezone_id": config.timezone_id,
+ "extra_http_headers": config.extra_headers,
}
)
+ # The default useragent in the headful is always correct now in the current versions of Playwright
+ if config.useragent:
+ self._context_options["user_agent"] = config.useragent
+ elif not config.useragent and config.headless:
+ self._context_options["user_agent"] = (
+ __default_chrome_useragent__ if config.real_chrome else __default_useragent__
+ )
+
+ if not config.cdp_url:
+ self._launch_options |= self._context_options
+ self._context_options = {}
+ flags = self._launch_options["args"]
+ if config.extra_flags or extra_flags:
+ flags = list(set(flags + (config.extra_flags or extra_flags)))
+
+ self._launch_options.update(
+ {
+ "args": flags,
+ "headless": config.headless,
+ "user_data_dir": config.user_data_dir,
+ "channel": "chrome" if config.real_chrome else "chromium",
+ }
+ )
+
+ if config.additional_args:
+ self._launch_options.update(config.additional_args)
+ else:
+ # while `context_options` is left to be used when cdp mode is enabled
+ self._launch_options = dict()
+ if config.additional_args:
+ self._context_options.update(config.additional_args)
+
+
+class DynamicSessionMixin(BaseSessionMixin):
+ def __validate__(self, **params):
+ self._config = self.__validate_routine__(params, model=PlaywrightConfig)
+ self.__generate_options__()
+
+
+class StealthySessionMixin(BaseSessionMixin):
+ def __validate__(self, **params):
+ self._config: StealthConfig = self.__validate_routine__(params, model=StealthConfig)
+ self._context_options.update(
+ {
+ "is_mobile": False,
+ "has_touch": False,
+ # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
+ "service_workers": "allow",
+ "ignore_https_errors": True,
+ "screen": {"width": 1920, "height": 1080},
+ "viewport": {"width": 1920, "height": 1080},
+ "permissions": ["geolocation", "notifications"],
+ }
+ )
+ self.__generate_stealth_options()
+
+ def __generate_stealth_options(self) -> None:
+ flags = tuple()
+ if not self._config.cdp_url:
+ flags = DEFAULT_FLAGS + DEFAULT_STEALTH_FLAGS
+
+ if self._config.block_webrtc:
+ flags += (
+ "--webrtc-ip-handling-policy=disable_non_proxied_udp",
+ "--force-webrtc-ip-handling-policy", # Ensures the policy is enforced
+ )
+ if not self._config.allow_webgl:
+ flags += (
+ "--disable-webgl",
+ "--disable-webgl-image-chromium",
+ "--disable-webgl2",
+ )
+ if self._config.hide_canvas:
+ flags += ("--fingerprinting-canvas-image-data-noise",)
+
+ super(StealthySessionMixin, self).__generate_options__(flags)
@staticmethod
def _detect_cloudflare(page_content: str) -> str | None:
diff --git a/scrapling/engines/_browsers/_config_tools.py b/scrapling/engines/_browsers/_config_tools.py
index eaf28d6..2d7691b 100644
--- a/scrapling/engines/_browsers/_config_tools.py
+++ b/scrapling/engines/_browsers/_config_tools.py
@@ -1,15 +1,10 @@
from functools import lru_cache
-from scrapling.core._types import Tuple
-from scrapling.engines.constants import (
- DEFAULT_STEALTH_FLAGS,
- HARMFUL_DEFAULT_ARGS,
- DEFAULT_FLAGS,
-)
from scrapling.engines.toolbelt.navigation import js_bypass_path
from scrapling.engines.toolbelt.fingerprints import generate_headers
__default_useragent__ = generate_headers(browser_mode=True).get("User-Agent")
+__default_chrome_useragent__ = generate_headers(browser_mode="chrome").get("User-Agent")
@lru_cache(1)
@@ -41,99 +36,3 @@ def _compiled_stealth_scripts():
with open(script_path, "r") as f:
scripts.append(f.read())
return tuple(scripts)
-
-
-@lru_cache(2, typed=True)
-def _set_flags(hide_canvas, disable_webgl): # pragma: no cover
- """Returns the flags that will be used while launching the browser if stealth mode is enabled"""
- flags = DEFAULT_STEALTH_FLAGS
- if hide_canvas:
- flags += ("--fingerprinting-canvas-image-data-noise",)
- if disable_webgl:
- flags += (
- "--disable-webgl",
- "--disable-webgl-image-chromium",
- "--disable-webgl2",
- )
-
- return flags
-
-
-@lru_cache(2, typed=True)
-def _launch_kwargs(
- headless,
- proxy: Tuple,
- locale,
- extra_headers,
- useragent,
- real_chrome,
- stealth,
- hide_canvas,
- disable_webgl,
- timezone_id,
- extra_flags: Tuple,
-) -> Tuple:
- """Creates the arguments we will use while launching playwright's browser"""
- base_args = DEFAULT_FLAGS
- if extra_flags:
- base_args = base_args + extra_flags
-
- launch_kwargs = {
- "locale": locale,
- "timezone_id": timezone_id or None,
- "headless": headless,
- "args": base_args,
- "color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
- "proxy": proxy or tuple(),
- "device_scale_factor": 2,
- "ignore_default_args": HARMFUL_DEFAULT_ARGS,
- "channel": "chrome" if real_chrome else "chromium",
- "extra_http_headers": extra_headers or tuple(),
- "user_agent": useragent or __default_useragent__,
- }
- if stealth:
- stealth_args = base_args + _set_flags(hide_canvas, disable_webgl)
- launch_kwargs.update(
- {
- "args": stealth_args,
- "chromium_sandbox": True,
- "is_mobile": False,
- "has_touch": False,
- # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
- "service_workers": "allow",
- "ignore_https_errors": True,
- "screen": {"width": 1920, "height": 1080},
- "viewport": {"width": 1920, "height": 1080},
- "permissions": ["geolocation", "notifications"],
- }
- )
-
- return tuple(launch_kwargs.items())
-
-
-@lru_cache(2, typed=True)
-def _context_kwargs(proxy, locale, extra_headers, useragent, stealth) -> Tuple:
- """Creates the arguments for the browser context"""
- context_kwargs = {
- "proxy": proxy or tuple(),
- "locale": locale,
- "color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
- "device_scale_factor": 2,
- "extra_http_headers": extra_headers or tuple(),
- "user_agent": useragent or __default_useragent__,
- }
- if stealth:
- context_kwargs.update(
- {
- "is_mobile": False,
- "has_touch": False,
- # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
- "service_workers": "allow",
- "ignore_https_errors": True,
- "screen": {"width": 1920, "height": 1080},
- "viewport": {"width": 1920, "height": 1080},
- "permissions": ["geolocation", "notifications"],
- }
- )
-
- return tuple(context_kwargs.items())
diff --git a/scrapling/engines/_browsers/_controllers.py b/scrapling/engines/_browsers/_controllers.py
index 9e48675..ff21438 100644
--- a/scrapling/engines/_browsers/_controllers.py
+++ b/scrapling/engines/_browsers/_controllers.py
@@ -9,8 +9,6 @@ from playwright.async_api import (
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
-from patchright.sync_api import sync_playwright as sync_patchright
-from patchright.async_api import async_playwright as async_patchright
from scrapling.core.utils import log
from scrapling.core._types import Unpack, TYPE_CHECKING
@@ -21,53 +19,27 @@ from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
-class DynamicSession(DynamicSessionMixin, SyncSession):
+class DynamicSession(SyncSession, DynamicSessionMixin):
"""A Browser session manager with page pooling."""
__slots__ = (
- "_max_pages",
- "_headless",
- "_hide_canvas",
- "_disable_webgl",
- "_real_chrome",
- "_stealth",
- "_google_search",
- "_proxy",
- "_locale",
- "_extra_headers",
- "_useragent",
- "_timeout",
- "_cookies",
- "_disable_resources",
- "_network_idle",
- "_load_dom",
- "_wait_selector",
- "_init_script",
- "_wait_selector_state",
- "_wait",
- "playwright",
- "browser",
- "context",
+ "_config",
+ "_context_options",
+ "_launch_options",
+ "max_pages",
"page_pool",
+ "_max_wait_for_page",
+ "playwright",
+ "context",
"_closed",
- "_selector_config",
- "_page_action",
- "launch_options",
- "context_options",
- "_cdp_url",
- "_headers_keys",
- "_extra_flags",
- "_additional_args",
- "_user_data_dir",
)
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -76,13 +48,11 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- :param timezone_id: Set the timezone for the browser if wanted.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
@@ -94,27 +64,24 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- sync_context = sync_patchright if self._stealth else sync_playwright
+ self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- self.playwright: Playwright = sync_context().start() # pyright: ignore [reportAttributeAccessIssue]
-
- if self._cdp_url: # pragma: no cover
- self.context = self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url).new_context(
- **self.context_options
- )
+ if self._config.cdp_url: # pragma: no cover
+ browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context = browser.new_context(**self._context_options)
else:
- self.context = self.playwright.chromium.launch_persistent_context(**self.launch_options)
+ self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
- if self._init_script: # pragma: no cover
- self.context.add_init_script(path=self._init_script)
+ if self._config.init_script: # pragma: no cover
+ self.context.add_init_script(path=self._config.init_script)
- if self._cookies: # pragma: no cover
- self.context.add_cookies(self._cookies)
+ if self._config.cookies: # pragma: no cover
+ self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -122,24 +89,21 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
-
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -193,16 +157,15 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
raise e
-class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
+class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
"""An async Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory."""
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -212,13 +175,11 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- :param timezone_id: Set the timezone for the browser if wanted.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -230,28 +191,26 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- async_context = async_patchright if self._stealth else async_playwright
+ self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- self.playwright: AsyncPlaywright = await async_context().start() # pyright: ignore [reportAttributeAccessIssue]
-
- if self._cdp_url:
- browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url)
- self.context: AsyncBrowserContext = await browser.new_context(**self.context_options)
+ if self._config.cdp_url:
+ browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
else:
self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
- **self.launch_options
+ **self._launch_options
)
- if self._init_script: # pragma: no cover
- await self.context.add_init_script(path=self._init_script)
+ if self._config.init_script: # pragma: no cover
+ await self.context.add_init_script(path=self._config.init_script)
- if self._cookies:
- await self.context.add_cookies(self._cookies) # pyright: ignore
+ if self._config.cookies:
+ await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
@@ -259,20 +218,18 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
diff --git a/scrapling/engines/_browsers/_camoufox.py b/scrapling/engines/_browsers/_stealth.py
similarity index 62%
rename from scrapling/engines/_browsers/_camoufox.py
rename to scrapling/engines/_browsers/_stealth.py
index e3cbc67..912d38e 100644
--- a/scrapling/engines/_browsers/_camoufox.py
+++ b/scrapling/engines/_browsers/_stealth.py
@@ -2,117 +2,101 @@ from random import randint
from re import compile as re_compile
from playwright.sync_api import (
- Page,
Locator,
- sync_playwright,
+ Page,
+ Playwright,
)
from playwright.async_api import (
- async_playwright,
Page as async_Page,
Locator as AsyncLocator,
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
+from patchright.sync_api import sync_playwright
+from patchright.async_api import async_playwright
from scrapling.core.utils import log
-from ._types import CamoufoxSession, CamoufoxFetchParams
-from scrapling.core._types import Any, Unpack, TYPE_CHECKING
+from scrapling.core._types import Any, Unpack
+from ._config_tools import _compiled_stealth_scripts
+from ._types import StealthSession, StealthFetchParams
from ._base import SyncSession, AsyncSession, StealthySessionMixin
-from ._validators import validate_fetch as _validate, CamoufoxConfig
+from ._validators import validate_fetch as _validate, StealthConfig
from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
__CF_PATTERN__ = re_compile("challenges.cloudflare.com/cdn-cgi/challenge-platform/.*")
-class StealthySession(StealthySessionMixin, SyncSession):
- """A Stealthy session manager with page pooling."""
+class StealthySession(SyncSession, StealthySessionMixin):
+ """A Stealthy Browser session manager with page pooling."""
__slots__ = (
- "_max_pages",
- "_headless",
- "_block_images",
- "_disable_resources",
- "_block_webrtc",
- "_allow_webgl",
- "_network_idle",
- "_load_dom",
- "_humanize",
- "_solve_cloudflare",
- "_wait",
- "_timeout",
- "_page_action",
- "_wait_selector",
- "_init_script",
- "_addons",
- "_wait_selector_state",
- "_cookies",
- "_google_search",
- "_extra_headers",
- "_proxy",
- "_os_randomize",
- "_disable_ads",
- "_geoip",
- "_selector_config",
- "_additional_args",
- "playwright",
- "browser",
- "context",
+ "_config",
+ "_context_options",
+ "_launch_options",
+ "max_pages",
"page_pool",
+ "_max_wait_for_page",
+ "playwright",
+ "context",
"_closed",
- "launch_options",
- "_headers_keys",
- "_user_data_dir",
)
- def __init__(self, **kwargs: Unpack[CamoufoxSession]):
- """A Browser session manager with page pooling
+ def __init__(self, **kwargs: Unpack[StealthSession]):
+ """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
- :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- self.playwright = sync_playwright().start()
- self.context = self.playwright.firefox.launch_persistent_context(**self.launch_options)
+ self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- if self._init_script: # pragma: no cover
- self.context.add_init_script(path=self._init_script)
+ if self._config.cdp_url: # pragma: no cover
+ browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context = browser.new_context(**self._context_options)
+ else:
+ self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
- if self._cookies: # pragma: no cover
- self.context.add_cookies(self._cookies)
+ for script in _compiled_stealth_scripts():
+ self.context.add_init_script(script=script)
+
+ if self._config.init_script: # pragma: no cover
+ self.context.add_init_script(path=self._config.init_script)
+
+ if self._config.cookies: # pragma: no cover
+ self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -148,22 +132,27 @@ class StealthySession(StealthySessionMixin, SyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
- self._wait_for_page_stability(iframe, True, True)
+ self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not iframe.frame_element().is_visible():
# Double-checking that the iframe is loaded
page.wait_for_timeout(500)
+
outer_box: Any = iframe.frame_element().bounding_box()
if not iframe or not outer_box:
+ if "Just a moment... " not in (ResponseFactory._get_page_content(page)):
+ log.info("Cloudflare captcha is solved")
+ return
+
outer_box: Any = page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
- page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
+ page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -182,29 +171,26 @@ class StealthySession(StealthySessionMixin, SyncSession):
log.info("Cloudflare captcha is solved")
return
- def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
+ def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
- params = _validate(kwargs, self, CamoufoxConfig)
-
+ params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -233,7 +219,7 @@ class StealthySession(StealthySessionMixin, SyncSession):
if params.page_action:
try:
_ = params.page_action(page_info.page)
- except Exception as e:
+ except Exception as e: # pragma: no cover
log.error(f"Error executing page_action: {e}")
if params.wait_selector:
@@ -242,10 +228,12 @@ class StealthySession(StealthySessionMixin, SyncSession):
waiter.first.wait_for(state=params.wait_selector_state)
# Wait again after waiting for the selector, helpful with protections like Cloudflare
self._wait_for_page_stability(page_info.page, params.load_dom, params.network_idle)
- except Exception as e:
+ except Exception as e: # pragma: no cover
log.error(f"Error waiting for selector {params.wait_selector}: {e}")
page_info.page.wait_for_timeout(params.wait)
+
+ # Create response object
response = ResponseFactory.from_playwright_response(
page_info.page, first_response, final_response[0], params.selector_config
)
@@ -256,72 +244,78 @@ class StealthySession(StealthySessionMixin, SyncSession):
return response
- except Exception as e: # pragma: no cover
+ except Exception as e:
page_info.mark_error()
raise e
-class AsyncStealthySession(StealthySessionMixin, AsyncSession):
- """A Stealthy session manager with page pooling."""
+class AsyncStealthySession(AsyncSession, StealthySessionMixin):
+ """An async Stealthy Browser session manager with page pooling."""
- def __init__(self, **kwargs: Unpack[CamoufoxSession]):
- """A Browser session manager with page pooling
+ def __init__(self, **kwargs: Unpack[StealthSession]):
+ """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
- :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param max_pages: The maximum number of tabs to be opened at the same time. It will be used in rotation through a PagePool.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- self.playwright: AsyncPlaywright = await async_playwright().start()
- self.context: AsyncBrowserContext = await self.playwright.firefox.launch_persistent_context(
- **self.launch_options
- )
+ self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- if self._init_script: # pragma: no cover
- await self.context.add_init_script(path=self._init_script)
+ if self._config.cdp_url:
+ browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
+ else:
+ self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
+ **self._launch_options
+ )
- if self._cookies:
- await self.context.add_cookies(self._cookies) # pyright: ignore [reportArgumentType]
+ for script in _compiled_stealth_scripts():
+ await self.context.add_init_script(script=script)
+
+ if self._config.init_script: # pragma: no cover
+ await self.context.add_init_script(path=self._config.init_script)
+
+ if self._config.cookies:
+ await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
- async def _cloudflare_solver(self, page: async_Page): # pragma: no cover
- """Solve the cloudflare challenge displayed on the playwright page passed. The async version
+ async def _cloudflare_solver(self, page: async_Page) -> None: # pragma: no cover
+ """Solve the cloudflare challenge displayed on the playwright page passed
- :param page: The async targeted page
+ :param page: The targeted page
:return:
"""
await self._wait_for_networkidle(page, timeout=5000)
@@ -331,7 +325,7 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
return
else:
log.info(f'The turnstile version discovered is "{challenge_type}"')
- if challenge_type == "non-interactive": # pragma: no cover
+ if challenge_type == "non-interactive":
while "Just a moment... " in (await ResponseFactory._get_async_page_content(page)):
log.info("Waiting for Cloudflare wait page to disappear.")
await page.wait_for_timeout(1000)
@@ -350,22 +344,27 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
- await self._wait_for_page_stability(iframe, True, True)
+ await self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not await (await iframe.frame_element()).is_visible():
# Double-checking that the iframe is loaded
await page.wait_for_timeout(500)
+
outer_box: Any = await (await iframe.frame_element()).bounding_box()
if not iframe or not outer_box:
+ if "Just a moment... " not in (await ResponseFactory._get_async_page_content(page)):
+ log.info("Cloudflare captcha is solved")
+ return
+
outer_box: Any = await page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
- await page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
+ await page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
await self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -377,35 +376,33 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
await page.wait_for_timeout(100)
attempts += 1
if challenge_type != "embedded":
- await page.locator(box_selector).wait_for(state="detached")
+ await page.locator(box_selector).last.wait_for(state="detached")
await page.locator(".zone-name-title").wait_for(state="hidden")
await self._wait_for_page_stability(page, True, False)
log.info("Cloudflare captcha is solved")
return
- async def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
+ async def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
- params = _validate(kwargs, self, CamoufoxConfig)
+ params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -418,10 +415,6 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
final_response = [None]
handle_response = self._create_response_handler(page_info, final_response)
- if TYPE_CHECKING:
- if not isinstance(page_info.page, async_Page):
- raise TypeError
-
try:
# Navigate to URL and wait for a specified state
page_info.page.on("response", handle_response)
@@ -461,9 +454,8 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
# Close the page to free up resources
await page_info.page.close()
self.page_pool.pages.remove(page_info)
-
return response
- except Exception as e:
+ except Exception as e: # pragma: no cover
page_info.mark_error()
raise e
diff --git a/scrapling/engines/_browsers/_types.py b/scrapling/engines/_browsers/_types.py
index d1edca5..cb8d760 100644
--- a/scrapling/engines/_browsers/_types.py
+++ b/scrapling/engines/_browsers/_types.py
@@ -11,9 +11,10 @@ from scrapling.core._types import (
Mapping,
Optional,
Callable,
- Iterable,
+ Sequence,
TypedDict,
TypeAlias,
+ SetCookieParam,
SelectorWaitStates,
TYPE_CHECKING,
)
@@ -53,7 +54,7 @@ if TYPE_CHECKING: # pragma: no cover
json: Optional[Dict | List]
# Types for browser session
- class BrowserSession(TypedDict, total=False):
+ class PlaywrightSession(TypedDict, total=False):
max_pages: int
headless: bool
disable_resources: bool
@@ -61,9 +62,10 @@ if TYPE_CHECKING: # pragma: no cover
load_dom: bool
wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates
- cookies: Optional[Iterable[Dict]]
+ cookies: Sequence[SetCookieParam] | None
google_search: bool
wait: int | float
+ timezone_id: str | None
page_action: Optional[Callable]
proxy: Optional[str | Dict[str, str] | Tuple]
extra_headers: Optional[Dict[str, str]]
@@ -72,42 +74,32 @@ if TYPE_CHECKING: # pragma: no cover
user_data_dir: str
selector_config: Optional[Dict]
additional_args: Optional[Dict]
-
- class PlaywrightSession(BrowserSession, total=False):
- cdp_url: Optional[str]
- hide_canvas: bool
- disable_webgl: bool
+ locale: Optional[str]
real_chrome: bool
- stealth: bool
- locale: str
+ cdp_url: Optional[str]
useragent: Optional[str]
extra_flags: Optional[List[str]]
class PlaywrightFetchParams(TypedDict, total=False):
+ load_dom: bool
+ wait: int | float
+ network_idle: bool
google_search: bool
timeout: int | float
- wait: int | float
- page_action: Optional[Callable]
- extra_headers: Optional[Dict[str, str]]
disable_resources: bool
wait_selector: Optional[str]
- wait_selector_state: SelectorWaitStates
- network_idle: bool
- load_dom: bool
+ page_action: Optional[Callable]
selector_config: Optional[Dict]
+ extra_headers: Optional[Dict[str, str]]
+ wait_selector_state: SelectorWaitStates
- class CamoufoxSession(BrowserSession, total=False):
- block_images: bool
- block_webrtc: bool
+ class StealthSession(PlaywrightSession, total=False):
allow_webgl: bool
- humanize: bool | float
+ hide_canvas: bool
+ block_webrtc: bool
solve_cloudflare: bool
- addons: Optional[List[str]]
- os_randomize: bool
- disable_ads: bool
- geoip: bool
- class CamoufoxFetchParams(PlaywrightFetchParams, total=False):
+ class StealthFetchParams(PlaywrightFetchParams, total=False):
solve_cloudflare: bool
else: # pragma: no cover
@@ -116,5 +108,5 @@ else: # pragma: no cover
DataRequestParams = TypedDict
PlaywrightSession = TypedDict
PlaywrightFetchParams = TypedDict
- CamoufoxSession = TypedDict
- CamoufoxFetchParams = TypedDict
+ StealthSession = TypedDict
+ StealthFetchParams = TypedDict
diff --git a/scrapling/engines/_browsers/_validators.py b/scrapling/engines/_browsers/_validators.py
index f8b9875..82270ef 100644
--- a/scrapling/engines/_browsers/_validators.py
+++ b/scrapling/engines/_browsers/_validators.py
@@ -13,12 +13,13 @@ from scrapling.core._types import (
Tuple,
Optional,
Callable,
- Iterable,
- SelectorWaitStates,
+ Sequence,
overload,
+ SetCookieParam,
+ SelectorWaitStates,
)
from scrapling.engines.toolbelt.navigation import construct_proxy_dict
-from scrapling.engines._browsers._types import PlaywrightFetchParams, CamoufoxFetchParams
+from scrapling.engines._browsers._types import PlaywrightFetchParams, StealthFetchParams
# Custom validators for msgspec
@@ -35,15 +36,6 @@ def _is_invalid_file_path(value: str) -> bool | str: # pragma: no cover
return False
-def _validate_addon_path(value: str) -> None: # pragma: no cover
- """Fast addon path validation"""
- path = Path(value)
- if not path.exists():
- raise FileNotFoundError(f"Addon path not found: {value}")
- if not path.is_dir():
- raise ValueError(f"Addon path must be a directory of the extracted addon: {value}")
-
-
@lru_cache(2)
def _is_invalid_cdp_url(cdp_url: str) -> bool | str:
"""Fast CDP URL validation"""
@@ -65,39 +57,36 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
"""Configuration struct for validation"""
max_pages: PagesCount = 1
- cdp_url: Optional[str] = None
headless: bool = True
- google_search: bool = True
- hide_canvas: bool = False
- disable_webgl: bool = False
- real_chrome: bool = False
- stealth: bool = False
- wait: Seconds = 0
- page_action: Optional[Callable] = None
- proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
- locale: str = "en-US"
- extra_headers: Optional[Dict[str, str]] = None
- useragent: Optional[str] = None
- timeout: Seconds = 30000
- init_script: Optional[str] = None
disable_resources: bool = False
- wait_selector: Optional[str] = None
- cookies: Optional[Iterable[Dict]] = None
network_idle: bool = False
load_dom: bool = True
+ wait_selector: Optional[str] = None
wait_selector_state: SelectorWaitStates = "attached"
+ cookies: Sequence[SetCookieParam] | None = []
+ google_search: bool = True
+ wait: Seconds = 0
+ timezone_id: str | None = ""
+ page_action: Optional[Callable] = None
+ proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
+ extra_headers: Optional[Dict[str, str]] = None
+ timeout: Seconds = 30000
+ init_script: Optional[str] = None
user_data_dir: str = ""
- timezone_id: str = ""
- extra_flags: Optional[List[str]] = None
selector_config: Optional[Dict] = {}
additional_args: Optional[Dict] = {}
+ locale: str | None = None
+ real_chrome: bool = False
+ cdp_url: Optional[str] = None
+ useragent: Optional[str] = None
+ extra_flags: Optional[List[str]] = None
def __post_init__(self): # pragma: no cover
"""Custom validation after msgspec validation"""
if self.page_action and not callable(self.page_action):
raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
if self.proxy:
- self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
+ self.proxy = construct_proxy_dict(self.proxy)
if self.cdp_url:
cdp_msg = _is_invalid_cdp_url(self.cdp_url)
if cdp_msg:
@@ -118,64 +107,18 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
raise ValueError(validation_msg)
-class CamoufoxConfig(Struct, kw_only=True, frozen=False, weakref=True):
- """Configuration struct for validation"""
-
- max_pages: PagesCount = 1
- headless: bool = True # noqa: F821
- block_images: bool = False
- disable_resources: bool = False
- block_webrtc: bool = False
+class StealthConfig(PlaywrightConfig, kw_only=True, frozen=False, weakref=True):
allow_webgl: bool = True
- network_idle: bool = False
- load_dom: bool = True
- humanize: bool | float = True
+ hide_canvas: bool = False
+ block_webrtc: bool = False
solve_cloudflare: bool = False
- wait: Seconds = 0
- timeout: Seconds = 30000
- init_script: Optional[str] = None
- page_action: Optional[Callable] = None
- wait_selector: Optional[str] = None
- addons: Optional[List[str]] = None
- wait_selector_state: SelectorWaitStates = "attached"
- cookies: Optional[Iterable[Dict]] = None
- google_search: bool = True
- extra_headers: Optional[Dict[str, str]] = None
- proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
- os_randomize: bool = False
- disable_ads: bool = False
- geoip: bool = False
- user_data_dir: str = ""
- selector_config: Optional[Dict] = {}
- additional_args: Optional[Dict] = {}
def __post_init__(self):
"""Custom validation after msgspec validation"""
- if self.page_action and not callable(self.page_action):
- raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
- if self.proxy:
- self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
-
- if self.addons:
- for addon in self.addons:
- _validate_addon_path(addon)
- else:
- self.addons = []
-
- if self.init_script is not None:
- validation_msg = _is_invalid_file_path(self.init_script)
- if validation_msg:
- raise ValueError(validation_msg)
-
- if not self.cookies:
- self.cookies = []
+ super(StealthConfig, self).__post_init__()
# Cloudflare timeout adjustment
if self.solve_cloudflare and self.timeout < 60_000:
self.timeout = 60_000
- if not self.selector_config:
- self.selector_config = {}
- if not self.additional_args:
- self.additional_args = {}
@dataclass
@@ -197,9 +140,9 @@ class _fetch_params:
def validate_fetch(
- method_kwargs: Dict | PlaywrightFetchParams | CamoufoxFetchParams,
+ method_kwargs: Dict | PlaywrightFetchParams | StealthFetchParams,
session: Any,
- model: type[PlaywrightConfig] | type[CamoufoxConfig],
+ model: type[PlaywrightConfig] | type[StealthConfig],
) -> _fetch_params: # pragma: no cover
result = {}
overrides = {}
@@ -210,21 +153,20 @@ def validate_fetch(
for key in fetch_param_fields:
if key in method_kwargs:
overrides[key] = method_kwargs[key]
- else:
- # Check for underscore-prefixed attribute (private)
- attr_name = f"_{key}"
- if hasattr(session, attr_name):
- result[key] = getattr(session, attr_name)
+ elif hasattr(session, "_config") and hasattr(session._config, key):
+ result[key] = getattr(session._config, key)
if overrides:
validated_config = validate(overrides, model)
- # Extract only the fields that _fetch_params needs from validated_config
+ # Extract ONLY the fields that were actually overridden (not all fields)
+ # This prevents validated defaults from overwriting session config values
validated_dict = {
- f.name: getattr(validated_config, f.name)
- for f in fields(_fetch_params)
- if hasattr(validated_config, f.name)
+ field: getattr(validated_config, field) for field in overrides.keys() if hasattr(validated_config, field)
}
- validated_dict.setdefault("solve_cloudflare", False)
+
+ # Preserve solve_cloudflare if the user explicitly provided it, even if the model doesn't have it
+ if "solve_cloudflare" in overrides:
+ validated_dict["solve_cloudflare"] = overrides["solve_cloudflare"]
# Start with session defaults, then overwrite with validated overrides
result.update(validated_dict)
@@ -238,7 +180,7 @@ def validate_fetch(
# Cache default values for each model to reduce validation overhead
models_default_values = {}
-for _model in (CamoufoxConfig, PlaywrightConfig):
+for _model in (StealthConfig, PlaywrightConfig):
_defaults = {}
if hasattr(_model, "__struct_defaults__") and hasattr(_model, "__struct_fields__"):
for field_name, default_value in zip(_model.__struct_fields__, _model.__struct_defaults__): # type: ignore
@@ -256,14 +198,14 @@ def _filter_defaults(params: Dict, model: str) -> Dict:
@overload
-def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
+def validate(params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
@overload
-def validate(params: Dict, model: type[CamoufoxConfig]) -> CamoufoxConfig: ...
+def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
-def validate(params: Dict, model: type[PlaywrightConfig] | type[CamoufoxConfig]) -> PlaywrightConfig | CamoufoxConfig:
+def validate(params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]) -> PlaywrightConfig | StealthConfig:
try:
# Filter out params with the default values (no need to validate them) to speed up validation
filtered = _filter_defaults(params, model.__name__)
diff --git a/scrapling/engines/constants.py b/scrapling/engines/constants.py
index df12ee3..db8618c 100644
--- a/scrapling/engines/constants.py
+++ b/scrapling/engines/constants.py
@@ -74,7 +74,6 @@ DEFAULT_STEALTH_FLAGS = (
"--disable-domain-reliability",
"--disable-threaded-animation",
"--disable-threaded-scrolling",
- # '--disable-reading-from-canvas', # For Firefox
"--enable-simple-cache-backend",
"--disable-background-networking",
"--enable-surface-synchronization",
@@ -94,9 +93,7 @@ DEFAULT_STEALTH_FLAGS = (
"--autoplay-policy=no-user-gesture-required",
"--disable-offer-store-unmasked-wallet-cards",
"--disable-blink-features=AutomationControlled",
- "--webrtc-ip-handling-policy=disable_non_proxied_udp",
"--disable-component-extensions-with-background-pages",
- "--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
"--enable-features=NetworkService,NetworkServiceInProcess,TrustTokens,TrustTokensAlwaysAllowIssuance",
"--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4",
"--disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process,TranslateUI,BlinkGenPropertyTrees",
diff --git a/scrapling/engines/toolbelt/fingerprints.py b/scrapling/engines/toolbelt/fingerprints.py
index c006ec2..2ddbbe5 100644
--- a/scrapling/engines/toolbelt/fingerprints.py
+++ b/scrapling/engines/toolbelt/fingerprints.py
@@ -13,19 +13,32 @@ from scrapling.core._types import Dict, Literal, Tuple
__OS_NAME__ = platform_system()
OSName = Literal["linux", "macos", "windows"]
+# Current versions hardcoded for now (Playwright doesn't allow to know the version of a browser without launching it)
+chromium_version = 141
+chrome_version = 143
@lru_cache(10, typed=True)
-def generate_convincing_referer(url: str) -> str:
+def generate_convincing_referer(url: str) -> str | None:
"""Takes the domain from the URL without the subdomain/suffix and make it look like you were searching Google for this website
>>> generate_convincing_referer('https://www.somewebsite.com/blah')
'https://www.google.com/search?q=somewebsite'
:param url: The URL you are about to fetch.
- :return: Google's search URL of the domain name
+ :return: Google's search URL of the domain name, or None for localhost/IP addresses
"""
- website_name = extract(url).domain
+ extracted = extract(url)
+ website_name = extracted.domain
+
+ # Skip generating referer for localhost, IP addresses, or when there's no valid domain
+ if not website_name or not extracted.suffix or website_name in ("localhost", "127.0.0.1", "::1"):
+ return None
+
+ # Check if it's an IP address (simple check for IPv4)
+ if all(part.isdigit() for part in website_name.split(".") if part):
+ return None
+
return f"https://www.google.com/search?q={website_name}"
@@ -46,7 +59,7 @@ def get_os_name() -> OSName | Tuple:
return SUPPORTED_OPERATING_SYSTEMS
-def generate_headers(browser_mode: bool = False) -> Dict:
+def generate_headers(browser_mode: bool | str = False) -> Dict:
"""Generate real browser-like headers using browserforge's generator
:param browser_mode: If enabled, the headers created are used for playwright, so it has to match everything
@@ -55,13 +68,14 @@ def generate_headers(browser_mode: bool = False) -> Dict:
# In the browser mode, we don't care about anything other than matching the OS and the browser type with the browser we are using,
# So we don't raise any inconsistency red flags while websites fingerprinting us
os_name = get_os_name()
- browsers = [Browser(name="chrome", min_version=130)]
+ ver = chrome_version if browser_mode and browser_mode == "chrome" else chromium_version
+ browsers = [Browser(name="chrome", min_version=ver, max_version=ver)]
if not browser_mode:
os_name = ("windows", "macos", "linux")
browsers.extend(
[
- Browser(name="firefox", min_version=130),
- Browser(name="edge", min_version=130),
+ Browser(name="firefox", min_version=142),
+ Browser(name="edge", min_version=140),
]
)
return HeaderGenerator(browser=browsers, os=os_name, device="desktop").generate()
diff --git a/scrapling/engines/toolbelt/navigation.py b/scrapling/engines/toolbelt/navigation.py
index 959aa5f..1b44f5a 100644
--- a/scrapling/engines/toolbelt/navigation.py
+++ b/scrapling/engines/toolbelt/navigation.py
@@ -11,7 +11,7 @@ from msgspec import Struct, structs, convert, ValidationError
from playwright.sync_api import Route
from scrapling.core.utils import log
-from scrapling.core._types import Dict, Tuple, overload, Literal
+from scrapling.core._types import Dict, Tuple
from scrapling.engines.constants import DEFAULT_DISABLED_RESOURCES
__BYPASSES_DIR__ = Path(__file__).parent / "bypasses"
@@ -49,20 +49,11 @@ async def async_intercept_route(route: async_Route):
await route.continue_()
-@overload
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[True]) -> Tuple: ...
-
-
-@overload
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[False] = False) -> Dict: ...
-
-
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: bool = False) -> Dict | Tuple:
+def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple) -> Dict:
"""Validate a proxy and return it in the acceptable format for Playwright
Reference: https://playwright.dev/python/docs/network#http-proxy
:param proxy_string: A string or a dictionary representation of the proxy.
- :param as_tuple: Return the proxy dictionary as a tuple to be cachable
:return:
"""
if isinstance(proxy_string, str):
@@ -78,7 +69,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
}
if proxy.port:
result["server"] += f":{proxy.port}"
- return tuple(result.items()) if as_tuple else result
+ return result
except ValueError:
# Urllib will say that one of the parameters above can't be casted to the correct type like `int` for port etc...
raise ValueError("The proxy argument's string is in invalid format!")
@@ -87,7 +78,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
try:
validated = convert(proxy_string, ProxyDict)
result_dict = structs.asdict(validated)
- return tuple(result_dict.items()) if as_tuple else result_dict
+ return result_dict
except ValidationError as e:
raise TypeError(f"Invalid proxy dictionary: {e}")
diff --git a/scrapling/fetchers/__init__.py b/scrapling/fetchers/__init__.py
index c135273..e2e5866 100644
--- a/scrapling/fetchers/__init__.py
+++ b/scrapling/fetchers/__init__.py
@@ -3,7 +3,7 @@ from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from scrapling.fetchers.requests import Fetcher, AsyncFetcher, FetcherSession
from scrapling.fetchers.chrome import DynamicFetcher, DynamicSession, AsyncDynamicSession
- from scrapling.fetchers.firefox import StealthyFetcher, StealthySession, AsyncStealthySession
+ from scrapling.fetchers.stealth_chrome import StealthyFetcher, StealthySession, AsyncStealthySession
# Lazy import mapping
@@ -14,9 +14,9 @@ _LAZY_IMPORTS = {
"DynamicFetcher": ("scrapling.fetchers.chrome", "DynamicFetcher"),
"DynamicSession": ("scrapling.fetchers.chrome", "DynamicSession"),
"AsyncDynamicSession": ("scrapling.fetchers.chrome", "AsyncDynamicSession"),
- "StealthyFetcher": ("scrapling.fetchers.firefox", "StealthyFetcher"),
- "StealthySession": ("scrapling.fetchers.firefox", "StealthySession"),
- "AsyncStealthySession": ("scrapling.fetchers.firefox", "AsyncStealthySession"),
+ "StealthyFetcher": ("scrapling.fetchers.stealth_chrome", "StealthyFetcher"),
+ "StealthySession": ("scrapling.fetchers.stealth_chrome", "StealthySession"),
+ "AsyncStealthySession": ("scrapling.fetchers.stealth_chrome", "AsyncStealthySession"),
}
__all__ = [
diff --git a/scrapling/fetchers/chrome.py b/scrapling/fetchers/chrome.py
index a706826..0d50c19 100644
--- a/scrapling/fetchers/chrome.py
+++ b/scrapling/fetchers/chrome.py
@@ -5,51 +5,37 @@ from scrapling.engines._browsers._controllers import DynamicSession, AsyncDynami
class DynamicFetcher(BaseFetcher):
- """A `Fetcher` class type that provide many options, all of them are based on PlayWright.
-
- Using this Fetcher class, you can do requests with:
- - Vanilla Playwright without any modifications other than the ones you chose.
- - Stealthy Playwright with the stealth mode I wrote for it. It's still a work in progress, but it bypasses many online tests like bot.sannysoft.com
- Some of the things stealth mode does include:
- 1) Patches the CDP runtime fingerprint.
- 2) Mimics some of the real browsers' properties by injecting several JS files and using custom options.
- 3) Using custom flags on launch to hide Playwright even more and make it faster.
- 4) Generates real browser's headers of the same type and same user OS, then append it to the request.
- - Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
-
- > Note that these are the main options with PlayWright, but it can be mixed.
- """
+ """A `Fetcher` that provide many options to fetch/load websites' pages through chromium-based browsers."""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[PlaywrightSession]) -> Response:
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- - cookies: Set cookies for the next request.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - locale: Set the locale for the browser if wanted. The default value is `en-US`.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - extra_flags: A list of additional browser flags to pass to the browser on launch.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request.
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
@@ -68,31 +54,30 @@ class DynamicFetcher(BaseFetcher):
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- - cookies: Set cookies for the next request.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - locale: Set the locale for the browser if wanted. The default value is `en-US`.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - extra_flags: A list of additional browser flags to pass to the browser on launch.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request.
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
diff --git a/scrapling/fetchers/firefox.py b/scrapling/fetchers/firefox.py
deleted file mode 100644
index a9361d6..0000000
--- a/scrapling/fetchers/firefox.py
+++ /dev/null
@@ -1,102 +0,0 @@
-from scrapling.core._types import Unpack
-from scrapling.engines._browsers._types import CamoufoxSession
-from scrapling.engines.toolbelt.custom import BaseFetcher, Response
-from scrapling.engines._browsers._camoufox import StealthySession, AsyncStealthySession
-
-
-class StealthyFetcher(BaseFetcher):
- """A `Fetcher` class type that is a completely stealthy fetcher that uses a modified version of Firefox.
-
- It works as real browsers passing almost all online tests/protections based on Camoufox.
- Other added flavors include setting the faked OS fingerprints to match the user's OS, and the referer of every request is set as if this request came from Google's search of this URL's domain.
- """
-
- @classmethod
- def fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
- """
- Opens up a browser and do your request based on your chosen options below.
-
- :param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - block_images: Prevent the loading of images through Firefox preferences.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - block_webrtc: Blocks WebRTC entirely.
- - allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - addons: List of Firefox addons to use. Must be paths to extracted addons.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - cookies: Set cookies for the next request.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- - disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- - geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Camoufox as additional settings.
- :return: A `Response` object.
- """
- selector_config = kwargs.get("selector_config", {}) or kwargs.get(
- "custom_config", {}
- ) # Checking `custom_config` for backward compatibility
- if not isinstance(selector_config, dict):
- raise TypeError("Argument `selector_config` must be a dictionary.")
-
- kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
-
- with StealthySession(**kwargs) as engine:
- return engine.fetch(url)
-
- @classmethod
- async def async_fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
- """
- Opens up a browser and do your request based on your chosen options below.
-
- :param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - block_images: Prevent the loading of images through Firefox preferences.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - block_webrtc: Blocks WebRTC entirely.
- - allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - addons: List of Firefox addons to use. Must be paths to extracted addons.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - cookies: Set cookies for the next request.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- - disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- - geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Camoufox as additional settings.
- :return: A `Response` object.
- """
- selector_config = kwargs.get("selector_config", {}) or kwargs.get(
- "custom_config", {}
- ) # Checking `custom_config` for backward compatibility
- if not isinstance(selector_config, dict):
- raise TypeError("Argument `selector_config` must be a dictionary.")
-
- kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
-
- async with AsyncStealthySession(**kwargs) as engine:
- return await engine.fetch(url)
diff --git a/scrapling/fetchers/stealth_chrome.py b/scrapling/fetchers/stealth_chrome.py
new file mode 100644
index 0000000..58574d3
--- /dev/null
+++ b/scrapling/fetchers/stealth_chrome.py
@@ -0,0 +1,107 @@
+from scrapling.core._types import Unpack
+from scrapling.engines._browsers._types import StealthSession
+from scrapling.engines.toolbelt.custom import BaseFetcher, Response
+from scrapling.engines._browsers._stealth import StealthySession, AsyncStealthySession
+
+
+class StealthyFetcher(BaseFetcher):
+ """A `Fetcher` class type which is a completely stealthy built on top of Chromium.
+
+ It works as real browsers passing almost all online tests/protections with many customization options.
+ """
+
+ @classmethod
+ def fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
+ """
+ Opens up a browser and do your request based on your chosen options below.
+
+ :param url: Target url.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
+ :return: A `Response` object.
+ """
+ selector_config = kwargs.get("selector_config", {}) or kwargs.get(
+ "custom_config", {}
+ ) # Checking `custom_config` for backward compatibility
+ if not isinstance(selector_config, dict):
+ raise TypeError("Argument `selector_config` must be a dictionary.")
+
+ kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
+
+ with StealthySession(**kwargs) as engine:
+ return engine.fetch(url)
+
+ @classmethod
+ async def async_fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
+ """
+ Opens up a browser and do your request based on your chosen options below.
+
+ :param url: Target url.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
+ :return: A `Response` object.
+ """
+ selector_config = kwargs.get("selector_config", {}) or kwargs.get(
+ "custom_config", {}
+ ) # Checking `custom_config` for backward compatibility
+ if not isinstance(selector_config, dict):
+ raise TypeError("Argument `selector_config` must be a dictionary.")
+
+ kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
+
+ async with AsyncStealthySession(**kwargs) as engine:
+ return await engine.fetch(url)
diff --git a/setup.cfg b/setup.cfg
index beef70a..1a02999 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
[metadata]
name = scrapling
-version = 0.3.12
+version = 0.3.13
author = Karim Shoair
author_email = karim.shoair@pm.me
description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!
diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py
index 4a79c25..e2f86f3 100644
--- a/tests/cli/test_cli.py
+++ b/tests/cli/test_cli.py
@@ -156,7 +156,6 @@ class TestCLI:
html_url,
str(output_file),
'--headless',
- '--stealth',
'--timeout', '60000'
]
)
diff --git a/tests/fetchers/async/test_dynamic.py b/tests/fetchers/async/test_dynamic.py
index 0d46365..3fe38cc 100644
--- a/tests/fetchers/async/test_dynamic.py
+++ b/tests/fetchers/async/test_dynamic.py
@@ -53,10 +53,7 @@ class TestDynamicFetcherAsync:
@pytest.mark.parametrize(
"kwargs",
[
- {"disable_webgl": True, "hide_canvas": False},
- {"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
- {"stealth": True}, # causes issues with GitHub Actions
- {"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
+ {"real_chrome": True, "disable_resources": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
diff --git a/tests/fetchers/async/test_dynamic_session.py b/tests/fetchers/async/test_dynamic_session.py
index 234854c..c403017 100644
--- a/tests/fetchers/async/test_dynamic_session.py
+++ b/tests/fetchers/async/test_dynamic_session.py
@@ -71,7 +71,6 @@ class TestAsyncDynamicSession:
"""Test AsyncDynamicSession with various options"""
async with AsyncDynamicSession(
headless=False,
- stealth=True,
disable_resources=True,
extra_headers={"X-Test": "value"}
) as session:
diff --git a/tests/fetchers/async/test_requests_session.py b/tests/fetchers/async/test_requests_session.py
index c4e1355..3846e69 100644
--- a/tests/fetchers/async/test_requests_session.py
+++ b/tests/fetchers/async/test_requests_session.py
@@ -1,4 +1,3 @@
-import pytest
from scrapling.engines.static import AsyncFetcherClient
diff --git a/tests/fetchers/async/test_camoufox.py b/tests/fetchers/async/test_stealth.py
similarity index 81%
rename from tests/fetchers/async/test_camoufox.py
rename to tests/fetchers/async/test_stealth.py
index e8bbd3f..bba39d8 100644
--- a/tests/fetchers/async/test_camoufox.py
+++ b/tests/fetchers/async/test_stealth.py
@@ -1,4 +1,3 @@
-from playwright._impl._errors import TimeoutError
import pytest
import pytest_httpbin
@@ -30,8 +29,8 @@ class TestStealthyFetcher:
async def test_basic_fetch(self, fetcher, urls):
"""Test doing a basic fetch request with multiple statuses"""
assert (await fetcher.async_fetch(urls["status_200"])).status == 200
- assert (await fetcher.async_fetch(urls["status_404"])).status == 404
- assert (await fetcher.async_fetch(urls["status_501"])).status == 501
+ # assert (await fetcher.async_fetch(urls["status_404"])).status == 404
+ # assert (await fetcher.async_fetch(urls["status_501"])).status == 501
async def test_cookies_loading(self, fetcher, urls):
"""Test if cookies are set after the request"""
@@ -55,10 +54,9 @@ class TestStealthyFetcher:
@pytest.mark.parametrize(
"kwargs",
[
- {"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
- {"block_webrtc": False, "allow_webgl": True, "block_images": True},
+ {"block_webrtc": True, "allow_webgl": True},
+ {"block_webrtc": False, "allow_webgl": True},
{"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
- {"block_images": True, "disable_resources": True, },
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
@@ -67,11 +65,8 @@ class TestStealthyFetcher:
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"google_search": True,
"extra_headers": {"ayo": ""},
- "os_randomize": True,
- "disable_ads": True,
- # "geoip": True,
"selector_config": {"keep_comments": False, "keep_cdata": False},
- "additional_args": {"window": (1920, 1080)},
+ "additional_args": {},
},
],
)
diff --git a/tests/fetchers/async/test_camoufox_session.py b/tests/fetchers/async/test_stealth_session.py
similarity index 96%
rename from tests/fetchers/async/test_camoufox_session.py
rename to tests/fetchers/async/test_stealth_session.py
index 05f7953..140742d 100644
--- a/tests/fetchers/async/test_camoufox_session.py
+++ b/tests/fetchers/async/test_stealth_session.py
@@ -72,9 +72,8 @@ class TestAsyncStealthySession:
"""Test AsyncStealthySession with various options"""
async with AsyncStealthySession(
max_pages=1,
- block_images=True,
- disable_ads=True,
- humanize=True
+ block_webrtc=True,
+ allow_webgl=True
) as session:
response = await session.fetch(urls["html"])
assert response.status == 200
diff --git a/tests/fetchers/sync/test_camoufox.py b/tests/fetchers/sync/test_camoufox.py
deleted file mode 100644
index 269238f..0000000
--- a/tests/fetchers/sync/test_camoufox.py
+++ /dev/null
@@ -1,79 +0,0 @@
-import pytest
-import pytest_httpbin
-
-from scrapling import StealthyFetcher
-StealthyFetcher.adaptive = True
-
-
-@pytest_httpbin.use_class_based_httpbin
-class TestStealthyFetcher:
- @pytest.fixture(scope="class")
- def fetcher(self):
- """Fixture to create a StealthyFetcher instance for the entire test class"""
- return StealthyFetcher
-
- @pytest.fixture(autouse=True)
- def setup_urls(self, httpbin):
- """Fixture to set up URLs for testing"""
- self.status_200 = f"{httpbin.url}/status/200"
- self.status_404 = f"{httpbin.url}/status/404"
- self.status_501 = f"{httpbin.url}/status/501"
- self.basic_url = f"{httpbin.url}/get"
- self.html_url = f"{httpbin.url}/html"
- self.delayed_url = f"{httpbin.url}/delay/10" # 10 Seconds delay response
- self.cookies_url = f"{httpbin.url}/cookies/set/test/value"
-
- def test_basic_fetch(self, fetcher):
- """Test doing a basic fetch request with multiple statuses"""
- assert fetcher.fetch(self.status_200).status == 200
- assert fetcher.fetch(self.status_404).status == 404
- assert fetcher.fetch(self.status_501).status == 501
-
- def test_cookies_loading(self, fetcher):
- """Test if cookies are set after the request"""
- response = fetcher.fetch(self.cookies_url)
- cookies = {response.cookies[0]['name']: response.cookies[0]['value']}
- assert cookies == {"test": "value"}
-
- def test_automation(self, fetcher):
- """Test if automation breaks the code or not"""
-
- def scroll_page(page):
- page.mouse.wheel(10, 0)
- page.mouse.move(100, 400)
- page.mouse.up()
- return page
-
- assert fetcher.fetch(self.html_url, page_action=scroll_page).status == 200
-
- @pytest.mark.parametrize(
- "kwargs",
- [
- {"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
- {"block_webrtc": False, "allow_webgl": True, "block_images": True},
- {"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
- {"block_images": True, "disable_resources": True, },
- {"wait_selector": "h1", "wait_selector_state": "attached"},
- {"wait_selector": "h1", "wait_selector_state": "visible"},
- {
- "network_idle": True,
- "wait": 10,
- "timeout": 30_000,
- "cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
- "google_search": True,
- "extra_headers": {"ayo": ""},
- "os_randomize": True,
- "disable_ads": True,
- # "geoip": True,
- "selector_config": {"keep_comments": False, "keep_cdata": False},
- "additional_args": {"window": (1920, 1080)},
- },
- ],
- )
- def test_properties(self, fetcher, kwargs):
- """Test if different arguments break the code or not"""
- response = fetcher.fetch(
- self.html_url,
- **kwargs
- )
- assert response.status == 200
diff --git a/tests/fetchers/sync/test_dynamic.py b/tests/fetchers/sync/test_dynamic.py
index e044e33..cac8a73 100644
--- a/tests/fetchers/sync/test_dynamic.py
+++ b/tests/fetchers/sync/test_dynamic.py
@@ -51,10 +51,7 @@ class TestDynamicFetcher:
@pytest.mark.parametrize(
"kwargs",
[
- {"disable_webgl": True, "hide_canvas": False},
- {"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
- {"stealth": True}, # causes issues with GitHub Actions
- {"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
+ {"disable_resources": True, "real_chrome": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
diff --git a/tests/fetchers/sync/test_camoufox_session.py b/tests/fetchers/sync/test_stealth_session.py
similarity index 86%
rename from tests/fetchers/sync/test_camoufox_session.py
rename to tests/fetchers/sync/test_stealth_session.py
index e282d98..e740b01 100644
--- a/tests/fetchers/sync/test_camoufox_session.py
+++ b/tests/fetchers/sync/test_stealth_session.py
@@ -2,11 +2,11 @@ import re
import pytest
import pytest_httpbin
-from scrapling.engines._browsers._camoufox import StealthySession, __CF_PATTERN__
+from scrapling.engines._browsers._stealth import StealthySession, __CF_PATTERN__
-class TestCamoufoxConstants:
- """Test Camoufox constants and patterns"""
+class TestStealthConstants:
+ """Test Stealth constants and patterns"""
def test_cf_pattern_regex(self):
"""Test __CF_PATTERN__ regex compilation"""
@@ -54,7 +54,6 @@ class TestStealthySession:
with StealthySession(
headless=True,
- block_images=True,
disable_resources=True,
solve_cloudflare=True,
wait=1000,
@@ -63,12 +62,11 @@ class TestStealthySession:
) as session:
assert session.max_pages == 1
- assert session._headless is True
- assert session._block_images is True
- assert session._disable_resources is True
- assert session._solve_cloudflare is True
- assert session._wait == 1000
- assert session._timeout == 60000
+ assert session._config.headless is True
+ assert session._config.disable_resources is True
+ assert session._config.solve_cloudflare is True
+ assert session._config.wait == 1000
+ assert session._config.timeout == 60000
assert session.context is not None
# Test Cloudflare detection
diff --git a/tests/fetchers/test_utils.py b/tests/fetchers/test_utils.py
index 0028902..fea3f3f 100644
--- a/tests/fetchers/test_utils.py
+++ b/tests/fetchers/test_utils.py
@@ -190,14 +190,6 @@ class TestConstructProxyDict:
}
assert result == expected
- def test_proxy_as_tuple(self):
- """Test returning proxy as a tuple"""
- result = construct_proxy_dict("http://proxy.example.com:8080", as_tuple=True)
-
- assert isinstance(result, tuple)
- result_dict = dict(result)
- assert result_dict["server"] == "http://proxy.example.com:8080"
-
def test_invalid_proxy_string(self):
"""Test invalid proxy string"""
with pytest.raises(ValueError):
diff --git a/tests/fetchers/test_validator.py b/tests/fetchers/test_validator.py
index 118554c..209fae0 100644
--- a/tests/fetchers/test_validator.py
+++ b/tests/fetchers/test_validator.py
@@ -1,8 +1,8 @@
import pytest
from scrapling.engines._browsers._validators import (
validate,
+ StealthConfig,
PlaywrightConfig,
- CamoufoxConfig
)
@@ -23,7 +23,7 @@ class TestValidators:
assert config.max_pages == 2
assert config.headless is True
assert config.timeout == 30000
- assert isinstance(config.proxy, tuple) # Should be converted to tuple
+ assert isinstance(config.proxy, dict)
def test_playwright_config_invalid_max_pages(self):
"""Test PlaywrightConfig with invalid max_pages"""
@@ -51,8 +51,8 @@ class TestValidators:
with pytest.raises(TypeError):
validate(params, PlaywrightConfig)
- def test_camoufox_config_valid(self):
- """Test valid CamoufoxConfig"""
+ def test_stealth_config_valid(self):
+ """Test valid StealthConfig"""
params = {
"max_pages": 1,
"headless": True,
@@ -60,20 +60,20 @@ class TestValidators:
"timeout": 30000
}
- config = validate(params, CamoufoxConfig)
+ config = validate(params, StealthConfig)
assert config.max_pages == 1
assert config.headless is True
assert config.solve_cloudflare is False
assert config.timeout == 30000
- def test_camoufox_config_cloudflare_timeout(self):
- """Test CamoufoxConfig timeout adjustment for Cloudflare"""
+ def test_stealth_config_cloudflare_timeout(self):
+ """Test StealthConfig timeout adjustment for Cloudflare"""
params = {
"solve_cloudflare": True,
"timeout": 10000 # Less than the required 60,000
}
- config = validate(params, CamoufoxConfig)
+ config = validate(params, StealthConfig)
assert config.timeout == 60000 # Should be increased
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 52f672c..7e9a957 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,7 +1,6 @@
pytest>=2.8.0,<9
pytest-cov
playwright
-camoufox
werkzeug<3.0.0
pytest-httpbin==2.1.0
pytest-asyncio
diff --git a/tox.ini b/tox.ini
index 2a24c42..c2a30ae 100644
--- a/tox.ini
+++ b/tox.ini
@@ -10,9 +10,8 @@ envlist = pre-commit,py{310,311,312,313}
usedevelop = True
changedir = tests
deps =
- playwright>=1.56.0
- patchright>=1.56.0
- camoufox>=0.4.11
+ playwright==1.56.0
+ patchright==1.56.0
-r{toxinidir}/tests/requirements.txt
extras = ai,shell
commands =
In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.
@@ -310,7 +310,7 @@ In the [Selector](#selector) class, all methods/properties that should return a Let's see what [Selectors](#selectors) class adds to the table with that out of the way. ### Properties -Apart from the normal operations on Python lists, such as iteration and slicing, etc. +Apart from the standard operations on Python lists, such as iteration and slicing. You can do the following: @@ -334,7 +334,7 @@ Execute CSS and XPath selectors directly on the [Selector](#selector) instances
@@ -374,7 +374,7 @@ If you are too lazy like me and want to know the number of [Selector](#selector)
```python
page.css('.product_pod').length
```
-instead of this
+which is equivalent to
```python
len(page.css('.product_pod'))
```
@@ -389,15 +389,15 @@ Of course, TextHandler provides extra methods and properties that standard Pytho
### Usage
First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well.
-First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they will accept the same arguments as well.
+First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they accept the same arguments.
-- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
+- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but, as you probably figured out from the name, it returns only the first result as a `TextHandler` instance.
Also, it takes other helpful arguments, which are:
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- - **clean_match**: It's disabled by default. This causes the method to ignore all whitespace and consecutive spaces while matching.
- - **case_sensitive**: It's enabled by default. As the name implies, disabling it will cause the regex to ignore the case of letters while compiling.
+ - **clean_match**: It's disabled by default. This causes the method to ignore all whitespace, including consecutive spaces, while matching.
+ - **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
```python
@@ -471,14 +471,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
```python
>>> page.css_first('div::text').json()
```
- You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have direct text content at all.
+ You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.
In this case, the `get_all_text` method comes to the rescue, so you can do something like that ```python >>> page.css_first('div').get_all_text(ignore_tags=[]).json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} ``` I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.
- Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example: + Another related behavior to be aware of occurs when using any fetcher, which we will explain later. If you have a JSON response like this example: ```python >>> page = Selector("""{"some_key": "some_value"}""") ``` @@ -493,14 +493,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods {'some_key': 'some_value'} ``` You might wonder how this happened, given that the `html` tag doesn't contain direct text.
- Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.
This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions. + Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it uses the `get_all_text` method directly.
- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance ```python >>> TextHandler('\n wonderful idea, \reh?').clean() 'wonderful idea, eh?' ``` -Also, you can pass `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters. +Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters. - Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists ```python @@ -518,10 +518,10 @@ Other methods and properties will be added over time, but remember that this cla ## TextHandlers You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods. -The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing new needs to be explained here, but new methods will be added over time. +The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. Nothing new needs to be explained here, but new methods will be added over time. ## AttributesHandler -This is a read-only version of Python's standard dictionary, or `dict`, that is used solely to store the attributes of each element or each [Selector](#selector) instance. +This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element or [Selector](#selector) instance. ```python >>> print(page.find('script').attrib) {'id': 'page-data', 'type': 'application/json'} @@ -534,7 +534,7 @@ It currently adds two extra simple methods: - The `search_values` method - In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will require some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item. + In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item. A simple example would be ```python @@ -555,13 +555,13 @@ It currently adds two extra simple methods:,
]
```
- All these elements have 'product' as a value for the attribute `class`.
+ All these elements have 'product' as the value for the `class` attribute.
Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
- The `json_string` property
- This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error
+ This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error.
```python
>>>page.find('script').attrib.json_string
diff --git a/docs/parsing/selection.md b/docs/parsing/selection.md
index a262f82..0b3877d 100644
--- a/docs/parsing/selection.md
+++ b/docs/parsing/selection.md
@@ -22,7 +22,7 @@ Scrapling implements CSS3 selectors as described in the [W3C specification](http
Also, Scrapling implements some non-standard pseudo-elements like:
-* To select text nodes, use ``::text``
+* To select text nodes, use ``::text``.
* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
@@ -36,19 +36,19 @@ In short, it is the same situation as CSS Selectors; if you come from Scrapy/Par
To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above, and `xpath_first` is faster.
-> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
+> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here, as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
### Selectors examples
Let's see some shared examples of using CSS and XPath Selectors.
-Select all elements with the class `product`
+Select all elements with the class `product`.
```python
products = page.css('.product')
products = page.xpath('//*[@class="product"]')
```
Note: The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class**
-Select the first element with the class `product`
+Select the first element with the class `product`.
```python
product = page.css_first('.product')
product = page.xpath_first('//*[@class="product"]')
@@ -73,12 +73,12 @@ Get the `href` attribute of the first element with the `a` tag name
link = page.css_first('a::attr(href)')
link = page.xpath_first('//a/@href')
```
-Select the text of the first element with the `h1` tag name, which contains 'Phone', and under an element with class 'product'
+Select the text of the first element with the `h1` tag name, which contains `Phone`, and under an element with class `product`.
```python
title = page.css_first('.product h1:contains("Phone")::text')
title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
```
-You can nest and chain selectors as you want, given that it returns results
+You can nest and chain selectors as you want, given that they return results
```python
page.css_first('.product').css_first('h1:contains("Phone")::text')
page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
@@ -104,7 +104,7 @@ Scrapling provides the ability to select elements based on their direct text con
What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
-With `find_by_text`, you will pass the text as the first argument; with the `find_by_regex` method, the regex pattern is the first argument. Both methods share the following arguments:
+With `find_by_text`, you pass the text as the first argument; with `find_by_regex`, the regex pattern is the first argument. Both methods share the following arguments:
* **first_match**: If `True` (the default), the method used will return the first result it finds.
* **case_sensitive**: If `True`, the case of the letters will be considered.
@@ -117,7 +117,7 @@ By default, Scrapling searches for the exact matching of the text/pattern you pa
Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
### Finding Similar Elements
-One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
+One of the most remarkable new features Scrapling puts on the table is the ability to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
So, how does it work?
@@ -129,8 +129,8 @@ Imagine a scenario where you found a product by its title, for example, and you
That's a lot of talking, I know, but I had to go deep. I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
-* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the values of the attributes of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what the other arguments do first.
-* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change a lot between elements, making them unreliable.
+* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the tag attributes of both elements should be at least 20% similar. If you want to turn off this check (basically Step 3), you can set this attribute to 0, but I recommend you read what the other arguments do first.
+* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change significantly across elements, making them unreliable.
* **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends.
Now, let's check out the examples below.
@@ -148,7 +148,7 @@ Find the first element whose text fully matches this text
>>> page.find_by_text('Tipping the Velvet')
```
-Combining it with `page.urljoin` to return the full URL from the relative `href`
+Combining it with `page.urljoin` to return the full URL from the relative `href`.
```python
>>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html'
@@ -174,7 +174,7 @@ Get all elements that contain the word `the` (Partial matching)
'Mesaerion: The Best Science ...',
"It's Only the Himalayas"]
```
-The search is case-insensitive, so those results have `The`, not only the lowercase one `the`; let's limit the search to the elements with `the` only.
+The search is case-insensitive, so those results include `The`, not just the lowercase `the`; let's limit the search to elements with `the` only.
```python
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
@@ -332,14 +332,14 @@ def extract_reviews(page):
]
```
## Filters-based searching
-This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier to learn for newcomers to Web Scraping than writing selectors.
+This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier for newcomers to Web Scraping to learn than writing selectors.
-Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
+Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can accept multiple filters and return all elements on the pages where all these filters apply.
To be more specific:
* Any string passed is considered a tag name.
-* Any iterable passed, like List/Tuple/Set, is considered an iterable of tag names.
+* Any iterable passed, like List/Tuple/Set, will be considered as an iterable of tag names.
* Any dictionary is considered a mapping of HTML element(s), attribute names, and attribute values.
* Any regex patterns passed are used to filter elements by content, like the `find_by_regex` method
* Any functions passed are used to filter elements
@@ -357,7 +357,7 @@ It filters all elements in the current page/element in the following order:
Notes:
1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on.
-2. The order in which you pass the arguments doesn't matter. The only order taken into consideration is the order explained above.
+2. The order in which you pass the arguments doesn't matter. The only order considered is the one explained above.
Check examples to clear any confusion :)
@@ -509,4 +509,4 @@ We will have a deep look at it while explaining the [TextHandler](main_classes.m
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']
```
-And so on. You get the idea. We will explain this in more detail on the next page while explaining the [TextHandler](main_classes.md#texthandler) class.
\ No newline at end of file
+And so on. You get the idea. We will explain this in more detail on the next page, along with the [TextHandler](main_classes.md#texthandler) class.
\ No newline at end of file
diff --git a/docs/requirements.txt b/docs/requirements.txt
index e996a0b..c2c5d54 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1,5 +1,8 @@
-mkdocs-material
-mkdocstrings
-mkdocstrings-python
-mkdocs-material[imaging]
-black
\ No newline at end of file
+mkdocstrings>=1.0.0
+mkdocstrings-python>=2.0.1
+griffe-inherited-docstrings
+griffe-runtime-objects
+griffe-sphinx
+mkdocs-material[imaging]>=9.7.1
+black>=25.12.0
+pngquant
\ No newline at end of file
diff --git a/docs/tutorials/migrating_from_beautifulsoup.md b/docs/tutorials/migrating_from_beautifulsoup.md
index d1b7019..e5474bb 100644
--- a/docs/tutorials/migrating_from_beautifulsoup.md
+++ b/docs/tutorials/migrating_from_beautifulsoup.md
@@ -1,10 +1,10 @@
# Migrating from BeautifulSoup to Scrapling
-If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is incredibly faster, provides the same parsing capabilities, adds more parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
+If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is much faster, provides the same parsing capabilities as BS, adds additional parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
-Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to accomplish a specific task using BeautifulSoup and the corresponding method in Scrapling.
+Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to achieve a specific task using BeautifulSoup and the corresponding method in Scrapling.
-You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, but that's one of the reasons why BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short oneliner, there is no need to sacrifice performance to shorten that short line :)
+You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, which is one of the reasons BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short one-liner, there is no need to sacrifice performance to shorten that short line :)
| Task | BeautifulSoup Code | Scrapling Code |
@@ -56,7 +56,7 @@ Here's a simple example of scraping a web page to extract all the links using Be
import requests
from bs4 import BeautifulSoup
-url = 'http://example.com'
+url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
@@ -70,7 +70,7 @@ for link in links:
```python
from scrapling import Fetcher
-url = 'http://example.com'
+url = 'https://example.com'
page = Fetcher.get(url)
links = page.css('a::attr(href)')
@@ -78,12 +78,12 @@ for link in links:
print(link)
```
-As you can see, Scrapling simplifies the process by handling the fetching and parsing in a single step, making your code cleaner and more efficient.
+As you can see, Scrapling simplifies the process by combining fetching and parsing into a single step, making your code cleaner and more efficient.
**Additional Notes:**
- **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons.
-- **Element Types**: In BeautifulSoup, elements are `Tag` objects, while in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
+- **Element Types**: In BeautifulSoup, elements are `Tag` objects; in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.css_first()`). To avoid errors, check for `None` before accessing properties.
- **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can help remove extra whitespace, consecutive spaces, or unwanted characters. Please check out the documentation for the complete list.
diff --git a/docs/tutorials/replacing_ai.md b/docs/tutorials/replacing_ai.md
index 733e88b..200073c 100644
--- a/docs/tutorials/replacing_ai.md
+++ b/docs/tutorials/replacing_ai.md
@@ -26,13 +26,13 @@ How will you solve that manually? I'm referring to generic web scraping of vario
## AI to the rescue, but at a high cost
-Of course, the AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
+Of course, AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
This approach is, of course, beautiful. I love AI and find it very fascinating, especially Generative AI. You will probably spend a lot of time on prompt engineering and tweaking the prompts, but if that's cool with you, you will soon hit the real issue with using AI here.
Most websites have vast amounts of content per page, which you will need to pass to the AI somehow so it can do its magic. This will burn through tokens like fire in a haystack, quickly accumulating high costs.
-Unless money is irrelevant to you, you will try to find less expensive approaches, and that's why I made Scrapling :smile:
+Unless money is irrelevant to you, you will try to find less expensive approaches, and that's where Scrapling comes into play :smile:
## Scrapling got you covered
@@ -41,11 +41,11 @@ Scrapling can handle almost all issues you will face during Web Scraping, and th
### Solving issue T1: Rapidly changing website structures
That's why the [adaptive](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/) feature was made. You knew I would talk about it, and here we are :)
-While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties to find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
+While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties so you can find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
That's how the adaptive feature works: it stores everything unique about an element. When the website structure changes, it returns the element with the highest similarity score of the previous element.
-I have already explained that in more detail and with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
+I have already explained this in more detail, with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
### Solving issue T2: Unstable selectors
If you have been doing Web scraping for a long enough time, you have likely experienced this once. I'm referring to a website that employs poor design patterns, built on raw HTML without any IDs/classes, or uses random class names with nothing else to rely on, etc...
@@ -59,16 +59,16 @@ In these cases, standard selection methods with CSS/XPath selectors won't be opt
There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this.
### Solving issue T3: Increasingly complex anti-bot measures
-It's known that making an undetectable spider takes more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
+It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
-1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides many flexible options, like stealth mode suitable for small to medium protections and using your real browser.
-2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher utilizes our version of a modified Firefox browser, called [Camoufox](https://camoufox.com/stealth/), which nearly passes all known tests and incorporates additional tricks. **With v0.3, this fetcher can bypass Cloudflare for you automatically as well!**
+1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
+2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
-These two will be improved a lot with the upcoming updates, so stay tuned :)
+We keep improving these two with each update, so stay tuned :)
### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data
-This one is tough to handle, but it's possible with Scrapling's flexibility.
+This one is tough to handle, but Scrapling's flexibility makes it possible.
I talked with someone who uses AI to extract prices from different websites. He is only interested in prices and titles, so he uses AI to find the price for him.
@@ -94,7 +94,7 @@ It will be a bit boring, but it's definitely less expensive than AI.
This example illustrates the point I aim to convey here. Not every challenge will need AI to be solved, but sometimes you need to be creative, and that might save you a lot of money.
### Solving issue B3: Pagination variations
-This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the following updates :)
+This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :)
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea.
@@ -112,6 +112,6 @@ For a quick comparison.
This table is based on pricing from [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing)
## Conclusion
-While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit designed to tackle the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
+While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit for tackling the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects!
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
index ea2dd7a..c8b9709 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -29,6 +29,7 @@ theme:
- navigation.sections
- navigation.tracking
- navigation.instant
+ - navigation.instant.prefetch
- navigation.instant.progress
# - navigation.tabs
# - navigation.expand
@@ -41,6 +42,7 @@ theme:
- content.action.view
- content.action.edit
- content.code.copy
+ - content.code.select
- content.code.annotate
- content.code.annotation
@@ -103,10 +105,13 @@ markdown_extensions:
plugins:
- search
-# - social:
-# cards_layout_options:
-# background_color: "#1f1f1f"
-# font_family: Roboto
+ - privacy:
+ links: false
+ - optimize
+ - social:
+ cards_layout_options:
+ background_color: "#1f1f1f"
+ font_family: Roboto
- mkdocstrings:
handlers:
python:
@@ -122,11 +127,19 @@ plugins:
unwrap_annotated: true
filters:
- '!^_'
+ - "^__"
merge_init_into_class: true
docstring_section_style: spacy
signature_crossrefs: true
show_symbol_type_heading: true
show_symbol_type_toc: true
+ show_inheritance_diagram: true
+ modernize_annotations: true
+ extensions:
+ - griffe_runtime_objects
+ - griffe_sphinx
+ - griffe_inherited_docstrings:
+ merge: true
extra:
homepage: https://scrapling.readthedocs.io/en/latest/
diff --git a/pyproject.toml b/pyproject.toml
index 0dd663b..a5b6f5c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "scrapling"
# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
-version = "0.3.12"
+version = "0.3.13"
description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
readme = {file = "docs/README.md", content-type = "text/markdown"}
license = {file = "LICENSE"}
@@ -60,7 +60,7 @@ dependencies = [
"lxml>=6.0.2",
"cssselect>=1.3.0",
"orjson>=3.11.5",
- "tldextract>=5.3.0",
+ "tldextract>=5.3.1",
]
[project.optional-dependencies]
@@ -69,8 +69,7 @@ fetchers = [
"curl_cffi>=0.14.0",
"playwright==1.56.0",
"patchright==1.56.0",
- "camoufox>=0.4.11",
- "geoip2>=5.2.0",
+ "browserforge>=1.2.1",
"msgspec>=0.20.0",
]
ai = [
diff --git a/scrapling/__init__.py b/scrapling/__init__.py
index 23f917b..0eca2f2 100644
--- a/scrapling/__init__.py
+++ b/scrapling/__init__.py
@@ -1,5 +1,5 @@
__author__ = "Karim Shoair (karim.shoair@pm.me)"
-__version__ = "0.3.12"
+__version__ = "0.3.13"
__copyright__ = "Copyright (c) 2024 Karim Shoair"
from typing import Any, TYPE_CHECKING
diff --git a/scrapling/cli.py b/scrapling/cli.py
index 99a2c63..68470a5 100644
--- a/scrapling/cli.py
+++ b/scrapling/cli.py
@@ -125,14 +125,9 @@ def install(force): # pragma: no cover
"playwright",
"install-deps",
"chromium",
- "firefox",
],
"Playwright dependencies",
)
- __Execute(
- [python_executable, "-m", "camoufox", "fetch", "--browserforge"],
- "Camoufox browser and databases",
- )
# if no errors raised by the above commands, then we add the below file
__PACKAGE_DIR__.joinpath(".scrapling_dependencies_installed").touch()
else:
@@ -610,17 +605,11 @@ def delete(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
-@option("--locale", default="en-US", help="Browser locale (default: en-US)")
-@option("--stealth/--no-stealth", default=False, help="Enable stealth mode (default: False)")
+@option("--locale", default=None, help="Specify user locale. Defaults to the system default locale.")
@option(
- "--hide-canvas/--show-canvas",
+ "--real-chrome/--no-real-chrome",
default=False,
- help="Add noise to canvas operations (default: False)",
-)
-@option(
- "--disable-webgl/--enable-webgl",
- default=False,
- help="Disable WebGL support (default: False)",
+ help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
@@ -640,9 +629,7 @@ def fetch(
css_selector,
wait_selector,
locale,
- stealth,
- hide_canvas,
- disable_webgl,
+ real_chrome,
proxy,
extra_headers,
):
@@ -659,9 +646,7 @@ def fetch(
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser.
- :param stealth: Enables stealth mode.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -676,9 +661,7 @@ def fetch(
"network_idle": network_idle,
"timeout": timeout,
"locale": locale,
- "stealth": stealth,
- "hide_canvas": hide_canvas,
- "disable_webgl": disable_webgl,
+ "real_chrome": real_chrome,
}
if wait > 0:
@@ -703,11 +686,6 @@ def fetch(
default=True,
help="Run browser in headless mode (default: True)",
)
-@option(
- "--block-images/--allow-images",
- default=False,
- help="Block image loading (default: False)",
-)
@option(
"--disable-resources/--enable-resources",
default=False,
@@ -718,11 +696,6 @@ def fetch(
default=False,
help="Block WebRTC entirely (default: False)",
)
-@option(
- "--humanize/--no-humanize",
- default=False,
- help="Humanize cursor movement (default: False)",
-)
@option(
"--solve-cloudflare/--no-solve-cloudflare",
default=False,
@@ -735,9 +708,14 @@ def fetch(
help="Wait for network idle (default: False)",
)
@option(
- "--disable-ads/--allow-ads",
+ "--real-chrome/--no-real-chrome",
default=False,
- help="Install uBlock Origin addon (default: False)",
+ help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
+)
+@option(
+ "--hide-canvas/--show-canvas",
+ default=False,
+ help="Add noise to canvas operations (default: False)",
)
@option(
"--timeout",
@@ -757,11 +735,6 @@ def fetch(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
-@option(
- "--geoip/--no-geoip",
- default=False,
- help="Use IP geolocation for timezone/locale (default: False)",
-)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
"--extra-headers",
@@ -773,19 +746,17 @@ def stealthy_fetch(
url,
output_file,
headless,
- block_images,
disable_resources,
block_webrtc,
- humanize,
solve_cloudflare,
allow_webgl,
network_idle,
- disable_ads,
+ real_chrome,
+ hide_canvas,
timeout,
wait,
css_selector,
wait_selector,
- geoip,
proxy,
extra_headers,
):
@@ -795,19 +766,17 @@ def stealthy_fetch(
:param url: Target url.
:param output_file: Output file path (.md for Markdown, .html for HTML).
:param headless: Run the browser in headless/hidden, or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
:param disable_resources: Drop requests of unnecessary resources for a speed boost.
:param block_webrtc: Blocks WebRTC entirely.
- :param humanize: Humanize the cursor movement.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges.
:param allow_webgl: Allow WebGL (recommended to keep enabled).
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Install the uBlock Origin addon on the browser.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before returning.
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Automatically use IP's longitude, latitude, timezone, country, locale.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -818,16 +787,14 @@ def stealthy_fetch(
# Build request arguments
kwargs = {
"headless": headless,
- "block_images": block_images,
"disable_resources": disable_resources,
"block_webrtc": block_webrtc,
- "humanize": humanize,
"solve_cloudflare": solve_cloudflare,
"allow_webgl": allow_webgl,
"network_idle": network_idle,
- "disable_ads": disable_ads,
+ "real_chrome": real_chrome,
+ "hide_canvas": hide_canvas,
"timeout": timeout,
- "geoip": geoip,
}
if wait > 0:
diff --git a/scrapling/core/_shell_signatures.py b/scrapling/core/_shell_signatures.py
index 803e6d5..d9a42fc 100644
--- a/scrapling/core/_shell_signatures.py
+++ b/scrapling/core/_shell_signatures.py
@@ -1,9 +1,13 @@
from scrapling.core._types import (
- Dict,
Any,
+ Dict,
List,
Tuple,
+ Sequence,
+ Callable,
Optional,
+ SetCookieParam,
+ SelectorWaitStates,
)
# Parameter definitions for shell function signatures (defined once at module level)
@@ -31,57 +35,58 @@ _REQUESTS_PARAMS = {
_FETCH_PARAMS = {
"headless": bool,
- "google_search": bool,
- "hide_canvas": bool,
- "disable_webgl": bool,
- "real_chrome": bool,
- "stealth": bool,
- "wait": int | float,
- "page_action": Optional[Any],
- "proxy": Optional[str | Dict],
- "locale": str,
- "extra_headers": Optional[Dict[str, str]],
- "useragent": Optional[str],
- "cdp_url": Optional[str],
- "timeout": int | float,
"disable_resources": bool,
- "wait_selector": Optional[str],
- "init_script": Optional[str],
- "cookies": Optional[List[Dict]],
"network_idle": bool,
"load_dom": bool,
- "wait_selector_state": Any,
- "extra_flags": Optional[List[str]],
- "additional_args": Optional[Dict],
+ "wait_selector": Optional[str],
+ "wait_selector_state": SelectorWaitStates,
+ "cookies": Sequence[SetCookieParam],
+ "google_search": bool,
+ "wait": int | float,
+ "timezone_id": str | None,
+ "page_action": Optional[Callable],
+ "proxy": Optional[str | Dict[str, str] | Tuple],
+ "extra_headers": Optional[Dict[str, str]],
+ "timeout": int | float,
+ "init_script": Optional[str],
+ "user_data_dir": str,
"selector_config": Optional[Dict],
+ "additional_args": Optional[Dict],
+ "locale": Optional[str],
+ "real_chrome": bool,
+ "cdp_url": Optional[str],
+ "useragent": Optional[str],
+ "extra_flags": Optional[List[str]],
}
_STEALTHY_FETCH_PARAMS = {
"headless": bool,
- "block_images": bool,
"disable_resources": bool,
- "block_webrtc": bool,
- "allow_webgl": bool,
"network_idle": bool,
"load_dom": bool,
- "humanize": bool | float,
- "solve_cloudflare": bool,
- "wait": int | float,
- "timeout": int | float,
- "page_action": Optional[Any],
"wait_selector": Optional[str],
- "init_script": Optional[str],
- "addons": Optional[List[str]],
- "wait_selector_state": Any,
- "cookies": Optional[List[Dict]],
+ "wait_selector_state": SelectorWaitStates,
+ "cookies": Sequence[SetCookieParam],
"google_search": bool,
+ "wait": int | float,
+ "timezone_id": str | None,
+ "page_action": Optional[Callable],
+ "proxy": Optional[str | Dict[str, str] | Tuple],
"extra_headers": Optional[Dict[str, str]],
- "proxy": Optional[str | Dict],
- "os_randomize": bool,
- "disable_ads": bool,
- "geoip": bool,
+ "timeout": int | float,
+ "init_script": Optional[str],
+ "user_data_dir": str,
"selector_config": Optional[Dict],
"additional_args": Optional[Dict],
+ "locale": Optional[str],
+ "real_chrome": bool,
+ "cdp_url": Optional[str],
+ "useragent": Optional[str],
+ "extra_flags": Optional[List[str]],
+ "allow_webgl": bool,
+ "hide_canvas": bool,
+ "block_webrtc": bool,
+ "solve_cloudflare": bool,
}
# Mapping of function names to their parameter definitions
diff --git a/scrapling/core/_types.py b/scrapling/core/_types.py
index ac19a91..d7c1f8b 100644
--- a/scrapling/core/_types.py
+++ b/scrapling/core/_types.py
@@ -57,3 +57,17 @@ except ImportError: # pragma: no cover
from typing_extensions import Self # Backport
except ImportError:
Self = object
+
+
+# Copied from `playwright._impl._api_structures.SetCookieParam`
+class SetCookieParam(TypedDict, total=False):
+ name: str
+ value: str
+ url: Optional[str]
+ domain: Optional[str]
+ path: Optional[str]
+ expires: Optional[float]
+ httpOnly: Optional[bool]
+ secure: Optional[bool]
+ sameSite: Optional[Literal["Lax", "None", "Strict"]]
+ partitionKey: Optional[str]
diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py
index 4d5277a..4d5929b 100644
--- a/scrapling/core/ai.py
+++ b/scrapling/core/ai.py
@@ -17,13 +17,15 @@ from scrapling.fetchers import (
from scrapling.core._types import (
Optional,
Tuple,
- extraction_types,
Mapping,
Dict,
List,
Any,
- SelectorWaitStates,
Generator,
+ Sequence,
+ SetCookieParam,
+ extraction_types,
+ SelectorWaitStates,
)
@@ -213,20 +215,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
- hide_canvas: bool = False,
- disable_webgl: bool = False,
real_chrome: bool = False,
- stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
- locale: str = "en-US",
+ timezone_id: str | None = None,
+ locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
- cookies: Optional[List[Dict]] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> ResponseModel:
@@ -242,21 +242,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -269,15 +267,13 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
- stealth=stealth,
cdp_url=cdp_url,
headless=headless,
useragent=useragent,
- hide_canvas=hide_canvas,
+ timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
- disable_webgl=disable_webgl,
extra_headers=extra_headers,
google_search=google_search,
disable_resources=disable_resources,
@@ -301,20 +297,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
- hide_canvas: bool = False,
- disable_webgl: bool = False,
real_chrome: bool = False,
- stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
- locale: str = "en-US",
+ timezone_id: str | None = None,
+ locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
- cookies: Optional[List[Dict]] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> List[ResponseModel]:
@@ -330,21 +324,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -356,17 +348,15 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
- stealth=stealth,
cdp_url=cdp_url,
headless=headless,
max_pages=len(urls),
useragent=useragent,
- hide_canvas=hide_canvas,
+ timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
google_search=google_search,
- disable_webgl=disable_webgl,
extra_headers=extra_headers,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
@@ -393,29 +383,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
- block_images: bool = False,
+ google_search: bool = True,
+ real_chrome: bool = False,
+ wait: int | float = 0,
+ proxy: Optional[str | Dict[str, str]] = None,
+ timezone_id: str | None = None,
+ locale: str | None = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ useragent: Optional[str] = None,
+ hide_canvas: bool = False,
+ cdp_url: Optional[str] = None,
+ timeout: int | float = 30000,
disable_resources: bool = False,
+ wait_selector: Optional[str] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
+ network_idle: bool = False,
+ wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
- network_idle: bool = False,
- humanize: bool | float = True,
solve_cloudflare: bool = False,
- wait: int | float = 0,
- timeout: int | float = 30000,
- wait_selector: Optional[str] = None,
- addons: Optional[List[str]] = None,
- wait_selector_state: SelectorWaitStates = "attached",
- cookies: Optional[List[Dict]] = None,
- google_search: bool = True,
- extra_headers: Optional[Dict[str, str]] = None,
- proxy: Optional[str | Dict[str, str]] = None,
- os_randomize: bool = False,
- disable_ads: bool = False,
- geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> ResponseModel:
- """Use Scrapling's version of the Camoufox browser to fetch a URL and return a structured output of the result.
- Note: This is best suitable for high protection levels. It's slower than the other tools.
+ """Use the stealthy fetcher to fetch a URL and return a structured output of the result.
+ Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param url: The URL to request.
@@ -426,54 +416,52 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
page = await StealthyFetcher.async_fetch(
url,
wait=wait,
proxy=proxy,
- geoip=geoip,
- addons=addons,
+ locale=locale,
+ cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
- humanize=humanize,
+ useragent=useragent,
+ timezone_id=timezone_id,
+ real_chrome=real_chrome,
+ hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
- disable_ads=disable_ads,
network_idle=network_idle,
- block_images=block_images,
block_webrtc=block_webrtc,
- os_randomize=os_randomize,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
+ additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- additional_args=additional_args,
)
return _ContentTranslator(
Convertor._extract_content(
@@ -492,29 +480,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
- block_images: bool = False,
+ google_search: bool = True,
+ real_chrome: bool = False,
+ wait: int | float = 0,
+ proxy: Optional[str | Dict[str, str]] = None,
+ timezone_id: str | None = None,
+ locale: str | None = None,
+ extra_headers: Optional[Dict[str, str]] = None,
+ useragent: Optional[str] = None,
+ hide_canvas: bool = False,
+ cdp_url: Optional[str] = None,
+ timeout: int | float = 30000,
disable_resources: bool = False,
+ wait_selector: Optional[str] = None,
+ cookies: Sequence[SetCookieParam] | None = None,
+ network_idle: bool = False,
+ wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
- network_idle: bool = False,
- humanize: bool | float = True,
solve_cloudflare: bool = False,
- wait: int | float = 0,
- timeout: int | float = 30000,
- wait_selector: Optional[str] = None,
- addons: Optional[List[str]] = None,
- wait_selector_state: SelectorWaitStates = "attached",
- cookies: Optional[List[Dict]] = None,
- google_search: bool = True,
- extra_headers: Optional[Dict[str, str]] = None,
- proxy: Optional[str | Dict[str, str]] = None,
- os_randomize: bool = False,
- disable_ads: bool = False,
- geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> List[ResponseModel]:
- """Use Scrapling's version of the Camoufox browser to fetch a group of URLs at the same time, and for each page return a structured output of the result.
- Note: This is best suitable for high protection levels. It's slower than the other tools.
+ """Use the stealthy fetcher to fetch a group of URLs at the same time, and for each page return a structured output of the result.
+ Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param urls: A tuple of the URLs to request.
@@ -525,54 +513,51 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
async with AsyncStealthySession(
wait=wait,
proxy=proxy,
- geoip=geoip,
- addons=addons,
+ locale=locale,
+ cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
- humanize=humanize,
- max_pages=len(urls),
+ useragent=useragent,
+ timezone_id=timezone_id,
+ real_chrome=real_chrome,
+ hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
- disable_ads=disable_ads,
- block_images=block_images,
- block_webrtc=block_webrtc,
network_idle=network_idle,
- os_randomize=os_randomize,
+ block_webrtc=block_webrtc,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
+ additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
- additional_args=additional_args,
) as session:
tasks = [session.fetch(url) for url in urls]
responses = await gather(*tasks)
diff --git a/scrapling/core/translator.py b/scrapling/core/translator.py
index bb6d405..c34ed7d 100644
--- a/scrapling/core/translator.py
+++ b/scrapling/core/translator.py
@@ -130,5 +130,5 @@ translator = HTMLTranslator()
@lru_cache(maxsize=256)
def css_to_xpath(query: str) -> str:
- """Return translated XPath version of a given CSS query"""
+ """Return the translated XPath version of a given CSS query"""
return translator.css_to_xpath(query)
diff --git a/scrapling/engines/_browsers/_base.py b/scrapling/engines/_browsers/_base.py
index fdb298c..bb14b5f 100644
--- a/scrapling/engines/_browsers/_base.py
+++ b/scrapling/engines/_browsers/_base.py
@@ -1,7 +1,6 @@
from time import time
from asyncio import sleep as asyncio_sleep, Lock
-from camoufox import DefaultAddons
from playwright.sync_api._generated import Page
from playwright.sync_api import (
Frame,
@@ -17,18 +16,18 @@ from playwright.async_api import (
BrowserContext as AsyncBrowserContext,
)
from playwright._impl._errors import Error as PlaywrightError
-from camoufox.pkgman import installed_verstr as camoufox_version
-from camoufox.utils import launch_options as generate_launch_options
from ._page import PageInfo, PagePool
from scrapling.parser import Selector
-from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING
-from scrapling.engines.toolbelt.fingerprints import get_os_name
-from ._validators import validate, PlaywrightConfig, CamoufoxConfig
-from ._config_tools import _compiled_stealth_scripts, _launch_kwargs, _context_kwargs
+from ._validators import validate, PlaywrightConfig, StealthConfig
+from ._config_tools import __default_chrome_useragent__, __default_useragent__
from scrapling.engines.toolbelt.navigation import intercept_route, async_intercept_route
-
-__ff_version_str__ = camoufox_version().split(".", 1)[0]
+from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING, overload, Tuple
+from scrapling.engines.constants import (
+ DEFAULT_STEALTH_FLAGS,
+ HARMFUL_DEFAULT_ARGS,
+ DEFAULT_FLAGS,
+)
class SyncSession:
@@ -84,10 +83,6 @@ class SyncSession:
if disable_resources:
page.route("**/*", intercept_route)
- if getattr(self, "stealth", False):
- for script in _compiled_stealth_scripts():
- page.add_init_script(script=script)
-
page_info = self.page_pool.add_page(page)
page_info.mark_busy()
return page_info
@@ -202,10 +197,6 @@ class AsyncSession:
if disable_resources:
await page.route("**/*", async_intercept_route)
- if getattr(self, "stealth", False):
- for script in _compiled_stealth_scripts():
- await page.add_init_script(script=script)
-
return self.page_pool.add_page(page)
def get_pool_stats(self) -> Dict[str, int]:
@@ -251,151 +242,118 @@ class AsyncSession:
return handle_response
-class DynamicSessionMixin:
- def __validate__(self, **params):
+class BaseSessionMixin:
+ @overload
+ def __validate_routine__(self, params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
+
+ @overload
+ def __validate_routine__(self, params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
+
+ def __validate_routine__(
+ self, params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]
+ ) -> PlaywrightConfig | StealthConfig:
+ # Dark color scheme bypasses the 'prefersLightColor' check in creepjs
+ self._context_options: Dict[str, Any] = {"color_scheme": "dark", "device_scale_factor": 2}
+ self._launch_options: Dict[str, Any] = self._context_options | {
+ "args": DEFAULT_FLAGS,
+ "ignore_default_args": HARMFUL_DEFAULT_ARGS,
+ }
if "__max_pages" in params:
params["max_pages"] = params.pop("__max_pages")
- config = validate(params, model=PlaywrightConfig)
+ config = validate(params, model=model)
+ self._headers_keys = (
+ {header.lower() for header in config.extra_headers.keys()} if config.extra_headers else set()
+ )
- self._max_pages = config.max_pages
- self._headless = config.headless
- self._hide_canvas = config.hide_canvas
- self._disable_webgl = config.disable_webgl
- self._real_chrome = config.real_chrome
- self._stealth = config.stealth
- self._google_search = config.google_search
- self._wait = config.wait
- self._proxy = config.proxy
- self._locale = config.locale
- self._extra_headers = config.extra_headers
- self._useragent = config.useragent
- self._timeout = config.timeout
- self._cookies = config.cookies
- self._disable_resources = config.disable_resources
- self._cdp_url = config.cdp_url
- self._network_idle = config.network_idle
- self._load_dom = config.load_dom
- self._wait_selector = config.wait_selector
- self._init_script = config.init_script
- self._wait_selector_state = config.wait_selector_state
- self._extra_flags = config.extra_flags
- self._selector_config = config.selector_config
- self._timezone_id = config.timezone_id
- self._additional_args = config.additional_args
- self._page_action = config.page_action
- self._user_data_dir = config.user_data_dir
- self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
- self.__initiate_browser_options__()
+ return config
- def __initiate_browser_options__(self):
- if TYPE_CHECKING:
- assert isinstance(self._proxy, tuple)
-
- if not self._cdp_url:
- # `launch_options` is used with persistent context
- self.launch_options = dict(
- _launch_kwargs(
- self._headless,
- self._proxy,
- self._locale,
- tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
- self._useragent,
- self._real_chrome,
- self._stealth,
- self._hide_canvas,
- self._disable_webgl,
- self._timezone_id,
- tuple(self._extra_flags) if self._extra_flags else tuple(),
- )
- )
- self.launch_options["extra_http_headers"] = dict(self.launch_options["extra_http_headers"])
- self.launch_options["proxy"] = dict(self.launch_options["proxy"]) or None
- self.launch_options["user_data_dir"] = self._user_data_dir
- self.launch_options.update(cast(Dict, self._additional_args))
- self.context_options = dict()
- else:
- # while `context_options` is left to be used when cdp mode is enabled
- self.launch_options = dict()
- self.context_options = dict(
- _context_kwargs(
- self._proxy,
- self._locale,
- tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
- self._useragent,
- self._stealth,
- )
- )
- self.context_options["extra_http_headers"] = dict(self.context_options["extra_http_headers"])
- self.context_options["proxy"] = dict(self.context_options["proxy"]) or None
- self.context_options.update(cast(Dict, self._additional_args))
-
-
-class StealthySessionMixin:
- def __validate__(self, **params):
- if "__max_pages" in params:
- params["max_pages"] = params.pop("__max_pages")
-
- config: CamoufoxConfig = validate(params, model=CamoufoxConfig)
-
- self._max_pages = config.max_pages
- self._headless = config.headless
- self._block_images = config.block_images
- self._disable_resources = config.disable_resources
- self._block_webrtc = config.block_webrtc
- self._allow_webgl = config.allow_webgl
- self._network_idle = config.network_idle
- self._load_dom = config.load_dom
- self._humanize = config.humanize
- self._solve_cloudflare = config.solve_cloudflare
- self._wait = config.wait
- self._timeout = config.timeout
- self._page_action = config.page_action
- self._wait_selector = config.wait_selector
- self._init_script = config.init_script
- self._addons = config.addons
- self._wait_selector_state = config.wait_selector_state
- self._cookies = config.cookies
- self._google_search = config.google_search
- self._extra_headers = config.extra_headers
- self._proxy = config.proxy
- self._os_randomize = config.os_randomize
- self._disable_ads = config.disable_ads
- self._geoip = config.geoip
- self._selector_config = config.selector_config
- self._additional_args = config.additional_args
- self._user_data_dir = config.user_data_dir
- self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
- self.__initiate_browser_options__()
-
- def __initiate_browser_options__(self):
- """Initiate browser options."""
- self.launch_options: Dict[str, Any] = generate_launch_options(
- **{
- "geoip": self._geoip,
- "proxy": dict(self._proxy) if self._proxy and isinstance(self._proxy, tuple) else self._proxy,
- "addons": self._addons,
- "exclude_addons": [] if self._disable_ads else [DefaultAddons.UBO],
- "headless": self._headless,
- "humanize": True if self._solve_cloudflare else self._humanize,
- "i_know_what_im_doing": True, # To turn warnings off with the user configurations
- "allow_webgl": self._allow_webgl,
- "block_webrtc": self._block_webrtc,
- "block_images": self._block_images, # Careful! it makes some websites don't finish loading at all like stackoverflow even in headful mode.
- "os": None if self._os_randomize else get_os_name(),
- "user_data_dir": self._user_data_dir,
- "ff_version": __ff_version_str__,
- "firefox_user_prefs": {
- # This is what enabling `enable_cache` does internally, so we do it from here instead
- "browser.sessionhistory.max_entries": 10,
- "browser.sessionhistory.max_total_viewers": -1,
- "browser.cache.memory.enable": True,
- "browser.cache.disk_cache_ssl": True,
- "browser.cache.disk.smart_size.enabled": True,
- },
- **cast(Dict, self._additional_args),
+ def __generate_options__(self, extra_flags: Tuple | None = None) -> None:
+ config = cast(PlaywrightConfig, getattr(self, "_config", None))
+ self._context_options.update(
+ {
+ "proxy": config.proxy,
+ "locale": config.locale,
+ "timezone_id": config.timezone_id,
+ "extra_http_headers": config.extra_headers,
}
)
+ # The default useragent in the headful is always correct now in the current versions of Playwright
+ if config.useragent:
+ self._context_options["user_agent"] = config.useragent
+ elif not config.useragent and config.headless:
+ self._context_options["user_agent"] = (
+ __default_chrome_useragent__ if config.real_chrome else __default_useragent__
+ )
+
+ if not config.cdp_url:
+ self._launch_options |= self._context_options
+ self._context_options = {}
+ flags = self._launch_options["args"]
+ if config.extra_flags or extra_flags:
+ flags = list(set(flags + (config.extra_flags or extra_flags)))
+
+ self._launch_options.update(
+ {
+ "args": flags,
+ "headless": config.headless,
+ "user_data_dir": config.user_data_dir,
+ "channel": "chrome" if config.real_chrome else "chromium",
+ }
+ )
+
+ if config.additional_args:
+ self._launch_options.update(config.additional_args)
+ else:
+ # while `context_options` is left to be used when cdp mode is enabled
+ self._launch_options = dict()
+ if config.additional_args:
+ self._context_options.update(config.additional_args)
+
+
+class DynamicSessionMixin(BaseSessionMixin):
+ def __validate__(self, **params):
+ self._config = self.__validate_routine__(params, model=PlaywrightConfig)
+ self.__generate_options__()
+
+
+class StealthySessionMixin(BaseSessionMixin):
+ def __validate__(self, **params):
+ self._config: StealthConfig = self.__validate_routine__(params, model=StealthConfig)
+ self._context_options.update(
+ {
+ "is_mobile": False,
+ "has_touch": False,
+ # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
+ "service_workers": "allow",
+ "ignore_https_errors": True,
+ "screen": {"width": 1920, "height": 1080},
+ "viewport": {"width": 1920, "height": 1080},
+ "permissions": ["geolocation", "notifications"],
+ }
+ )
+ self.__generate_stealth_options()
+
+ def __generate_stealth_options(self) -> None:
+ flags = tuple()
+ if not self._config.cdp_url:
+ flags = DEFAULT_FLAGS + DEFAULT_STEALTH_FLAGS
+
+ if self._config.block_webrtc:
+ flags += (
+ "--webrtc-ip-handling-policy=disable_non_proxied_udp",
+ "--force-webrtc-ip-handling-policy", # Ensures the policy is enforced
+ )
+ if not self._config.allow_webgl:
+ flags += (
+ "--disable-webgl",
+ "--disable-webgl-image-chromium",
+ "--disable-webgl2",
+ )
+ if self._config.hide_canvas:
+ flags += ("--fingerprinting-canvas-image-data-noise",)
+
+ super(StealthySessionMixin, self).__generate_options__(flags)
@staticmethod
def _detect_cloudflare(page_content: str) -> str | None:
diff --git a/scrapling/engines/_browsers/_config_tools.py b/scrapling/engines/_browsers/_config_tools.py
index eaf28d6..2d7691b 100644
--- a/scrapling/engines/_browsers/_config_tools.py
+++ b/scrapling/engines/_browsers/_config_tools.py
@@ -1,15 +1,10 @@
from functools import lru_cache
-from scrapling.core._types import Tuple
-from scrapling.engines.constants import (
- DEFAULT_STEALTH_FLAGS,
- HARMFUL_DEFAULT_ARGS,
- DEFAULT_FLAGS,
-)
from scrapling.engines.toolbelt.navigation import js_bypass_path
from scrapling.engines.toolbelt.fingerprints import generate_headers
__default_useragent__ = generate_headers(browser_mode=True).get("User-Agent")
+__default_chrome_useragent__ = generate_headers(browser_mode="chrome").get("User-Agent")
@lru_cache(1)
@@ -41,99 +36,3 @@ def _compiled_stealth_scripts():
with open(script_path, "r") as f:
scripts.append(f.read())
return tuple(scripts)
-
-
-@lru_cache(2, typed=True)
-def _set_flags(hide_canvas, disable_webgl): # pragma: no cover
- """Returns the flags that will be used while launching the browser if stealth mode is enabled"""
- flags = DEFAULT_STEALTH_FLAGS
- if hide_canvas:
- flags += ("--fingerprinting-canvas-image-data-noise",)
- if disable_webgl:
- flags += (
- "--disable-webgl",
- "--disable-webgl-image-chromium",
- "--disable-webgl2",
- )
-
- return flags
-
-
-@lru_cache(2, typed=True)
-def _launch_kwargs(
- headless,
- proxy: Tuple,
- locale,
- extra_headers,
- useragent,
- real_chrome,
- stealth,
- hide_canvas,
- disable_webgl,
- timezone_id,
- extra_flags: Tuple,
-) -> Tuple:
- """Creates the arguments we will use while launching playwright's browser"""
- base_args = DEFAULT_FLAGS
- if extra_flags:
- base_args = base_args + extra_flags
-
- launch_kwargs = {
- "locale": locale,
- "timezone_id": timezone_id or None,
- "headless": headless,
- "args": base_args,
- "color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
- "proxy": proxy or tuple(),
- "device_scale_factor": 2,
- "ignore_default_args": HARMFUL_DEFAULT_ARGS,
- "channel": "chrome" if real_chrome else "chromium",
- "extra_http_headers": extra_headers or tuple(),
- "user_agent": useragent or __default_useragent__,
- }
- if stealth:
- stealth_args = base_args + _set_flags(hide_canvas, disable_webgl)
- launch_kwargs.update(
- {
- "args": stealth_args,
- "chromium_sandbox": True,
- "is_mobile": False,
- "has_touch": False,
- # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
- "service_workers": "allow",
- "ignore_https_errors": True,
- "screen": {"width": 1920, "height": 1080},
- "viewport": {"width": 1920, "height": 1080},
- "permissions": ["geolocation", "notifications"],
- }
- )
-
- return tuple(launch_kwargs.items())
-
-
-@lru_cache(2, typed=True)
-def _context_kwargs(proxy, locale, extra_headers, useragent, stealth) -> Tuple:
- """Creates the arguments for the browser context"""
- context_kwargs = {
- "proxy": proxy or tuple(),
- "locale": locale,
- "color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
- "device_scale_factor": 2,
- "extra_http_headers": extra_headers or tuple(),
- "user_agent": useragent or __default_useragent__,
- }
- if stealth:
- context_kwargs.update(
- {
- "is_mobile": False,
- "has_touch": False,
- # I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
- "service_workers": "allow",
- "ignore_https_errors": True,
- "screen": {"width": 1920, "height": 1080},
- "viewport": {"width": 1920, "height": 1080},
- "permissions": ["geolocation", "notifications"],
- }
- )
-
- return tuple(context_kwargs.items())
diff --git a/scrapling/engines/_browsers/_controllers.py b/scrapling/engines/_browsers/_controllers.py
index 9e48675..ff21438 100644
--- a/scrapling/engines/_browsers/_controllers.py
+++ b/scrapling/engines/_browsers/_controllers.py
@@ -9,8 +9,6 @@ from playwright.async_api import (
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
-from patchright.sync_api import sync_playwright as sync_patchright
-from patchright.async_api import async_playwright as async_patchright
from scrapling.core.utils import log
from scrapling.core._types import Unpack, TYPE_CHECKING
@@ -21,53 +19,27 @@ from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
-class DynamicSession(DynamicSessionMixin, SyncSession):
+class DynamicSession(SyncSession, DynamicSessionMixin):
"""A Browser session manager with page pooling."""
__slots__ = (
- "_max_pages",
- "_headless",
- "_hide_canvas",
- "_disable_webgl",
- "_real_chrome",
- "_stealth",
- "_google_search",
- "_proxy",
- "_locale",
- "_extra_headers",
- "_useragent",
- "_timeout",
- "_cookies",
- "_disable_resources",
- "_network_idle",
- "_load_dom",
- "_wait_selector",
- "_init_script",
- "_wait_selector_state",
- "_wait",
- "playwright",
- "browser",
- "context",
+ "_config",
+ "_context_options",
+ "_launch_options",
+ "max_pages",
"page_pool",
+ "_max_wait_for_page",
+ "playwright",
+ "context",
"_closed",
- "_selector_config",
- "_page_action",
- "launch_options",
- "context_options",
- "_cdp_url",
- "_headers_keys",
- "_extra_flags",
- "_additional_args",
- "_user_data_dir",
)
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -76,13 +48,11 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- :param timezone_id: Set the timezone for the browser if wanted.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
@@ -94,27 +64,24 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- sync_context = sync_patchright if self._stealth else sync_playwright
+ self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- self.playwright: Playwright = sync_context().start() # pyright: ignore [reportAttributeAccessIssue]
-
- if self._cdp_url: # pragma: no cover
- self.context = self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url).new_context(
- **self.context_options
- )
+ if self._config.cdp_url: # pragma: no cover
+ browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context = browser.new_context(**self._context_options)
else:
- self.context = self.playwright.chromium.launch_persistent_context(**self.launch_options)
+ self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
- if self._init_script: # pragma: no cover
- self.context.add_init_script(path=self._init_script)
+ if self._config.init_script: # pragma: no cover
+ self.context.add_init_script(path=self._config.init_script)
- if self._cookies: # pragma: no cover
- self.context.add_cookies(self._cookies)
+ if self._config.cookies: # pragma: no cover
+ self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -122,24 +89,21 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
-
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -193,16 +157,15 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
raise e
-class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
+class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
"""An async Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory."""
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -212,13 +175,11 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- :param timezone_id: Set the timezone for the browser if wanted.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -230,28 +191,26 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- async_context = async_patchright if self._stealth else async_playwright
+ self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- self.playwright: AsyncPlaywright = await async_context().start() # pyright: ignore [reportAttributeAccessIssue]
-
- if self._cdp_url:
- browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url)
- self.context: AsyncBrowserContext = await browser.new_context(**self.context_options)
+ if self._config.cdp_url:
+ browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
else:
self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
- **self.launch_options
+ **self._launch_options
)
- if self._init_script: # pragma: no cover
- await self.context.add_init_script(path=self._init_script)
+ if self._config.init_script: # pragma: no cover
+ await self.context.add_init_script(path=self._config.init_script)
- if self._cookies:
- await self.context.add_cookies(self._cookies) # pyright: ignore
+ if self._config.cookies:
+ await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
@@ -259,20 +218,18 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
diff --git a/scrapling/engines/_browsers/_camoufox.py b/scrapling/engines/_browsers/_stealth.py
similarity index 62%
rename from scrapling/engines/_browsers/_camoufox.py
rename to scrapling/engines/_browsers/_stealth.py
index e3cbc67..912d38e 100644
--- a/scrapling/engines/_browsers/_camoufox.py
+++ b/scrapling/engines/_browsers/_stealth.py
@@ -2,117 +2,101 @@ from random import randint
from re import compile as re_compile
from playwright.sync_api import (
- Page,
Locator,
- sync_playwright,
+ Page,
+ Playwright,
)
from playwright.async_api import (
- async_playwright,
Page as async_Page,
Locator as AsyncLocator,
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
+from patchright.sync_api import sync_playwright
+from patchright.async_api import async_playwright
from scrapling.core.utils import log
-from ._types import CamoufoxSession, CamoufoxFetchParams
-from scrapling.core._types import Any, Unpack, TYPE_CHECKING
+from scrapling.core._types import Any, Unpack
+from ._config_tools import _compiled_stealth_scripts
+from ._types import StealthSession, StealthFetchParams
from ._base import SyncSession, AsyncSession, StealthySessionMixin
-from ._validators import validate_fetch as _validate, CamoufoxConfig
+from ._validators import validate_fetch as _validate, StealthConfig
from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
__CF_PATTERN__ = re_compile("challenges.cloudflare.com/cdn-cgi/challenge-platform/.*")
-class StealthySession(StealthySessionMixin, SyncSession):
- """A Stealthy session manager with page pooling."""
+class StealthySession(SyncSession, StealthySessionMixin):
+ """A Stealthy Browser session manager with page pooling."""
__slots__ = (
- "_max_pages",
- "_headless",
- "_block_images",
- "_disable_resources",
- "_block_webrtc",
- "_allow_webgl",
- "_network_idle",
- "_load_dom",
- "_humanize",
- "_solve_cloudflare",
- "_wait",
- "_timeout",
- "_page_action",
- "_wait_selector",
- "_init_script",
- "_addons",
- "_wait_selector_state",
- "_cookies",
- "_google_search",
- "_extra_headers",
- "_proxy",
- "_os_randomize",
- "_disable_ads",
- "_geoip",
- "_selector_config",
- "_additional_args",
- "playwright",
- "browser",
- "context",
+ "_config",
+ "_context_options",
+ "_launch_options",
+ "max_pages",
"page_pool",
+ "_max_wait_for_page",
+ "playwright",
+ "context",
"_closed",
- "launch_options",
- "_headers_keys",
- "_user_data_dir",
)
- def __init__(self, **kwargs: Unpack[CamoufoxSession]):
- """A Browser session manager with page pooling
+ def __init__(self, **kwargs: Unpack[StealthSession]):
+ """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
- :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- self.playwright = sync_playwright().start()
- self.context = self.playwright.firefox.launch_persistent_context(**self.launch_options)
+ self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- if self._init_script: # pragma: no cover
- self.context.add_init_script(path=self._init_script)
+ if self._config.cdp_url: # pragma: no cover
+ browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context = browser.new_context(**self._context_options)
+ else:
+ self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
- if self._cookies: # pragma: no cover
- self.context.add_cookies(self._cookies)
+ for script in _compiled_stealth_scripts():
+ self.context.add_init_script(script=script)
+
+ if self._config.init_script: # pragma: no cover
+ self.context.add_init_script(path=self._config.init_script)
+
+ if self._config.cookies: # pragma: no cover
+ self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -148,22 +132,27 @@ class StealthySession(StealthySessionMixin, SyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
- self._wait_for_page_stability(iframe, True, True)
+ self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not iframe.frame_element().is_visible():
# Double-checking that the iframe is loaded
page.wait_for_timeout(500)
+
outer_box: Any = iframe.frame_element().bounding_box()
if not iframe or not outer_box:
+ if "Just a moment... " not in (ResponseFactory._get_page_content(page)):
+ log.info("Cloudflare captcha is solved")
+ return
+
outer_box: Any = page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
- page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
+ page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -182,29 +171,26 @@ class StealthySession(StealthySessionMixin, SyncSession):
log.info("Cloudflare captcha is solved")
return
- def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
+ def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
- params = _validate(kwargs, self, CamoufoxConfig)
-
+ params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -233,7 +219,7 @@ class StealthySession(StealthySessionMixin, SyncSession):
if params.page_action:
try:
_ = params.page_action(page_info.page)
- except Exception as e:
+ except Exception as e: # pragma: no cover
log.error(f"Error executing page_action: {e}")
if params.wait_selector:
@@ -242,10 +228,12 @@ class StealthySession(StealthySessionMixin, SyncSession):
waiter.first.wait_for(state=params.wait_selector_state)
# Wait again after waiting for the selector, helpful with protections like Cloudflare
self._wait_for_page_stability(page_info.page, params.load_dom, params.network_idle)
- except Exception as e:
+ except Exception as e: # pragma: no cover
log.error(f"Error waiting for selector {params.wait_selector}: {e}")
page_info.page.wait_for_timeout(params.wait)
+
+ # Create response object
response = ResponseFactory.from_playwright_response(
page_info.page, first_response, final_response[0], params.selector_config
)
@@ -256,72 +244,78 @@ class StealthySession(StealthySessionMixin, SyncSession):
return response
- except Exception as e: # pragma: no cover
+ except Exception as e:
page_info.mark_error()
raise e
-class AsyncStealthySession(StealthySessionMixin, AsyncSession):
- """A Stealthy session manager with page pooling."""
+class AsyncStealthySession(AsyncSession, StealthySessionMixin):
+ """An async Stealthy Browser session manager with page pooling."""
- def __init__(self, **kwargs: Unpack[CamoufoxSession]):
- """A Browser session manager with page pooling
+ def __init__(self, **kwargs: Unpack[StealthSession]):
+ """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- :param block_images: Prevent the loading of images through Firefox preferences.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- :param block_webrtc: Blocks WebRTC entirely.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
- :param addons: List of Firefox addons to use. Must be paths to extracted addons.
- :param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
- :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- :param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- :param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
- :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
- :param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- :param max_pages: The maximum number of tabs to be opened at the same time. It will be used in rotation through a PagePool.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- :param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
- super().__init__(max_pages=self._max_pages)
+ super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
- self.playwright: AsyncPlaywright = await async_playwright().start()
- self.context: AsyncBrowserContext = await self.playwright.firefox.launch_persistent_context(
- **self.launch_options
- )
+ self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
- if self._init_script: # pragma: no cover
- await self.context.add_init_script(path=self._init_script)
+ if self._config.cdp_url:
+ browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
+ self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
+ else:
+ self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
+ **self._launch_options
+ )
- if self._cookies:
- await self.context.add_cookies(self._cookies) # pyright: ignore [reportArgumentType]
+ for script in _compiled_stealth_scripts():
+ await self.context.add_init_script(script=script)
+
+ if self._config.init_script: # pragma: no cover
+ await self.context.add_init_script(path=self._config.init_script)
+
+ if self._config.cookies:
+ await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
- async def _cloudflare_solver(self, page: async_Page): # pragma: no cover
- """Solve the cloudflare challenge displayed on the playwright page passed. The async version
+ async def _cloudflare_solver(self, page: async_Page) -> None: # pragma: no cover
+ """Solve the cloudflare challenge displayed on the playwright page passed
- :param page: The async targeted page
+ :param page: The targeted page
:return:
"""
await self._wait_for_networkidle(page, timeout=5000)
@@ -331,7 +325,7 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
return
else:
log.info(f'The turnstile version discovered is "{challenge_type}"')
- if challenge_type == "non-interactive": # pragma: no cover
+ if challenge_type == "non-interactive":
while "Just a moment... " in (await ResponseFactory._get_async_page_content(page)):
log.info("Waiting for Cloudflare wait page to disappear.")
await page.wait_for_timeout(1000)
@@ -350,22 +344,27 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
- await self._wait_for_page_stability(iframe, True, True)
+ await self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not await (await iframe.frame_element()).is_visible():
# Double-checking that the iframe is loaded
await page.wait_for_timeout(500)
+
outer_box: Any = await (await iframe.frame_element()).bounding_box()
if not iframe or not outer_box:
+ if "Just a moment... " not in (await ResponseFactory._get_async_page_content(page)):
+ log.info("Cloudflare captcha is solved")
+ return
+
outer_box: Any = await page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
- await page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
+ await page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
await self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -377,35 +376,33 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
await page.wait_for_timeout(100)
attempts += 1
if challenge_type != "embedded":
- await page.locator(box_selector).wait_for(state="detached")
+ await page.locator(box_selector).last.wait_for(state="detached")
await page.locator(".zone-name-title").wait_for(state="hidden")
await self._wait_for_page_stability(page, True, False)
log.info("Cloudflare captcha is solved")
return
- async def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
+ async def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
- :param kwargs: Additional keyword arguments including:
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- - disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
- Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
- This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
- params = _validate(kwargs, self, CamoufoxConfig)
+ params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -418,10 +415,6 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
final_response = [None]
handle_response = self._create_response_handler(page_info, final_response)
- if TYPE_CHECKING:
- if not isinstance(page_info.page, async_Page):
- raise TypeError
-
try:
# Navigate to URL and wait for a specified state
page_info.page.on("response", handle_response)
@@ -461,9 +454,8 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
# Close the page to free up resources
await page_info.page.close()
self.page_pool.pages.remove(page_info)
-
return response
- except Exception as e:
+ except Exception as e: # pragma: no cover
page_info.mark_error()
raise e
diff --git a/scrapling/engines/_browsers/_types.py b/scrapling/engines/_browsers/_types.py
index d1edca5..cb8d760 100644
--- a/scrapling/engines/_browsers/_types.py
+++ b/scrapling/engines/_browsers/_types.py
@@ -11,9 +11,10 @@ from scrapling.core._types import (
Mapping,
Optional,
Callable,
- Iterable,
+ Sequence,
TypedDict,
TypeAlias,
+ SetCookieParam,
SelectorWaitStates,
TYPE_CHECKING,
)
@@ -53,7 +54,7 @@ if TYPE_CHECKING: # pragma: no cover
json: Optional[Dict | List]
# Types for browser session
- class BrowserSession(TypedDict, total=False):
+ class PlaywrightSession(TypedDict, total=False):
max_pages: int
headless: bool
disable_resources: bool
@@ -61,9 +62,10 @@ if TYPE_CHECKING: # pragma: no cover
load_dom: bool
wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates
- cookies: Optional[Iterable[Dict]]
+ cookies: Sequence[SetCookieParam] | None
google_search: bool
wait: int | float
+ timezone_id: str | None
page_action: Optional[Callable]
proxy: Optional[str | Dict[str, str] | Tuple]
extra_headers: Optional[Dict[str, str]]
@@ -72,42 +74,32 @@ if TYPE_CHECKING: # pragma: no cover
user_data_dir: str
selector_config: Optional[Dict]
additional_args: Optional[Dict]
-
- class PlaywrightSession(BrowserSession, total=False):
- cdp_url: Optional[str]
- hide_canvas: bool
- disable_webgl: bool
+ locale: Optional[str]
real_chrome: bool
- stealth: bool
- locale: str
+ cdp_url: Optional[str]
useragent: Optional[str]
extra_flags: Optional[List[str]]
class PlaywrightFetchParams(TypedDict, total=False):
+ load_dom: bool
+ wait: int | float
+ network_idle: bool
google_search: bool
timeout: int | float
- wait: int | float
- page_action: Optional[Callable]
- extra_headers: Optional[Dict[str, str]]
disable_resources: bool
wait_selector: Optional[str]
- wait_selector_state: SelectorWaitStates
- network_idle: bool
- load_dom: bool
+ page_action: Optional[Callable]
selector_config: Optional[Dict]
+ extra_headers: Optional[Dict[str, str]]
+ wait_selector_state: SelectorWaitStates
- class CamoufoxSession(BrowserSession, total=False):
- block_images: bool
- block_webrtc: bool
+ class StealthSession(PlaywrightSession, total=False):
allow_webgl: bool
- humanize: bool | float
+ hide_canvas: bool
+ block_webrtc: bool
solve_cloudflare: bool
- addons: Optional[List[str]]
- os_randomize: bool
- disable_ads: bool
- geoip: bool
- class CamoufoxFetchParams(PlaywrightFetchParams, total=False):
+ class StealthFetchParams(PlaywrightFetchParams, total=False):
solve_cloudflare: bool
else: # pragma: no cover
@@ -116,5 +108,5 @@ else: # pragma: no cover
DataRequestParams = TypedDict
PlaywrightSession = TypedDict
PlaywrightFetchParams = TypedDict
- CamoufoxSession = TypedDict
- CamoufoxFetchParams = TypedDict
+ StealthSession = TypedDict
+ StealthFetchParams = TypedDict
diff --git a/scrapling/engines/_browsers/_validators.py b/scrapling/engines/_browsers/_validators.py
index f8b9875..82270ef 100644
--- a/scrapling/engines/_browsers/_validators.py
+++ b/scrapling/engines/_browsers/_validators.py
@@ -13,12 +13,13 @@ from scrapling.core._types import (
Tuple,
Optional,
Callable,
- Iterable,
- SelectorWaitStates,
+ Sequence,
overload,
+ SetCookieParam,
+ SelectorWaitStates,
)
from scrapling.engines.toolbelt.navigation import construct_proxy_dict
-from scrapling.engines._browsers._types import PlaywrightFetchParams, CamoufoxFetchParams
+from scrapling.engines._browsers._types import PlaywrightFetchParams, StealthFetchParams
# Custom validators for msgspec
@@ -35,15 +36,6 @@ def _is_invalid_file_path(value: str) -> bool | str: # pragma: no cover
return False
-def _validate_addon_path(value: str) -> None: # pragma: no cover
- """Fast addon path validation"""
- path = Path(value)
- if not path.exists():
- raise FileNotFoundError(f"Addon path not found: {value}")
- if not path.is_dir():
- raise ValueError(f"Addon path must be a directory of the extracted addon: {value}")
-
-
@lru_cache(2)
def _is_invalid_cdp_url(cdp_url: str) -> bool | str:
"""Fast CDP URL validation"""
@@ -65,39 +57,36 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
"""Configuration struct for validation"""
max_pages: PagesCount = 1
- cdp_url: Optional[str] = None
headless: bool = True
- google_search: bool = True
- hide_canvas: bool = False
- disable_webgl: bool = False
- real_chrome: bool = False
- stealth: bool = False
- wait: Seconds = 0
- page_action: Optional[Callable] = None
- proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
- locale: str = "en-US"
- extra_headers: Optional[Dict[str, str]] = None
- useragent: Optional[str] = None
- timeout: Seconds = 30000
- init_script: Optional[str] = None
disable_resources: bool = False
- wait_selector: Optional[str] = None
- cookies: Optional[Iterable[Dict]] = None
network_idle: bool = False
load_dom: bool = True
+ wait_selector: Optional[str] = None
wait_selector_state: SelectorWaitStates = "attached"
+ cookies: Sequence[SetCookieParam] | None = []
+ google_search: bool = True
+ wait: Seconds = 0
+ timezone_id: str | None = ""
+ page_action: Optional[Callable] = None
+ proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
+ extra_headers: Optional[Dict[str, str]] = None
+ timeout: Seconds = 30000
+ init_script: Optional[str] = None
user_data_dir: str = ""
- timezone_id: str = ""
- extra_flags: Optional[List[str]] = None
selector_config: Optional[Dict] = {}
additional_args: Optional[Dict] = {}
+ locale: str | None = None
+ real_chrome: bool = False
+ cdp_url: Optional[str] = None
+ useragent: Optional[str] = None
+ extra_flags: Optional[List[str]] = None
def __post_init__(self): # pragma: no cover
"""Custom validation after msgspec validation"""
if self.page_action and not callable(self.page_action):
raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
if self.proxy:
- self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
+ self.proxy = construct_proxy_dict(self.proxy)
if self.cdp_url:
cdp_msg = _is_invalid_cdp_url(self.cdp_url)
if cdp_msg:
@@ -118,64 +107,18 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
raise ValueError(validation_msg)
-class CamoufoxConfig(Struct, kw_only=True, frozen=False, weakref=True):
- """Configuration struct for validation"""
-
- max_pages: PagesCount = 1
- headless: bool = True # noqa: F821
- block_images: bool = False
- disable_resources: bool = False
- block_webrtc: bool = False
+class StealthConfig(PlaywrightConfig, kw_only=True, frozen=False, weakref=True):
allow_webgl: bool = True
- network_idle: bool = False
- load_dom: bool = True
- humanize: bool | float = True
+ hide_canvas: bool = False
+ block_webrtc: bool = False
solve_cloudflare: bool = False
- wait: Seconds = 0
- timeout: Seconds = 30000
- init_script: Optional[str] = None
- page_action: Optional[Callable] = None
- wait_selector: Optional[str] = None
- addons: Optional[List[str]] = None
- wait_selector_state: SelectorWaitStates = "attached"
- cookies: Optional[Iterable[Dict]] = None
- google_search: bool = True
- extra_headers: Optional[Dict[str, str]] = None
- proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
- os_randomize: bool = False
- disable_ads: bool = False
- geoip: bool = False
- user_data_dir: str = ""
- selector_config: Optional[Dict] = {}
- additional_args: Optional[Dict] = {}
def __post_init__(self):
"""Custom validation after msgspec validation"""
- if self.page_action and not callable(self.page_action):
- raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
- if self.proxy:
- self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
-
- if self.addons:
- for addon in self.addons:
- _validate_addon_path(addon)
- else:
- self.addons = []
-
- if self.init_script is not None:
- validation_msg = _is_invalid_file_path(self.init_script)
- if validation_msg:
- raise ValueError(validation_msg)
-
- if not self.cookies:
- self.cookies = []
+ super(StealthConfig, self).__post_init__()
# Cloudflare timeout adjustment
if self.solve_cloudflare and self.timeout < 60_000:
self.timeout = 60_000
- if not self.selector_config:
- self.selector_config = {}
- if not self.additional_args:
- self.additional_args = {}
@dataclass
@@ -197,9 +140,9 @@ class _fetch_params:
def validate_fetch(
- method_kwargs: Dict | PlaywrightFetchParams | CamoufoxFetchParams,
+ method_kwargs: Dict | PlaywrightFetchParams | StealthFetchParams,
session: Any,
- model: type[PlaywrightConfig] | type[CamoufoxConfig],
+ model: type[PlaywrightConfig] | type[StealthConfig],
) -> _fetch_params: # pragma: no cover
result = {}
overrides = {}
@@ -210,21 +153,20 @@ def validate_fetch(
for key in fetch_param_fields:
if key in method_kwargs:
overrides[key] = method_kwargs[key]
- else:
- # Check for underscore-prefixed attribute (private)
- attr_name = f"_{key}"
- if hasattr(session, attr_name):
- result[key] = getattr(session, attr_name)
+ elif hasattr(session, "_config") and hasattr(session._config, key):
+ result[key] = getattr(session._config, key)
if overrides:
validated_config = validate(overrides, model)
- # Extract only the fields that _fetch_params needs from validated_config
+ # Extract ONLY the fields that were actually overridden (not all fields)
+ # This prevents validated defaults from overwriting session config values
validated_dict = {
- f.name: getattr(validated_config, f.name)
- for f in fields(_fetch_params)
- if hasattr(validated_config, f.name)
+ field: getattr(validated_config, field) for field in overrides.keys() if hasattr(validated_config, field)
}
- validated_dict.setdefault("solve_cloudflare", False)
+
+ # Preserve solve_cloudflare if the user explicitly provided it, even if the model doesn't have it
+ if "solve_cloudflare" in overrides:
+ validated_dict["solve_cloudflare"] = overrides["solve_cloudflare"]
# Start with session defaults, then overwrite with validated overrides
result.update(validated_dict)
@@ -238,7 +180,7 @@ def validate_fetch(
# Cache default values for each model to reduce validation overhead
models_default_values = {}
-for _model in (CamoufoxConfig, PlaywrightConfig):
+for _model in (StealthConfig, PlaywrightConfig):
_defaults = {}
if hasattr(_model, "__struct_defaults__") and hasattr(_model, "__struct_fields__"):
for field_name, default_value in zip(_model.__struct_fields__, _model.__struct_defaults__): # type: ignore
@@ -256,14 +198,14 @@ def _filter_defaults(params: Dict, model: str) -> Dict:
@overload
-def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
+def validate(params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
@overload
-def validate(params: Dict, model: type[CamoufoxConfig]) -> CamoufoxConfig: ...
+def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
-def validate(params: Dict, model: type[PlaywrightConfig] | type[CamoufoxConfig]) -> PlaywrightConfig | CamoufoxConfig:
+def validate(params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]) -> PlaywrightConfig | StealthConfig:
try:
# Filter out params with the default values (no need to validate them) to speed up validation
filtered = _filter_defaults(params, model.__name__)
diff --git a/scrapling/engines/constants.py b/scrapling/engines/constants.py
index df12ee3..db8618c 100644
--- a/scrapling/engines/constants.py
+++ b/scrapling/engines/constants.py
@@ -74,7 +74,6 @@ DEFAULT_STEALTH_FLAGS = (
"--disable-domain-reliability",
"--disable-threaded-animation",
"--disable-threaded-scrolling",
- # '--disable-reading-from-canvas', # For Firefox
"--enable-simple-cache-backend",
"--disable-background-networking",
"--enable-surface-synchronization",
@@ -94,9 +93,7 @@ DEFAULT_STEALTH_FLAGS = (
"--autoplay-policy=no-user-gesture-required",
"--disable-offer-store-unmasked-wallet-cards",
"--disable-blink-features=AutomationControlled",
- "--webrtc-ip-handling-policy=disable_non_proxied_udp",
"--disable-component-extensions-with-background-pages",
- "--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
"--enable-features=NetworkService,NetworkServiceInProcess,TrustTokens,TrustTokensAlwaysAllowIssuance",
"--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4",
"--disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process,TranslateUI,BlinkGenPropertyTrees",
diff --git a/scrapling/engines/toolbelt/fingerprints.py b/scrapling/engines/toolbelt/fingerprints.py
index c006ec2..2ddbbe5 100644
--- a/scrapling/engines/toolbelt/fingerprints.py
+++ b/scrapling/engines/toolbelt/fingerprints.py
@@ -13,19 +13,32 @@ from scrapling.core._types import Dict, Literal, Tuple
__OS_NAME__ = platform_system()
OSName = Literal["linux", "macos", "windows"]
+# Current versions hardcoded for now (Playwright doesn't allow to know the version of a browser without launching it)
+chromium_version = 141
+chrome_version = 143
@lru_cache(10, typed=True)
-def generate_convincing_referer(url: str) -> str:
+def generate_convincing_referer(url: str) -> str | None:
"""Takes the domain from the URL without the subdomain/suffix and make it look like you were searching Google for this website
>>> generate_convincing_referer('https://www.somewebsite.com/blah')
'https://www.google.com/search?q=somewebsite'
:param url: The URL you are about to fetch.
- :return: Google's search URL of the domain name
+ :return: Google's search URL of the domain name, or None for localhost/IP addresses
"""
- website_name = extract(url).domain
+ extracted = extract(url)
+ website_name = extracted.domain
+
+ # Skip generating referer for localhost, IP addresses, or when there's no valid domain
+ if not website_name or not extracted.suffix or website_name in ("localhost", "127.0.0.1", "::1"):
+ return None
+
+ # Check if it's an IP address (simple check for IPv4)
+ if all(part.isdigit() for part in website_name.split(".") if part):
+ return None
+
return f"https://www.google.com/search?q={website_name}"
@@ -46,7 +59,7 @@ def get_os_name() -> OSName | Tuple:
return SUPPORTED_OPERATING_SYSTEMS
-def generate_headers(browser_mode: bool = False) -> Dict:
+def generate_headers(browser_mode: bool | str = False) -> Dict:
"""Generate real browser-like headers using browserforge's generator
:param browser_mode: If enabled, the headers created are used for playwright, so it has to match everything
@@ -55,13 +68,14 @@ def generate_headers(browser_mode: bool = False) -> Dict:
# In the browser mode, we don't care about anything other than matching the OS and the browser type with the browser we are using,
# So we don't raise any inconsistency red flags while websites fingerprinting us
os_name = get_os_name()
- browsers = [Browser(name="chrome", min_version=130)]
+ ver = chrome_version if browser_mode and browser_mode == "chrome" else chromium_version
+ browsers = [Browser(name="chrome", min_version=ver, max_version=ver)]
if not browser_mode:
os_name = ("windows", "macos", "linux")
browsers.extend(
[
- Browser(name="firefox", min_version=130),
- Browser(name="edge", min_version=130),
+ Browser(name="firefox", min_version=142),
+ Browser(name="edge", min_version=140),
]
)
return HeaderGenerator(browser=browsers, os=os_name, device="desktop").generate()
diff --git a/scrapling/engines/toolbelt/navigation.py b/scrapling/engines/toolbelt/navigation.py
index 959aa5f..1b44f5a 100644
--- a/scrapling/engines/toolbelt/navigation.py
+++ b/scrapling/engines/toolbelt/navigation.py
@@ -11,7 +11,7 @@ from msgspec import Struct, structs, convert, ValidationError
from playwright.sync_api import Route
from scrapling.core.utils import log
-from scrapling.core._types import Dict, Tuple, overload, Literal
+from scrapling.core._types import Dict, Tuple
from scrapling.engines.constants import DEFAULT_DISABLED_RESOURCES
__BYPASSES_DIR__ = Path(__file__).parent / "bypasses"
@@ -49,20 +49,11 @@ async def async_intercept_route(route: async_Route):
await route.continue_()
-@overload
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[True]) -> Tuple: ...
-
-
-@overload
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[False] = False) -> Dict: ...
-
-
-def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: bool = False) -> Dict | Tuple:
+def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple) -> Dict:
"""Validate a proxy and return it in the acceptable format for Playwright
Reference: https://playwright.dev/python/docs/network#http-proxy
:param proxy_string: A string or a dictionary representation of the proxy.
- :param as_tuple: Return the proxy dictionary as a tuple to be cachable
:return:
"""
if isinstance(proxy_string, str):
@@ -78,7 +69,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
}
if proxy.port:
result["server"] += f":{proxy.port}"
- return tuple(result.items()) if as_tuple else result
+ return result
except ValueError:
# Urllib will say that one of the parameters above can't be casted to the correct type like `int` for port etc...
raise ValueError("The proxy argument's string is in invalid format!")
@@ -87,7 +78,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
try:
validated = convert(proxy_string, ProxyDict)
result_dict = structs.asdict(validated)
- return tuple(result_dict.items()) if as_tuple else result_dict
+ return result_dict
except ValidationError as e:
raise TypeError(f"Invalid proxy dictionary: {e}")
diff --git a/scrapling/fetchers/__init__.py b/scrapling/fetchers/__init__.py
index c135273..e2e5866 100644
--- a/scrapling/fetchers/__init__.py
+++ b/scrapling/fetchers/__init__.py
@@ -3,7 +3,7 @@ from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from scrapling.fetchers.requests import Fetcher, AsyncFetcher, FetcherSession
from scrapling.fetchers.chrome import DynamicFetcher, DynamicSession, AsyncDynamicSession
- from scrapling.fetchers.firefox import StealthyFetcher, StealthySession, AsyncStealthySession
+ from scrapling.fetchers.stealth_chrome import StealthyFetcher, StealthySession, AsyncStealthySession
# Lazy import mapping
@@ -14,9 +14,9 @@ _LAZY_IMPORTS = {
"DynamicFetcher": ("scrapling.fetchers.chrome", "DynamicFetcher"),
"DynamicSession": ("scrapling.fetchers.chrome", "DynamicSession"),
"AsyncDynamicSession": ("scrapling.fetchers.chrome", "AsyncDynamicSession"),
- "StealthyFetcher": ("scrapling.fetchers.firefox", "StealthyFetcher"),
- "StealthySession": ("scrapling.fetchers.firefox", "StealthySession"),
- "AsyncStealthySession": ("scrapling.fetchers.firefox", "AsyncStealthySession"),
+ "StealthyFetcher": ("scrapling.fetchers.stealth_chrome", "StealthyFetcher"),
+ "StealthySession": ("scrapling.fetchers.stealth_chrome", "StealthySession"),
+ "AsyncStealthySession": ("scrapling.fetchers.stealth_chrome", "AsyncStealthySession"),
}
__all__ = [
diff --git a/scrapling/fetchers/chrome.py b/scrapling/fetchers/chrome.py
index a706826..0d50c19 100644
--- a/scrapling/fetchers/chrome.py
+++ b/scrapling/fetchers/chrome.py
@@ -5,51 +5,37 @@ from scrapling.engines._browsers._controllers import DynamicSession, AsyncDynami
class DynamicFetcher(BaseFetcher):
- """A `Fetcher` class type that provide many options, all of them are based on PlayWright.
-
- Using this Fetcher class, you can do requests with:
- - Vanilla Playwright without any modifications other than the ones you chose.
- - Stealthy Playwright with the stealth mode I wrote for it. It's still a work in progress, but it bypasses many online tests like bot.sannysoft.com
- Some of the things stealth mode does include:
- 1) Patches the CDP runtime fingerprint.
- 2) Mimics some of the real browsers' properties by injecting several JS files and using custom options.
- 3) Using custom flags on launch to hide Playwright even more and make it faster.
- 4) Generates real browser's headers of the same type and same user OS, then append it to the request.
- - Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
-
- > Note that these are the main options with PlayWright, but it can be mixed.
- """
+ """A `Fetcher` that provide many options to fetch/load websites' pages through chromium-based browsers."""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[PlaywrightSession]) -> Response:
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- - cookies: Set cookies for the next request.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - locale: Set the locale for the browser if wanted. The default value is `en-US`.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - extra_flags: A list of additional browser flags to pass to the browser on launch.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request.
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
@@ -68,31 +54,30 @@ class DynamicFetcher(BaseFetcher):
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- - cookies: Set cookies for the next request.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - locale: Set the locale for the browser if wanted. The default value is `en-US`.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- - real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- - hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- - disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- - cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - extra_flags: A list of additional browser flags to pass to the browser on launch.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Playwright's context as additional settings.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests of unnecessary resources for a speed boost.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
+ :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request.
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
diff --git a/scrapling/fetchers/firefox.py b/scrapling/fetchers/firefox.py
deleted file mode 100644
index a9361d6..0000000
--- a/scrapling/fetchers/firefox.py
+++ /dev/null
@@ -1,102 +0,0 @@
-from scrapling.core._types import Unpack
-from scrapling.engines._browsers._types import CamoufoxSession
-from scrapling.engines.toolbelt.custom import BaseFetcher, Response
-from scrapling.engines._browsers._camoufox import StealthySession, AsyncStealthySession
-
-
-class StealthyFetcher(BaseFetcher):
- """A `Fetcher` class type that is a completely stealthy fetcher that uses a modified version of Firefox.
-
- It works as real browsers passing almost all online tests/protections based on Camoufox.
- Other added flavors include setting the faked OS fingerprints to match the user's OS, and the referer of every request is set as if this request came from Google's search of this URL's domain.
- """
-
- @classmethod
- def fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
- """
- Opens up a browser and do your request based on your chosen options below.
-
- :param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - block_images: Prevent the loading of images through Firefox preferences.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - block_webrtc: Blocks WebRTC entirely.
- - allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - addons: List of Firefox addons to use. Must be paths to extracted addons.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - cookies: Set cookies for the next request.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- - disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- - geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Camoufox as additional settings.
- :return: A `Response` object.
- """
- selector_config = kwargs.get("selector_config", {}) or kwargs.get(
- "custom_config", {}
- ) # Checking `custom_config` for backward compatibility
- if not isinstance(selector_config, dict):
- raise TypeError("Argument `selector_config` must be a dictionary.")
-
- kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
-
- with StealthySession(**kwargs) as engine:
- return engine.fetch(url)
-
- @classmethod
- async def async_fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
- """
- Opens up a browser and do your request based on your chosen options below.
-
- :param url: Target url.
- :param kwargs: Browser session configuration options including:
- - headless: Run the browser in headless/hidden (default), or headful/visible mode.
- - block_images: Prevent the loading of images through Firefox preferences.
- - disable_resources: Drop requests of unnecessary resources for a speed boost.
- - block_webrtc: Blocks WebRTC entirely.
- - allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- - network_idle: Wait for the page until there are no network connections for at least 500 ms.
- - load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- - humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- - solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- - wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- - timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- - page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- - wait_selector: Wait for a specific CSS selector to be in a specific state.
- - init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- - addons: List of Firefox addons to use. Must be paths to extracted addons.
- - wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- - cookies: Set cookies for the next request.
- - google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- - extra_headers: A dictionary of extra headers to add to the request.
- - proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- - os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- - disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- - geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- - selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- - additional_args: Additional arguments to be passed to Camoufox as additional settings.
- :return: A `Response` object.
- """
- selector_config = kwargs.get("selector_config", {}) or kwargs.get(
- "custom_config", {}
- ) # Checking `custom_config` for backward compatibility
- if not isinstance(selector_config, dict):
- raise TypeError("Argument `selector_config` must be a dictionary.")
-
- kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
-
- async with AsyncStealthySession(**kwargs) as engine:
- return await engine.fetch(url)
diff --git a/scrapling/fetchers/stealth_chrome.py b/scrapling/fetchers/stealth_chrome.py
new file mode 100644
index 0000000..58574d3
--- /dev/null
+++ b/scrapling/fetchers/stealth_chrome.py
@@ -0,0 +1,107 @@
+from scrapling.core._types import Unpack
+from scrapling.engines._browsers._types import StealthSession
+from scrapling.engines.toolbelt.custom import BaseFetcher, Response
+from scrapling.engines._browsers._stealth import StealthySession, AsyncStealthySession
+
+
+class StealthyFetcher(BaseFetcher):
+ """A `Fetcher` class type which is a completely stealthy built on top of Chromium.
+
+ It works as real browsers passing almost all online tests/protections with many customization options.
+ """
+
+ @classmethod
+ def fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
+ """
+ Opens up a browser and do your request based on your chosen options below.
+
+ :param url: Target url.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
+ :return: A `Response` object.
+ """
+ selector_config = kwargs.get("selector_config", {}) or kwargs.get(
+ "custom_config", {}
+ ) # Checking `custom_config` for backward compatibility
+ if not isinstance(selector_config, dict):
+ raise TypeError("Argument `selector_config` must be a dictionary.")
+
+ kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
+
+ with StealthySession(**kwargs) as engine:
+ return engine.fetch(url)
+
+ @classmethod
+ async def async_fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
+ """
+ Opens up a browser and do your request based on your chosen options below.
+
+ :param url: Target url.
+ :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
+ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
+ Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
+ :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
+ :param cookies: Set cookies for the next request.
+ :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
+ :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
+ :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
+ :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
+ :param wait_selector: Wait for a specific CSS selector to be in a specific state.
+ :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
+ :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
+ rules. Defaults to the system default locale.
+ :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
+ :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
+ :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
+ :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
+ :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
+ :param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
+ :param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
+ :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
+ :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
+ :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
+ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
+ :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
+ :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
+ :param extra_flags: A list of additional browser flags to pass to the browser on launch.
+ :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
+ :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
+ :return: A `Response` object.
+ """
+ selector_config = kwargs.get("selector_config", {}) or kwargs.get(
+ "custom_config", {}
+ ) # Checking `custom_config` for backward compatibility
+ if not isinstance(selector_config, dict):
+ raise TypeError("Argument `selector_config` must be a dictionary.")
+
+ kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
+
+ async with AsyncStealthySession(**kwargs) as engine:
+ return await engine.fetch(url)
diff --git a/setup.cfg b/setup.cfg
index beef70a..1a02999 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
[metadata]
name = scrapling
-version = 0.3.12
+version = 0.3.13
author = Karim Shoair
author_email = karim.shoair@pm.me
description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!
diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py
index 4a79c25..e2f86f3 100644
--- a/tests/cli/test_cli.py
+++ b/tests/cli/test_cli.py
@@ -156,7 +156,6 @@ class TestCLI:
html_url,
str(output_file),
'--headless',
- '--stealth',
'--timeout', '60000'
]
)
diff --git a/tests/fetchers/async/test_dynamic.py b/tests/fetchers/async/test_dynamic.py
index 0d46365..3fe38cc 100644
--- a/tests/fetchers/async/test_dynamic.py
+++ b/tests/fetchers/async/test_dynamic.py
@@ -53,10 +53,7 @@ class TestDynamicFetcherAsync:
@pytest.mark.parametrize(
"kwargs",
[
- {"disable_webgl": True, "hide_canvas": False},
- {"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
- {"stealth": True}, # causes issues with GitHub Actions
- {"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
+ {"real_chrome": True, "disable_resources": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
diff --git a/tests/fetchers/async/test_dynamic_session.py b/tests/fetchers/async/test_dynamic_session.py
index 234854c..c403017 100644
--- a/tests/fetchers/async/test_dynamic_session.py
+++ b/tests/fetchers/async/test_dynamic_session.py
@@ -71,7 +71,6 @@ class TestAsyncDynamicSession:
"""Test AsyncDynamicSession with various options"""
async with AsyncDynamicSession(
headless=False,
- stealth=True,
disable_resources=True,
extra_headers={"X-Test": "value"}
) as session:
diff --git a/tests/fetchers/async/test_requests_session.py b/tests/fetchers/async/test_requests_session.py
index c4e1355..3846e69 100644
--- a/tests/fetchers/async/test_requests_session.py
+++ b/tests/fetchers/async/test_requests_session.py
@@ -1,4 +1,3 @@
-import pytest
from scrapling.engines.static import AsyncFetcherClient
diff --git a/tests/fetchers/async/test_camoufox.py b/tests/fetchers/async/test_stealth.py
similarity index 81%
rename from tests/fetchers/async/test_camoufox.py
rename to tests/fetchers/async/test_stealth.py
index e8bbd3f..bba39d8 100644
--- a/tests/fetchers/async/test_camoufox.py
+++ b/tests/fetchers/async/test_stealth.py
@@ -1,4 +1,3 @@
-from playwright._impl._errors import TimeoutError
import pytest
import pytest_httpbin
@@ -30,8 +29,8 @@ class TestStealthyFetcher:
async def test_basic_fetch(self, fetcher, urls):
"""Test doing a basic fetch request with multiple statuses"""
assert (await fetcher.async_fetch(urls["status_200"])).status == 200
- assert (await fetcher.async_fetch(urls["status_404"])).status == 404
- assert (await fetcher.async_fetch(urls["status_501"])).status == 501
+ # assert (await fetcher.async_fetch(urls["status_404"])).status == 404
+ # assert (await fetcher.async_fetch(urls["status_501"])).status == 501
async def test_cookies_loading(self, fetcher, urls):
"""Test if cookies are set after the request"""
@@ -55,10 +54,9 @@ class TestStealthyFetcher:
@pytest.mark.parametrize(
"kwargs",
[
- {"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
- {"block_webrtc": False, "allow_webgl": True, "block_images": True},
+ {"block_webrtc": True, "allow_webgl": True},
+ {"block_webrtc": False, "allow_webgl": True},
{"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
- {"block_images": True, "disable_resources": True, },
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
@@ -67,11 +65,8 @@ class TestStealthyFetcher:
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"google_search": True,
"extra_headers": {"ayo": ""},
- "os_randomize": True,
- "disable_ads": True,
- # "geoip": True,
"selector_config": {"keep_comments": False, "keep_cdata": False},
- "additional_args": {"window": (1920, 1080)},
+ "additional_args": {},
},
],
)
diff --git a/tests/fetchers/async/test_camoufox_session.py b/tests/fetchers/async/test_stealth_session.py
similarity index 96%
rename from tests/fetchers/async/test_camoufox_session.py
rename to tests/fetchers/async/test_stealth_session.py
index 05f7953..140742d 100644
--- a/tests/fetchers/async/test_camoufox_session.py
+++ b/tests/fetchers/async/test_stealth_session.py
@@ -72,9 +72,8 @@ class TestAsyncStealthySession:
"""Test AsyncStealthySession with various options"""
async with AsyncStealthySession(
max_pages=1,
- block_images=True,
- disable_ads=True,
- humanize=True
+ block_webrtc=True,
+ allow_webgl=True
) as session:
response = await session.fetch(urls["html"])
assert response.status == 200
diff --git a/tests/fetchers/sync/test_camoufox.py b/tests/fetchers/sync/test_camoufox.py
deleted file mode 100644
index 269238f..0000000
--- a/tests/fetchers/sync/test_camoufox.py
+++ /dev/null
@@ -1,79 +0,0 @@
-import pytest
-import pytest_httpbin
-
-from scrapling import StealthyFetcher
-StealthyFetcher.adaptive = True
-
-
-@pytest_httpbin.use_class_based_httpbin
-class TestStealthyFetcher:
- @pytest.fixture(scope="class")
- def fetcher(self):
- """Fixture to create a StealthyFetcher instance for the entire test class"""
- return StealthyFetcher
-
- @pytest.fixture(autouse=True)
- def setup_urls(self, httpbin):
- """Fixture to set up URLs for testing"""
- self.status_200 = f"{httpbin.url}/status/200"
- self.status_404 = f"{httpbin.url}/status/404"
- self.status_501 = f"{httpbin.url}/status/501"
- self.basic_url = f"{httpbin.url}/get"
- self.html_url = f"{httpbin.url}/html"
- self.delayed_url = f"{httpbin.url}/delay/10" # 10 Seconds delay response
- self.cookies_url = f"{httpbin.url}/cookies/set/test/value"
-
- def test_basic_fetch(self, fetcher):
- """Test doing a basic fetch request with multiple statuses"""
- assert fetcher.fetch(self.status_200).status == 200
- assert fetcher.fetch(self.status_404).status == 404
- assert fetcher.fetch(self.status_501).status == 501
-
- def test_cookies_loading(self, fetcher):
- """Test if cookies are set after the request"""
- response = fetcher.fetch(self.cookies_url)
- cookies = {response.cookies[0]['name']: response.cookies[0]['value']}
- assert cookies == {"test": "value"}
-
- def test_automation(self, fetcher):
- """Test if automation breaks the code or not"""
-
- def scroll_page(page):
- page.mouse.wheel(10, 0)
- page.mouse.move(100, 400)
- page.mouse.up()
- return page
-
- assert fetcher.fetch(self.html_url, page_action=scroll_page).status == 200
-
- @pytest.mark.parametrize(
- "kwargs",
- [
- {"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
- {"block_webrtc": False, "allow_webgl": True, "block_images": True},
- {"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
- {"block_images": True, "disable_resources": True, },
- {"wait_selector": "h1", "wait_selector_state": "attached"},
- {"wait_selector": "h1", "wait_selector_state": "visible"},
- {
- "network_idle": True,
- "wait": 10,
- "timeout": 30_000,
- "cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
- "google_search": True,
- "extra_headers": {"ayo": ""},
- "os_randomize": True,
- "disable_ads": True,
- # "geoip": True,
- "selector_config": {"keep_comments": False, "keep_cdata": False},
- "additional_args": {"window": (1920, 1080)},
- },
- ],
- )
- def test_properties(self, fetcher, kwargs):
- """Test if different arguments break the code or not"""
- response = fetcher.fetch(
- self.html_url,
- **kwargs
- )
- assert response.status == 200
diff --git a/tests/fetchers/sync/test_dynamic.py b/tests/fetchers/sync/test_dynamic.py
index e044e33..cac8a73 100644
--- a/tests/fetchers/sync/test_dynamic.py
+++ b/tests/fetchers/sync/test_dynamic.py
@@ -51,10 +51,7 @@ class TestDynamicFetcher:
@pytest.mark.parametrize(
"kwargs",
[
- {"disable_webgl": True, "hide_canvas": False},
- {"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
- {"stealth": True}, # causes issues with GitHub Actions
- {"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
+ {"disable_resources": True, "real_chrome": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
diff --git a/tests/fetchers/sync/test_camoufox_session.py b/tests/fetchers/sync/test_stealth_session.py
similarity index 86%
rename from tests/fetchers/sync/test_camoufox_session.py
rename to tests/fetchers/sync/test_stealth_session.py
index e282d98..e740b01 100644
--- a/tests/fetchers/sync/test_camoufox_session.py
+++ b/tests/fetchers/sync/test_stealth_session.py
@@ -2,11 +2,11 @@ import re
import pytest
import pytest_httpbin
-from scrapling.engines._browsers._camoufox import StealthySession, __CF_PATTERN__
+from scrapling.engines._browsers._stealth import StealthySession, __CF_PATTERN__
-class TestCamoufoxConstants:
- """Test Camoufox constants and patterns"""
+class TestStealthConstants:
+ """Test Stealth constants and patterns"""
def test_cf_pattern_regex(self):
"""Test __CF_PATTERN__ regex compilation"""
@@ -54,7 +54,6 @@ class TestStealthySession:
with StealthySession(
headless=True,
- block_images=True,
disable_resources=True,
solve_cloudflare=True,
wait=1000,
@@ -63,12 +62,11 @@ class TestStealthySession:
) as session:
assert session.max_pages == 1
- assert session._headless is True
- assert session._block_images is True
- assert session._disable_resources is True
- assert session._solve_cloudflare is True
- assert session._wait == 1000
- assert session._timeout == 60000
+ assert session._config.headless is True
+ assert session._config.disable_resources is True
+ assert session._config.solve_cloudflare is True
+ assert session._config.wait == 1000
+ assert session._config.timeout == 60000
assert session.context is not None
# Test Cloudflare detection
diff --git a/tests/fetchers/test_utils.py b/tests/fetchers/test_utils.py
index 0028902..fea3f3f 100644
--- a/tests/fetchers/test_utils.py
+++ b/tests/fetchers/test_utils.py
@@ -190,14 +190,6 @@ class TestConstructProxyDict:
}
assert result == expected
- def test_proxy_as_tuple(self):
- """Test returning proxy as a tuple"""
- result = construct_proxy_dict("http://proxy.example.com:8080", as_tuple=True)
-
- assert isinstance(result, tuple)
- result_dict = dict(result)
- assert result_dict["server"] == "http://proxy.example.com:8080"
-
def test_invalid_proxy_string(self):
"""Test invalid proxy string"""
with pytest.raises(ValueError):
diff --git a/tests/fetchers/test_validator.py b/tests/fetchers/test_validator.py
index 118554c..209fae0 100644
--- a/tests/fetchers/test_validator.py
+++ b/tests/fetchers/test_validator.py
@@ -1,8 +1,8 @@
import pytest
from scrapling.engines._browsers._validators import (
validate,
+ StealthConfig,
PlaywrightConfig,
- CamoufoxConfig
)
@@ -23,7 +23,7 @@ class TestValidators:
assert config.max_pages == 2
assert config.headless is True
assert config.timeout == 30000
- assert isinstance(config.proxy, tuple) # Should be converted to tuple
+ assert isinstance(config.proxy, dict)
def test_playwright_config_invalid_max_pages(self):
"""Test PlaywrightConfig with invalid max_pages"""
@@ -51,8 +51,8 @@ class TestValidators:
with pytest.raises(TypeError):
validate(params, PlaywrightConfig)
- def test_camoufox_config_valid(self):
- """Test valid CamoufoxConfig"""
+ def test_stealth_config_valid(self):
+ """Test valid StealthConfig"""
params = {
"max_pages": 1,
"headless": True,
@@ -60,20 +60,20 @@ class TestValidators:
"timeout": 30000
}
- config = validate(params, CamoufoxConfig)
+ config = validate(params, StealthConfig)
assert config.max_pages == 1
assert config.headless is True
assert config.solve_cloudflare is False
assert config.timeout == 30000
- def test_camoufox_config_cloudflare_timeout(self):
- """Test CamoufoxConfig timeout adjustment for Cloudflare"""
+ def test_stealth_config_cloudflare_timeout(self):
+ """Test StealthConfig timeout adjustment for Cloudflare"""
params = {
"solve_cloudflare": True,
"timeout": 10000 # Less than the required 60,000
}
- config = validate(params, CamoufoxConfig)
+ config = validate(params, StealthConfig)
assert config.timeout == 60000 # Should be increased
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 52f672c..7e9a957 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,7 +1,6 @@
pytest>=2.8.0,<9
pytest-cov
playwright
-camoufox
werkzeug<3.0.0
pytest-httpbin==2.1.0
pytest-asyncio
diff --git a/tox.ini b/tox.ini
index 2a24c42..c2a30ae 100644
--- a/tox.ini
+++ b/tox.ini
@@ -10,9 +10,8 @@ envlist = pre-commit,py{310,311,312,313}
usedevelop = True
changedir = tests
deps =
- playwright>=1.56.0
- patchright>=1.56.0
- camoufox>=0.4.11
+ playwright==1.56.0
+ patchright==1.56.0
-r{toxinidir}/tests/requirements.txt
extras = ai,shell
commands =
+ You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.
In this case, the `get_all_text` method comes to the rescue, so you can do something like that ```python >>> page.css_first('div').get_all_text(ignore_tags=[]).json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} ``` I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.
- Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example: + Another related behavior to be aware of occurs when using any fetcher, which we will explain later. If you have a JSON response like this example: ```python >>> page = Selector("""{"some_key": "some_value"}""") ``` @@ -493,14 +493,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods {'some_key': 'some_value'} ``` You might wonder how this happened, given that the `html` tag doesn't contain direct text.
- Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.
This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions. + Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it uses the `get_all_text` method directly.
- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance ```python >>> TextHandler('\n wonderful idea, \reh?').clean() 'wonderful idea, eh?' ``` -Also, you can pass `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters. +Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters. - Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists ```python @@ -518,10 +518,10 @@ Other methods and properties will be added over time, but remember that this cla ## TextHandlers You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods. -The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing new needs to be explained here, but new methods will be added over time. +The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. Nothing new needs to be explained here, but new methods will be added over time. ## AttributesHandler -This is a read-only version of Python's standard dictionary, or `dict`, that is used solely to store the attributes of each element or each [Selector](#selector) instance. +This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element or [Selector](#selector) instance. ```python >>> print(page.find('script').attrib) {'id': 'page-data', 'type': 'application/json'} @@ -534,7 +534,7 @@ It currently adds two extra simple methods: - The `search_values` method - In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will require some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item. + In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item. A simple example would be ```python @@ -555,13 +555,13 @@ It currently adds two extra simple methods: