docs: update the parsing section
This commit is contained in:
Karim shoair
+15
-15
@@ -65,7 +65,7 @@ if not element: # One day website changes?
|
||||
element = page.css('#p1', adaptive=True) # Scrapling still finds it!
|
||||
# the rest of your code...
|
||||
```
|
||||
Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
|
||||
Below, I will show you an example of how to use this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
|
||||
|
||||
## Real-World Scenario
|
||||
Let's use a real website as an example and use one of the fetchers to fetch its source. To achieve this, we need to identify a website that is about to update its design/structure, copy its source, and then wait for the website to change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
|
||||
@@ -98,7 +98,7 @@ Note that I introduced a new argument called `adaptive_domain`. This is because,
|
||||
|
||||
The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `adaptive_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
|
||||
|
||||
Hence, in the two examples above, I used both the `Selector` class and the `Fetcher` class to show you that the logic for adaptive is the same.
|
||||
Hence, in the two examples above, I used both the `Selector` and `Fetcher` classes to show that the adaptive logic is the same.
|
||||
|
||||
## How the adaptive scraping feature works
|
||||
Adaptive scraping works in two phases:
|
||||
@@ -106,19 +106,19 @@ Adaptive scraping works in two phases:
|
||||
1. **Save Phase**: Store unique properties of elements
|
||||
2. **Match Phase**: Find elements with similar properties later
|
||||
|
||||
Let's say you've got an element through selection or any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
|
||||
Let's say you've selected an element through any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
|
||||
|
||||
With as few technical details as possible, the general logic goes as follows:
|
||||
|
||||
1. You tell Scrapling to save that element's unique properties in one of the ways we will show below.
|
||||
2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
|
||||
3. Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
|
||||
1. The domain of the current website. If you are using the `Selector` class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically.
|
||||
2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this.
|
||||
3. Now, because everything about the element can be changed or removed by the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
|
||||
1. The domain of the current website. If you are using the `Selector` class, pass it when initializing the class; if you are using one of the fetchers, the domain will be taken from the URL automatically.
|
||||
2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself; we'll discuss this later.
|
||||
|
||||
Together, they will be used to retrieve the element's unique properties from the database later.
|
||||
Together, they will later be used to retrieve the element's unique properties from the database.
|
||||
|
||||
4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity with the desired element. In that comparison, everything is taken into consideration, as you will see later
|
||||
4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated based on their similarity to the desired element. In that comparison, everything is taken into consideration, as you will see later
|
||||
5. The element(s) with the highest similarity score to the wanted element are returned.
|
||||
|
||||
### The unique properties
|
||||
@@ -129,7 +129,7 @@ For Scrapling, the unique elements we are relying on are:
|
||||
- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
|
||||
- Element's parent tag name, attributes (names and values), and text.
|
||||
|
||||
But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
|
||||
But you need to understand that the comparison between elements isn't exact; it's more about how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
|
||||
|
||||
## How to use adaptive feature
|
||||
The adaptive feature can be applied to any found element, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.
|
||||
@@ -146,11 +146,11 @@ Examples:
|
||||
```
|
||||
If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain.
|
||||
|
||||
If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and the `adaptive` feature only uses the latest saved properties.
|
||||
If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you use the same identifier later for a different website and don't pass the URL parameter when initializing it. The save process overwrites previous data, and the `adaptive` feature uses only the latest saved properties.
|
||||
|
||||
Besides those arguments, we have `storage` and `storage_args`. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
|
||||
Besides those arguments, we have `storage` and `storage_args`. Both are for the class to connect to the database; by default, it's set to the SQLite class the library uses. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
|
||||
|
||||
Now, after enabling the `adaptive` feature globally, you have two main ways to use it.
|
||||
Now that you've enabled the `adaptive` feature globally, you have two main ways to use it.
|
||||
|
||||
### The CSS/XPath Selection way
|
||||
As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page, like below
|
||||
@@ -163,7 +163,7 @@ element = page.css('#p1', adaptive=True)
|
||||
```
|
||||
Pretty simple, eh?
|
||||
|
||||
Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
|
||||
Well, a lot happened under the hood here. Remember the identifier we mentioned before that you need to set to retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
|
||||
|
||||
Additionally, for all these methods, you can pass the `identifier` argument to set it yourself. This is useful in some instances, or you can use it to save properties with the `auto_save` argument.
|
||||
|
||||
@@ -174,7 +174,7 @@ First, let's say you got an element like this by text:
|
||||
```python
|
||||
>>> element = page.find_by_text('Tipping the Velvet', first_match=True)
|
||||
```
|
||||
You can save its unique properties with the `save` method, like below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
|
||||
You can save its unique properties using the `save` method, as shown below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
|
||||
```python
|
||||
>>> page.save(element, 'my_special_element')
|
||||
```
|
||||
@@ -221,7 +221,7 @@ page.save(product, 'specific_product')
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
In the `adaptive` save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
|
||||
In the `adaptive` save process, only the unique properties of the first element in the selection results are saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors are separated and each is executed alone.
|
||||
|
||||
## Final thoughts
|
||||
Explaining this feature in detail without complications turned out to be challenging. However, still, if there's something left unclear, you can head out to the [discussions section](https://github.com/D4Vinci/Scrapling/discussions), and I will reply to you ASAP, or the Discord server, or reach out to me privately and have a chat :)
|
||||
@@ -41,7 +41,7 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th
|
||||
I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.
|
||||
You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested.
|
||||
|
||||
After that, for the main page and elements within, most properties are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
|
||||
After that, most properties on the main page and its elements are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
|
||||
|
||||
### Properties
|
||||
You have already seen much of this on the [overview](../overview.md) page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the [traversal](#traversal) section.
|
||||
@@ -111,7 +111,7 @@ But if you try to get the direct text content, it will be empty because it doesn
|
||||
```
|
||||
The `get_all_text` method has the following optional arguments:
|
||||
|
||||
1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'
|
||||
1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'.
|
||||
2. **strip**: If enabled, strings will be stripped before concatenation. Disabled by default.
|
||||
3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results and ignore any elements nested within them. The default is `('script', 'style',)`.
|
||||
4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
|
||||
@@ -132,7 +132,7 @@ If you use it on the page directly, you will find that you are operating on the
|
||||
>>> page.tag
|
||||
'html'
|
||||
```
|
||||
Now, I think I hammered the (`page`/`element`) idea, so I won't return to it again.
|
||||
Now, I think I've hammered the (`page`/`element`) idea, so I won't return to it.
|
||||
|
||||
Getting the attributes of the element
|
||||
```python
|
||||
@@ -196,7 +196,7 @@ Same case with XPath
|
||||
### Traversal
|
||||
Using the elements we found above, we will go over the properties/methods for moving on the page in detail.
|
||||
|
||||
If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding.
|
||||
If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online to better understand them.
|
||||
|
||||
If you are too lazy to search about it, here's a quick explanation to give you a good idea.<br/>
|
||||
In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.<br/>
|
||||
@@ -310,7 +310,7 @@ In the [Selector](#selector) class, all methods/properties that should return a
|
||||
|
||||
Let's see what [Selectors](#selectors) class adds to the table with that out of the way.
|
||||
### Properties
|
||||
Apart from the normal operations on Python lists, such as iteration and slicing, etc.
|
||||
Apart from the standard operations on Python lists, such as iteration and slicing.
|
||||
|
||||
You can do the following:
|
||||
|
||||
@@ -334,7 +334,7 @@ Execute CSS and XPath selectors directly on the [Selector](#selector) instances
|
||||
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
|
||||
...]
|
||||
```
|
||||
Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will still leave these methods to be explained in the [TextHandler](#texthandler) section below.
|
||||
Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will leave the explanation of these methods to the [TextHandler](#texthandler) section below.
|
||||
|
||||
However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance.
|
||||
```python
|
||||
@@ -355,7 +355,7 @@ However, in this class, the `re_first` behaves differently as it runs `re` on ea
|
||||
```
|
||||
With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`.
|
||||
```python
|
||||
# Find all the products with price '53.23'
|
||||
# Find all the products with price '53.23'.
|
||||
>>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
|
||||
>>> page.css('.product_pod').search(search_function)
|
||||
<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>
|
||||
@@ -374,7 +374,7 @@ If you are too lazy like me and want to know the number of [Selector](#selector)
|
||||
```python
|
||||
page.css('.product_pod').length
|
||||
```
|
||||
instead of this
|
||||
which is equivalent to
|
||||
```python
|
||||
len(page.css('.product_pod'))
|
||||
```
|
||||
@@ -389,15 +389,15 @@ Of course, TextHandler provides extra methods and properties that standard Pytho
|
||||
### Usage
|
||||
First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well.
|
||||
|
||||
First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they will accept the same arguments as well.
|
||||
First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they accept the same arguments.
|
||||
|
||||
- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
|
||||
- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but, as you probably figured out from the name, it returns only the first result as a `TextHandler` instance.
|
||||
|
||||
Also, it takes other helpful arguments, which are:
|
||||
|
||||
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
|
||||
- **clean_match**: It's disabled by default. This causes the method to ignore all whitespace and consecutive spaces while matching.
|
||||
- **case_sensitive**: It's enabled by default. As the name implies, disabling it will cause the regex to ignore the case of letters while compiling.
|
||||
- **clean_match**: It's disabled by default. This causes the method to ignore all whitespace, including consecutive spaces, while matching.
|
||||
- **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
|
||||
|
||||
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
|
||||
```python
|
||||
@@ -471,14 +471,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
|
||||
```python
|
||||
>>> page.css_first('div::text').json()
|
||||
```
|
||||
You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have direct text content at all.<br/><br/>
|
||||
You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.<br/><br/>
|
||||
In this case, the `get_all_text` method comes to the rescue, so you can do something like that
|
||||
```python
|
||||
>>> page.css_first('div').get_all_text(ignore_tags=[]).json()
|
||||
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
|
||||
```
|
||||
I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.<br/><br/>
|
||||
Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example:
|
||||
Another related behavior to be aware of occurs when using any fetcher, which we will explain later. If you have a JSON response like this example:
|
||||
```python
|
||||
>>> page = Selector("""{"some_key": "some_value"}""")
|
||||
```
|
||||
@@ -493,14 +493,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
|
||||
{'some_key': 'some_value'}
|
||||
```
|
||||
You might wonder how this happened, given that the `html` tag doesn't contain direct text.<br/>
|
||||
Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.<br/><br/>This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
|
||||
Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it uses the `get_all_text` method directly.<br/>
|
||||
|
||||
- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance
|
||||
```python
|
||||
>>> TextHandler('\n wonderful idea, \reh?').clean()
|
||||
'wonderful idea, eh?'
|
||||
```
|
||||
Also, you can pass `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
|
||||
Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
|
||||
|
||||
- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists
|
||||
```python
|
||||
@@ -518,10 +518,10 @@ Other methods and properties will be added over time, but remember that this cla
|
||||
## TextHandlers
|
||||
You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
|
||||
|
||||
The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing new needs to be explained here, but new methods will be added over time.
|
||||
The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. Nothing new needs to be explained here, but new methods will be added over time.
|
||||
|
||||
## AttributesHandler
|
||||
This is a read-only version of Python's standard dictionary, or `dict`, that is used solely to store the attributes of each element or each [Selector](#selector) instance.
|
||||
This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element or [Selector](#selector) instance.
|
||||
```python
|
||||
>>> print(page.find('script').attrib)
|
||||
{'id': 'page-data', 'type': 'application/json'}
|
||||
@@ -534,7 +534,7 @@ It currently adds two extra simple methods:
|
||||
|
||||
- The `search_values` method
|
||||
|
||||
In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will require some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
|
||||
In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
|
||||
|
||||
A simple example would be
|
||||
```python
|
||||
@@ -555,13 +555,13 @@ It currently adds two extra simple methods:
|
||||
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
|
||||
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
|
||||
```
|
||||
All these elements have 'product' as a value for the attribute `class`.
|
||||
All these elements have 'product' as the value for the `class` attribute.
|
||||
|
||||
Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
|
||||
|
||||
- The `json_string` property
|
||||
|
||||
This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error
|
||||
This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error.
|
||||
|
||||
```python
|
||||
>>>page.find('script').attrib.json_string
|
||||
|
||||
+17
-17
@@ -22,7 +22,7 @@ Scrapling implements CSS3 selectors as described in the [W3C specification](http
|
||||
|
||||
Also, Scrapling implements some non-standard pseudo-elements like:
|
||||
|
||||
* To select text nodes, use ``::text``
|
||||
* To select text nodes, use ``::text``.
|
||||
* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
|
||||
|
||||
In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
|
||||
@@ -36,19 +36,19 @@ In short, it is the same situation as CSS Selectors; if you come from Scrapy/Par
|
||||
|
||||
To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above, and `xpath_first` is faster.
|
||||
|
||||
> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
|
||||
> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here, as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
|
||||
|
||||
### Selectors examples
|
||||
Let's see some shared examples of using CSS and XPath Selectors.
|
||||
|
||||
Select all elements with the class `product`
|
||||
Select all elements with the class `product`.
|
||||
```python
|
||||
products = page.css('.product')
|
||||
products = page.xpath('//*[@class="product"]')
|
||||
```
|
||||
Note: The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class**
|
||||
|
||||
Select the first element with the class `product`
|
||||
Select the first element with the class `product`.
|
||||
```python
|
||||
product = page.css_first('.product')
|
||||
product = page.xpath_first('//*[@class="product"]')
|
||||
@@ -73,12 +73,12 @@ Get the `href` attribute of the first element with the `a` tag name
|
||||
link = page.css_first('a::attr(href)')
|
||||
link = page.xpath_first('//a/@href')
|
||||
```
|
||||
Select the text of the first element with the `h1` tag name, which contains 'Phone', and under an element with class 'product'
|
||||
Select the text of the first element with the `h1` tag name, which contains `Phone`, and under an element with class `product`.
|
||||
```python
|
||||
title = page.css_first('.product h1:contains("Phone")::text')
|
||||
title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
|
||||
```
|
||||
You can nest and chain selectors as you want, given that it returns results
|
||||
You can nest and chain selectors as you want, given that they return results
|
||||
```python
|
||||
page.css_first('.product').css_first('h1:contains("Phone")::text')
|
||||
page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
|
||||
@@ -104,7 +104,7 @@ Scrapling provides the ability to select elements based on their direct text con
|
||||
|
||||
What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
|
||||
|
||||
With `find_by_text`, you will pass the text as the first argument; with the `find_by_regex` method, the regex pattern is the first argument. Both methods share the following arguments:
|
||||
With `find_by_text`, you pass the text as the first argument; with `find_by_regex`, the regex pattern is the first argument. Both methods share the following arguments:
|
||||
|
||||
* **first_match**: If `True` (the default), the method used will return the first result it finds.
|
||||
* **case_sensitive**: If `True`, the case of the letters will be considered.
|
||||
@@ -117,7 +117,7 @@ By default, Scrapling searches for the exact matching of the text/pattern you pa
|
||||
Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
|
||||
|
||||
### Finding Similar Elements
|
||||
One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
|
||||
One of the most remarkable new features Scrapling puts on the table is the ability to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
|
||||
|
||||
So, how does it work?
|
||||
|
||||
@@ -129,8 +129,8 @@ Imagine a scenario where you found a product by its title, for example, and you
|
||||
|
||||
That's a lot of talking, I know, but I had to go deep. I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
|
||||
|
||||
* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the values of the attributes of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what the other arguments do first.
|
||||
* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change a lot between elements, making them unreliable.
|
||||
* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the tag attributes of both elements should be at least 20% similar. If you want to turn off this check (basically Step 3), you can set this attribute to 0, but I recommend you read what the other arguments do first.
|
||||
* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change significantly across elements, making them unreliable.
|
||||
* **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends.
|
||||
|
||||
Now, let's check out the examples below.
|
||||
@@ -148,7 +148,7 @@ Find the first element whose text fully matches this text
|
||||
>>> page.find_by_text('Tipping the Velvet')
|
||||
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>
|
||||
```
|
||||
Combining it with `page.urljoin` to return the full URL from the relative `href`
|
||||
Combining it with `page.urljoin` to return the full URL from the relative `href`.
|
||||
```python
|
||||
>>> page.find_by_text('Tipping the Velvet').attrib['href']
|
||||
'catalogue/tipping-the-velvet_999/index.html'
|
||||
@@ -174,7 +174,7 @@ Get all elements that contain the word `the` (Partial matching)
|
||||
'Mesaerion: The Best Science ...',
|
||||
"It's Only the Himalayas"]
|
||||
```
|
||||
The search is case-insensitive, so those results have `The`, not only the lowercase one `the`; let's limit the search to the elements with `the` only.
|
||||
The search is case-insensitive, so those results include `The`, not just the lowercase `the`; let's limit the search to elements with `the` only.
|
||||
```python
|
||||
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
|
||||
>>> [i.text for i in results]
|
||||
@@ -332,14 +332,14 @@ def extract_reviews(page):
|
||||
]
|
||||
```
|
||||
## Filters-based searching
|
||||
This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier to learn for newcomers to Web Scraping than writing selectors.
|
||||
This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier for newcomers to Web Scraping to learn than writing selectors.
|
||||
|
||||
Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
|
||||
Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can accept multiple filters and return all elements on the pages where all these filters apply.
|
||||
|
||||
To be more specific:
|
||||
|
||||
* Any string passed is considered a tag name.
|
||||
* Any iterable passed, like List/Tuple/Set, is considered an iterable of tag names.
|
||||
* Any iterable passed, like List/Tuple/Set, will be considered as an iterable of tag names.
|
||||
* Any dictionary is considered a mapping of HTML element(s), attribute names, and attribute values.
|
||||
* Any regex patterns passed are used to filter elements by content, like the `find_by_regex` method
|
||||
* Any functions passed are used to filter elements
|
||||
@@ -357,7 +357,7 @@ It filters all elements in the current page/element in the following order:
|
||||
Notes:
|
||||
|
||||
1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on.
|
||||
2. The order in which you pass the arguments doesn't matter. The only order taken into consideration is the order explained above.
|
||||
2. The order in which you pass the arguments doesn't matter. The only order considered is the one explained above.
|
||||
|
||||
Check examples to clear any confusion :)
|
||||
|
||||
@@ -509,4 +509,4 @@ We will have a deep look at it while explaining the [TextHandler](main_classes.m
|
||||
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
|
||||
['tipping-the-velvet_999']
|
||||
```
|
||||
And so on. You get the idea. We will explain this in more detail on the next page while explaining the [TextHandler](main_classes.md#texthandler) class.
|
||||
And so on. You get the idea. We will explain this in more detail on the next page, along with the [TextHandler](main_classes.md#texthandler) class.
|
||||
Reference in New Issue
Block a user