,
...]
```
+If you are too lazy like me and want to know the number of [Selector](#selector) instances in a [Selectors](#selectors) instance. You can do this:
+```python
+page.css('.product_pod').length
+```
+instead of this
+```python
+len(page.css('.product_pod'))
+```
+Yup, like JavaScript :)
## TextHandler
This class is mandatory to understand, as all methods/properties that should return a string for you will return `TextHandler`, and the ones that should return a list of strings will return [TextHandlers](#texthandlers) instead.
-TextHandler is a subclass of the standard Python string, so you can do anything with it. So, what is the difference that requires a different naming?
+TextHandler is a subclass of the standard Python string, so you can do anything with it that you can do with a Python string. So, what is the difference that requires a different naming?
-Of course, TextHandler provides extra methods and properties that the standard Python strings can't do. We will review them now, but remember that all methods and properties in all classes that return string(s) are returning TextHandler, which opens the door for creativity and makes the code shorter and cleaner, as you will see. Also, you can import it directly and use it on any string, which we will explain later.
+Of course, TextHandler provides extra methods and properties that standard Python strings can't do. We will review them now, but remember that all methods and properties in all classes that return string(s) return TextHandler, which opens the door for creativity and makes the code shorter and cleaner, as you will see. Also, you can import it directly and use it on any string, which we will explain [later](../development/scrapling_custom_types.md).
### Usage
-First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a TextHandler again, so you can chain them as you want. If you find a method or property that returns a standard string instead of TextHandler, please open an issue, and we will override it as well.
+First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well.
-First, we start with the `re` and `re_first` methods. These are the same methods that exist in the rest of the classes ([Adaptor](#adaptor), [Adaptors](#adaptors), and [TextHandlers](#texthandlers)), so they will take the same arguments as well.
+First, we start with the `re` and `re_first` methods. These are the same methods that exist in the rest of the classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they will take the same arguments as well.
- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
+- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
Also, it takes other helpful arguments, which are:
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- - **clean_match**: It's disabled by default. This makes the method ignore all whitespaces and consecutive spaces while matching.
- - **case_sensitive**: It's enabled by default. As the name implies, disabling it will make the regex ignore letters case while compiling it.
-
+ - **clean_match**: It's disabled by default. This makes the method ignore all whitespaces and consecutive spaces while matching.
+ - **case_sensitive**: It's enabled by default. As the name implies, disabling it will make the regex ignore the case of letters while compiling it.
+
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
```python
>>> page.css('.price_color').re(r'[\d\.]+')
@@ -405,25 +425,25 @@ First, we start with the `re` and `re_first` methods. These are the same methods
>>> test_string.re('hi there', clean_match=True, case_sensitive=False)
['hi There']
```
- Another use of the idea of replacing strings with `TextHandler` everywhere is a property like `html_content` returns `TextHandler` so you can do regex on the HTML content if you want:
+ Another use of the idea of replacing strings with `TextHandler` everywhere is that a property like `html_content` returns `TextHandler`, so you can do regex on the HTML content if you want:
```python
>>> page.html_content.re('div class=".*">(.*)>> page.css_first('#page-data::text')
'\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n '
>>> page.css_first('#page-data::text').json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
- Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically like this
+ Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically, like this
```python
>>> page.css_first('#page-data').json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
- The [Adaptor](#adaptor) class adds one thing here, too; let's say this is the page we are working with:
+ The [Selector](#selector) class adds one thing here, too; let's say this is the page we are working with:
```html
@@ -438,42 +458,42 @@ First, we start with the `re` and `re_first` methods. These are the same methods
```
- The [Adaptor](#adaptor) class has the `get_all_text` method, which you should be aware of by now. This method returns a `TextHandler`, of course.
+ The [Selector](#selector) class has the `get_all_text` method, which you should be aware of by now. This method returns a `TextHandler`, of course.
So, as you know here, if you did something like this
```python
>>> page.css_first('div::text').json()
```
- You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have text content at all.
+ You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have direct text content at all.
In this case, the `get_all_text` method comes to the rescue, so you can do something like that
```python
>>> page.css_first('div').get_all_text(ignore_tags=[]).json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.
- Another related behavior you should be aware of is the case while using any of the fetchers, which we will explain later. If you have a JSON response like this example:
+ Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example:
```python
- >>> page = Adaptor("""{"some_key": "some_value"}""")
+ >>> page = Selector("""{"some_key": "some_value"}""")
```
- Because the [Adaptor](#adaptor) class is optimized to deal with HTML pages, it will deal with it as a broken HTML response and fix it, so if you used the `html_content` property, you get this
+ Because the [Selector](#selector) class is optimized to deal with HTML pages, it will deal with it as a broken HTML response and fix it, so if you used the `html_content` property, you get this
```python
>>> page.html_content
'
{"some_key": "some_value"}
'
```
- Here, you can use `json` method directly, and it will work
+ Here, you can use the `json` method directly, and it will work
```python
>>> page.json()
{'some_key': 'some_value'}
```
- You might wonder how this happened while the `html` tag lacks direct text?
- Well, for these cases like JSON responses, I made the `.json()` method inside the [Adaptor](#adaptor) class to check if the current element doesn't have text content; it will use the `get_all_text` method directly.
It might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
+ You might wonder how this happened while the `html` tag doesn't have direct text?
+ Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.
This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
-- Another handy method is `.clean()`, this will remove all white spaces and consecutive spaces for you and return a new `TextHandler`, wonderful
+- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance
```python
>>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?'
```
-- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you as you do with lists
+- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists
```python
>>> TextHandler('acb').sort()
'abc'
@@ -487,19 +507,19 @@ Or do it in reverse:
Other methods and properties will be added over time, but remember that this class is returned in place of strings nearly everywhere in the library.
## TextHandlers
-You probably guessed it: This class is similar to [Adaptors](#adaptors) and [Adaptor](#adaptor), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
+You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
-The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing is new to explain here, but new methods will be added here with time.
+The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing is new to explain here, but new methods will be added over time.
## AttributesHandler
-This is a read-only version of Python's standard dictionary or `dict` that's only used to store the attributes of each element or each [Adaptor](#adaptor) instance, in other words.
+This is a read-only version of Python's standard dictionary or `dict` that's only used to store the attributes of each element or each [Selector](#selector) instance, in other words.
```python
>>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'}
>>> type(page.find('script').attrib).__name__
'AttributesHandler'
```
-Because it's read-only, it will use fewer resources than the standard dictionary. Still, it has the same dictionary method/properties other than those allowing you to modify/override the data.
+Because it's read-only, it will use fewer resources than the standard dictionary. Still, it has the same dictionary method and properties, except those that allow you to modify/override the data.
It currently adds two extra simple methods:
@@ -530,10 +550,10 @@ It currently adds two extra simple methods:
Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
- - The `json_string` property
+- The `json_string` property
- This property converts current attributes to JSON string if the attributes are JSON serializable; otherwise, it throws an error
- ```python
- >>> page.find('script').attrib.json_string
- b'{"id":"page-data","type":"application/json"}'
- ```
\ No newline at end of file
+ This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error
+ ```python
+ >>>page.find('script').attrib.json_string
+ b'{"id":"page-data","type":"application/json"}'
+ ```
\ No newline at end of file