diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md index 40a66f3..d71b26f 100644 --- a/docs/parsing/main_classes.md +++ b/docs/parsing/main_classes.md @@ -165,10 +165,10 @@ print(article.prettify()) ``` -Use the `.body` property to get the raw content of the page +Use the `.body` property to get the raw content of the page. Starting from v0.4, when used on a `Response` object from fetchers, `.body` always returns `bytes`. ```python >>> page.body -'\n \n Some page\n \n \n
\n
\n

Product 1

\n

This is product 1

\n $10.99\n \n
\n\n
\n

Product 2

\n

This is product 2

\n $20.99\n \n
\n\n
\n

Product 3

\n

This is product 3

\n $15.99\n \n
\n
\n\n \n \n' +'\n \n Some page\n \n ...' ``` To get all the ancestors in the DOM tree of this element ```python @@ -233,7 +233,7 @@ This element returns the same result as the `children` property because its chil Another example of using the element with the `product-list` class will clear the difference between the `children` property and the `below_elements` property ```python ->>> products_list = page.css_first('.product-list') +>>> products_list = page.css('.product-list')[0] >>> products_list.children [, , @@ -262,7 +262,7 @@ Get the next element of the current element The same logic applies to the `previous` property ```python >>> article.previous # It's the first child, so it doesn't have a previous element ->>> second_article = page.css_first('.product[data-id="2"]') +>>> second_article = page.css('.product[data-id="2"]')[0] >>> second_article.previous ``` @@ -287,27 +287,57 @@ You can search for a specific ancestor of an element that satisfies a search fun ## Selectors The class `Selectors` is the "List" version of the [Selector](#selector) class. It inherits from the Python standard `List` type, so it shares all `List` properties and methods while adding more methods to make the operations you want to execute on the [Selector](#selector) instances within more straightforward. -In the [Selector](#selector) class, all methods/properties that should return a group of elements return them as a [Selectors](#selectors) class instance. The only exceptions are when you use the CSS/XPath methods as follows: +In the [Selector](#selector) class, all methods/properties that should return a group of elements return them as a [Selectors](#selectors) class instance. -- If you selected a text node with the selector, then the return type will be [TextHandler](#texthandler)/[TextHandlers](#texthandlers).
Examples: - ```python - >>> page.css('a::text') # -> TextHandlers - >>> page.xpath('//a/text()') # -> TextHandlers - >>> page.css_first('a::text') # -> TextHandler - >>> page.xpath_first('//a/text()') # -> TextHandler - >>> page.css('a::attr(href)') # -> TextHandlers - >>> page.xpath('//a/@href') # -> TextHandlers - >>> page.css_first('a::attr(href)') # -> TextHandler - >>> page.xpath_first('//a/@href') # -> TextHandler - ``` -- If you used a combined selector that returns mixed types, the result will be a Python standard `List`.
Examples: - ```python - >>> page.css('.price_color') # -> Selectors - >>> page.css('.product_pod a::attr(href)') # -> TextHandlers - >>> page.css('.price_color, .product_pod a::attr(href)') # -> List - ``` +Starting with v0.4, all selection methods consistently return [Selector](#selector)/[Selectors](#selectors) objects, even for text nodes and attribute values. Text nodes (selected via `::text`, `/text()`, `::attr()`, `/@attr`) are wrapped in [Selector](#selector) objects. These text node selectors have `tag` set to `"#text"`, and their `text` property returns the text value. You can still access the text value directly, and all other properties gracefully return empty/default values. -Let's see what [Selectors](#selectors) class adds to the table with that out of the way. +```python +>>> page.css('a::text') # -> Selectors (of text node Selectors) +>>> page.xpath('//a/text()') # -> Selectors +>>> page.css('a::text').get() # -> TextHandler (the first text value) +>>> page.css('a::text').getall() # -> TextHandlers (all text values) +>>> page.css('a::attr(href)') # -> Selectors +>>> page.xpath('//a/@href') # -> Selectors +>>> page.css('.price_color') # -> Selectors +``` + +### Data extraction methods +Starting with v0.4, [Selector](#selector) and [Selectors](#selectors) both provide `get()`, `getall()`, and their aliases `extract_first` and `extract` (following Scrapy conventions). The old `get_all()` method has been removed. + +**On a [Selector](#selector) object:** + +- `get()` returns a `TextHandler` — for text node selectors, it returns the text value; for HTML element selectors, it returns the serialized outer HTML. +- `getall()` returns a `TextHandlers` list containing the single serialized string. +- `extract_first` is an alias for `get()`, and `extract` is an alias for `getall()`. + +```python +>>> page.css('h3')[0].get() # Outer HTML of the element +'

Product 1

' + +>>> page.css('h3::text')[0].get() # Text value of the text node +'Product 1' +``` + +**On a [Selectors](#selectors) object:** + +- `get(default=None)` returns the serialized string of the **first** element, or `default` if the list is empty. +- `getall()` serializes **all** elements and returns a `TextHandlers` list. +- `extract_first` is an alias for `get()`, and `extract` is an alias for `getall()`. + +```python +>>> page.css('.price::text').get() # First price text +'$10.99' + +>>> page.css('.price::text').getall() # All price texts +['$10.99', '$20.99', '$15.99'] + +>>> page.css('.price::text').get('') # With default value +'$10.99' +``` + +These methods work seamlessly with all selection types (CSS, XPath, `find`, etc.) and are the recommended way to extract text and attribute values in a Scrapy-compatible style. + +Now, let's see what [Selectors](#selectors) class adds to the table with that out of the way. ### Properties Apart from the standard operations on Python lists, such as iteration and slicing. @@ -369,6 +399,15 @@ You can use the `filter` method, too, which takes a function like the `search` m
, ...] ``` +You can safely access the first or last element without worrying about index errors: +```python +>>> page.css('.product').first # First Selector or None + +>>> page.css('.product').last # Last Selector or None + +>>> page.css('.nonexistent').first # Returns None instead of raising IndexError +``` + If you are too lazy like me and want to know the number of [Selector](#selector) instances in a [Selectors](#selectors) instance. You can do this: ```python page.css('.product_pod').length @@ -440,14 +479,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods - You also have the `.json()` method, which tries to convert the content to a JSON object quickly if possible; otherwise, it throws an error ```python - >>> page.css_first('#page-data::text') + >>> page.css('#page-data::text').get() '\n {\n "lastUpdated": "2024-09-22T10:30:00Z",\n "totalProducts": 3\n }\n ' - >>> page.css_first('#page-data::text').json() + >>> page.css('#page-data::text').get().json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} ``` Hence, if you didn't specify a text node while selecting an element (like the text content or an attribute text content), the text content will be selected automatically, like this ```python - >>> page.css_first('#page-data').json() + >>> page.css('#page-data')[0].json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} ``` The [Selector](#selector) class adds one thing here, too; let's say this is the page we are working with: @@ -468,12 +507,12 @@ First, we start with the `re` and `re_first` methods. These are the same methods The [Selector](#selector) class has the `get_all_text` method, which you should be aware of by now. This method returns a `TextHandler`, of course.

So, as you know here, if you did something like this ```python - >>> page.css_first('div::text').json() + >>> page.css('div::text').get().json() ``` You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.

In this case, the `get_all_text` method comes to the rescue, so you can do something like that ```python - >>> page.css_first('div').get_all_text(ignore_tags=[]).json() + >>> page.css('div')[0].get_all_text(ignore_tags=[]).json() {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} ``` I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.