>> quote.children.css_first(".author::text")
+'Albert Einstein'
>>> quote.has_class('quote')
True
# Generate new selectors for any element
->>> quote.css_selector
+>>> quote.generate_css_selector
'body > div > div:nth-of-type(2) > div > div'
-# Test these selectors on your favorite browser or reuse them again in the library in other methods!
->>> quote.xpath_selector
+# Test these selectors on your favorite browser or reuse them again in the library's methods!
+>>> quote.generate_xpath_selector
'//body/div/div[2]/div/div'
```
If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element like below
@@ -164,11 +341,9 @@ You can search for a specific ancestor of an element that satisfies a function,
### Content-based Selection & Finding Similar Elements
You can select elements by their text content in multiple ways, here's a full example on another website:
```python
->>> response = requests.get('https://books.toscrape.com/index.html')
+>>> page = Fetcher().get('https://books.toscrape.com/index.html').adaptor
->>> page = Adaptor(response.text, url=response.url)
-
->>> page.find_by_text('Tipping the Velvet') # Find the first element that its text fully matches this text
+>>> page.find_by_text('Tipping the Velvet') # Find the first element whose text fully matches this text
>>> page.find_by_text('Tipping the Velvet', first_match=False) # Get all matches if there are more
@@ -208,8 +383,8 @@ To increase the complexity a little bit, let's say we want to get all books' dat
```python
>>> for product in page.find_by_text('Tipping the Velvet').parent.parent.find_similar():
print({
- "name": product.css('h3 a::text')[0],
- "price": product.css('.price_color')[0].re_first(r'[\d\.]+'),
+ "name": product.css_first('h3 a::text'),
+ "price": product.css_first('.price_color').re_first(r'[\d\.]+'),
"stock": product.css('.availability::text')[-1].clean()
})
{'name': 'A Light in the ...', 'price': '51.77', 'stock': 'In stock'}
@@ -220,8 +395,6 @@ To increase the complexity a little bit, let's say we want to get all books' dat
The [documentation](https://github.com/D4Vinci/Scrapling/tree/main/docs/Examples) will provide more advanced examples.
### Handling Structural Changes
-> Because [the internet archive](https://web.archive.org/) is down at the time of writing this, I can't use real websites as examples even though I tested that before (I mean browsing an old version of a website and then counting the current version of the website as structural changes)
-
Let's say you are scraping a page with a structure like this:
```html
@@ -237,7 +410,7 @@ Let's say you are scraping a page with a structure like this:
```
-and you want to scrape the first product, the one with the `p1` ID. You will probably write a selector like this
+And you want to scrape the first product, the one with the `p1` ID. You will probably write a selector like this
```python
page.css('#p1')
```
@@ -262,34 +435,147 @@ When website owners implement structural changes like
```
-The selector will no longer function and your code needs maintenance. That's where Scrapling auto-matching feature comes into play.
+The selector will no longer function and your code needs maintenance. That's where Scrapling's auto-matching feature comes into play.
```python
+from scrapling import Adaptor
# Before the change
-page = Adaptor(page_source, url='example.com', auto_match=True)
+page = Adaptor(page_source, url='example.com')
element = page.css('#p1' auto_save=True)
if not element: # One day website changes?
- element = page.css('#p1', auto_match=True) # Still finds it!
+ element = page.css('#p1', auto_match=True) # Scrapling still finds it!
# the rest of the code...
```
-> How does the auto-matching work? Check the [FAQs](#FAQs) section for that and other possible issues while auto-matching.
+> How does the auto-matching work? Check the [FAQs](#-enlightening-questions-and-faqs) section for that and other possible issues while auto-matching.
+
+#### Real-World Scenario
+Let's use a real website as an example and use one of the fetchers to fetch its source. To do this we need to find a website that will change its design/structure soon, take a copy of its source then wait for the website to make the change. Of course, that's nearly impossible to know unless I know the website's owner but that will make it a staged test haha.
+
+To solve this issue, I will use [The Web Archive](https://archive.org/)'s [Wayback Machine](https://web.archive.org/). Here is a copy of [StackOverFlow's website in 2010](https://web.archive.org/web/20100102003420/http://stackoverflow.com/), pretty old huh?Let's test if the automatch feature can extract the same button in the old design from 2010 and the current design using the same selector :)
+
+If I want to extract the Questions button from the old design I can use a selector like this `#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a` This selector is too specific because it was generated by Google Chrome.
+Now let's test the same selector in both versions
+```python
+>> from scrapling import Fetcher
+>> selector = '#hmenus > div:nth-child(1) > ul > li:nth-child(1) > a'
+>> old_url = "https://web.archive.org/web/20100102003420/http://stackoverflow.com/"
+>> new_url = "https://stackoverflow.com/"
+>>
+>> page = Fetcher(automatch_domain='stackoverflow.com').get(old_url, timeout=30).adaptor
+>> element1 = page.css_first(selector, auto_save=True)
+>>
+>> # Same selector but used in the updated website
+>> page = Fetcher(automatch_domain="stackoverflow.com").get(new_url).adaptor
+>> element2 = page.css_first(selector, auto_match=True)
+>>
+>> if element1.text == element2.text:
+... print('Scrapling found the same element in the old design and the new design!')
+'Scrapling found the same element in the old design and the new design!'
+```
+Note that I used a new argument called `automatch_domain`, this is because for Scrapling these are two different URLs, not the website so it isolates their data. To tell Scrapling they are the same website, we then pass the domain we want to use for saving auto-match data for them both so Scrapling doesn't isolate them.
+
+In a real-world scenario, the code will be the same except it will use the same URL for both requests so you won't need to use the `automatch_domain` argument. This is the closest example I can give to real-world cases so I hope it didn't confuse you :)
**Notes:**
-1. Passing the `auto_save` argument without setting `auto_match` to `True` while initializing the Adaptor object will only result in ignoring the `auto_save` argument value and the following warning message
+1. For the two examples above I used one time the `Adaptor` class and the second time the `Fetcher` class just to show you that you can create the `Adaptor` object by yourself if you have the source or fetch the source using any `Fetcher` class then it will create the `Adaptor` object for you on the `.adaptor` property.
+2. Passing the `auto_save` argument with the `auto_match` argument set to `False` while initializing the Adaptor/Fetcher object will only result in ignoring the `auto_save` argument value and the following warning message
```text
Argument `auto_save` will be ignored because `auto_match` wasn't enabled on initialization. Check docs for more info.
```
This behavior is purely for performance reasons so the database gets created/connected only when you are planning to use the auto-matching features. Same case with the `auto_match` argument.
-2. The `auto_match` parameter works only for `Adaptor` instances not `Adaptors` so if you do something like this you will get an error
+3. The `auto_match` parameter works only for `Adaptor` instances not `Adaptors` so if you do something like this you will get an error
```python
page.css('body').css('#p1', auto_match=True)
```
because you can't auto-match a whole list, you have to be specific and do something like
```python
- page.css('body')[0].css('#p1', auto_match=True)
+ page.css_first('body').css('#p1', auto_match=True)
```
+### Find elements by filters
+Inspired by BeautifulSoup's `find_all` function you can find elements by using `find_all`/`find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
+
+* To be more specific:
+ * Any string passed is considered a tag name
+ * Any iterable passed like List/Tuple/Set is considered an iterable of tag names.
+ * Any dictionary is considered a mapping of HTML element(s) attribute names and attribute values.
+ * Any regex patterns passed are used as filters
+ * Any functions passed are used as filters
+ * Any keyword argument passed is considered as an HTML element attribute with its value.
+
+So the way it works is after collecting all passed arguments and keywords, each filter passes its results to the following filter in a waterfall-like filtering system.
+
It filters all elements in the current page/element in the following order:
+
+1. All elements with the passed tag name(s).
+2. All elements that match all passed attribute(s).
+3. All elements that match all passed regex patterns.
+4. All elements that fulfill all passed function(s).
+
+Note: The filtering process always starts from the first filter it finds in the filtering order above so if no tag name(s) are passed but attributes are passed, the process starts from that layer and so on. **But the order in which you pass the arguments doesn't matter.**
+
+Examples to clear any confusion :)
+
+```python
+>> from scrapling import Fetcher
+>> page = Fetcher().get('https://quotes.toscrape.com/').adaptor
+# Find all elements with tag name `div`.
+>> page.find_all('div')
+[