Karim shoair
328 lines
17 KiB
Markdown
328 lines
17 KiB
Markdown
We will start by quickly reviewing the parsing capabilities. Then, we will fetch websites with custom browsers, make requests, and parse the response.
|
|
|
|
Here's an HTML document generated by ChatGPT we will be using as an example throughout this page:
|
|
```html
|
|
<html>
|
|
<head>
|
|
<title>Complex Web Page</title>
|
|
<style>
|
|
.hidden { display: none; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<nav>
|
|
<ul>
|
|
<li> <a href="#home">Home</a> </li>
|
|
<li> <a href="#about">About</a> </li>
|
|
<li> <a href="#contact">Contact</a> </li>
|
|
</ul>
|
|
</nav>
|
|
</header>
|
|
<main>
|
|
<section id="products" schema='{"jsonable": "data"}'>
|
|
<h2>Products</h2>
|
|
<div class="product-list">
|
|
<article class="product" data-id="1">
|
|
<h3>Product 1</h3>
|
|
<p class="description">This is product 1</p>
|
|
<span class="price">$10.99</span>
|
|
<div class="hidden stock">In stock: 5</div>
|
|
</article>
|
|
|
|
<article class="product" data-id="2">
|
|
<h3>Product 2</h3>
|
|
<p class="description">This is product 2</p>
|
|
<span class="price">$20.99</span>
|
|
<div class="hidden stock">In stock: 3</div>
|
|
</article>
|
|
|
|
<article class="product" data-id="3">
|
|
<h3>Product 3</h3>
|
|
<p class="description">This is product 3</p>
|
|
<span class="price">$15.99</span>
|
|
<div class="hidden stock">Out of stock</div>
|
|
</article>
|
|
</div>
|
|
</section>
|
|
|
|
<section id="reviews">
|
|
<h2>Customer Reviews</h2>
|
|
<div class="review-list">
|
|
<div class="review" data-rating="5">
|
|
<p class="review-text">Great product!</p>
|
|
<span class="reviewer">John Doe</span>
|
|
</div>
|
|
<div class="review" data-rating="4">
|
|
<p class="review-text">Good value for money.</p>
|
|
<span class="reviewer">Jane Smith</span>
|
|
</div>
|
|
</div>
|
|
</section>
|
|
</main>
|
|
<script id="page-data" type="application/json">
|
|
{
|
|
"lastUpdated": "2024-09-22T10:30:00Z",
|
|
"totalProducts": 3
|
|
}
|
|
</script>
|
|
</body>
|
|
</html>
|
|
```
|
|
Starting with loading raw HTML above like this
|
|
```python
|
|
from scrapling.parser import Adaptor
|
|
page = Adaptor(html_doc)
|
|
page # <data='<html><head><title>Complex Web Page</tit...'>
|
|
```
|
|
Get all text content on the page recursively
|
|
```python
|
|
page.get_all_text(ignore_tags=('script', 'style'))
|
|
# 'Complex Web Page\nHome\nAbout\nContact\nProducts\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock\nCustomer Reviews\nGreat product!\nJohn Doe\nGood value for money.\nJane Smith'
|
|
```
|
|
|
|
## Finding elements
|
|
If there's an element you want to find on the page, you will! Your creativity level is the only limitation!
|
|
|
|
Finding the first HTML `section` element
|
|
```python
|
|
section_element = page.find('section')
|
|
# <data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>
|
|
```
|
|
Find all `section` elements
|
|
```python
|
|
section_elements = page.find_all('section')
|
|
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
|
|
```
|
|
Find all `section` elements whose `id` attribute value is `products`
|
|
```python
|
|
section_elements = page.find_all('section', {'id':"products"})
|
|
# Same as
|
|
section_elements = page.find_all('section', id="products")
|
|
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
|
|
```
|
|
Find all `section` elements that its `id` attribute value contains `product`
|
|
```python
|
|
section_elements = page.find_all('section', {'id*':"product"})
|
|
```
|
|
Find all `h3` elements whose text content matches this regex `Product \d`
|
|
```python
|
|
page.find_all('h3', re.compile(r'Product \d'))
|
|
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
|
|
```
|
|
Find all `h3` and `h2` elements whose text content matches regex `Product` only
|
|
```python
|
|
page.find_all(['h3', 'h2'], re.compile(r'Product'))
|
|
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>, <data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
|
|
```
|
|
Find all elements that its text content matches exactly `Products` (Whitespaces are not taken into consideration)
|
|
```python
|
|
page.find_by_text('Products', first_match=False)
|
|
# [<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
|
|
```
|
|
Or find all elements whose text content matches regex `Product \d`
|
|
```python
|
|
page.find_by_regex(r'Product \d', first_match=False)
|
|
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
|
|
```
|
|
Find all elements that are similar to the element you want
|
|
```python
|
|
target_element = page.find_by_regex(r'Product \d', first_match=True)
|
|
# <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>
|
|
target_element.find_similar()
|
|
# [<data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
|
|
```
|
|
Find the first element that matches a CSS selector
|
|
```python
|
|
page.css_first('.product-list [data-id="1"]')
|
|
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
|
|
```
|
|
Find all elements that match a CSS selector
|
|
```python
|
|
page.css('.product-list article')
|
|
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
|
|
```
|
|
Find the first element that matches an XPath selector
|
|
```python
|
|
page.xpath_first("//*[@id='products']/div/article")
|
|
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
|
|
```
|
|
Find all elements that match an XPath selector
|
|
```python
|
|
page.xpath("//*[@id='products']/div/article")
|
|
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
|
|
```
|
|
|
|
With this, we just scratched the surface of these functions; more advanced options with these selection methods are shown later.
|
|
## Accessing elements' data
|
|
It's as simple as
|
|
```python
|
|
>>> section_element.tag
|
|
'section'
|
|
>>> print(section_element.attrib)
|
|
{'id': 'products', 'schema': '{"jsonable": "data"}'}
|
|
>>> section_element.attrib['schema'].json() # If an attribute value can be converted to json, then use `.json()` to convert it
|
|
{'jsonable': 'data'}
|
|
>>> section_element.text # Direct text content
|
|
''
|
|
>>> section_element.get_all_text() # All text content recursively
|
|
'Products\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
|
|
>>> section_element.html_content # The HTML content of the element
|
|
'<section id="products" schema=\'{"jsonable": "data"}\'><h2>Products</h2>\n <div class="product-list">\n <article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article><article class="product" data-id="2"><h3>Product 2</h3>\n <p class="description">This is product 2</p>\n <span class="price">$20.99</span>\n <div class="hidden stock">In stock: 3</div>\n </article><article class="product" data-id="3"><h3>Product 3</h3>\n <p class="description">This is product 3</p>\n <span class="price">$15.99</span>\n <div class="hidden stock">Out of stock</div>\n </article></div>\n </section>'
|
|
>>> print(section_element.prettify()) # The prettified version
|
|
'''
|
|
<section id="products" schema='{"jsonable": "data"}'><h2>Products</h2>
|
|
<div class="product-list">
|
|
<article class="product" data-id="1"><h3>Product 1</h3>
|
|
<p class="description">This is product 1</p>
|
|
<span class="price">$10.99</span>
|
|
<div class="hidden stock">In stock: 5</div>
|
|
</article><article class="product" data-id="2"><h3>Product 2</h3>
|
|
<p class="description">This is product 2</p>
|
|
<span class="price">$20.99</span>
|
|
<div class="hidden stock">In stock: 3</div>
|
|
</article><article class="product" data-id="3"><h3>Product 3</h3>
|
|
<p class="description">This is product 3</p>
|
|
<span class="price">$15.99</span>
|
|
<div class="hidden stock">Out of stock</div>
|
|
</article>
|
|
</div>
|
|
</section>
|
|
'''
|
|
>>> section_element.path # All the ancestors in the DOM tree of this element
|
|
[<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>,
|
|
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>,
|
|
<data='<html><head><title>Complex Web Page</tit...'>]
|
|
>>> section_element.generate_css_selector
|
|
'#products'
|
|
>>> section_element.generate_full_css_selector
|
|
'body > main > #products > #products'
|
|
>>> section_element.generate_xpath_selector
|
|
"//*[@id='products']"
|
|
>>> section_element.generate_full_xpath_selector
|
|
"//body/main/*[@id='products']"
|
|
```
|
|
|
|
## Navigation
|
|
Using the elements we found above
|
|
|
|
```python
|
|
>>> section_element.parent
|
|
<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>
|
|
>>> section_element.parent.tag
|
|
'main'
|
|
>>> section_element.parent.parent.tag
|
|
'body'
|
|
>>> section_element.children
|
|
[<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>,
|
|
<data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
|
|
>>> section_element.siblings
|
|
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
|
|
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`
|
|
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
|
|
>>> section_element.children.css('h2::text')
|
|
['Products']
|
|
>>> page.css_first('[data-id="1"]').has_class('product')
|
|
True
|
|
```
|
|
If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element like the one below
|
|
```python
|
|
for ancestor in quote.iterancestors():
|
|
# do something with it...
|
|
```
|
|
You can search for a specific ancestor of an element that satisfies a function; all you need to do is to pass a function that takes an `Adaptor` object as an argument and return `True` if the condition satisfies or `False` otherwise like below:
|
|
```python
|
|
>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))
|
|
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>
|
|
```
|
|
|
|
## Fetching websites
|
|
Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers.
|
|
|
|
A fetcher is made for every use case.
|
|
|
|
### HTTP Requests
|
|
For simple HTTP requests, there's a `Fetcher` class that can be imported as below:
|
|
```python
|
|
from scrapling.fetchers import Fetcher
|
|
```
|
|
But that's class, so you will need to create an instance of the Fetcher first like this:
|
|
```python
|
|
from scrapling.fetchers import Fetcher
|
|
fetcher = Fetcher()
|
|
page = fetcher.get('https://httpbin.org/get')
|
|
```
|
|
This is intended, and you will find it with all fetchers because there are settings you can pass to `Fetcher()` initialization, but more on this later.
|
|
|
|
If you are going to use the default settings anyway, you can do this instead for a cleaner approach:
|
|
```python
|
|
from scrapling.fetchers import Fetcher
|
|
page = Fetcher.get('https://httpbin.org/get')
|
|
```
|
|
With that out of the way, here's how to do all HTTP methods:
|
|
```python
|
|
>>> from scrapling.fetchers import Fetcher
|
|
>>> page = Fetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
|
|
>>> page = Fetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
|
|
>>> page = Fetcher.put('https://httpbin.org/put', data={'key': 'value'})
|
|
>>> page = Fetcher.delete('https://httpbin.org/delete')
|
|
```
|
|
For Async requests, you will just replace the import like below:
|
|
```python
|
|
>>> from scrapling.fetchers import AsyncFetcher
|
|
>>> page = await AsyncFetcher.get('https://httpbin.org/get', stealthy_headers=True, follow_redirects=True)
|
|
>>> page = await AsyncFetcher.post('https://httpbin.org/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
|
|
>>> page = await AsyncFetcher.put('https://httpbin.org/put', data={'key': 'value'})
|
|
>>> page = await AsyncFetcher.delete('https://httpbin.org/delete')
|
|
```
|
|
|
|
> Note: You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from Google's search of this URL's domain. It's enabled by default.
|
|
|
|
This is just the tip of this fetcher; check the full page from [here](fetching/static.md)
|
|
|
|
### Dynamic loading
|
|
We have you covered if you deal with dynamic websites like most today!
|
|
|
|
The `PlayWrightFetcher` class provides many options to fetch/load websites' pages through browsers.
|
|
```python
|
|
>>> from scrapling.fetchers import PlayWrightFetcher
|
|
>>> page = PlayWrightFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
|
|
>>> page.css_first("#search a::attr(href)")
|
|
'https://github.com/D4Vinci/Scrapling'
|
|
>>> # The async version of fetch
|
|
>>> page = await PlayWrightFetcher.async_fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True)
|
|
>>> page.css_first("#search a::attr(href)")
|
|
'https://github.com/D4Vinci/Scrapling'
|
|
```
|
|
It's named like that because it's built on top of [Playwright](https://playwright.dev/python/), and it currently provides 4 main run options that can be mixed as you want:
|
|
|
|
- Vanilla Playwright without any modifications other than the ones you chose.
|
|
- Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode but bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/). Check out the `StealthyFetcher` class below for more advanced stealth mode.
|
|
- Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
|
|
- [NSTBrowser](https://app.nstbrowser.io/r/1vO5e5)'s [docker browserless](https://hub.docker.com/r/nstbrowser/browserless) option by passing the CDP URL and enabling `nstbrowser_mode` option.
|
|
|
|
> Note: All requests done by this fetcher are waited by default for all javascript to be fully loaded and executed. In detail, it waits for the `load` and `domcontentloaded` load states to be reached; you can make it wait for the `networkidle` load state by passing 'network_idle=True', as you will see later.
|
|
|
|
Again, this is just the tip of this fetcher. Check out the full page from [here](fetching/dynamic.md) for all details and the complete list of arguments.
|
|
|
|
### Dynamic anti-protection loading
|
|
We also have you covered if you deal with dynamic websites with annoying anti-protections!
|
|
|
|
The `StealthyFetcher` class uses a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), bypassing most anti-bot protections by default. Scrapling adds extra layers of flavors and configurations to further increase performance and undetectability.
|
|
```python
|
|
>>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection') # Running headless by default
|
|
>>> page.status == 200
|
|
True
|
|
>>> page = StealthyFetcher().fetch('https://www.browserscan.net/bot-detection', humanize=True, os_randomize=True) # and the rest of arguments...
|
|
>>> # The async version of fetch
|
|
>>> page = await StealthyFetcher().async_fetch('https://www.browserscan.net/bot-detection')
|
|
>>> page.status == 200
|
|
True
|
|
```
|
|
> Note: All requests done by this fetcher are waited by default for all javascript to be fully loaded and executed. In detail, it waits for the `load` and `domcontentloaded` load states to be reached; you can make it wait for the `networkidle` load state by passing 'network_idle=True', as you will see later.
|
|
|
|
Again, this is just the tip of this fetcher. Check out the full page from [here](fetching/dynamic.md) for all details and the complete list of arguments.
|
|
|
|
---
|
|
|
|
That's Scrapling at a glance. If you want to learn more about it, continue to the next section. |