First version of Scrapling full documentation
This commit is contained in:
+107
-2
@@ -1,2 +1,107 @@
|
||||
# This section is still under work but any help is highly appreciated
|
||||
## I will try to make full detailed documentation with Sphinx ASAP.
|
||||
# Scrapling
|
||||
|
||||
Scrapling is an Undetectable, high-performance, intelligent Web scraping library for Python 3 to make Web Scraping easy!
|
||||
|
||||
Scrapling isn't only about making undetectable requests or fetching pages under the radar!
|
||||
|
||||
It has its own parser that adapts to website changes and provides many element selection/querying options other than traditional selectors, powerful DOM traversal API, and many other features while significantly outperforming popular parsing alternatives.
|
||||
|
||||
Scrapling is built from the ground up by Web scraping experts for beginners and experts. The goal is to provide powerful features while maintaining simplicity and minimal boilerplate code.
|
||||
|
||||
```python
|
||||
>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, PlayWrightFetcher
|
||||
>> StealthyFetcher.auto_match = True
|
||||
# Fetch websites' source under the radar!
|
||||
>> page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True)
|
||||
>> print(page.status)
|
||||
200
|
||||
>> products = page.css('.product', auto_save=True) # Scrape data that survives website design changes!
|
||||
>> # Later, if the website structure changes, pass `auto_match=True`
|
||||
>> products = page.css('.product', auto_match=True) # and Scrapling still finds them!
|
||||
```
|
||||
## Key Features
|
||||
### Fetch websites as you prefer with async support
|
||||
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class.
|
||||
- **Dynamic Loading & Automation**: Fetch dynamic websites with the `PlayWrightFetcher` class through your real browser, Scrapling's stealth mode, Playwright's Chromium browser, or [NSTbrowser](https://app.nstbrowser.io/r/1vO5e5)'s browserless!
|
||||
- **Anti-bot Protections Bypass**: Easily bypass protections with the `StealthyFetcher` and `PlayWrightFetcher` classes.
|
||||
|
||||
### Easy Scraping
|
||||
- **Smart Element Tracking**: Relocate elements after website changes using an intelligent similarity system and integrated storage.
|
||||
- **Flexible Selection**: CSS selectors, XPath selectors, filters-based search, text search, regex search, and more.
|
||||
- **Find Similar Elements**: Automatically locate elements similar to the element you found!
|
||||
- **Smart Content Scraping**: Extract data from multiple websites without specific selectors using Scrapling powerful features.
|
||||
|
||||
### High Performance
|
||||
- **Lightning Fast**: Built from the ground up with performance in mind, outperforming most popular Python scraping libraries.
|
||||
- **Memory Efficient**: Optimized data structures for minimal memory footprint.
|
||||
- **Fast JSON serialization**: 10x faster than standard library.
|
||||
|
||||
### Developer Friendly
|
||||
- **Powerful Navigation API**: Easy DOM traversal in all directions.
|
||||
- **Rich Text Processing**: All strings have built-in regex, cleaning methods, and more. All elements' attributes are optimized dictionaries that use less memory than standard dictionaries with added methods.
|
||||
- **Auto Selectors Generation**: Generate robust short and full CSS/XPath selectors for any element.
|
||||
- **Familiar API**: Similar to Scrapy/BeautifulSoup and the same CSS pseudo-elements used in Scrapy.
|
||||
- **Type hints**: Complete type/doc-strings coverage for future-proofing and best autocompletion support.
|
||||
|
||||
## Star History
|
||||
Scrapling’s GitHub stars have grown steadily since its release (see chart below).
|
||||
|
||||
<div id="chartContainer">
|
||||
<a href="https://github.com/D4Vinci/Scrapling">
|
||||
<img id="chartImage" alt="Star History Chart" src="https://api.star-history.com/svg?repos=D4Vinci/Scrapling&type=Date" height="400"/>
|
||||
</a>
|
||||
</div>
|
||||
|
||||
<script>
|
||||
const observer = new MutationObserver((mutations) => {
|
||||
mutations.forEach((mutation) => {
|
||||
if (mutation.attributeName === 'data-md-color-media') {
|
||||
const colorMedia = document.body.getAttribute('data-md-color-media');
|
||||
const isDarkScheme = document.body.getAttribute('data-md-color-scheme') === 'slate';
|
||||
const chartImg = document.querySelector('#chartImage');
|
||||
const baseUrl = 'https://api.star-history.com/svg?repos=D4Vinci/Scrapling&type=Date';
|
||||
|
||||
if (colorMedia === '(prefers-color-scheme)' ? isDarkScheme : colorMedia.includes('dark')) {
|
||||
chartImg.src = `${baseUrl}&theme=dark`;
|
||||
} else {
|
||||
chartImg.src = baseUrl;
|
||||
}
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
observer.observe(document.body, {
|
||||
attributes: true,
|
||||
attributeFilter: ['data-md-color-media', 'data-md-color-scheme']
|
||||
});
|
||||
</script>
|
||||
|
||||
## Installation
|
||||
Scrapling is a breeze to get started with!<br/>Starting from version 0.2.9, we require at least Python 3.9 to work.
|
||||
|
||||
Run this command to install it with Python's pip.
|
||||
```bash
|
||||
pip3 install scrapling
|
||||
```
|
||||
You are ready if you plan to use the parser only (the `Adaptor` class).
|
||||
|
||||
But if you are going to make requests or fetch pages with Scrapling, then run this command to install browsers' dependencies needed to use the Fetchers
|
||||
```bash
|
||||
scrapling install
|
||||
```
|
||||
If you have any installation issues, please open an [issue](https://github.com/D4Vinci/Scrapling/issues/new/choose).
|
||||
|
||||
## How the documentation is organized
|
||||
Scrapling has a lot of documentation, so we try to follow a guideline called the [Diátaxis documentation framework](https://diataxis.fr/).
|
||||
|
||||
## Support
|
||||
|
||||
If you like Scrapling and want to support its development:
|
||||
|
||||
- ⭐ Star the [GitHub repository](https://github.com/D4Vinci/Scrapling)
|
||||
- 💝 Consider [sponsoring the project or buying me a coffe](donate.md) :wink:
|
||||
- 🐛 Report bugs and suggest features through [GitHub Issues](https://github.com/D4Vinci/Scrapling/issues)
|
||||
|
||||
## License
|
||||
|
||||
This project is licensed under BSD-3 License. See the [LICENSE](https://github.com/D4Vinci/Scrapling/blob/main/LICENSE) file for details.
|
||||
Reference in New Issue
Block a user