freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.3.13

Browse Source
This commit is contained in:
Karim shoair
2026-01-01 22:07:00 +02:00
committed by GitHub
parent 2e5152f790 b5fdb71b52
commit 6ad2ecb9f2
66 changed files with 1257 additions and 1686 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/tests.yml
+2 -30
View File
@@ -58,7 +58,7 @@ jobs:
- name: Install all browsers dependencies
run: |
python3 -m pip install --upgrade pip
python3 -m pip install playwright==1.56.0 patchright==1.56.0 camoufox>=0.4.11
python3 -m pip install playwright==1.56.0 patchright==1.56.0
- name: Get Playwright version
id: playwright-version
@@ -88,35 +88,7 @@ jobs:
else
echo "Skipping install - using cached Playwright browsers"
fi
python3 -m playwright install-deps chromium firefox
- name: Get Camoufox version
id: camoufox-version
run: |
CAMOUFOX_VERSION=$(python3 -c "import importlib.metadata; print(importlib.metadata.version('camoufox'))")
echo "version=$CAMOUFOX_VERSION" >> $GITHUB_OUTPUT
echo "Camoufox version: $CAMOUFOX_VERSION"
- name: Retrieve Camoufox browser from cache if any
id: camoufox-cache
uses: actions/cache@v4
with:
path: |
~/.cache/camoufox
~/Library/Caches/camoufox
key: ${{ runner.os }}-camoufox-${{ steps.camoufox-version.outputs.version }}-v1
restore-keys: |
${{ runner.os }}-camoufox-${{ steps.camoufox-version.outputs.version }}-
${{ runner.os }}-camoufox-
- name: Install Camoufox browser
run: |
echo "Cache hit: ${{ steps.camoufox-cache.outputs.cache-hit }}"
if [ "${{ steps.camoufox-cache.outputs.cache-hit }}" != "true" ]; then
python3 -m camoufox fetch --browserforge
else
echo "Skipping fetch - using cached Camoufox browser"
fi
python3 -m playwright install-deps chromium
# Cache tox environments
- name: Cache tox environments
.readthedocs.yaml
+2
View File
@@ -7,6 +7,8 @@ version: 2
# Set the OS, Python version, and other tools you might need
build:
os: ubuntu-24.04
apt_packages:
- pngquant
tools:
python: "3.13"
CONTRIBUTING.md
+4 -4
View File
@@ -4,7 +4,7 @@ Thank you for your interest in contributing to Scrapling!
Everybody is invited and welcome to contribute to Scrapling.
Minor changes have a better chance of being included promptly. Adding unit tests for new features or test cases for bugs you've fixed helps us ensure that the Pull Request (PR) is acceptable.
Minor changes are more likely to be included promptly. Adding unit tests for new features or test cases for bugs you've fixed helps us ensure that the Pull Request (PR) is acceptable.
There are many ways to contribute to Scrapling. Here are some of them:
@@ -18,7 +18,7 @@ There are many ways to contribute to Scrapling. Here are some of them:
If you have decided to make a contribution to Scrapling, but you do not know what to contribute, here are some ways to find pending work:
- Check out the [contribution](https://github.com/D4Vinci/Scrapling/contribute) GitHub page, which lists open issues tagged as good first issue. These issues provide a good starting point.
- Check out the [contribution](https://github.com/D4Vinci/Scrapling/contribute) GitHub page, which lists open issues tagged as `good first issue`. These issues provide a good starting point.
- There are also the [help wanted](https://github.com/D4Vinci/Scrapling/issues?q=is%3Aissue%20label%3A%22help%20wanted%22%20state%3Aopen) issues, but know that some may require familiarity with the Scrapling code base first. You can also target any other issue, provided it is not tagged as `invalid`, `wontfix`, or similar tags.
- If you enjoy writing automated tests, you can work on increasing our test coverage. Currently, the test coverage is around 90–92%.
- Join the [Discord community](https://discord.gg/EMgGbDceNQ) and ask questions in the `#help` channel.
@@ -38,7 +38,7 @@ Please follow these coding conventions as we do when writing code for Scrapling:
| `refactor:` | Code refactoring |
| `chore:` | Maintenance tasks |
Then include the details of the change in the body/description of the commit message.
Then include the details of the change in the commit message body/description.
Example:
```
@@ -99,7 +99,7 @@ pytest --cov=scrapling tests/
To ensure that your PR gets accepted, please make sure that your PR is based on the latest changes from the dev branch and that it satisfies the following requirements:
- The PR should be made against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling. Any PR made against the main branch will be rejected.
- The code should be passing all available tests. We are using tox with GitHub's CI to run the current tests on all supported Python versions with every commit.
- The code should be passing all available tests. We use tox with GitHub's CI to run the current tests on all supported Python versions for every code-related commit.
- The code should be passing all code quality checks we mentioned above. We are using GitHub's CI to enforce the code style checks performed by pre-commit. If you were using the pre-commit hooks we discussed above, you should not see any issues when committing your changes.
- Make your changes, keep the code clean with an explanation of any part that might be vague, and remember to create a separate virtual environment for this project.
- If you are adding a new feature, please add tests for it.
Dockerfile
+1 -2
View File
@@ -23,9 +23,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=cache,target=/var/cache/apt \
--mount=type=cache,target=/var/lib/apt \
apt-get update && \
uv run playwright install-deps chromium firefox && \
uv run playwright install-deps chromium && \
uv run playwright install chromium && \
uv run camoufox fetch --browserforge && \
uv sync --all-extras --compile-bytecode && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
docs/README.md
+12 -13
View File
@@ -90,8 +90,8 @@ Built for the modern Web, Scrapling features **its own rapid parsing engine** an
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP3.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, real Chrome, and custom stealth mode.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` using a modified version of Firefox and fingerprint spoofing. Can bypass all types of Cloudflare's Turnstile and Interstitial with automation easily.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium and Google's Chrome.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes.
@@ -99,7 +99,7 @@ Built for the modern Web, Scrapling features **its own rapid parsing engine** an
- 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
- 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
@@ -153,7 +153,7 @@ data = page.css('.quote .text::text')
```
> [!NOTE]
> There's a wonderful guide to get you started quickly with Scraping [here](https://substack.thewebscraping.club/p/scrapling-hands-on-guide) written by The Web Scraping Club. In case you find it easier to get you started than the [documentation website](https://scrapling.readthedocs.io/en/latest/).
> There's a wonderful guide to get you started quickly with Scrapling [here](https://substack.thewebscraping.club/p/scrapling-hands-on-guide) written by The Web Scraping Club. In case you find it easier to get you started than the [documentation website](https://scrapling.readthedocs.io/en/latest/).
### Advanced Parsing & Navigation
```python
@@ -222,13 +222,12 @@ Scrapling v0.3 includes a powerful command-line interface:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Launch the interactive Web Scraping shell
```bash
# Launch interactive Web Scraping shell
scrapling shell
# Extract pages to a file directly without programming (Extracts the content inside `body` tag by default)
# If the output file ends with `.txt`, then the text content of the target will be extracted.
# If ended with `.md`, it will be a markdown representation of the HTML content, and `.html` will be the HTML content right away.
```
Extract pages to a file directly without programming (Extracts the content inside the `body` tag by default). If the output file ends with `.txt`, then the text content of the target will be extracted. If it ends in `.md`, it will be a Markdown representation of the HTML content; if it ends in `.html`, it will be the HTML content itself.
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -240,7 +239,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Performance Benchmarks
Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 have delivered exceptional performance improvements across all operations.
Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 have delivered exceptional performance improvements across all operations. The following benchmarks compare Scrapling's parser with other popular libraries.
### Text Extraction Speed Test (5000 nested elements)
@@ -280,14 +279,14 @@ Starting with v0.3.2, this installation only includes the parser engine and its
### Optional Dependencies
1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies and then install their browser dependencies with
1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:
```bash
pip install "scrapling[fetchers]"
scrapling install
```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
2. Extra features:
- Install the MCP server feature:
@@ -313,7 +312,7 @@ Or download it from the GitHub registry:
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
This image is automatically built and pushed through GitHub actions on the repository's main branch.
This image is automatically built and pushed using GitHub Actions and the repository's main branch.
## Contributing
docs/README_AR.md
+11 -12
View File
@@ -87,8 +87,8 @@ Scrapling ليست مجرد مكتبة أخرى لاستخراج بيانات ا
### جلب متقدم للمواقع مع دعم الجلسات
- **طلبات HTTP**: طلبات HTTP سريعة وخفية مع فئة `Fetcher`. يمكنها تقليد بصمة TLS للمتصفح والرؤوس واستخدام HTTP3.
- **التحميل الديناميكي**: جلب المواقع الديناميكية مع أتمتة كاملة للمتصفح من خلال فئة `DynamicFetcher` التي تدعم Chromium من Playwright، وChrome الحقيقي، ووضع التخفي المخصص.
- **تجاوز مكافحة الروبوتات**: قدرات تخفي متقدمة مع `StealthyFetcher` باستخدام نسخة معدلة من Firefox وانتحال البصمات. يمكنه تجاوز جميع أنواع Turnstile وInterstitial من Cloudflare بسهولة بالأتمتة.
- **التحميل الديناميكي**: جلب المواقع الديناميكية مع أتمتة كاملة للمتصفح من خلال فئة `DynamicFetcher` التي تدعم Chromium من Playwright و Google Chrome.
- **تجاوز مكافحة الروبوتات**: قدرات تخفي متقدمة مع `StealthyFetcher` وانتحال البصمات. يمكنه تجاوز جميع أنواع Turnstile/Interstitial من Cloudflare بسهولة بالأتمتة.
- **إدارة الجلسات**: دعم الجلسات المستمرة مع فئات `FetcherSession` و`StealthySession` و`DynamicSession` لإدارة ملفات تعريف الارتباط والحالة عبر الطلبات.
- **دعم Async**: دعم async كامل عبر جميع الجوالب وفئات الجلسات async المخصصة.
@@ -96,7 +96,7 @@ Scrapling ليست مجرد مكتبة أخرى لاستخراج بيانات ا
- 🔄 **تتبع العناصر الذكي**: إعادة تحديد موقع العناصر بعد تغييرات الموقع باستخدام خوارزميات التشابه الذكية.
- 🎯 **الاختيار المرن الذكي**: محددات CSS، محددات XPath، البحث القائم على الفلاتر، البحث النصي، البحث بالتعبيرات العادية والمزيد.
- 🔍 **البحث عن عناصر مشابهة**: تحديد العناصر المشابهة للعناصر الموجودة تلقائياً.
- 🤖 **خادم MCP للاستخدام مع الذكاء الاصطناعي**: خادم MCP مدمج لاستخراج بيانات الويب بمساعدة الذكاء الاصطناعي واستخراج البيانات. يتميز خادم MCP بقدرات مخصصة قوية تستخدم Scrapling لاستخراج المحتوى المستهدف قبل تمريره إلى الذكاء الاصطناعي (Claude/Cursor/إلخ)، وبالتالي تسريع العمليات وتقليل التكاليف عن طريق تقليل استخدام الرموز. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🤖 **خادم MCP للاستخدام مع الذكاء الاصطناعي**: خادم MCP مدمج لاستخراج بيانات الويب بمساعدة الذكاء الاصطناعي واستخراج البيانات. يتميز خادم MCP بقدرات قوية مخصصة تستفيد من Scrapling لاستخراج المحتوى المستهدف قبل تمريره إلى الذكاء الاصطناعي (Claude/Cursor/إلخ)، وبالتالي تسريع العمليات وتقليل التكاليف عن طريق تقليل استخدام الرموز. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### بنية عالية الأداء ومختبرة في المعارك
- 🚀 **سريع كالبرق**: أداء محسّن يتفوق على معظم مكتبات استخراج Python.
@@ -212,13 +212,12 @@ async with AsyncStealthySession(max_pages=2) as session:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
تشغيل غلاف استخراج الويب التفاعلي
```bash
# تشغيل غلاف استخراج الويب التفاعلي
scrapling shell
# استخراج الصفحات إلى ملف مباشرة دون برمجة (يستخرج المحتوى داخل وسم `body` افتراضياً)
# إذا انتهى ملف الإخراج بـ `.txt`، فسيتم استخراج محتوى النص للهدف.
# إذا انتهى بـ `.md`، فسيكون تمثيل markdown لمحتوى HTML، و`.html` سيكون محتوى HTML مباشرة.
```
استخراج الصفحات إلى ملف مباشرة دون برمجة (يستخرج المحتوى داخل وسم `body` افتراضياً). إذا انتهى ملف الإخراج بـ `.txt`، فسيتم استخراج محتوى النص للهدف. إذا انتهى بـ `.md`، فسيكون تمثيل Markdown لمحتوى HTML؛ إذا انتهى بـ `.html`، فسيكون محتوى HTML نفسه.
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # جميع العناصر المطابقة لمحدد CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## معايير الأداء
Scrapling ليس قوياً فقط - إنه أيضاً سريع بشكل مذهل، والتحديثات منذ الإصدار 0.3 قدمت تحسينات أداء استثنائية عبر جميع العمليات.
Scrapling ليس قوياً فقط - إنه أيضاً سريع بشكل مذهل، والتحديثات منذ الإصدار 0.3 قدمت تحسينات أداء استثنائية عبر جميع العمليات. تقارن المعايير التالية محلل Scrapling مع المكتبات الشائعة الأخرى.
### اختبار سرعة استخراج النص (5000 عنصر متداخل)
@@ -270,14 +269,14 @@ pip install scrapling
### التبعيات الاختيارية
1. إذا كنت ستستخدم أياً من الميزات الإضافية أدناه، أو الجوالب، أو فئاتها، فأنت بحاجة إلى تثبيت تبعيات الجوالب ثم تثبيت تبعيات المتصفح الخاصة بها بـ
1. إذا كنت ستستخدم أياً من الميزات الإضافية أدناه، أو الجوالب، أو فئاتها، فستحتاج إلى تثبيت تبعيات الجوالب وتبعيات المتصفح الخاصة بها على النحو التالي:
```bash
pip install "scrapling[fetchers]"
scrapling install
```
يقوم هذا بتنزيل جميع المتصفحات مع تبعيات النظام وتبعيات معالجة البصمات الخاصة بها.
يقوم هذا بتنزيل جميع المتصفحات، إلى جانب تبعيات النظام وتبعيات معالجة البصمات الخاصة بها.
2. ميزات إضافية:
- تثبيت ميزة خادم MCP:
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
يتم بناء هذه الصورة ودفعها تلقائياً من خلال إجراءات GitHub على الفرع الرئيسي للمستودع.
يتم بناء هذه الصورة ودفعها تلقائياً باستخدام GitHub Actions والفرع الرئيسي للمستودع.
## المساهمة
docs/README_CN.md
+11 -12
View File
@@ -87,8 +87,8 @@ Scrapling不仅仅是另一个网页抓取库。它是第一个**自适应**抓
### 支持会话的高级网站获取
- **HTTP请求**：使用`Fetcher`类进行快速和隐秘的HTTP请求。可以模拟浏览器的TLS指纹、标头并使用HTTP3。
- **动态加载**：通过`DynamicFetcher`类使用完整的浏览器自动化获取动态网站，支持Playwright的Chromium、真实Chrome和自定义隐秘模式。
- **反机器人绕过**：使用`StealthyFetcher`的高级隐秘功能，使用修改版Firefox和指纹伪装。可以轻松自动绕过所有类型的Cloudflare的Turnstile和Interstitial。
- **动态加载**：通过`DynamicFetcher`类使用完整的浏览器自动化获取动态网站，支持Playwright的Chromium和Google Chrome。
- **反机器人绕过**：使用`StealthyFetcher`的高级隐秘功能和指纹伪装。可以轻松自动绕过所有类型的Cloudflare的Turnstile/Interstitial。
- **会话管理**：使用`FetcherSession`、`StealthySession`和`DynamicSession`类持久化会话支持，用于跨请求的cookie和状态管理。
- **异步支持**：所有获取器和专用异步会话类的完整异步支持。
@@ -96,7 +96,7 @@ Scrapling不仅仅是另一个网页抓取库。它是第一个**自适应**抓
- 🔄 **智能元素跟踪**：使用智能相似性算法在网站更改后重新定位元素。
- 🎯 **智能灵活选择**：CSS选择器、XPath选择器、基于过滤器的搜索、文本搜索、正则表达式搜索等。
- 🔍 **查找相似元素**：自动定位与找到的元素相似的元素。
- 🤖 **与AI一起使用的MCP服务器**：内置MCP服务器用于AI辅助网页抓取和数据提取。MCP服务器具有自定义的强大功能，利用Scrapling在将内容传递给AI（Claude/Cursor等）之前提取目标内容，从而加快操作并通过最小化令牌使用来降低成本。（[演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE)）
- 🤖 **与AI一起使用的MCP服务器**：内置MCP服务器用于AI辅助网页抓取和数据提取。MCP服务器具有强大的自定义功能，利用Scrapling在将内容传递给AI（Claude/Cursor等）之前提取目标内容，从而加快操作并通过最小化令牌使用来降低成本。（[演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE)）
### 高性能和经过实战测试的架构
- 🚀 **闪电般快速**：优化性能超越大多数Python抓取库。
@@ -212,13 +212,12 @@ Scrapling v0.3包含强大的命令行界面：
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
启动交互式网页抓取shell
```bash
# 启动交互式网页抓取shell
scrapling shell
# 直接将页面提取到文件而无需编程（默认提取`body`标签内的内容）
# 如果输出文件以`.txt`结尾，则将提取目标的文本内容。
# 如果以`.md`结尾，它将是HTML内容的markdown表示，`.html`将直接是HTML内容。
```
直接将页面提取到文件而无需编程（默认提取`body`标签内的内容）。如果输出文件以`.txt`结尾，则将提取目标的文本内容。如果以`.md`结尾，它将是HTML内容的Markdown表示；如果以`.html`结尾，它将是HTML内容本身。
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # 所有匹配CSS选择器'#fromSkipToProducts'的元素
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## 性能基准
Scrapling不仅功能强大——它还速度极快，自0.3版本以来的更新在所有操作中都提供了卓越的性能改进。
Scrapling不仅功能强大——它还速度极快，自0.3版本以来的更新在所有操作中都提供了卓越的性能改进。以下基准测试将Scrapling的解析器与其他流行库进行了比较。
### 文本提取速度测试（5000个嵌套元素）
@@ -270,14 +269,14 @@ pip install scrapling
### 可选依赖项
1. 如果您要使用以下任何额外功能、获取器或它们的类，那么您需要安装获取器的依赖项，然后使用以下命令安装它们的浏览器依赖项
1. 如果您要使用以下任何额外功能、获取器或它们的类，您将需要安装获取器的依赖项和它们的浏览器依赖项，如下所示：
```bash
pip install "scrapling[fetchers]"
scrapling install
```
这会下载所有浏览器及其系统依赖项和指纹操作依赖项。
这会下载所有浏览器，以及它们的系统依赖项和指纹操作依赖项。
2. 额外功能：
- 安装MCP服务器功能：
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
此镜像通过仓库主分支上的GitHub actions自动构建和推送。
此镜像使用GitHub Actions和仓库主分支自动构建和推送。
## 贡献
docs/README_DE.md
+11 -12
View File
@@ -87,8 +87,8 @@ Für das moderne Web entwickelt, bietet Scrapling **seine eigene schnelle Parsin
### Erweiterte Website-Abruf mit Sitzungsunterstützung
- **HTTP-Anfragen**: Schnelle und heimliche HTTP-Anfragen mit der `Fetcher`-Klasse. Kann Browser-TLS-Fingerabdrücke, Header imitieren und HTTP3 verwenden.
- **Dynamisches Laden**: Abrufen dynamischer Websites mit vollständiger Browser-Automatisierung über die `DynamicFetcher`-Klasse, die Playwrights Chromium, echtes Chrome und benutzerdefinierten Stealth-Modus unterstützt.
- **Anti-Bot-Umgehung**: Erweiterte Stealth-Fähigkeiten mit `StealthyFetcher` unter Verwendung einer modifizierten Firefox-Version und Fingerabdruck-Spoofing. Kann alle Arten von Cloudflares Turnstile und Interstitial einfach mit Automatisierung umgehen.
- **Dynamisches Laden**: Abrufen dynamischer Websites mit vollständiger Browser-Automatisierung über die `DynamicFetcher`-Klasse, die Playwrights Chromium und Google Chrome unterstützt.
- **Anti-Bot-Umgehung**: Erweiterte Stealth-Fähigkeiten mit `StealthyFetcher` und Fingerabdruck-Spoofing. Kann alle Arten von Cloudflares Turnstile/Interstitial einfach mit Automatisierung umgehen.
- **Sitzungsverwaltung**: Persistente Sitzungsunterstützung mit den Klassen `FetcherSession`, `StealthySession` und `DynamicSession` für Cookie- und Zustandsverwaltung über Anfragen hinweg.
- **Async-Unterstützung**: Vollständige Async-Unterstützung über alle Fetcher und dedizierte Async-Sitzungsklassen hinweg.
@@ -96,7 +96,7 @@ Für das moderne Web entwickelt, bietet Scrapling **seine eigene schnelle Parsin
- 🔄 **Intelligente Element-Verfolgung**: Elemente nach Website-Änderungen mit intelligenten Ähnlichkeitsalgorithmen neu lokalisieren.
- 🎯 **Intelligente flexible Auswahl**: CSS-Selektoren, XPath-Selektoren, filterbasierte Suche, Textsuche, Regex-Suche und mehr.
- 🔍 **Ähnliche Elemente finden**: Elemente, die gefundenen Elementen ähnlich sind, automatisch lokalisieren.
- 🤖 **MCP-Server für die Verwendung mit KI**: Integrierter MCP-Server für KI-unterstütztes Web Scraping und Datenextraktion. Der MCP-Server verfügt über benutzerdefinierte, leistungsstarke Funktionen, die Scrapling nutzen, um gezielten Inhalt zu extrahieren, bevor er an die KI (Claude/Cursor/etc.) übergeben wird, wodurch Vorgänge beschleunigt und Kosten durch Minimierung der Token-Nutzung gesenkt werden. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🤖 **MCP-Server für die Verwendung mit KI**: Integrierter MCP-Server für KI-unterstütztes Web Scraping und Datenextraktion. Der MCP-Server verfügt über leistungsstarke, benutzerdefinierte Funktionen, die Scrapling nutzen, um gezielten Inhalt zu extrahieren, bevor er an die KI (Claude/Cursor/etc.) übergeben wird, wodurch Vorgänge beschleunigt und Kosten durch Minimierung der Token-Nutzung gesenkt werden. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### Hochleistungs- und praxiserprobte Architektur
- 🚀 **Blitzschnell**: Optimierte Leistung, die die meisten Python-Scraping-Bibliotheken übertrifft.
@@ -212,13 +212,12 @@ Scrapling v0.3 enthält eine leistungsstarke Befehlszeilenschnittstelle:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Interaktive Web-Scraping-Shell starten
```bash
# Interaktive Web-Scraping-Shell starten
scrapling shell
# Seiten direkt ohne Programmierung in eine Datei extrahieren (Extrahiert standardmäßig den Inhalt im `body`-Tag)
# Wenn die Ausgabedatei mit `.txt` endet, wird der Textinhalt des Ziels extrahiert.
# Wenn sie mit `.md` endet, ist es eine Markdown-Darstellung des HTML-Inhalts, und `.html` ist direkt der HTML-Inhalt.
```
Seiten direkt ohne Programmierung in eine Datei extrahieren (Extrahiert standardmäßig den Inhalt im `body`-Tag). Wenn die Ausgabedatei mit `.txt` endet, wird der Textinhalt des Ziels extrahiert. Wenn sie mit `.md` endet, ist es eine Markdown-Darstellung des HTML-Inhalts; wenn sie mit `.html` endet, ist es der HTML-Inhalt selbst.
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Alle Elemente, die dem CSS-Selektor '#fromSkipToProducts' entsprechen
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Leistungsbenchmarks
Scrapling ist nicht nur leistungsstark – es ist auch blitzschnell, und die Updates seit Version 0.3 haben außergewöhnliche Leistungsverbesserungen bei allen Operationen gebracht.
Scrapling ist nicht nur leistungsstark – es ist auch blitzschnell, und die Updates seit Version 0.3 haben außergewöhnliche Leistungsverbesserungen bei allen Operationen gebracht. Die folgenden Benchmarks vergleichen den Parser von Scrapling mit anderen beliebten Bibliotheken.
### Textextraktions-Geschwindigkeitstest (5000 verschachtelte Elemente)
@@ -270,14 +269,14 @@ Ab v0.3.2 enthält diese Installation nur die Parser-Engine und ihre Abhängigke
### Optionale Abhängigkeiten
1. Wenn Sie eine der folgenden zusätzlichen Funktionen, die Fetcher oder ihre Klassen verwenden möchten, müssen Sie die Abhängigkeiten der Fetcher installieren und dann ihre Browser-Abhängigkeiten mit
1. Wenn Sie eine der folgenden zusätzlichen Funktionen, die Fetcher oder ihre Klassen verwenden möchten, müssen Sie die Abhängigkeiten der Fetcher und ihre Browser-Abhängigkeiten wie folgt installieren:
```bash
pip install "scrapling[fetchers]"
scrapling install
```
Dies lädt alle Browser mit ihren Systemabhängigkeiten und Fingerabdruck-Manipulationsabhängigkeiten herunter.
Dies lädt alle Browser zusammen mit ihren Systemabhängigkeiten und Fingerabdruck-Manipulationsabhängigkeiten herunter.
2. Zusätzliche Funktionen:
- MCP-Server-Funktion installieren:
@@ -303,7 +302,7 @@ Oder laden Sie es aus der GitHub-Registry herunter:
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
Dieses Image wird automatisch über GitHub Actions im Hauptzweig des Repositorys erstellt und gepusht.
Dieses Image wird automatisch mit GitHub Actions und dem Hauptzweig des Repositorys erstellt und gepusht.
## Beitragen
docs/README_ES.md
+11 -12
View File
@@ -87,8 +87,8 @@ Construido para la Web moderna, Scrapling presenta **su propio motor de análisi
### Obtención Avanzada de Sitios Web con Soporte de Sesión
- **Solicitudes HTTP**: Solicitudes HTTP rápidas y sigilosas con la clase `Fetcher`. Puede imitar la huella TLS de los navegadores, encabezados y usar HTTP3.
- **Carga Dinámica**: Obtén sitios web dinámicos con automatización completa del navegador a través de la clase `DynamicFetcher` compatible con Chromium de Playwright, Chrome real y modo sigiloso personalizado.
- **Evasión Anti-bot**: Capacidades de sigilo avanzadas con `StealthyFetcher` usando una versión modificada de Firefox y falsificación de huellas digitales. Puede evadir todos los tipos de Turnstile e Interstitial de Cloudflare con automatización fácilmente.
- **Carga Dinámica**: Obtén sitios web dinámicos con automatización completa del navegador a través de la clase `DynamicFetcher` compatible con Chromium de Playwright y Google Chrome.
- **Evasión Anti-bot**: Capacidades de sigilo avanzadas con `StealthyFetcher` y falsificación de huellas digitales. Puede evadir fácilmente todos los tipos de Turnstile/Interstitial de Cloudflare con automatización.
- **Gestión de Sesión**: Soporte de sesión persistente con las clases `FetcherSession`, `StealthySession` y `DynamicSession` para la gestión de cookies y estado entre solicitudes.
- **Soporte Async**: Soporte async completo en todos los fetchers y clases de sesión async dedicadas.
@@ -96,7 +96,7 @@ Construido para la Web moderna, Scrapling presenta **su propio motor de análisi
- 🔄 **Seguimiento Inteligente de Elementos**: Relocaliza elementos después de cambios en el sitio web usando algoritmos inteligentes de similitud.
- 🎯 **Selección Flexible Inteligente**: Selectores CSS, selectores XPath, búsqueda basada en filtros, búsqueda de texto, búsqueda regex y más.
- 🔍 **Encontrar Elementos Similares**: Localiza automáticamente elementos similares a los elementos encontrados.
- 🤖 **Servidor MCP para usar con IA**: Servidor MCP integrado para Web Scraping asistido por IA y extracción de datos. El servidor MCP presenta capacidades personalizadas y poderosas que utilizan Scrapling para extraer contenido específico antes de pasarlo a la IA (Claude/Cursor/etc), acelerando así las operaciones y reduciendo costos al minimizar el uso de tokens. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🤖 **Servidor MCP para usar con IA**: Servidor MCP integrado para Web Scraping asistido por IA y extracción de datos. El servidor MCP presenta capacidades poderosas y personalizadas que aprovechan Scrapling para extraer contenido específico antes de pasarlo a la IA (Claude/Cursor/etc), acelerando así las operaciones y reduciendo costos al minimizar el uso de tokens. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### Arquitectura de Alto Rendimiento y Probada en Batalla
- 🚀 **Ultrarrápido**: Rendimiento optimizado que supera a la mayoría de las bibliotecas de scraping de Python.
@@ -212,13 +212,12 @@ Scrapling v0.3 incluye una poderosa interfaz de línea de comandos:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Lanzar shell interactivo de Web Scraping
```bash
# Lanzar shell interactivo de Web Scraping
scrapling shell
# Extraer páginas a un archivo directamente sin programar (Extrae el contenido dentro de la etiqueta `body` por defecto)
# Si el archivo de salida termina con `.txt`, entonces se extraerá el contenido de texto del objetivo.
# Si termina con `.md`, será una representación markdown del contenido HTML, y `.html` será el contenido HTML directamente.
```
Extraer páginas a un archivo directamente sin programar (Extrae el contenido dentro de la etiqueta `body` por defecto). Si el archivo de salida termina con `.txt`, entonces se extraerá el contenido de texto del objetivo. Si termina con `.md`, será una representación Markdown del contenido HTML; si termina con `.html`, será el contenido HTML en sí mismo.
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Todos los elementos que coinciden con el selector CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Benchmarks de Rendimiento
Scrapling no solo es poderoso, también es increíblemente rápido, y las actualizaciones desde la versión 0.3 han brindado mejoras de rendimiento excepcionales en todas las operaciones.
Scrapling no solo es poderoso, también es increíblemente rápido, y las actualizaciones desde la versión 0.3 han brindado mejoras de rendimiento excepcionales en todas las operaciones. Los siguientes benchmarks comparan el analizador de Scrapling con otras bibliotecas populares.
### Prueba de Velocidad de Extracción de Texto (5000 elementos anidados)
@@ -270,14 +269,14 @@ A partir de v0.3.2, esta instalación solo incluye el motor de análisis y sus d
### Dependencias Opcionales
1. Si vas a usar alguna de las características adicionales a continuación, los fetchers, o sus clases, entonces necesitas instalar las dependencias de los fetchers y luego instalar sus dependencias del navegador con
1. Si vas a usar alguna de las características adicionales a continuación, los fetchers, o sus clases, necesitarás instalar las dependencias de los fetchers y sus dependencias del navegador de la siguiente manera:
```bash
pip install "scrapling[fetchers]"
scrapling install
```
Esto descarga todos los navegadores con sus dependencias del sistema y dependencias de manipulación de huellas digitales.
Esto descarga todos los navegadores, junto con sus dependencias del sistema y dependencias de manipulación de huellas digitales.
2. Características adicionales:
- Instalar la característica del servidor MCP:
@@ -303,7 +302,7 @@ O descárgala desde el registro de GitHub:
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
Esta imagen se construye y publica automáticamente a través de GitHub actions en la rama principal del repositorio.
Esta imagen se construye y publica automáticamente usando GitHub Actions y la rama principal del repositorio.
## Contribuir
docs/README_JP.md
+11 -12
View File
@@ -87,8 +87,8 @@ Scraplingは単なるウェブスクレイピングライブラリではあり
### セッションサポート付き高度なウェブサイト取得
- **HTTPリクエスト**：`Fetcher`クラスで高速でステルスなHTTPリクエスト。ブラウザのTLSフィンガープリント、ヘッダーを模倣し、HTTP3を使用できます。
- **動的読み込み**：Playwright's Chromium、実際のChrome、カスタムステルスモードをサポートする`DynamicFetcher`クラスを通じた完全なブラウザ自動化で動的ウェブサイトを取得。
- **アンチボット回避**：修正されたFirefoxとフィンガープリント偽装を使用した`StealthyFetcher`による高度なステルス機能。自動化でCloudflareのTurnstileとInterstitialのすべてのタイプを簡単に回避できます。
- **動的読み込み**：Playwright's ChromiumとGoogle Chromeをサポートする`DynamicFetcher`クラスを通じた完全なブラウザ自動化で動的ウェブサイトを取得。
- **アンチボット回避**：`StealthyFetcher`とフィンガープリント偽装による高度なステルス機能。自動化でCloudflareのTurnstile/Interstitialのすべてのタイプを簡単に回避できます。
- **セッション管理**：リクエスト間でCookieと状態を管理するための`FetcherSession`、`StealthySession`、`DynamicSession`クラスによる永続的なセッションサポート。
- **非同期サポート**：すべてのフェッチャーと専用非同期セッションクラス全体での完全な非同期サポート。
@@ -96,7 +96,7 @@ Scraplingは単なるウェブスクレイピングライブラリではあり
- 🔄 **スマート要素追跡**：インテリジェントな類似性アルゴリズムを使用してウェブサイトの変更後に要素を再配置。
- 🎯 **スマート柔軟選択**：CSSセレクタ、XPathセレクタ、フィルタベース検索、テキスト検索、正規表現検索など。
- 🔍 **類似要素を見つける**：見つかった要素に類似した要素を自動的に特定。
- 🤖 **AIと使用するMCPサーバー**：AI支援ウェブスクレイピングとデータ抽出のための組み込みMCPサーバー。MCPサーバーは、AI（Claude/Cursorなど）に渡す前にScraplingを利用してターゲットコンテンツを抽出するカスタムで強力な機能を備えており、操作を高速化し、トークン使用量を最小限に抑えることでコストを削減します。（[デモビデオ](https://www.youtube.com/watch?v=qyFk3ZNwOxE)）
- 🤖 **AIと使用するMCPサーバー**：AI支援ウェブスクレイピングとデータ抽出のための組み込みMCPサーバー。MCPサーバーは、AI（Claude/Cursorなど）に渡す前にScraplingを活用してターゲットコンテンツを抽出する強力でカスタムな機能を備えており、操作を高速化し、トークン使用量を最小限に抑えることでコストを削減します。（[デモビデオ](https://www.youtube.com/watch?v=qyFk3ZNwOxE)）
### 高性能で実戦テスト済みのアーキテクチャ
- 🚀 **高速**：ほとんどのPythonスクレイピングライブラリを上回る最適化されたパフォーマンス。
@@ -212,13 +212,12 @@ Scrapling v0.3には強力なコマンドラインインターフェースが含
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
インタラクティブウェブスクレイピングシェルを起動
```bash
# インタラクティブウェブスクレイピングシェルを起動
scrapling shell
# プログラミングせずに直接ページをファイルに抽出（デフォルトで`body`タグ内のコンテンツを抽出）
# 出力ファイルが`.txt`で終わる場合、ターゲットのテキストコンテンツが抽出されます。
# `.md`で終わる場合、HTMLコンテンツのMarkdown表現になり、`.html`は直接HTMLコンテンツになります。
```
プログラミングせずに直接ページをファイルに抽出（デフォルトで`body`タグ内のコンテンツを抽出）。出力ファイルが`.txt`で終わる場合、ターゲットのテキストコンテンツが抽出されます。`.md`で終わる場合、HTMLコンテンツのMarkdown表現になります；`.html`で終わる場合、HTMLコンテンツそのものになります。
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # CSSセレクタ'#fromSkipToProducts'に一致するすべての要素
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## パフォーマンスベンチマーク
Scraplingは強力であるだけでなく、驚くほど高速で、バージョン0.3以降のアップデートはすべての操作で優れたパフォーマンス向上を実現しています。
Scraplingは強力であるだけでなく、驚くほど高速で、バージョン0.3以降のアップデートはすべての操作で優れたパフォーマンス向上を実現しています。以下のベンチマークは、Scraplingのパーサーを他の人気のあるライブラリと比較しています。
### テキスト抽出速度テスト（5000個のネストされた要素）
@@ -270,14 +269,14 @@ v0.3.2以降、このインストールにはパーサーエンジンとその
### オプションの依存関係
1. 以下の追加機能、フェッチャー、またはそれらのクラスのいずれかを使用する場合は、フェッチャーの依存関係をインストールしてから、次のコマンドでブラウザの依存関係をインストールする必要があります
1. 以下の追加機能、フェッチャー、またはそれらのクラスのいずれかを使用する場合は、フェッチャーの依存関係とブラウザの依存関係を次のようにインストールする必要があります：
```bash
pip install "scrapling[fetchers]"
scrapling install
```
これにより、すべてのブラウザとそのシステム依存関係およびフィンガープリント操作依存関係がダウンロードされます。
これにより、すべてのブラウザ、およびそれらのシステム依存関係とフィンガープリント操作依存関係がダウンロードされます。
2. 追加機能：
- MCPサーバー機能をインストール：
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
このイメージは、リポジトリのメインブランチでGitHub actionsを通じて自動的にビルドおよびプッシュされます。
このイメージは、GitHub Actionsとリポジトリのメインブランチを使用して自動的にビルドおよびプッシュされます。
## 貢献
docs/README_RU.md
+11 -12
View File
@@ -87,8 +87,8 @@ Scrapling - это не просто очередная библиотека д
### Продвинутая загрузка сайтов с поддержкой сессий
- **HTTP-запросы**: Быстрые и скрытные HTTP-запросы с классом `Fetcher`. Может имитировать TLS-отпечаток браузера, заголовки и использовать HTTP3.
- **Динамическая загрузка**: Загрузка динамических сайтов с полной автоматизацией браузера через класс `DynamicFetcher`, поддерживающий Chromium от Playwright, настоящий Chrome и пользовательский режим скрытности.
- **Обход анти-ботов**: Расширенные возможности скрытности с `StealthyFetcher`, использующим модифицированную версию Firefox и подмену отпечатков. Может легко обойти все типы Turnstile и Interstitial от Cloudflare с помощью автоматизации.
- **Динамическая загрузка**: Загрузка динамических сайтов с полной автоматизацией браузера через класс `DynamicFetcher`, поддерживающий Chromium от Playwright и Google Chrome.
- **Обход анти-ботов**: Расширенные возможности скрытности с `StealthyFetcher` и подмену отпечатков. Может легко обойти все типы Turnstile/Interstitial от Cloudflare с помощью автоматизации.
- **Управление сессиями**: Поддержка постоянных сессий с классами `FetcherSession`, `StealthySession` и `DynamicSession` для управления cookie и состоянием между запросами.
- **Поддержка асинхронности**: Полная асинхронная поддержка во всех фетчерах и выделенных асинхронных классах сессий.
@@ -96,7 +96,7 @@ Scrapling - это не просто очередная библиотека д
- 🔄 **Умное отслеживание элементов**: Перемещайте элементы после изменений сайта с помощью интеллектуальных алгоритмов подобия.
- 🎯 **Умный гибкий выбор**: CSS-селекторы, XPath-селекторы, поиск на основе фильтров, текстовый поиск, поиск по регулярным выражениям и многое другое.
- 🔍 **Поиск похожих элементов**: Автоматически находите элементы, похожие на найденные элементы.
- 🤖 **MCP-сервер для использования с ИИ**: Встроенный MCP-сервер для веб-скрапинга с помощью ИИ и извлечения данных. MCP-сервер обладает пользовательскими, мощными возможностями, которые используют Scrapling для извлечения целевого контента перед передачей его ИИ (Claude/Cursor/и т.д.), тем самым ускоряя операции и снижая затраты за счет минимизации использования токенов. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
- 🤖 **MCP-сервер для использования с ИИ**: Встроенный MCP-сервер для веб-скрапинга с помощью ИИ и извлечения данных. MCP-сервер обладает мощными, пользовательскими возможностями, которые используют Scrapling для извлечения целевого контента перед передачей его ИИ (Claude/Cursor/и т.д.), тем самым ускоряя операции и снижая затраты за счет минимизации использования токенов. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### Высокопроизводительная и проверенная в боях архитектура
- 🚀 **Молниеносно быстро**: Оптимизированная производительность превосходит большинство библиотек скрапинга Python.
@@ -212,13 +212,12 @@ Scrapling v0.3 включает мощный интерфейс командно
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Запустить интерактивную оболочку веб-скрапинга
```bash
# Запустить интерактивную оболочку веб-скрапинга
scrapling shell
# Извлечь страницы в файл напрямую без программирования (Извлекает содержимое внутри тега `body` по умолчанию)
# Если выходной файл заканчивается на `.txt`, то будет извлечено текстовое содержимое цели.
# Если заканчивается на `.md`, это будет markdown-представление HTML-содержимого, а `.html` будет непосредственно HTML-содержимым.
```
Извлечь страницы в файл напрямую без программирования (Извлекает содержимое внутри тега `body` по умолчанию). Если выходной файл заканчивается на `.txt`, то будет извлечено текстовое содержимое цели. Если заканчивается на `.md`, это будет Markdown-представление HTML-содержимого; если заканчивается на `.html`, это будет само HTML-содержимое.
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Все элементы, соответствующие CSS-селектору '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Тесты производительности
Scrapling не только мощный - он также невероятно быстрый, и обновления с версии 0.3 обеспечили исключительные улучшения производительности во всех операциях.
Scrapling не только мощный - он также невероятно быстрый, и обновления с версии 0.3 обеспечили исключительные улучшения производительности во всех операциях. Следующие тесты производительности сравнивают парсер Scrapling с другими популярными библиотеками.
### Тест скорости извлечения текста (5000 вложенных элементов)
@@ -270,14 +269,14 @@ pip install scrapling
### Опциональные зависимости
1. Если вы собираетесь использовать какие-либо из дополнительных функций ниже, фетчеры или их классы, то вам нужно установить зависимости фетчеров, а затем установить их зависимости браузера с помощью
1. Если вы собираетесь использовать какие-либо из дополнительных функций ниже, фетчеры или их классы, вам необходимо установить зависимости фетчеров и их зависимости браузера следующим образом:
```bash
pip install "scrapling[fetchers]"
scrapling install
```
Это загрузит все браузеры с их системными зависимостями и зависимостями манипуляции отпечатками.
Это загрузит все браузеры вместе с их системными зависимостями и зависимостями манипуляции отпечатками.
2. Дополнительные функции:
- Установить функцию MCP-сервера:
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
Этот образ автоматически создается и отправляется через GitHub actions в основной ветке репозитория.
Этот образ автоматически создается и отправляется с использованием GitHub Actions и основной ветки репозитория.
## Вклад
docs/ai/mcp-server.md
+20 -16
View File
@@ -2,7 +2,7 @@
<iframe width="560" height="315" src="https://www.youtube.com/embed/qyFk3ZNwOxE?si=3FHzgcYCb66iJ6e3" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful Web Scraping capabilities directly to your favorite AI chatbot or AI agent. This integration allows you to scrape websites, extract data, and bypass anti-bot protections conversationally through Claude's AI interface or any other chatbot that supports MCP.
The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful Web Scraping capabilities directly to your favorite AI chatbot or AI agent. This integration allows you to scrape websites, extract data, and bypass anti-bot protections conversationally through Claude's AI interface or any interface that supports MCP.
## Features
@@ -13,11 +13,11 @@ The Scrapling MCP Server provides six powerful tools for web scraping:
- **`bulk_get`**: An async version of the above tool that allows scraping of multiple URLs at the same time!
### 🌐 Dynamic Content Scraping
- **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, stealth mode, and more!
- **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, and more!
- **`bulk_fetch`**: An async version of the above tool that allows scraping of multiple URLs in different browser tabs at the same time!
### 🔒 Stealth Scraping
- **`stealthy_fetch`**: Uses our modified version of Camoufox browser to bypass Cloudflare Turnstile/Interstitial and other anti-bot systems with complete control over the request/browser!
- **`stealthy_fetch`**: Uses our Stealthy browser to bypass Cloudflare Turnstile/Interstitial and other anti-bot systems with complete control over the request/browser!
- **`bulk_stealthy_fetch`**: An async version of the above tool that allows stealth scraping of multiple URLs in different browser tabs at the same time!
### Key Capabilities
@@ -30,9 +30,9 @@ The Scrapling MCP Server provides six powerful tools for web scraping:
#### But why use Scrapling MCP Server instead of other available tools?
Aside from its stealth capabilities and ability to bypass Cloudflare Turnstile/Interstitial, Scrapling's server is the only one that allows you to pass a CSS selector in the prompt to extract specific elements before handing the content to the AI.
Aside from its stealth capabilities and ability to bypass Cloudflare Turnstile/Interstitial, Scrapling's server is the only one that lets you select specific elements to pass to the AI, saving a lot of time and tokens!
The way other servers work is that they extract the content, then pass it all to the AI to extract the fields you want. This causes the AI to consume a lot more tokens that are not needed (from irrelevant content). Scrapling solves this problem by allowing you to pass a CSS selector to narrow down the content you want before passing it to the AI, which makes the whole process much faster and more efficient.
The way other servers work is that they extract the content, then pass it all to the AI to extract the fields you want. This causes the AI to consume far more tokens than needed (from irrelevant content). Scrapling solves this problem by allowing you to pass a CSS selector to narrow down the content you want before passing it to the AI, which makes the whole process much faster and more efficient.
If you don't know how to write/use CSS selectors, don't worry. You can tell the AI in the prompt to write selectors to match possible fields for you and watch it try different combinations until it finds the right one, as we will show in the examples section.
@@ -48,10 +48,14 @@ pip install "scrapling[ai]"
scrapling install
```
Or use the Docker image directly:
Or use the Docker image directly from the Docker registry:
```bash
docker pull pyd4vinci/scrapling
```
Or download it from the GitHub registry:
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
## Setting up the MCP Server
@@ -83,12 +87,12 @@ If that's the first MCP server you're adding, set the content of the file to thi
}
}
```
As per the [official article](https://modelcontextprotocol.io/quickstart/user), this action creates a new configuration file if one doesn’t exist or opens your existing configuration. The file is located at
As per the [official article](https://modelcontextprotocol.io/quickstart/user), this action either creates a new configuration file if none exists or opens your existing configuration. The file is located at
1. **MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
2. **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
To ensure it's working, it's best to use the full path to the `scrapling` executable. Open the terminal and execute the following command:
To ensure it's working, use the full path to the `scrapling` executable. Open the terminal and execute the following command:
1. **MacOS**: `which scrapling`
2. **Windows**: `where scrapling`
@@ -150,7 +154,7 @@ Use the following to enable 'Streamable HTTP' transport mode:
```bash
scrapling mcp --http
```
Hence, the default value for the host the server is listening on is '0.0.0.0' and the port is 8000, which both can be configured as below:
Hence, the default value for the host the server is listening to is '0.0.0.0' and the port is 8000, which both can be configured as below:
```bash
scrapling mcp --http --host '127.0.0.1' --port 8000
```
@@ -169,13 +173,13 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Scrape the main content from https://example.com and convert it to markdown format.
```
Claude will use the `get` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for three attempts, unless you instruct it to do otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
Claude will use the `get` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
A more optimized version of the same prompt would be:
```
Use regular requests to scrape the main content from https://example.com and convert it to markdown format.
```
This tells Claude about the right tool to use here, so it doesn't have to guess. Sometimes it will start using normal requests on its own, and at other times, it will assume browsers are better suited for this website without any apparent reason. As a general rule of thumb, you should always tell Claude what tool to use if you want to save time, money, and get consistent results.
This tells Claude which tool to use here, so it doesn't have to guess. Sometimes it will start using normal requests on its own, and at other times, it will assume browsers are better suited for this website without any apparent reason. As a general rule of thumb, you should always tell Claude which tool to use if you want to save time and money and get consistent results.
2. **Targeted Data Extraction**
@@ -185,7 +189,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Get all product titles from https://shop.example.com using the CSS selector '.product-title'. If the request fails, retry up to 5 times every 10 seconds.
```
The server will extract only the elements matching your selector and return them as a structured list. Notice I told it to set the tool to only try three times in case the website has connection issues, but the default setting should be fine for most cases.
The server will extract only the elements matching your selector and return them as a structured list. Notice I told it to set the tool to try only 3 times in case the website has connection issues, but the default setting should be fine for most cases.
3. **E-commerce Data Collection**
@@ -199,7 +203,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Get the product names, prices, and descriptions from each page.
```
Claude will use `bulk_fetch` to scrape all URLs concurrently, then analyze the extracted data.
Claude will use `bulk_fetch` to concurrently scrape all URLs, then analyze the extracted data.
4. **More advanced workflow**
@@ -216,14 +220,14 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
And if you know how to write CSS selectors, you can instruct Claude to apply the selectors to the elements you want, and it will nearly complete the task immediately.
```
Use normal requests to extract the URLs of all games on the page below, then perform a bulk request to them and return a list of all action games.
The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in the second request is `[data-qa="gameInfo#releaseInformation#genre-value"]`
The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in the second request is `[data-qa="gameInfo#releaseInformation#genre-value"]`.
URL: https://store.playstation.com/en-us/pages/browse
```
5. **Get data from a website with Cloudflare protection**
If you think the website you are targeting has Cloudflare protection, you should tell Claude instead of letting it discover that on its own.
If you think the website you are targeting has Cloudflare protection, tell Claude instead of letting it discover it on its own.
```
What's the price of this product? Be cautious, as it utilizes Cloudflare's Turnstile protection. Make the browser visible while you work.
@@ -234,7 +238,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
You can, for example, use a prompt like this:
```
Extract all the product URLs in the following category, then return the prices and the details of the first three products.
Extract all product URLs for the following category, then return the prices and details for the first 3 products.
https://www.arnotts.ie/furniture/bedroom/bed-frames/
```
docs/benchmarks.md
+1 -1
View File
@@ -1,6 +1,6 @@
# Performance Benchmarks
Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 deliver exceptional performance improvements across all operations!
Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 have delivered exceptional performance improvements across all operations. The following benchmarks compare Scrapling's parser with other popular libraries.
## Benchmark Results
docs/cli/extract-commands.md
+17 -20
View File
@@ -2,7 +2,7 @@
**Web Scraping through the terminal without requiring any programming!**
The `scrapling extract` Command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction.
The `scrapling extract` command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction.
> 💡 **Prerequisites:**
>
@@ -30,7 +30,7 @@ The extract command is a set of simple terminal tools that:
```bash
scrapling extract get "https://example.com" page_content.txt
```
This does an HTTP GET request and saves the text content of the webpage to `page_content.txt`.
This makes an HTTP GET request and saves the webpage's text content to `page_content.txt`.
- **Save as Different Formats**
@@ -73,13 +73,13 @@ Commands:
stealthy-fetch Use StealthyFetcher to fetch content with advanced...
```
We will go through each Command in detail below.
We will go through each command in detail below.
### HTTP Requests
1. **GET Request**
The most common Command for downloading website content:
The most common command for downloading website content:
```bash
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]
@@ -105,7 +105,7 @@ We will go through each Command in detail below.
# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
```
Get the available options for the Command with `scrapling extract get --help` as follows:
Get the available options for the command with `scrapling extract get --help` as follows:
```bash
Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE
@@ -143,7 +143,7 @@ We will go through each Command in detail below.
# Send JSON data
scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the Command with `scrapling extract post --help` as follows:
Get the available options for the command with `scrapling extract post --help` as follows:
```bash
Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE
@@ -182,7 +182,7 @@ We will go through each Command in detail below.
# Send JSON data
scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the Command with `scrapling extract put --help` as follows:
Get the available options for the command with `scrapling extract put --help` as follows:
```bash
Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE
@@ -220,7 +220,7 @@ We will go through each Command in detail below.
# Send JSON data
scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"
```
Get the available options for the Command with `scrapling extract delete --help` as follows:
Get the available options for the command with `scrapling extract delete --help` as follows:
```bash
Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE
@@ -263,7 +263,7 @@ We will go through each Command in detail below.
# Run in visible browser mode (helpful for debugging)
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
```
Get the available options for the Command with `scrapling extract fetch --help` as follows:
Get the available options for the command with `scrapling extract fetch --help` as follows:
```bash
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
@@ -279,10 +279,8 @@ We will go through each Command in detail below.
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--locale TEXT Browser locale (default: en-US)
--stealth / --no-stealth Enable stealth mode (default: False)
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--disable-webgl / --enable-webgl Disable WebGL support (default: False)
--locale TEXT Specify user locale. Defaults to the system default locale.
---real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit.
@@ -304,10 +302,10 @@ We will go through each Command in detail below.
# Solve Cloudflare challenges
scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
# Use proxy for anonymity
# Use a proxy for anonymity.
scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
```
Get the available options for the Command with `scrapling extract stealthy-fetch --help` as follows:
Get the available options for the command with `scrapling extract stealthy-fetch --help` as follows:
```bash
Usage: scrapling extract stealthy-fetch [OPTIONS] URL OUTPUT_FILE
@@ -317,25 +315,24 @@ We will go through each Command in detail below.
Options:
--headless / --no-headless Run browser in headless mode (default: True)
--block-images / --allow-images Block image loading (default: False)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
--humanize / --no-humanize Humanize cursor movement (default: False)
--solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
--allow-webgl / --block-webgl Allow WebGL (default: True)
--network-idle / --no-network-idle Wait for network idle (default: False)
--disable-ads / --allow-ads Install uBlock Origin addon (default: False)
---real-chrome/--no-real-chrom If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--hide-canvas/--show-canvas Add noise to canvas operations (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--geoip / --no-geoip Use IP/Proxy geolocation for timezone/locale (default: False)
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit.
```
## When to use each Command
## When to use each command
If you are not a Web Scraping expert and can't decide what to choose, you can use the following formula to help you decide:
docs/cli/interactive-shell.md
+4 -2
View File
@@ -4,7 +4,7 @@
**Powerful Web Scraping REPL for Developers and Data Scientists**
The Scrapling Interactive Shell is an enhanced IPython-based environment designed specifically for Web Scraping tasks. It provides instant access to all Scrapling features, clever shortcuts, automatic page management, and advanced tools like curl command conversion.
The Scrapling Interactive Shell is an enhanced IPython-based environment designed specifically for Web Scraping tasks. It provides instant access to all Scrapling features, clever shortcuts, automatic page management, and advanced tools, such as conversion of the curl command.
> 💡 **Prerequisites:**
>
@@ -129,7 +129,9 @@ View scraped pages in your browser:
### Curl Command Integration
The shell provides a few functions to help you convert curl commands from the browser DevTools to `Fetcher` requests, which are `uncurl` and `curl2fetcher`. First, you need to copy a request as a curl command like the following:
The shell provides a few functions to help you convert curl commands from the browser DevTools to `Fetcher` requests: `uncurl` and `curl2fetcher`.
First, you need to copy a request as a curl command like the following:
<img src="../../assets/scrapling_shell_curl.png" title="Copying a request as a curl command from Chrome" alt="Copying a request as a curl command from Chrome" style="width: 70%;"/>
docs/cli/overview.md
+1 -1
View File
@@ -27,4 +27,4 @@ and the installation of the fetchers' dependencies with the following command
```bash
scrapling install
```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
docs/development/adaptive_storage_system.md
+3 -3
View File
@@ -1,6 +1,6 @@
Scrapling uses SQLite by default, but this tutorial covers writing your storage system to store element properties there for `adaptive` feature.
Scrapling uses SQLite by default, but this tutorial shows how to write your own storage system to store element properties for the `adaptive` feature.
You might want to use FireBase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because the spiders will share with each other.
You might want to use Firebase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because spiders can share adaptive data with each other.
So first, to make your storage class work, it must do the big 3:
@@ -8,7 +8,7 @@ So first, to make your storage class work, it must do the big 3:
2. Use the decorator `functools.lru_cache` on top of the class to follow the Singleton design pattern as other classes.
3. Implement methods `save` and `retrieve`, as you see from the type hints:
- The method `save` returns nothing and will get two arguments from the library
* The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the function `element_to_dict` in submodule `scrapling.core.utils._StorageTools` to keep the same format and save it to your database as you wish.
* The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the `element_to_dict` function in the submodule `scrapling.core.utils._StorageTools` to maintain the same format, and then saved to your database as you wish.
* The second one is a string, the identifier used for retrieval. The combination result of this identifier and the `url` argument from initialization must be unique for each row, or the `adaptive` data will be messed up.
- The method `retrieve` takes a string, which is the identifier; using it with the `url` passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns `None`.
docs/development/scrapling_custom_types.md
+5 -5
View File
@@ -1,6 +1,6 @@
> You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)
### All current types can be imported alone like below
### All current types can be imported alone, like below
```python
>>> from scrapling.core.custom_types import TextHandler, AttributesHandler
@@ -11,11 +11,11 @@
>>> somedict_2 = AttributesHandler(a=1)
```
Note that `TextHandler` is a subclass of Python's `str`, so all normal operations/methods that work with Python strings will work.
If you want to check for the type in your code, it's better to depend on Python's built-in function `issubclass`.
Note that `TextHandler` is a subclass of Python's `str`, so all standard operations/methods that work with Python strings will work.
If you want to check the type in your code, it's better to use Python's built-in `issubclass` function.
The class `AttributesHandler` is a subclass of `collections.abc.Mapping`, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the `_data` property, but be careful; it's of type `types.MappingProxyType`, so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds).
So, to make it simple for you if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with class `AttributesHandler` except the ones that try to modify the actual data.
So, to make it simple for you, if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with the class `AttributesHandler` except for the ones that try to modify the actual data.
If you want to modify the data inside `AttributesHandler,` you have to convert it to a dictionary first, like using the `dict` function, and then modify it outside.
If you want to modify the data inside `AttributesHandler`, you have to convert it to a dictionary first, e.g., using the `dict` function, and then change it outside.
docs/donate.md
+2 -3
View File
@@ -1,6 +1,6 @@
I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to provide these projects for free to the community. By becoming a sponsor, you would directly fund my coffee reserves, helping me continuously update existing projects and create new ones.
I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to make them available to the community for free. By becoming a sponsor, you would directly fund my coffee reserves, helping me continuously update existing projects and create new ones.
You can sponsor me directly through [GitHub sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me A Coffe](https://buymeacoffee.com/d4vinci).
You can sponsor me directly through the [GitHub Sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me a Coffee](https://buymeacoffee.com/d4vinci).
Thank you, stay curious, and hack the planet! ❤️
@@ -22,4 +22,3 @@ Perks:
- The same logo will be featured at [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling).
- Your logo will be featured as a top sponsor on [Scrapling's website](https://scrapling.readthedocs.io/en/latest/) main page.
- A Shoutout with each [Release note](https://github.com/D4Vinci/Scrapling/releases).
docs/fetching/choosing.md
+12 -12
View File
@@ -3,7 +3,7 @@ Fetchers are classes that can do requests or fetch pages for you easily in a sin
This feature was introduced because, before v0.2, Scrapling was only a parsing engine. The target here is to gradually become the one-stop shop for all Web Scraping needs.
> Fetchers are not wrappers built on top of other libraries. However, they utilize these libraries as an engine to request/fetch pages easily for you, while fully leveraging that engine and adding features for you. Some fetchers don't even use the official library for requests; instead, they use their own custom version. For example, `StealthyFetcher` utilizes `Camoufox` browser directly, without relying on its Python library for anything except launch options. This last part might change soon as well.
> Fetchers are not wrappers built on top of other libraries. However, they only use these libraries as an engine to request/fetch pages. To further clarify this, all fetchers have features that the underlying engines don't, while still fully leveraging those engines and optimizing them for Web Scraping.
## Fetchers Overview
@@ -12,17 +12,17 @@ Scrapling provides three different fetcher classes with their session classes; e
The following table compares them and can be quickly used for guidance.
| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
|--------------------|---------------------------------------------------|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇 |
| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| JavaScript loading | ❌ | ✅ | ✅ |
| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Slight protections | - Dynamically loaded websites <br/>- Small automation <br/>- Complicated protections |
| Browser(s) | ❌ | Chromium and Google Chrome | Modified Firefox |
| Browser API used | ❌ | PlayWright | PlayWright |
| Setup Complexity | Simple | Simple | Simple |
| Feature | Fetcher | DynamicFetcher | StealthyFetcher |
|--------------------|---------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇🐇 |
| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| JavaScript loading | ❌ | ✅ | ✅ |
| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Small-Mid protections | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
| Browser(s) | ❌ | Chromium and Google Chrome | Chromium and Google Chrome |
| Browser API used | ❌ | PlayWright | PlayWright |
| Setup Complexity | Simple | Simple | Simple |
In the following pages, we will talk about each one in detail.
docs/fetching/dynamic.md
+56 -94
View File
@@ -1,6 +1,6 @@
# Introduction
Here, we will discuss the `DynamicFetcher` class (previously known as `PlayWrightFetcher`). This class provides flexible browser automation with multiple configuration options and some stealth capabilities.
Here, we will discuss the `DynamicFetcher` class (formerly `PlayWrightFetcher`). This class provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
@@ -23,7 +23,7 @@ Now, we will review most of the arguments one by one, using examples. If you wan
> Note: The async version of the `fetch` method is the `async_fetch` method, of course.
This fetcher currently provides four main run options that can be combined as desired.
This fetcher currently provides three main run options that can be combined as desired.
Which are:
@@ -31,77 +31,70 @@ Which are:
```python
DynamicFetcher.fetch('https://example.com')
```
Using it in that manner will open a Chromium browser and load the page. There are no tricks or extra features unless you enable some; it's just a plain PlayWright API.
Using it in that manner will open a Chromium browser and load the page. There are optimizations for speed, and some stealth goes automatically under the hood, but other than that, there are no tricks or extra features unless you enable some; it's just a plain PlayWright API.
### 2. Stealth Mode
```python
DynamicFetcher.fetch('https://example.com', stealth=True)
```
It's the same as the vanilla Playwright option, but it provides a simple stealth mode suitable for websites with a small to medium protection layer(s).
Some of the things this fetcher's stealth mode does include:
* Patching the CDP runtime fingerprint by using PatchRight.
* Mimics some of the real browsers' properties by injecting several JS files and using custom options.
* Custom flags are used on launch to hide Playwright even more and make it faster.
* Generates real browser headers of the same type and user OS, then appends them to the request's headers.
### 3. Real Chrome
### 2. Real Chrome
```python
DynamicFetcher.fetch('https://example.com', real_chrome=True)
```
If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium.
If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium. This will make your requests look more authentic, so they're less detectable for better results.
This will make your requests look more authentic, so it's less detectable, and you can even use the `stealth=True` mode with it for better results, like below:
```python
DynamicFetcher.fetch('https://example.com', real_chrome=True, stealth=True)
```
If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
```commandline
playwright install chrome
```
### 4. CDP Connection
### 3. CDP Connection
```python
DynamicFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
```
Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
> Notes:
>
> * There was a `stealth` option here, but it was moved to the `StealthyFetcher` class, as explained on the next page, with additional features since version 0.3.13.<br/>
> * This makes it less confusing for new users, easier to maintain, and provides other benefits, as explained on the [StealthyFetcher page](../fetching/stealthy.md).
## Full list of arguments
Scrapling provides many options with this fetcher and its session classes. To make it as simple as possible, we will list the options here and give examples of how to use most of them.
| Argument | Description | Optional |
|:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
| url | Target url | ❌ |
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
| disable_resources | Drop requests for unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ |
| cookies | Set cookies for the next request. | ✔️ |
| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser.** | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ |
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
| disable_webgl | Disables WebGL and WebGL 2.0 support entirely. | ✔️ |
| stealth | Enables stealth mode; you should always check the documentation to see what the stealth mode does currently. | ✔️ |
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
| locale | Set the locale for the browser if wanted. The default value is `en-US`. | ✔️ |
| timezone_id | Set the timezone for the browser if wanted. | ✔️ |
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
| Argument | Description | Optional |
|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
| url | Target url | ❌ |
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
| cookies | Set cookies for the next request. | ✔️ |
| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ |
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, and `selector_config`.
> 🔍 Notes:
>
> 1. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
> 2. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
> 3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`.
> 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
## Examples
It's easier to understand with examples, so let's take a look.
@@ -110,10 +103,7 @@ It's easier to understand with examples, so let's take a look.
```python
# Disable unnecessary resources
page = DynamicFetcher.fetch(
'https://example.com',
disable_resources=True # Blocks fonts, images, media, etc...
)
page = DynamicFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
```
### Network Control
@@ -125,11 +115,8 @@ page = DynamicFetcher.fetch('https://example.com', network_idle=True)
# Custom timeout (in milliseconds)
page = DynamicFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
# Proxy support
page = DynamicFetcher.fetch(
'https://example.com',
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
)
# Proxy support (It can also be a dictionary with only the keys 'server', 'username', and 'password'.)
page = DynamicFetcher.fetch('https://example.com', proxy='http://username:password@host:port')
```
### Downloading Files
@@ -141,7 +128,7 @@ with open(file='poster.png', mode='wb') as f:
f.write(page.body)
```
The `body` attribute of the `Response` object is a `bytes` object containing the response body in case of Non-HTML responses.
The `body` attribute of the `Response` object is a `bytes` object containing the response body in case of non-HTML responses.
### Browser Automation
This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues.
@@ -157,10 +144,7 @@ def scroll_page(page: Page):
page.mouse.move(100, 400)
page.mouse.up()
page = DynamicFetcher.fetch(
'https://example.com',
page_action=scroll_page
)
page = DynamicFetcher.fetch('https://example.com', page_action=scroll_page)
```
Of course, if you use the async fetch version, the function must also be async.
```python
@@ -171,10 +155,7 @@ async def scroll_page(page: Page):
await page.mouse.move(100, 400)
await page.mouse.up()
page = await DynamicFetcher.async_fetch(
'https://example.com',
page_action=scroll_page
)
page = await DynamicFetcher.async_fetch('https://example.com', page_action=scroll_page)
```
### Wait Conditions
@@ -201,30 +182,13 @@ The states the fetcher can wait for can be any of the following ([source](https:
### Some Stealth Features
```python
# Full stealth mode
page = DynamicFetcher.fetch(
'https://example.com',
stealth=True,
hide_canvas=True,
disable_webgl=True,
google_search=True
)
# Custom user agent
page = DynamicFetcher.fetch(
'https://example.com',
useragent='Mozilla/5.0...'
)
# Set browser locale
page = DynamicFetcher.fetch(
'https://example.com',
locale='en-US'
google_search=True,
useragent='Mozilla/5.0...', # Custom user agent
locale='en-US', # Set browser locale
)
```
Hence, the `hide_canvas` argument doesn't disable the canvas; instead, it hides it by adding random noise to canvas operations, preventing fingerprinting. Also, if you didn't set a user agent (preferred), the fetcher will generate a real User Agent of the same browser and use it.
The `google_search` argument is enabled by default, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
### General example
```python
@@ -259,7 +223,6 @@ from scrapling.fetchers import DynamicSession
# Create a session with default configuration
with DynamicSession(
headless=True,
stealth=True,
disable_resources=True,
real_chrome=True
) as session:
@@ -279,7 +242,6 @@ from scrapling.fetchers import AsyncDynamicSession
async def scrape_multiple_sites():
async with AsyncDynamicSession(
stealth=True,
network_idle=True,
timeout=30000,
max_pages=3
@@ -298,7 +260,7 @@ You may have noticed the `max_pages` argument. This is a new argument that enabl
1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
This logic allows for multiple websites to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
@@ -317,6 +279,6 @@ Use DynamicFetcher when:
- Want multiple browser options
- Using a real Chrome browser
- Need custom browser config
- Want flexible stealth options
- Want a few stealth options
If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
docs/fetching/static.md
+6 -5
View File
@@ -25,7 +25,7 @@ All methods for making requests here share some arguments, so let's discuss them
- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.
- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.
- **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**.
- **impersonate**: Impersonate specific browsers' TLS fingerprints. Accepts browser strings like `"chrome110"`, `"firefox102"`, `"safari15_5"` to use specific versions or `"chrome"`, `"firefox"`, `"safari"`, `"edge"` to automatically use the latest version available. This makes your requests appear as if they're coming from real browsers at the TLS level. **Defaults to the latest available Chrome version.**
- **impersonate**: Impersonate specific browsers' TLS fingerprints. Accepts browser strings or a list of them like `"chrome110"`, `"firefox102"`, `"safari15_5"` to use specific versions or `"chrome"`, `"firefox"`, `"safari"`, `"edge"` to automatically use the latest version available. This makes your requests appear to come from real browsers at the TLS level. If you pass it a list of strings, it will choose a random one with each request. **Defaults to the latest available Chrome version.**
- **http3**: Use HTTP/3 protocol for requests. **Defaults to False**. It might be problematic if used with `impersonate`.
- **cookies**: Cookies to use in the request. Can be a dictionary of `name→value` or a list of dictionaries.
- **proxy**: As the name implies, the proxy for this request is used to route all traffic (HTTP and HTTPS). The format accepted here is `http://username:password@localhost:8030`.
@@ -39,14 +39,15 @@ All methods for making requests here share some arguments, so let's discuss them
> Note: <br/>
> 1. The currently available browsers to impersonate are (`"edge"`, `"chrome"`, `"chrome_android"`, `"safari"`, `"safari_beta"`, `"safari_ios"`, `"safari_ios_beta"`, `"firefox"`, `"tor"`)<br/>
> 2. The available browsers to impersonate and their corresponding versions are automatically displayed in the argument autocompletion and updated automatically with each `curl_cffi` update.
> 2. The available browsers to impersonate, along with their corresponding versions, are automatically displayed in the argument autocompletion and updated with each `curl_cffi` update.<br/>
> 3. If any of the arguments `impersonate` or `stealthy_headers` are enabled, the fetchers will automatically generate real browser headers that match the browser version used.
Other than this, for further customization, you can pass any arguments that `curl_cffi` supports for any method if that method doesn't already support it.
### HTTP Methods
There are additional arguments for each method, depending on the method, such as `params` for GET requests and `data`/`json` for POST/PUT/DELETE requests.
Examples are the best way to explain this, as follows.
Examples are the best way to explain this:
> Hence: `OPTIONS` and `HEAD` methods are not supported.
#### GET
@@ -166,7 +167,7 @@ And for asynchronous requests, it's a small adjustment
## Session Management
For making multiple requests with the same configuration, use the `FetcherSession` class. It can be used in both synchronous and asynchronous code without issue; the class detects and changes the session type automatically without requiring a different import.
For making multiple requests with the same configuration, use the `FetcherSession` class. It can be used in both synchronous and asynchronous code without issue; the class automatically detects and changes the session type, without requiring a different import.
The `FetcherSession` class can accept nearly all the arguments that the methods can take, which enables you to specify a config for the entire session and later choose a different config for one of the requests effortlessly, as you will see in the following examples.
@@ -181,7 +182,7 @@ with FetcherSession(
timeout=30,
retries=3
) as session:
# Make multiple requests with the same settings
# Make multiple requests with the same settings and the same cookies
page1 = session.get('https://scrapling.requestcatcher.com/get')
page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
page3 = session.get('https://api.github.com/events')
docs/fetching/stealthy.md
+160 -144
View File
@@ -1,14 +1,17 @@
# Introduction
Here, we will discuss the `StealthyFetcher` class. This class is similar to [DynamicFetcher](dynamic.md#introduction) in many ways, including browser automation and the use of [Playwright's API](https://playwright.dev/python/docs/intro). The main difference is that this class provides advanced anti-bot protection bypass capabilities and a custom version of a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), from which most stealth comes.
Here, we will discuss the `StealthyFetcher` class. This class is very similar to the [DynamicFetcher](dynamic.md#introduction) class, including the browsers, the automation, and the use of [Playwright's API](https://playwright.dev/python/docs/intro). The main difference is that this class provides advanced anti-bot protection bypass capabilities; most of them are handled automatically under the hood, and the rest is up to you to enable.
As with [DynamicFetcher](dynamic.md#introduction), you will need some knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) to automate the page, as we will explain later.
**Note:** _This fetcher was using a custom version of [Camoufox](https://github.com/daijro/camoufox) as an engine before version 0.3.13, which was replaced now with [patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright) for many reasons. See [this section](#using-camoufox-as-an-engine) for information if you still need to use [Camoufox](https://github.com/daijro/camoufox). We might switch back to [Camoufox](https://github.com/daijro/camoufox) in the future if its development continues._
> 💡 **Prerequisites:**
>
> 1. You’ve completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
> 2. You’ve completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
> 3. You’ve completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
> 1. You've completed or read the [DynamicFetcher](dynamic.md#introduction) page since this class builds upon it, and we won't repeat the same information here for that reason.
> 2. You’ve completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
> 3. You’ve completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
> 4. You’ve completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
## Basic Usage
You have one primary way to import this Fetcher, which is the same for all fetchers.
@@ -20,81 +23,80 @@ Check out how to configure the parsing options [here](choosing.md#parser-configu
> Note: The async version of the `fetch` method is the `async_fetch` method, of course.
## What does it do?
The `StealthyFetcher` class is a stealthy version of the [DynamicFetcher](dynamic.md#introduction) class, and here are some of the things it does:
1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
6. It makes requests look as if they came from Google's search page of the requested website.
7. and other anti-protection options...
## Full list of arguments
Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments
| Argument | Description | Optional |
|:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
| url | Target url | ❌ |
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
| block_images | Prevent the loading of images through Firefox preferences. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ |
| disable_resources | Drop requests for unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ |
| cookies | Set cookies for the next request. | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
| block_webrtc | Blocks WebRTC entirely. | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ |
| addons | List of Firefox addons to use. **Must be paths to extracted addons.** | ✔️ |
| humanize | Humanize the cursor movement. The cursor movement takes either True or the maximum duration in seconds. The cursor typically takes up to 1.5 seconds to move across the window. | ✔️ |
| allow_webgl | Enabled by default. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
| geoip | Recommended to use with proxies; Automatically use IPs' longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. | ✔️ |
| os_randomize | If enabled, Scrapling will randomize the OS fingerprints used. The default is matching the fingerprints with the current OS. | ✔️ |
| disable_ads | Disabled by default; this installs the `uBlock Origin` addon on the browser if enabled. | ✔️ |
| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
| timeout | The timeout used in all operations and waits through the page. It's in milliseconds, and the default is 30000. | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| additional_args | Additional arguments to be passed to Camoufox as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
| Argument | Description | Optional |
|:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
| url | Target url | ❌ |
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
| disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
| cookies | Set cookies for the next request. | ✔️ |
| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ |
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
| locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
| timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |
| block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |
| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
| allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, and `selector_config`.
> 🔍 Notes:
>
> 1. It's basically the same arguments as [DynamicFetcher](dynamic.md#introduction) class but with these additional arguments `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
> 2. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
> 3. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
> 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
## Examples
It's easier to understand with examples, so we will now review most of the arguments individually.
It's easier to understand with examples, so we will now review most of the arguments individually. Since it's the same class as the [DynamicFetcher](dynamic.md#introduction), you can refer to that page for more examples, as we won't repeat all the examples from there.
### Browser Modes
```python
# Headless/hidden mode (default)
page = StealthyFetcher.fetch('https://example.com', headless=True)
# Visible browser mode
page = StealthyFetcher.fetch('https://example.com', headless=False)
```
### Resource Control
```python
# Block images
page = StealthyFetcher.fetch('https://example.com', block_images=True)
# Disable unnecessary resources
page = StealthyFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
```
### Cloudflare Protection Bypass
### Cloudflare and stealth options
```python
# Automatic Cloudflare solver
page = StealthyFetcher.fetch(
'https://nopecha.com/demo/cloudflare',
solve_cloudflare=True # Automatically solve Cloudflare challenges
)
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
# Works with other stealth options
page = StealthyFetcher.fetch(
'https://protected-site.com',
solve_cloudflare=True,
humanize=True,
geoip=True,
os_randomize=True
block_webrtc=True,
real_chrome=True,
hide_canvas=True,
google_search=True,
proxy='http://username:password@host:port', # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
)
```
@@ -104,64 +106,13 @@ The `solve_cloudflare` parameter enables automatic detection and solving all typ
- Interactive challenges (clicking verification boxes)
- Invisible challenges (automatic background verification)
And even solves the custom pages.
And even solves the custom pages with embedded captcha.
**Important notes:**
- Sometimes, with websites that use custom implementations, you will need to use `wait_selector` to make sure Scrapling waits for the real website content to be loaded after solving the captcha. Some websites can be the real definition of an edge case while we are trying to make the solver as generic as possible.
- When `solve_cloudflare=True` is enabled, `humanize=True` is automatically activated for more realistic behavior
- The timeout should be at least 60 seconds when using the Cloudflare solver for sufficient challenge-solving time
- This feature works seamlessly with proxies and other stealth options
### Additional stealth options
```python
page = StealthyFetcher.fetch(
'https://example.com',
block_webrtc=True, # Block WebRTC
allow_webgl=False, # Disable WebGL
humanize=True, # Make the mouse move as a human would move it
geoip=True, # Use IP's longitude, latitude, timezone, country, and locale, then spoof the WebRTC IP address...
os_randomize=True, # Randomize the OS fingerprints used. The default is matching the fingerprints with the current OS.
disable_ads=True, # Block ads with uBlock Origin addon (enabled by default)
google_search=True
)
# Custom humanization duration
page = StealthyFetcher.fetch(
'https://example.com',
humanize=1.5 # Max 1.5 seconds for cursor movement
)
```
The `google_search` argument is enabled by default, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
### Network Control
```python
# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
page = StealthyFetcher.fetch('https://example.com', network_idle=True)
# Custom timeout (in milliseconds)
page = StealthyFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
# Proxy support
page = StealthyFetcher.fetch(
'https://example.com',
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
)
```
### Downloading Files
```python
page = StealthyFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/poster.png')
with open(file='poster.png', mode='wb') as f:
f.write(page.body)
```
The `body` attribute of the `Response` object is a `bytes` object containing the response body in case of Non-HTML responses.
> 🔍 **Important notes:**
>
> 1. Sometimes, with websites that use custom implementations, you will need to use `wait_selector` to make sure Scrapling waits for the real website content to be loaded after solving the captcha. Some websites can be the real definition of an edge case while we are trying to make the solver as generic as possible.
> 2. The timeout should be at least 60 seconds when using the Cloudflare solver for sufficient challenge-solving time.
> 3. This feature works seamlessly with proxies and other stealth options.
### Browser Automation
This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues.
@@ -177,10 +128,7 @@ def scroll_page(page: Page):
page.mouse.move(100, 400)
page.mouse.up()
page = StealthyFetcher.fetch(
'https://example.com',
page_action=scroll_page
)
page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)
```
Of course, if you use the async fetch version, the function must also be async.
```python
@@ -191,10 +139,7 @@ async def scroll_page(page: Page):
await page.mouse.move(100, 400)
await page.mouse.up()
page = await StealthyFetcher.async_fetch(
'https://example.com',
page_action=scroll_page
)
page = await StealthyFetcher.async_fetch('https://example.com', page_action=scroll_page)
```
### Wait Conditions
@@ -217,19 +162,9 @@ The states the fetcher can wait for can be any of the following ([source](https:
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
### Firefox Addons
```python
# Custom Firefox addons
page = StealthyFetcher.fetch(
'https://example.com',
addons=['/path/to/addon1', '/path/to/addon2']
)
```
The paths here must point to extracted addons that will be installed automatically upon browser launch.
### Real-world example (Amazon)
This is for educational purposes only; this example was generated by AI, which shows how easy it is to work with Scrapling through AI
This is for educational purposes only; this example was generated by AI, which also shows how easy it is to work with Scrapling through AI
```python
def scrape_amazon_product(url):
# Use StealthyFetcher to bypass protection
@@ -261,8 +196,8 @@ from scrapling.fetchers import StealthySession
# Create a session with default configuration
with StealthySession(
headless=True,
geoip=True,
humanize=True,
real_chrome=True,
block_webrtc=True,
solve_cloudflare=True
) as session:
# Make multiple requests with the same browser instance
@@ -281,8 +216,8 @@ from scrapling.fetchers import AsyncStealthySession
async def scrape_multiple_sites():
async with AsyncStealthySession(
geoip=True,
os_randomize=True,
real_chrome=True,
block_webrtc=True,
solve_cloudflare=True,
timeout=60000, # 60 seconds for Cloudflare challenges
max_pages=3
@@ -296,12 +231,12 @@ async def scrape_multiple_sites():
return pages
```
You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
This logic allows for multiple websites to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
@@ -312,6 +247,87 @@ In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resou
- **Consistent fingerprint**: Same browser fingerprint across all requests.
- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
## Using Camoufox as an engine
If you see that Camoufox is stable on your device, has no high memory issues, and want to continue using Camoufox as before v0.3.13. This section is for you.
First, you will need to install the Camoufox library, browser, and Firefox system dependencies if you didn't already:
```commandline
pip install camoufox
playwright install-deps firefox
camoufox fetch
```
Then you will inherit from `StealthySession` and set it as below:
```python
from scrapling.fetchers import StealthySession
from playwright.sync_api import sync_playwright
from camoufox.utils import launch_options as generate_launch_options
class StealthySession(StealthySession):
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
self.playwright = sync_playwright().start()
# Configure camoufox run options here
launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})
# Here's an example, part of what we have been doing before v0.3.13
launch_options = generate_launch_options(**{
"geoip": False,
"proxy": self._config.proxy,
"headless": self._config.headless,
"humanize": True if self._config.solve_cloudflare else False, # Better enable humanize for Cloudflare, otherwise it's up to you
"i_know_what_im_doing": True, # To turn warnings off with the user configurations
"allow_webgl": self._config.allow_webgl,
"block_webrtc": self._config.block_webrtc,
"os": None,
"user_data_dir": self._config.user_data_dir,
"firefox_user_prefs": {
# This is what enabling `enable_cache` does internally, so we do it from here instead
"browser.sessionhistory.max_entries": 10,
"browser.sessionhistory.max_total_viewers": -1,
"browser.cache.memory.enable": True,
"browser.cache.disk_cache_ssl": True,
"browser.cache.disk.smart_size.enabled": True,
},
# etc...
})
self.context = self.playwright.firefox.launch_persistent_context(**launch_options)
else:
raise RuntimeError("Session has been already started")
```
After that, you can use it normally as before, even for solving Cloudflare challenges:
```python
with StealthySession(solve_cloudflare=True, headless=True) as session:
page = session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')
if page.css('#page-not-found-404'):
print('Cloudflare challenge solved successfully!')
```
The same logic applies to the `AsyncStealthySession` class with a few differences:
```python
from scrapling.fetchers import AsyncStealthySession
from playwright.async_api import async_playwright
from camoufox.utils import launch_options as generate_launch_options
class AsyncStealthySession(AsyncStealthySession):
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
self.playwright = await async_playwright().start()
# Configure camoufox run options here
launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})
# or set the launch options as in the above example
self.context = await self.playwright.firefox.launch_persistent_context(**launch_options)
else:
raise RuntimeError("Session has been already started")
async with AsyncStealthySession(solve_cloudflare=True, headless=True) as session:
page = await session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')
if page.css('#page-not-found-404'):
print('Cloudflare challenge solved successfully!')
```
Enjoy! :)
## When to Use
Use StealthyFetcher when:
docs/index.md
+8 -8
View File
@@ -18,7 +18,7 @@
Scrapling isn't just another Web Scraping library. It's the first **adaptive** scraping library that learns from website changes and evolves with them. While other libraries break when websites update their structure, Scrapling automatically relocates your elements and keeps your scrapers running.
Built for the modern Web, Scrapling features its own rapid parsing engine and fetchers to handle all Web Scraping challenges you face or will face. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
Built for the modern Web, Scrapling features **its own rapid parsing engine** and fetchers to handle all Web Scraping challenges you face or will face. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
```python
>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
@@ -51,8 +51,8 @@ Built for the modern Web, Scrapling features its own rapid parsing engine and fe
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, real Chrome, and custom stealth mode.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` using a modified version of Firefox and fingerprint spoofing. Can bypass all types of Cloudflare's Turnstile/Interstitial with automation easily.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, and Google's Chrome.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can bypass all types of Cloudflare's Turnstile/Interstitial with automation easily.
- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes.
@@ -60,7 +60,7 @@ Built for the modern Web, Scrapling features its own rapid parsing engine and fe
- 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
- 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage.
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
@@ -100,14 +100,14 @@ Starting with v0.3.2, this installation only includes the parser engine and its
### Optional Dependencies
1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies, and then install their browser dependencies with
1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:
```bash
pip install "scrapling[fetchers]"
scrapling install
```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies.
This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
2. Extra features:
@@ -135,10 +135,10 @@ Or download it from the GitHub registry:
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
This image is automatically built and pushed through GitHub actions on the repository's main branch.
This image is automatically built and pushed using GitHub Actions and the repository's main branch.
## How the documentation is organized
Scrapling has a lot of documentation, so we try to follow a guideline called the [Diátaxis documentation framework](https://diataxis.fr/).
Scrapling has extensive documentation, so we try to follow the [Diátaxis documentation framework](https://diataxis.fr/).
## Support
docs/overview.md
+22 -12
View File
@@ -1,4 +1,4 @@
We will start by quickly reviewing the parsing capabilities. Then, we will fetch websites with custom browsers, make requests, and parse the response.
We will start by quickly reviewing the parsing capabilities. Then we will fetch websites using custom browsers, make requests, and parse the responses.
Here's an HTML document generated by ChatGPT that we will be using as an example throughout this page:
```html
@@ -82,7 +82,7 @@ page.get_all_text(ignore_tags=('script', 'style'))
```
## Finding elements
If there's an element you want to find on the page, you will! Your creativity level is the only limitation!
If there's an element you want to find on the page, you will find it! Your creativity level is the only limitation!
Finding the first HTML `section` element
```python
@@ -94,14 +94,14 @@ Find all `section` elements
section_elements = page.find_all('section')
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
```
Find all `section` elements whose `id` attribute value is `products`
Find all `section` elements whose `id` attribute value is `products`.
```python
section_elements = page.find_all('section', {'id':"products"})
# Same as
section_elements = page.find_all('section', id="products")
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
```
Find all `section` elements whose `id` attribute value contains `product`
Find all `section` elements whose `id` attribute value contains `product`.
```python
section_elements = page.find_all('section', {'id*':"product"})
```
@@ -218,7 +218,7 @@ Using the elements we found above
<data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
>>> section_element.siblings
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`.
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
>>> section_element.children.css('h2::text')
['Products']
@@ -237,7 +237,7 @@ You can search for a specific ancestor of an element that satisfies a function;
```
## Fetching websites
Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers.
Instead of passing the raw HTML to Scrapling, you can retrieve a website's response directly via HTTP requests or by fetching it in a browser.
A fetcher is made for every use case.
@@ -267,7 +267,7 @@ For Async requests, you will replace the import like below:
> Notes:
>
> 1. You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from a Google search of this domain. It's enabled by default.
> 2. The `impersonate` argument allows you to fake the TLS fingerprint for a specific version of a browser.
> 2. The `impersonate` argument lets you fake the TLS fingerprint for a specific browser version.
> 3. There's also the `http3` argument, which, when enabled, makes the fetcher use HTTP/3 for requests, which makes your requests more authentic
This is just the tip of the iceberg with this fetcher; check out the rest from [here](fetching/static.md)
@@ -275,7 +275,7 @@ This is just the tip of the iceberg with this fetcher; check out the rest from [
### Dynamic loading
We have you covered if you deal with dynamic websites like most today!
The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides many options to fetch/load websites' pages through browsers.
The `DynamicFetcher` class (formerly `PlayWrightFetcher`) offers many options for fetching and loading web pages using Chromium-based browsers.
```python
>>> from scrapling.fetchers import DynamicFetcher
>>> page = DynamicFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
@@ -286,10 +286,9 @@ The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides ma
>>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling'
```
It's built on top of [Playwright](https://playwright.dev/python/) and it's currently providing three main run options that can be mixed as you want:
It's built on top of [Playwright](https://playwright.dev/python/), and it's currently providing two main run options that can be mixed as you want:
- Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser.
- Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode, but it bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/). Check out the `StealthyFetcher` class below for more advanced stealth mode. It uses the Chromium browser.
- Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
@@ -298,7 +297,18 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
### Dynamic anti-protection loading
We also have you covered if you deal with dynamic websites with annoying anti-protections!
The `StealthyFetcher` class uses a custom version of a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), bypassing most bot detections by default. Scrapling offers a faster custom version, includes extra tools, and features easy configurations to further increase undetectability.
The `StealthyFetcher` class uses a stealthy version of the `DynamicFetcher` explained above.
Some of the things it does:
1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
6. It makes requests look as if they came from Google's search page of the requested website.
7. and other anti-protection options...
```python
>>> from scrapling.fetchers import StealthyFetcher
>>> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default
@@ -318,4 +328,4 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
---
That's Scrapling at a glance. If you want to learn more about it, continue to the next section.
That's Scrapling at a glance. If you want to learn more, continue to the next section.
docs/parsing/adaptive.md
+18 -16
View File
@@ -50,7 +50,7 @@ When website owners implement structural changes like
```
The selector will no longer function, and your code needs maintenance. That's where Scrapling's `adaptive` feature comes into play.
With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element and without AI :)
With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element, and without AI :)
```python
from scrapling import Selector, Fetcher
@@ -65,7 +65,7 @@ if not element: # One day website changes?
element = page.css('#p1', adaptive=True) # Scrapling still finds it!
# the rest of your code...
```
Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
Below, I will show you an example of how to use this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
## Real-World Scenario
Let's use a real website as an example and use one of the fetchers to fetch its source. To achieve this, we need to identify a website that is about to update its design/structure, copy its source, and then wait for the website to change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
@@ -98,7 +98,9 @@ Note that I introduced a new argument called `adaptive_domain`. This is because,
The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `adaptive_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
Hence, in the two examples above, I used both the `Selector` class and the `Fetcher` class to show you that the logic for adaptive is the same.
Hence, in the two examples above, I used both the `Selector` and `Fetcher` classes to show that the adaptive logic is the same.
> Note: the main reason for creating the `adaptive_domain` argument was to handle if the website changed its URL while changing the design/structure. In that case, you can use it to continue using the previously stored adaptive data for the new URL. Otherwise, scrapling will consider it a new website and discard the old data.
## How the adaptive scraping feature works
Adaptive scraping works in two phases:
@@ -106,19 +108,19 @@ Adaptive scraping works in two phases:
1. **Save Phase**: Store unique properties of elements
2. **Match Phase**: Find elements with similar properties later
Let's say you've got an element through selection or any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
Let's say you've selected an element through any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
With as few technical details as possible, the general logic goes as follows:
1. You tell Scrapling to save that element's unique properties in one of the ways we will show below.
2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
3. Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
1. The domain of the current website. If you are using the `Selector` class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically.
2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this.
3. Now, because everything about the element can be changed or removed by the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
1. The domain of the current website. If you are using the `Selector` class, pass it when initializing; if you are using a fetcher, the domain will be automatically taken from the URL.
2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself; we'll discuss this later.
Together, they will be used to retrieve the element's unique properties from the database later.
Together, they will later be used to retrieve the element's unique properties from the database.
4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity with the desired element. In that comparison, everything is taken into consideration, as you will see later
4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated based on their similarity to the desired element. In that comparison, everything is taken into consideration, as you will see later
5. The element(s) with the highest similarity score to the wanted element are returned.
### The unique properties
@@ -129,7 +131,7 @@ For Scrapling, the unique elements we are relying on are:
- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
- Element's parent tag name, attributes (names and values), and text.
But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
But you need to understand that the comparison between elements isn't exact; it's more about how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
## How to use adaptive feature
The adaptive feature can be applied to any found element, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.
@@ -146,11 +148,11 @@ Examples:
```
If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain.
If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and the `adaptive` feature only uses the latest saved properties.
If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you use the same identifier later for a different website and don't pass the URL parameter when initializing it. The save process overwrites previous data, and the `adaptive` feature uses only the latest saved properties.
Besides those arguments, we have `storage` and `storage_args`. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
Besides those arguments, we have `storage` and `storage_args`. Both are for the class to connect to the database; by default, it uses the SQLite class provided by the library. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
Now, after enabling the `adaptive` feature globally, you have two main ways to use it.
Now that you've enabled the `adaptive` feature globally, you have two main ways to use it.
### The CSS/XPath Selection way
As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page, like below
@@ -163,7 +165,7 @@ element = page.css('#p1', adaptive=True)
```
Pretty simple, eh?
Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
Well, a lot happened under the hood here. Remember the identifier we mentioned before that you need to set to retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
Additionally, for all these methods, you can pass the `identifier` argument to set it yourself. This is useful in some instances, or you can use it to save properties with the `auto_save` argument.
@@ -174,7 +176,7 @@ First, let's say you got an element like this by text:
```python
>>> element = page.find_by_text('Tipping the Velvet', first_match=True)
```
You can save its unique properties with the `save` method, like below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
You can save its unique properties using the `save` method, as shown below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
```python
>>> page.save(element, 'my_special_element')
```
@@ -221,7 +223,7 @@ page.save(product, 'specific_product')
```
## Known Issues
In the `adaptive` save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone.
In the `adaptive` save process, only the unique properties of the first element in the selection results are saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors are separated and each is executed alone.
## Final thoughts
Explaining this feature in detail without complications turned out to be challenging. However, still, if there's something left unclear, you can head out to the [discussions section](https://github.com/D4Vinci/Scrapling/discussions), and I will reply to you ASAP, or the Discord server, or reach out to me privately and have a chat :)
docs/parsing/main_classes.md
+21 -21
View File
@@ -41,7 +41,7 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th
I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.
You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested.
After that, for the main page and elements within, most properties are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
After that, most properties on the main page and its elements are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
### Properties
You have already seen much of this on the [overview](../overview.md) page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the [traversal](#traversal) section.
@@ -111,7 +111,7 @@ But if you try to get the direct text content, it will be empty because it doesn
```
The `get_all_text` method has the following optional arguments:
1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'
1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'.
2. **strip**: If enabled, strings will be stripped before concatenation. Disabled by default.
3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results and ignore any elements nested within them. The default is `('script', 'style',)`.
4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
@@ -132,7 +132,7 @@ If you use it on the page directly, you will find that you are operating on the
>>> page.tag
'html'
```
Now, I think I hammered the (`page`/`element`) idea, so I won't return to it again.
Now, I think I've hammered the (`page`/`element`) idea, so I won't return to it.
Getting the attributes of the element
```python
@@ -196,7 +196,7 @@ Same case with XPath
### Traversal
Using the elements we found above, we will go over the properties/methods for moving on the page in detail.
If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding.
If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online to better understand them.
If you are too lazy to search about it, here's a quick explanation to give you a good idea.<br/>
In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.<br/>
@@ -310,7 +310,7 @@ In the [Selector](#selector) class, all methods/properties that should return a
Let's see what [Selectors](#selectors) class adds to the table with that out of the way.
### Properties
Apart from the normal operations on Python lists, such as iteration and slicing, etc.
Apart from the standard operations on Python lists, such as iteration and slicing.
You can do the following:
@@ -334,7 +334,7 @@ Execute CSS and XPath selectors directly on the [Selector](#selector) instances
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
...]
```
Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will still leave these methods to be explained in the [TextHandler](#texthandler) section below.
Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will leave the explanation of these methods to the [TextHandler](#texthandler) section below.
However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance.
```python
@@ -355,7 +355,7 @@ However, in this class, the `re_first` behaves differently as it runs `re` on ea
```
With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`.
```python
# Find all the products with price '53.23'
# Find all the products with price '53.23'.
>>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
>>> page.css('.product_pod').search(search_function)
<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>
@@ -374,7 +374,7 @@ If you are too lazy like me and want to know the number of [Selector](#selector)
```python
page.css('.product_pod').length
```
instead of this
which is equivalent to
```python
len(page.css('.product_pod'))
```
@@ -389,15 +389,15 @@ Of course, TextHandler provides extra methods and properties that standard Pytho
### Usage
First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well.
First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they will accept the same arguments as well.
First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they accept the same arguments.
- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance.
- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but, as you probably figured out from the name, it returns only the first result as a `TextHandler` instance.
Also, it takes other helpful arguments, which are:
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- **clean_match**: It's disabled by default. This causes the method to ignore all whitespace and consecutive spaces while matching.
- **case_sensitive**: It's enabled by default. As the name implies, disabling it will cause the regex to ignore the case of letters while compiling.
- **clean_match**: It's disabled by default. This causes the method to ignore all whitespace, including consecutive spaces, while matching.
- **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
```python
@@ -471,14 +471,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
```python
>>> page.css_first('div::text').json()
```
You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have direct text content at all.<br/><br/>
You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.<br/><br/>
In this case, the `get_all_text` method comes to the rescue, so you can do something like that
```python
>>> page.css_first('div').get_all_text(ignore_tags=[]).json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
```
I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.<br/><br/>
Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example:
Another related behavior to be aware of occurs when using any fetcher, which we will explain later. If you have a JSON response like this example:
```python
>>> page = Selector("""{"some_key": "some_value"}""")
```
@@ -493,14 +493,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
{'some_key': 'some_value'}
```
You might wonder how this happened, given that the `html` tag doesn't contain direct text.<br/>
Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.<br/><br/>This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions.
Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it uses the `get_all_text` method directly.<br/>
- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance
```python
>>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?'
```
Also, you can pass `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists
```python
@@ -518,10 +518,10 @@ Other methods and properties will be added over time, but remember that this cla
## TextHandlers
You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing new needs to be explained here, but new methods will be added over time.
The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. Nothing new needs to be explained here, but new methods will be added over time.
## AttributesHandler
This is a read-only version of Python's standard dictionary, or `dict`, that is used solely to store the attributes of each element or each [Selector](#selector) instance.
This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element or [Selector](#selector) instance.
```python
>>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'}
@@ -534,7 +534,7 @@ It currently adds two extra simple methods:
- The `search_values` method
In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will require some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
A simple example would be
```python
@@ -555,13 +555,13 @@ It currently adds two extra simple methods:
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
```
All these elements have 'product' as a value for the attribute `class`.
All these elements have 'product' as the value for the `class` attribute.
Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
- The `json_string` property
This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error
This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error.
```python
>>>page.find('script').attrib.json_string
docs/parsing/selection.md
+17 -17
View File
@@ -22,7 +22,7 @@ Scrapling implements CSS3 selectors as described in the [W3C specification](http
Also, Scrapling implements some non-standard pseudo-elements like:
* To select text nodes, use ``::text``
* To select text nodes, use ``::text``.
* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
@@ -36,19 +36,19 @@ In short, it is the same situation as CSS Selectors; if you come from Scrapy/Par
To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above, and `xpath_first` is faster.
> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here, as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
### Selectors examples
Let's see some shared examples of using CSS and XPath Selectors.
Select all elements with the class `product`
Select all elements with the class `product`.
```python
products = page.css('.product')
products = page.xpath('//*[@class="product"]')
```
Note: The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class**
Select the first element with the class `product`
Select the first element with the class `product`.
```python
product = page.css_first('.product')
product = page.xpath_first('//*[@class="product"]')
@@ -73,12 +73,12 @@ Get the `href` attribute of the first element with the `a` tag name
link = page.css_first('a::attr(href)')
link = page.xpath_first('//a/@href')
```
Select the text of the first element with the `h1` tag name, which contains 'Phone', and under an element with class 'product'
Select the text of the first element with the `h1` tag name, which contains `Phone`, and under an element with class `product`.
```python
title = page.css_first('.product h1:contains("Phone")::text')
title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
```
You can nest and chain selectors as you want, given that it returns results
You can nest and chain selectors as you want, given that they return results
```python
page.css_first('.product').css_first('h1:contains("Phone")::text')
page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
@@ -104,7 +104,7 @@ Scrapling provides the ability to select elements based on their direct text con
What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
With `find_by_text`, you will pass the text as the first argument; with the `find_by_regex` method, the regex pattern is the first argument. Both methods share the following arguments:
With `find_by_text`, you pass the text as the first argument; with `find_by_regex`, the regex pattern is the first argument. Both methods share the following arguments:
* **first_match**: If `True` (the default), the method used will return the first result it finds.
* **case_sensitive**: If `True`, the case of the letters will be considered.
@@ -117,7 +117,7 @@ By default, Scrapling searches for the exact matching of the text/pattern you pa
Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
### Finding Similar Elements
One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
One of the most remarkable new features Scrapling puts on the table is the ability to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
So, how does it work?
@@ -129,8 +129,8 @@ Imagine a scenario where you found a product by its title, for example, and you
That's a lot of talking, I know, but I had to go deep. I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the values of the attributes of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what the other arguments do first.
* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change a lot between elements, making them unreliable.
* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the tag attributes of both elements should be at least 20% similar. If you want to turn off this check (basically Step 3), you can set this attribute to 0, but I recommend you read what the other arguments do first.
* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change significantly across elements, making them unreliable.
* **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends.
Now, let's check out the examples below.
@@ -148,7 +148,7 @@ Find the first element whose text fully matches this text
>>> page.find_by_text('Tipping the Velvet')
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>
```
Combining it with `page.urljoin` to return the full URL from the relative `href`
Combining it with `page.urljoin` to return the full URL from the relative `href`.
```python
>>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html'
@@ -174,7 +174,7 @@ Get all elements that contain the word `the` (Partial matching)
'Mesaerion: The Best Science ...',
"It's Only the Himalayas"]
```
The search is case-insensitive, so those results have `The`, not only the lowercase one `the`; let's limit the search to the elements with `the` only.
The search is case-insensitive, so those results include `The`, not just the lowercase `the`; let's limit the search to elements with `the` only.
```python
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results]
@@ -332,14 +332,14 @@ def extract_reviews(page):
]
```
## Filters-based searching
This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier to learn for newcomers to Web Scraping than writing selectors.
This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier for newcomers to Web Scraping to learn than writing selectors.
Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to.
Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can accept multiple filters and return all elements on the pages where all these filters apply.
To be more specific:
* Any string passed is considered a tag name.
* Any iterable passed, like List/Tuple/Set, is considered an iterable of tag names.
* Any iterable passed, like List/Tuple/Set, will be considered as an iterable of tag names.
* Any dictionary is considered a mapping of HTML element(s), attribute names, and attribute values.
* Any regex patterns passed are used to filter elements by content, like the `find_by_regex` method
* Any functions passed are used to filter elements
@@ -357,7 +357,7 @@ It filters all elements in the current page/element in the following order:
Notes:
1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on.
2. The order in which you pass the arguments doesn't matter. The only order taken into consideration is the order explained above.
2. The order in which you pass the arguments doesn't matter. The only order considered is the one explained above.
Check examples to clear any confusion :)
@@ -509,4 +509,4 @@ We will have a deep look at it while explaining the [TextHandler](main_classes.m
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999']
```
And so on. You get the idea. We will explain this in more detail on the next page while explaining the [TextHandler](main_classes.md#texthandler) class.
And so on. You get the idea. We will explain this in more detail on the next page, along with the [TextHandler](main_classes.md#texthandler) class.
docs/requirements.txt
+8 -5
View File
@@ -1,5 +1,8 @@
mkdocs-material
mkdocstrings
mkdocstrings-python
mkdocs-material[imaging]
black
mkdocstrings>=1.0.0
mkdocstrings-python>=2.0.1
griffe-inherited-docstrings
griffe-runtime-objects
griffe-sphinx
mkdocs-material[imaging]>=9.7.1
black>=25.12.0
pngquant
docs/tutorials/migrating_from_beautifulsoup.md
+7 -7
View File
@@ -1,10 +1,10 @@
# Migrating from BeautifulSoup to Scrapling
If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is incredibly faster, provides the same parsing capabilities, adds more parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is much faster, provides the same parsing capabilities as BS, adds additional parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to accomplish a specific task using BeautifulSoup and the corresponding method in Scrapling.
Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to achieve a specific task using BeautifulSoup and the corresponding method in Scrapling.
You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, but that's one of the reasons why BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short oneliner, there is no need to sacrifice performance to shorten that short line :)
You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, which is one of the reasons BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short one-liner, there is no need to sacrifice performance to shorten that short line :)
| Task | BeautifulSoup Code | Scrapling Code |
@@ -56,7 +56,7 @@ Here's a simple example of scraping a web page to extract all the links using Be
import requests
from bs4 import BeautifulSoup
url = 'http://example.com'
url = 'https://example.com'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
@@ -70,7 +70,7 @@ for link in links:
```python
from scrapling import Fetcher
url = 'http://example.com'
url = 'https://example.com'
page = Fetcher.get(url)
links = page.css('a::attr(href)')
@@ -78,12 +78,12 @@ for link in links:
print(link)
```
As you can see, Scrapling simplifies the process by handling the fetching and parsing in a single step, making your code cleaner and more efficient.
As you can see, Scrapling simplifies the process by combining fetching and parsing into a single step, making your code cleaner and more efficient.
**Additional Notes:**
- **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons.
- **Element Types**: In BeautifulSoup, elements are `Tag` objects, while in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
- **Element Types**: In BeautifulSoup, elements are `Tag` objects; in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.css_first()`). To avoid errors, check for `None` before accessing properties.
- **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can help remove extra whitespace, consecutive spaces, or unwanted characters. Please check out the documentation for the complete list.
docs/tutorials/replacing_ai.md
+11 -11
View File
@@ -26,13 +26,13 @@ How will you solve that manually? I'm referring to generic web scraping of vario
## AI to the rescue, but at a high cost
Of course, the AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
Of course, AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
This approach is, of course, beautiful. I love AI and find it very fascinating, especially Generative AI. You will probably spend a lot of time on prompt engineering and tweaking the prompts, but if that's cool with you, you will soon hit the real issue with using AI here.
Most websites have vast amounts of content per page, which you will need to pass to the AI somehow so it can do its magic. This will burn through tokens like fire in a haystack, quickly accumulating high costs.
Unless money is irrelevant to you, you will try to find less expensive approaches, and that's why I made Scrapling :smile:
Unless money is irrelevant to you, you will try to find less expensive approaches, and that's where Scrapling comes into play :smile:
## Scrapling got you covered
@@ -41,11 +41,11 @@ Scrapling can handle almost all issues you will face during Web Scraping, and th
### Solving issue T1: Rapidly changing website structures
That's why the [adaptive](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/) feature was made. You knew I would talk about it, and here we are :)
While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties to find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties so you can find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
That's how the adaptive feature works: it stores everything unique about an element. When the website structure changes, it returns the element with the highest similarity score of the previous element.
I have already explained that in more detail and with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
I have already explained this in more detail, with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
### Solving issue T2: Unstable selectors
If you have been doing Web scraping for a long enough time, you have likely experienced this once. I'm referring to a website that employs poor design patterns, built on raw HTML without any IDs/classes, or uses random class names with nothing else to rely on, etc...
@@ -59,16 +59,16 @@ In these cases, standard selection methods with CSS/XPath selectors won't be opt
There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this.
### Solving issue T3: Increasingly complex anti-bot measures
It's known that making an undetectable spider takes more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides many flexible options, like stealth mode suitable for small to medium protections and using your real browser.
2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher utilizes our version of a modified Firefox browser, called [Camoufox](https://camoufox.com/stealth/), which nearly passes all known tests and incorporates additional tricks. **With v0.3, this fetcher can bypass Cloudflare for you automatically as well!**
1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
These two will be improved a lot with the upcoming updates, so stay tuned :)
We keep improving these two with each update, so stay tuned :)
### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data
This one is tough to handle, but it's possible with Scrapling's flexibility.
This one is tough to handle, but Scrapling's flexibility makes it possible.
I talked with someone who uses AI to extract prices from different websites. He is only interested in prices and titles, so he uses AI to find the price for him.
@@ -94,7 +94,7 @@ It will be a bit boring, but it's definitely less expensive than AI.
This example illustrates the point I aim to convey here. Not every challenge will need AI to be solved, but sometimes you need to be creative, and that might save you a lot of money.
### Solving issue B3: Pagination variations
This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the following updates :)
This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :)
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea.
@@ -112,6 +112,6 @@ For a quick comparison.
This table is based on pricing from [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing)
## Conclusion
While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit designed to tackle the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit for tackling the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects!
mkdocs.yml
+17 -4
View File
@@ -29,6 +29,7 @@ theme:
- navigation.sections
- navigation.tracking
- navigation.instant
- navigation.instant.prefetch
- navigation.instant.progress
# - navigation.tabs
# - navigation.expand
@@ -41,6 +42,7 @@ theme:
- content.action.view
- content.action.edit
- content.code.copy
- content.code.select
- content.code.annotate
- content.code.annotation
@@ -103,10 +105,13 @@ markdown_extensions:
plugins:
- search
# - social:
# cards_layout_options:
# background_color: "#1f1f1f"
# font_family: Roboto
- privacy:
links: false
- optimize
- social:
cards_layout_options:
background_color: "#1f1f1f"
font_family: Roboto
- mkdocstrings:
handlers:
python:
@@ -122,11 +127,19 @@ plugins:
unwrap_annotated: true
filters:
- '!^_'
- "^__"
merge_init_into_class: true
docstring_section_style: spacy
signature_crossrefs: true
show_symbol_type_heading: true
show_symbol_type_toc: true
show_inheritance_diagram: true
modernize_annotations: true
extensions:
- griffe_runtime_objects
- griffe_sphinx
- griffe_inherited_docstrings:
merge: true
extra:
homepage: https://scrapling.readthedocs.io/en/latest/
pyproject.toml
+3 -4
View File
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "scrapling"
# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
version = "0.3.12"
version = "0.3.13"
description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
readme = {file = "docs/README.md", content-type = "text/markdown"}
license = {file = "LICENSE"}
@@ -60,7 +60,7 @@ dependencies = [
"lxml>=6.0.2",
"cssselect>=1.3.0",
"orjson>=3.11.5",
"tldextract>=5.3.0",
"tldextract>=5.3.1",
]
[project.optional-dependencies]
@@ -69,8 +69,7 @@ fetchers = [
"curl_cffi>=0.14.0",
"playwright==1.56.0",
"patchright==1.56.0",
"camoufox>=0.4.11",
"geoip2>=5.2.0",
"browserforge>=1.2.1",
"msgspec>=0.20.0",
]
ai = [
scrapling/__init__.py
+1 -1
View File
@@ -1,5 +1,5 @@
__author__ = "Karim Shoair (karim.shoair@pm.me)"
__version__ = "0.3.12"
__version__ = "0.3.13"
__copyright__ = "Copyright (c) 2024 Karim Shoair"
from typing import Any, TYPE_CHECKING
scrapling/cli.py
+19 -52
View File
@@ -125,14 +125,9 @@ def install(force): # pragma: no cover
"playwright",
"install-deps",
"chromium",
"firefox",
],
"Playwright dependencies",
)
__Execute(
[python_executable, "-m", "camoufox", "fetch", "--browserforge"],
"Camoufox browser and databases",
)
# if no errors raised by the above commands, then we add the below file
__PACKAGE_DIR__.joinpath(".scrapling_dependencies_installed").touch()
else:
@@ -610,17 +605,11 @@ def delete(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
@option("--locale", default="en-US", help="Browser locale (default: en-US)")
@option("--stealth/--no-stealth", default=False, help="Enable stealth mode (default: False)")
@option("--locale", default=None, help="Specify user locale. Defaults to the system default locale.")
@option(
"--hide-canvas/--show-canvas",
"--real-chrome/--no-real-chrome",
default=False,
help="Add noise to canvas operations (default: False)",
)
@option(
"--disable-webgl/--enable-webgl",
default=False,
help="Disable WebGL support (default: False)",
help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
@@ -640,9 +629,7 @@ def fetch(
css_selector,
wait_selector,
locale,
stealth,
hide_canvas,
disable_webgl,
real_chrome,
proxy,
extra_headers,
):
@@ -659,9 +646,7 @@ def fetch(
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser.
:param stealth: Enables stealth mode.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -676,9 +661,7 @@ def fetch(
"network_idle": network_idle,
"timeout": timeout,
"locale": locale,
"stealth": stealth,
"hide_canvas": hide_canvas,
"disable_webgl": disable_webgl,
"real_chrome": real_chrome,
}
if wait > 0:
@@ -703,11 +686,6 @@ def fetch(
default=True,
help="Run browser in headless mode (default: True)",
)
@option(
"--block-images/--allow-images",
default=False,
help="Block image loading (default: False)",
)
@option(
"--disable-resources/--enable-resources",
default=False,
@@ -718,11 +696,6 @@ def fetch(
default=False,
help="Block WebRTC entirely (default: False)",
)
@option(
"--humanize/--no-humanize",
default=False,
help="Humanize cursor movement (default: False)",
)
@option(
"--solve-cloudflare/--no-solve-cloudflare",
default=False,
@@ -735,9 +708,14 @@ def fetch(
help="Wait for network idle (default: False)",
)
@option(
"--disable-ads/--allow-ads",
"--real-chrome/--no-real-chrome",
default=False,
help="Install uBlock Origin addon (default: False)",
help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
)
@option(
"--hide-canvas/--show-canvas",
default=False,
help="Add noise to canvas operations (default: False)",
)
@option(
"--timeout",
@@ -757,11 +735,6 @@ def fetch(
help="CSS selector to extract specific content from the page. It returns all matches.",
)
@option("--wait-selector", help="CSS selector to wait for before proceeding")
@option(
"--geoip/--no-geoip",
default=False,
help="Use IP geolocation for timezone/locale (default: False)",
)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option(
"--extra-headers",
@@ -773,19 +746,17 @@ def stealthy_fetch(
url,
output_file,
headless,
block_images,
disable_resources,
block_webrtc,
humanize,
solve_cloudflare,
allow_webgl,
network_idle,
disable_ads,
real_chrome,
hide_canvas,
timeout,
wait,
css_selector,
wait_selector,
geoip,
proxy,
extra_headers,
):
@@ -795,19 +766,17 @@ def stealthy_fetch(
:param url: Target url.
:param output_file: Output file path (.md for Markdown, .html for HTML).
:param headless: Run the browser in headless/hidden, or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences.
:param disable_resources: Drop requests of unnecessary resources for a speed boost.
:param block_webrtc: Blocks WebRTC entirely.
:param humanize: Humanize the cursor movement.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges.
:param allow_webgl: Allow WebGL (recommended to keep enabled).
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param disable_ads: Install the uBlock Origin addon on the browser.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before returning.
:param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param geoip: Automatically use IP's longitude, latitude, timezone, country, locale.
:param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request.
"""
@@ -818,16 +787,14 @@ def stealthy_fetch(
# Build request arguments
kwargs = {
"headless": headless,
"block_images": block_images,
"disable_resources": disable_resources,
"block_webrtc": block_webrtc,
"humanize": humanize,
"solve_cloudflare": solve_cloudflare,
"allow_webgl": allow_webgl,
"network_idle": network_idle,
"disable_ads": disable_ads,
"real_chrome": real_chrome,
"hide_canvas": hide_canvas,
"timeout": timeout,
"geoip": geoip,
}
if wait > 0:
scrapling/core/_shell_signatures.py
+41 -36
View File
@@ -1,9 +1,13 @@
from scrapling.core._types import (
Dict,
Any,
Dict,
List,
Tuple,
Sequence,
Callable,
Optional,
SetCookieParam,
SelectorWaitStates,
)
# Parameter definitions for shell function signatures (defined once at module level)
@@ -31,57 +35,58 @@ _REQUESTS_PARAMS = {
_FETCH_PARAMS = {
"headless": bool,
"google_search": bool,
"hide_canvas": bool,
"disable_webgl": bool,
"real_chrome": bool,
"stealth": bool,
"wait": int | float,
"page_action": Optional[Any],
"proxy": Optional[str | Dict],
"locale": str,
"extra_headers": Optional[Dict[str, str]],
"useragent": Optional[str],
"cdp_url": Optional[str],
"timeout": int | float,
"disable_resources": bool,
"wait_selector": Optional[str],
"init_script": Optional[str],
"cookies": Optional[List[Dict]],
"network_idle": bool,
"load_dom": bool,
"wait_selector_state": Any,
"extra_flags": Optional[List[str]],
"additional_args": Optional[Dict],
"wait_selector": Optional[str],
"wait_selector_state": SelectorWaitStates,
"cookies": Sequence[SetCookieParam],
"google_search": bool,
"wait": int | float,
"timezone_id": str | None,
"page_action": Optional[Callable],
"proxy": Optional[str | Dict[str, str] | Tuple],
"extra_headers": Optional[Dict[str, str]],
"timeout": int | float,
"init_script": Optional[str],
"user_data_dir": str,
"selector_config": Optional[Dict],
"additional_args": Optional[Dict],
"locale": Optional[str],
"real_chrome": bool,
"cdp_url": Optional[str],
"useragent": Optional[str],
"extra_flags": Optional[List[str]],
}
_STEALTHY_FETCH_PARAMS = {
"headless": bool,
"block_images": bool,
"disable_resources": bool,
"block_webrtc": bool,
"allow_webgl": bool,
"network_idle": bool,
"load_dom": bool,
"humanize": bool | float,
"solve_cloudflare": bool,
"wait": int | float,
"timeout": int | float,
"page_action": Optional[Any],
"wait_selector": Optional[str],
"init_script": Optional[str],
"addons": Optional[List[str]],
"wait_selector_state": Any,
"cookies": Optional[List[Dict]],
"wait_selector_state": SelectorWaitStates,
"cookies": Sequence[SetCookieParam],
"google_search": bool,
"wait": int | float,
"timezone_id": str | None,
"page_action": Optional[Callable],
"proxy": Optional[str | Dict[str, str] | Tuple],
"extra_headers": Optional[Dict[str, str]],
"proxy": Optional[str | Dict],
"os_randomize": bool,
"disable_ads": bool,
"geoip": bool,
"timeout": int | float,
"init_script": Optional[str],
"user_data_dir": str,
"selector_config": Optional[Dict],
"additional_args": Optional[Dict],
"locale": Optional[str],
"real_chrome": bool,
"cdp_url": Optional[str],
"useragent": Optional[str],
"extra_flags": Optional[List[str]],
"allow_webgl": bool,
"hide_canvas": bool,
"block_webrtc": bool,
"solve_cloudflare": bool,
}
# Mapping of function names to their parameter definitions
scrapling/core/_types.py
+14
View File
@@ -57,3 +57,17 @@ except ImportError: # pragma: no cover
from typing_extensions import Self # Backport
except ImportError:
Self = object
# Copied from `playwright._impl._api_structures.SetCookieParam`
class SetCookieParam(TypedDict, total=False):
name: str
value: str
url: Optional[str]
domain: Optional[str]
path: Optional[str]
expires: Optional[float]
httpOnly: Optional[bool]
secure: Optional[bool]
sameSite: Optional[Literal["Lax", "None", "Strict"]]
partitionKey: Optional[str]
scrapling/core/ai.py
+89 -104
View File
@@ -17,13 +17,15 @@ from scrapling.fetchers import (
from scrapling.core._types import (
Optional,
Tuple,
extraction_types,
Mapping,
Dict,
List,
Any,
SelectorWaitStates,
Generator,
Sequence,
SetCookieParam,
extraction_types,
SelectorWaitStates,
)
@@ -213,20 +215,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
hide_canvas: bool = False,
disable_webgl: bool = False,
real_chrome: bool = False,
stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
locale: str = "en-US",
timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
cookies: Optional[List[Dict]] = None,
cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> ResponseModel:
@@ -242,21 +242,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -269,15 +267,13 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
stealth=stealth,
cdp_url=cdp_url,
headless=headless,
useragent=useragent,
hide_canvas=hide_canvas,
timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
disable_webgl=disable_webgl,
extra_headers=extra_headers,
google_search=google_search,
disable_resources=disable_resources,
@@ -301,20 +297,18 @@ class ScraplingMCPServer:
main_content_only: bool = True,
headless: bool = False,
google_search: bool = True,
hide_canvas: bool = False,
disable_webgl: bool = False,
real_chrome: bool = False,
stealth: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
locale: str = "en-US",
timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
cookies: Optional[List[Dict]] = None,
cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
) -> List[ResponseModel]:
@@ -330,21 +324,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -356,17 +348,15 @@ class ScraplingMCPServer:
locale=locale,
timeout=timeout,
cookies=cookies,
stealth=stealth,
cdp_url=cdp_url,
headless=headless,
max_pages=len(urls),
useragent=useragent,
hide_canvas=hide_canvas,
timezone_id=timezone_id,
real_chrome=real_chrome,
network_idle=network_idle,
wait_selector=wait_selector,
google_search=google_search,
disable_webgl=disable_webgl,
extra_headers=extra_headers,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
@@ -393,29 +383,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
block_images: bool = False,
google_search: bool = True,
real_chrome: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
hide_canvas: bool = False,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
network_idle: bool = False,
humanize: bool | float = True,
solve_cloudflare: bool = False,
wait: int | float = 0,
timeout: int | float = 30000,
wait_selector: Optional[str] = None,
addons: Optional[List[str]] = None,
wait_selector_state: SelectorWaitStates = "attached",
cookies: Optional[List[Dict]] = None,
google_search: bool = True,
extra_headers: Optional[Dict[str, str]] = None,
proxy: Optional[str | Dict[str, str]] = None,
os_randomize: bool = False,
disable_ads: bool = False,
geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> ResponseModel:
"""Use Scrapling's version of the Camoufox browser to fetch a URL and return a structured output of the result.
Note: This is best suitable for high protection levels. It's slower than the other tools.
"""Use the stealthy fetcher to fetch a URL and return a structured output of the result.
Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param url: The URL to request.
@@ -426,54 +416,52 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param block_webrtc: Blocks WebRTC entirely.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
page = await StealthyFetcher.async_fetch(
url,
wait=wait,
proxy=proxy,
geoip=geoip,
addons=addons,
locale=locale,
cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
humanize=humanize,
useragent=useragent,
timezone_id=timezone_id,
real_chrome=real_chrome,
hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
disable_ads=disable_ads,
network_idle=network_idle,
block_images=block_images,
block_webrtc=block_webrtc,
os_randomize=os_randomize,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
additional_args=additional_args,
)
return _ContentTranslator(
Convertor._extract_content(
@@ -492,29 +480,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None,
main_content_only: bool = True,
headless: bool = True, # noqa: F821
block_images: bool = False,
google_search: bool = True,
real_chrome: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
hide_canvas: bool = False,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False,
wait_selector: Optional[str] = None,
cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False,
allow_webgl: bool = True,
network_idle: bool = False,
humanize: bool | float = True,
solve_cloudflare: bool = False,
wait: int | float = 0,
timeout: int | float = 30000,
wait_selector: Optional[str] = None,
addons: Optional[List[str]] = None,
wait_selector_state: SelectorWaitStates = "attached",
cookies: Optional[List[Dict]] = None,
google_search: bool = True,
extra_headers: Optional[Dict[str, str]] = None,
proxy: Optional[str | Dict[str, str]] = None,
os_randomize: bool = False,
disable_ads: bool = False,
geoip: bool = False,
additional_args: Optional[Dict] = None,
) -> List[ResponseModel]:
"""Use Scrapling's version of the Camoufox browser to fetch a group of URLs at the same time, and for each page return a structured output of the result.
Note: This is best suitable for high protection levels. It's slower than the other tools.
"""Use the stealthy fetcher to fetch a group of URLs at the same time, and for each page return a structured output of the result.
Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param urls: A tuple of the URLs to request.
@@ -525,54 +513,51 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param block_webrtc: Blocks WebRTC entirely.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
async with AsyncStealthySession(
wait=wait,
proxy=proxy,
geoip=geoip,
addons=addons,
locale=locale,
cdp_url=cdp_url,
timeout=timeout,
cookies=cookies,
headless=headless,
humanize=humanize,
max_pages=len(urls),
useragent=useragent,
timezone_id=timezone_id,
real_chrome=real_chrome,
hide_canvas=hide_canvas,
allow_webgl=allow_webgl,
disable_ads=disable_ads,
block_images=block_images,
block_webrtc=block_webrtc,
network_idle=network_idle,
os_randomize=os_randomize,
block_webrtc=block_webrtc,
wait_selector=wait_selector,
google_search=google_search,
extra_headers=extra_headers,
additional_args=additional_args,
solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources,
wait_selector_state=wait_selector_state,
additional_args=additional_args,
) as session:
tasks = [session.fetch(url) for url in urls]
responses = await gather(*tasks)
scrapling/core/translator.py
+1 -1
View File
@@ -130,5 +130,5 @@ translator = HTMLTranslator()
@lru_cache(maxsize=256)
def css_to_xpath(query: str) -> str:
"""Return translated XPath version of a given CSS query"""
"""Return the translated XPath version of a given CSS query"""
return translator.css_to_xpath(query)
scrapling/engines/_browsers/_base.py
+113 -155
View File
@@ -1,7 +1,6 @@
from time import time
from asyncio import sleep as asyncio_sleep, Lock
from camoufox import DefaultAddons
from playwright.sync_api._generated import Page
from playwright.sync_api import (
Frame,
@@ -17,18 +16,18 @@ from playwright.async_api import (
BrowserContext as AsyncBrowserContext,
)
from playwright._impl._errors import Error as PlaywrightError
from camoufox.pkgman import installed_verstr as camoufox_version
from camoufox.utils import launch_options as generate_launch_options
from ._page import PageInfo, PagePool
from scrapling.parser import Selector
from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING
from scrapling.engines.toolbelt.fingerprints import get_os_name
from ._validators import validate, PlaywrightConfig, CamoufoxConfig
from ._config_tools import _compiled_stealth_scripts, _launch_kwargs, _context_kwargs
from ._validators import validate, PlaywrightConfig, StealthConfig
from ._config_tools import __default_chrome_useragent__, __default_useragent__
from scrapling.engines.toolbelt.navigation import intercept_route, async_intercept_route
__ff_version_str__ = camoufox_version().split(".", 1)[0]
from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING, overload, Tuple
from scrapling.engines.constants import (
DEFAULT_STEALTH_FLAGS,
HARMFUL_DEFAULT_ARGS,
DEFAULT_FLAGS,
)
class SyncSession:
@@ -84,10 +83,6 @@ class SyncSession:
if disable_resources:
page.route("**/*", intercept_route)
if getattr(self, "stealth", False):
for script in _compiled_stealth_scripts():
page.add_init_script(script=script)
page_info = self.page_pool.add_page(page)
page_info.mark_busy()
return page_info
@@ -202,10 +197,6 @@ class AsyncSession:
if disable_resources:
await page.route("**/*", async_intercept_route)
if getattr(self, "stealth", False):
for script in _compiled_stealth_scripts():
await page.add_init_script(script=script)
return self.page_pool.add_page(page)
def get_pool_stats(self) -> Dict[str, int]:
@@ -251,151 +242,118 @@ class AsyncSession:
return handle_response
class DynamicSessionMixin:
def __validate__(self, **params):
class BaseSessionMixin:
@overload
def __validate_routine__(self, params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
@overload
def __validate_routine__(self, params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
def __validate_routine__(
self, params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]
) -> PlaywrightConfig | StealthConfig:
# Dark color scheme bypasses the 'prefersLightColor' check in creepjs
self._context_options: Dict[str, Any] = {"color_scheme": "dark", "device_scale_factor": 2}
self._launch_options: Dict[str, Any] = self._context_options | {
"args": DEFAULT_FLAGS,
"ignore_default_args": HARMFUL_DEFAULT_ARGS,
}
if "__max_pages" in params:
params["max_pages"] = params.pop("__max_pages")
config = validate(params, model=PlaywrightConfig)
config = validate(params, model=model)
self._headers_keys = (
{header.lower() for header in config.extra_headers.keys()} if config.extra_headers else set()
)
self._max_pages = config.max_pages
self._headless = config.headless
self._hide_canvas = config.hide_canvas
self._disable_webgl = config.disable_webgl
self._real_chrome = config.real_chrome
self._stealth = config.stealth
self._google_search = config.google_search
self._wait = config.wait
self._proxy = config.proxy
self._locale = config.locale
self._extra_headers = config.extra_headers
self._useragent = config.useragent
self._timeout = config.timeout
self._cookies = config.cookies
self._disable_resources = config.disable_resources
self._cdp_url = config.cdp_url
self._network_idle = config.network_idle
self._load_dom = config.load_dom
self._wait_selector = config.wait_selector
self._init_script = config.init_script
self._wait_selector_state = config.wait_selector_state
self._extra_flags = config.extra_flags
self._selector_config = config.selector_config
self._timezone_id = config.timezone_id
self._additional_args = config.additional_args
self._page_action = config.page_action
self._user_data_dir = config.user_data_dir
self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
self.__initiate_browser_options__()
return config
def __initiate_browser_options__(self):
if TYPE_CHECKING:
assert isinstance(self._proxy, tuple)
if not self._cdp_url:
# `launch_options` is used with persistent context
self.launch_options = dict(
_launch_kwargs(
self._headless,
self._proxy,
self._locale,
tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
self._useragent,
self._real_chrome,
self._stealth,
self._hide_canvas,
self._disable_webgl,
self._timezone_id,
tuple(self._extra_flags) if self._extra_flags else tuple(),
)
)
self.launch_options["extra_http_headers"] = dict(self.launch_options["extra_http_headers"])
self.launch_options["proxy"] = dict(self.launch_options["proxy"]) or None
self.launch_options["user_data_dir"] = self._user_data_dir
self.launch_options.update(cast(Dict, self._additional_args))
self.context_options = dict()
else:
# while `context_options` is left to be used when cdp mode is enabled
self.launch_options = dict()
self.context_options = dict(
_context_kwargs(
self._proxy,
self._locale,
tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
self._useragent,
self._stealth,
)
)
self.context_options["extra_http_headers"] = dict(self.context_options["extra_http_headers"])
self.context_options["proxy"] = dict(self.context_options["proxy"]) or None
self.context_options.update(cast(Dict, self._additional_args))
class StealthySessionMixin:
def __validate__(self, **params):
if "__max_pages" in params:
params["max_pages"] = params.pop("__max_pages")
config: CamoufoxConfig = validate(params, model=CamoufoxConfig)
self._max_pages = config.max_pages
self._headless = config.headless
self._block_images = config.block_images
self._disable_resources = config.disable_resources
self._block_webrtc = config.block_webrtc
self._allow_webgl = config.allow_webgl
self._network_idle = config.network_idle
self._load_dom = config.load_dom
self._humanize = config.humanize
self._solve_cloudflare = config.solve_cloudflare
self._wait = config.wait
self._timeout = config.timeout
self._page_action = config.page_action
self._wait_selector = config.wait_selector
self._init_script = config.init_script
self._addons = config.addons
self._wait_selector_state = config.wait_selector_state
self._cookies = config.cookies
self._google_search = config.google_search
self._extra_headers = config.extra_headers
self._proxy = config.proxy
self._os_randomize = config.os_randomize
self._disable_ads = config.disable_ads
self._geoip = config.geoip
self._selector_config = config.selector_config
self._additional_args = config.additional_args
self._user_data_dir = config.user_data_dir
self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
self.__initiate_browser_options__()
def __initiate_browser_options__(self):
"""Initiate browser options."""
self.launch_options: Dict[str, Any] = generate_launch_options(
**{
"geoip": self._geoip,
"proxy": dict(self._proxy) if self._proxy and isinstance(self._proxy, tuple) else self._proxy,
"addons": self._addons,
"exclude_addons": [] if self._disable_ads else [DefaultAddons.UBO],
"headless": self._headless,
"humanize": True if self._solve_cloudflare else self._humanize,
"i_know_what_im_doing": True, # To turn warnings off with the user configurations
"allow_webgl": self._allow_webgl,
"block_webrtc": self._block_webrtc,
"block_images": self._block_images, # Careful! it makes some websites don't finish loading at all like stackoverflow even in headful mode.
"os": None if self._os_randomize else get_os_name(),
"user_data_dir": self._user_data_dir,
"ff_version": __ff_version_str__,
"firefox_user_prefs": {
# This is what enabling `enable_cache` does internally, so we do it from here instead
"browser.sessionhistory.max_entries": 10,
"browser.sessionhistory.max_total_viewers": -1,
"browser.cache.memory.enable": True,
"browser.cache.disk_cache_ssl": True,
"browser.cache.disk.smart_size.enabled": True,
},
**cast(Dict, self._additional_args),
def __generate_options__(self, extra_flags: Tuple | None = None) -> None:
config = cast(PlaywrightConfig, getattr(self, "_config", None))
self._context_options.update(
{
"proxy": config.proxy,
"locale": config.locale,
"timezone_id": config.timezone_id,
"extra_http_headers": config.extra_headers,
}
)
# The default useragent in the headful is always correct now in the current versions of Playwright
if config.useragent:
self._context_options["user_agent"] = config.useragent
elif not config.useragent and config.headless:
self._context_options["user_agent"] = (
__default_chrome_useragent__ if config.real_chrome else __default_useragent__
)
if not config.cdp_url:
self._launch_options |= self._context_options
self._context_options = {}
flags = self._launch_options["args"]
if config.extra_flags or extra_flags:
flags = list(set(flags + (config.extra_flags or extra_flags)))
self._launch_options.update(
{
"args": flags,
"headless": config.headless,
"user_data_dir": config.user_data_dir,
"channel": "chrome" if config.real_chrome else "chromium",
}
)
if config.additional_args:
self._launch_options.update(config.additional_args)
else:
# while `context_options` is left to be used when cdp mode is enabled
self._launch_options = dict()
if config.additional_args:
self._context_options.update(config.additional_args)
class DynamicSessionMixin(BaseSessionMixin):
def __validate__(self, **params):
self._config = self.__validate_routine__(params, model=PlaywrightConfig)
self.__generate_options__()
class StealthySessionMixin(BaseSessionMixin):
def __validate__(self, **params):
self._config: StealthConfig = self.__validate_routine__(params, model=StealthConfig)
self._context_options.update(
{
"is_mobile": False,
"has_touch": False,
# I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
"service_workers": "allow",
"ignore_https_errors": True,
"screen": {"width": 1920, "height": 1080},
"viewport": {"width": 1920, "height": 1080},
"permissions": ["geolocation", "notifications"],
}
)
self.__generate_stealth_options()
def __generate_stealth_options(self) -> None:
flags = tuple()
if not self._config.cdp_url:
flags = DEFAULT_FLAGS + DEFAULT_STEALTH_FLAGS
if self._config.block_webrtc:
flags += (
"--webrtc-ip-handling-policy=disable_non_proxied_udp",
"--force-webrtc-ip-handling-policy", # Ensures the policy is enforced
)
if not self._config.allow_webgl:
flags += (
"--disable-webgl",
"--disable-webgl-image-chromium",
"--disable-webgl2",
)
if self._config.hide_canvas:
flags += ("--fingerprinting-canvas-image-data-noise",)
super(StealthySessionMixin, self).__generate_options__(flags)
@staticmethod
def _detect_cloudflare(page_content: str) -> str | None:
scrapling/engines/_browsers/_config_tools.py
+1 -102
View File
@@ -1,15 +1,10 @@
from functools import lru_cache
from scrapling.core._types import Tuple
from scrapling.engines.constants import (
DEFAULT_STEALTH_FLAGS,
HARMFUL_DEFAULT_ARGS,
DEFAULT_FLAGS,
)
from scrapling.engines.toolbelt.navigation import js_bypass_path
from scrapling.engines.toolbelt.fingerprints import generate_headers
__default_useragent__ = generate_headers(browser_mode=True).get("User-Agent")
__default_chrome_useragent__ = generate_headers(browser_mode="chrome").get("User-Agent")
@lru_cache(1)
@@ -41,99 +36,3 @@ def _compiled_stealth_scripts():
with open(script_path, "r") as f:
scripts.append(f.read())
return tuple(scripts)
@lru_cache(2, typed=True)
def _set_flags(hide_canvas, disable_webgl): # pragma: no cover
"""Returns the flags that will be used while launching the browser if stealth mode is enabled"""
flags = DEFAULT_STEALTH_FLAGS
if hide_canvas:
flags += ("--fingerprinting-canvas-image-data-noise",)
if disable_webgl:
flags += (
"--disable-webgl",
"--disable-webgl-image-chromium",
"--disable-webgl2",
)
return flags
@lru_cache(2, typed=True)
def _launch_kwargs(
headless,
proxy: Tuple,
locale,
extra_headers,
useragent,
real_chrome,
stealth,
hide_canvas,
disable_webgl,
timezone_id,
extra_flags: Tuple,
) -> Tuple:
"""Creates the arguments we will use while launching playwright's browser"""
base_args = DEFAULT_FLAGS
if extra_flags:
base_args = base_args + extra_flags
launch_kwargs = {
"locale": locale,
"timezone_id": timezone_id or None,
"headless": headless,
"args": base_args,
"color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
"proxy": proxy or tuple(),
"device_scale_factor": 2,
"ignore_default_args": HARMFUL_DEFAULT_ARGS,
"channel": "chrome" if real_chrome else "chromium",
"extra_http_headers": extra_headers or tuple(),
"user_agent": useragent or __default_useragent__,
}
if stealth:
stealth_args = base_args + _set_flags(hide_canvas, disable_webgl)
launch_kwargs.update(
{
"args": stealth_args,
"chromium_sandbox": True,
"is_mobile": False,
"has_touch": False,
# I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
"service_workers": "allow",
"ignore_https_errors": True,
"screen": {"width": 1920, "height": 1080},
"viewport": {"width": 1920, "height": 1080},
"permissions": ["geolocation", "notifications"],
}
)
return tuple(launch_kwargs.items())
@lru_cache(2, typed=True)
def _context_kwargs(proxy, locale, extra_headers, useragent, stealth) -> Tuple:
"""Creates the arguments for the browser context"""
context_kwargs = {
"proxy": proxy or tuple(),
"locale": locale,
"color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
"device_scale_factor": 2,
"extra_http_headers": extra_headers or tuple(),
"user_agent": useragent or __default_useragent__,
}
if stealth:
context_kwargs.update(
{
"is_mobile": False,
"has_touch": False,
# I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
"service_workers": "allow",
"ignore_https_errors": True,
"screen": {"width": 1920, "height": 1080},
"viewport": {"width": 1920, "height": 1080},
"permissions": ["geolocation", "notifications"],
}
)
return tuple(context_kwargs.items())
scrapling/engines/_browsers/_controllers.py
+61 -104
View File
@@ -9,8 +9,6 @@ from playwright.async_api import (
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
from patchright.sync_api import sync_playwright as sync_patchright
from patchright.async_api import async_playwright as async_patchright
from scrapling.core.utils import log
from scrapling.core._types import Unpack, TYPE_CHECKING
@@ -21,53 +19,27 @@ from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
class DynamicSession(DynamicSessionMixin, SyncSession):
class DynamicSession(SyncSession, DynamicSessionMixin):
"""A Browser session manager with page pooling."""
__slots__ = (
"_max_pages",
"_headless",
"_hide_canvas",
"_disable_webgl",
"_real_chrome",
"_stealth",
"_google_search",
"_proxy",
"_locale",
"_extra_headers",
"_useragent",
"_timeout",
"_cookies",
"_disable_resources",
"_network_idle",
"_load_dom",
"_wait_selector",
"_init_script",
"_wait_selector_state",
"_wait",
"playwright",
"browser",
"context",
"_config",
"_context_options",
"_launch_options",
"max_pages",
"page_pool",
"_max_wait_for_page",
"playwright",
"context",
"_closed",
"_selector_config",
"_page_action",
"launch_options",
"context_options",
"_cdp_url",
"_headers_keys",
"_extra_flags",
"_additional_args",
"_user_data_dir",
)
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -76,13 +48,11 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param timezone_id: Set the timezone for the browser if wanted.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
@@ -94,27 +64,24 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages)
super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
sync_context = sync_patchright if self._stealth else sync_playwright
self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
self.playwright: Playwright = sync_context().start() # pyright: ignore [reportAttributeAccessIssue]
if self._cdp_url: # pragma: no cover
self.context = self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url).new_context(
**self.context_options
)
if self._config.cdp_url: # pragma: no cover
browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
self.context = browser.new_context(**self._context_options)
else:
self.context = self.playwright.chromium.launch_persistent_context(**self.launch_options)
self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
if self._init_script: # pragma: no cover
self.context.add_init_script(path=self._init_script)
if self._config.init_script: # pragma: no cover
self.context.add_init_script(path=self._config.init_script)
if self._cookies: # pragma: no cover
self.context.add_cookies(self._cookies)
if self._config.cookies: # pragma: no cover
self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -122,24 +89,21 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
:param kwargs: Additional keyword arguments including:
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -193,16 +157,15 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
raise e
class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
"""An async Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory."""
def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -212,13 +175,11 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param timezone_id: Set the timezone for the browser if wanted.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -230,28 +191,26 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages)
super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
async_context = async_patchright if self._stealth else async_playwright
self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
self.playwright: AsyncPlaywright = await async_context().start() # pyright: ignore [reportAttributeAccessIssue]
if self._cdp_url:
browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url)
self.context: AsyncBrowserContext = await browser.new_context(**self.context_options)
if self._config.cdp_url:
browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
else:
self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
**self.launch_options
**self._launch_options
)
if self._init_script: # pragma: no cover
await self.context.add_init_script(path=self._init_script)
if self._config.init_script: # pragma: no cover
await self.context.add_init_script(path=self._config.init_script)
if self._cookies:
await self.context.add_cookies(self._cookies) # pyright: ignore
if self._config.cookies:
await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
@@ -259,20 +218,18 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
:param kwargs: Additional keyword arguments including:
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, PlaywrightConfig)
scrapling/engines/_browsers/_camoufox.py → scrapling/engines/_browsers/_stealth.py
+140 -148
View File
@@ -2,117 +2,101 @@ from random import randint
from re import compile as re_compile
from playwright.sync_api import (
Page,
Locator,
sync_playwright,
Page,
Playwright,
)
from playwright.async_api import (
async_playwright,
Page as async_Page,
Locator as AsyncLocator,
Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext,
)
from patchright.sync_api import sync_playwright
from patchright.async_api import async_playwright
from scrapling.core.utils import log
from ._types import CamoufoxSession, CamoufoxFetchParams
from scrapling.core._types import Any, Unpack, TYPE_CHECKING
from scrapling.core._types import Any, Unpack
from ._config_tools import _compiled_stealth_scripts
from ._types import StealthSession, StealthFetchParams
from ._base import SyncSession, AsyncSession, StealthySessionMixin
from ._validators import validate_fetch as _validate, CamoufoxConfig
from ._validators import validate_fetch as _validate, StealthConfig
from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
__CF_PATTERN__ = re_compile("challenges.cloudflare.com/cdn-cgi/challenge-platform/.*")
class StealthySession(StealthySessionMixin, SyncSession):
"""A Stealthy session manager with page pooling."""
class StealthySession(SyncSession, StealthySessionMixin):
"""A Stealthy Browser session manager with page pooling."""
__slots__ = (
"_max_pages",
"_headless",
"_block_images",
"_disable_resources",
"_block_webrtc",
"_allow_webgl",
"_network_idle",
"_load_dom",
"_humanize",
"_solve_cloudflare",
"_wait",
"_timeout",
"_page_action",
"_wait_selector",
"_init_script",
"_addons",
"_wait_selector_state",
"_cookies",
"_google_search",
"_extra_headers",
"_proxy",
"_os_randomize",
"_disable_ads",
"_geoip",
"_selector_config",
"_additional_args",
"playwright",
"browser",
"context",
"_config",
"_context_options",
"_launch_options",
"max_pages",
"page_pool",
"_max_wait_for_page",
"playwright",
"context",
"_closed",
"launch_options",
"_headers_keys",
"_user_data_dir",
)
def __init__(self, **kwargs: Unpack[CamoufoxSession]):
"""A Browser session manager with page pooling
def __init__(self, **kwargs: Unpack[StealthSession]):
"""A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param block_webrtc: Blocks WebRTC entirely.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages)
super().__init__()
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
self.playwright = sync_playwright().start()
self.context = self.playwright.firefox.launch_persistent_context(**self.launch_options)
self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
if self._init_script: # pragma: no cover
self.context.add_init_script(path=self._init_script)
if self._config.cdp_url: # pragma: no cover
browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
self.context = browser.new_context(**self._context_options)
else:
self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
if self._cookies: # pragma: no cover
self.context.add_cookies(self._cookies)
for script in _compiled_stealth_scripts():
self.context.add_init_script(script=script)
if self._config.init_script: # pragma: no cover
self.context.add_init_script(path=self._config.init_script)
if self._config.cookies: # pragma: no cover
self.context.add_cookies(self._config.cookies)
else:
raise RuntimeError("Session has been already started")
@@ -148,22 +132,27 @@ class StealthySession(StealthySessionMixin, SyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
self._wait_for_page_stability(iframe, True, True)
self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not iframe.frame_element().is_visible():
# Double-checking that the iframe is loaded
page.wait_for_timeout(500)
outer_box: Any = iframe.frame_element().bounding_box()
if not iframe or not outer_box:
if "<title>Just a moment...</title>" not in (ResponseFactory._get_page_content(page)):
log.info("Cloudflare captcha is solved")
return
outer_box: Any = page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -182,29 +171,26 @@ class StealthySession(StealthySessionMixin, SyncSession):
log.info("Cloudflare captcha is solved")
return
def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
:param kwargs: Additional keyword arguments including:
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, CamoufoxConfig)
params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -233,7 +219,7 @@ class StealthySession(StealthySessionMixin, SyncSession):
if params.page_action:
try:
_ = params.page_action(page_info.page)
except Exception as e:
except Exception as e: # pragma: no cover
log.error(f"Error executing page_action: {e}")
if params.wait_selector:
@@ -242,10 +228,12 @@ class StealthySession(StealthySessionMixin, SyncSession):
waiter.first.wait_for(state=params.wait_selector_state)
# Wait again after waiting for the selector, helpful with protections like Cloudflare
self._wait_for_page_stability(page_info.page, params.load_dom, params.network_idle)
except Exception as e:
except Exception as e: # pragma: no cover
log.error(f"Error waiting for selector {params.wait_selector}: {e}")
page_info.page.wait_for_timeout(params.wait)
# Create response object
response = ResponseFactory.from_playwright_response(
page_info.page, first_response, final_response[0], params.selector_config
)
@@ -256,72 +244,78 @@ class StealthySession(StealthySessionMixin, SyncSession):
return response
except Exception as e: # pragma: no cover
except Exception as e:
page_info.mark_error()
raise e
class AsyncStealthySession(StealthySessionMixin, AsyncSession):
"""A Stealthy session manager with page pooling."""
class AsyncStealthySession(AsyncSession, StealthySessionMixin):
"""An async Stealthy Browser session manager with page pooling."""
def __init__(self, **kwargs: Unpack[CamoufoxSession]):
"""A Browser session manager with page pooling
def __init__(self, **kwargs: Unpack[StealthSession]):
"""A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param block_webrtc: Blocks WebRTC entirely.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param max_pages: The maximum number of tabs to be opened at the same time. It will be used in rotation through a PagePool.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
"""
self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages)
super().__init__(max_pages=self._config.max_pages)
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
self.playwright: AsyncPlaywright = await async_playwright().start()
self.context: AsyncBrowserContext = await self.playwright.firefox.launch_persistent_context(
**self.launch_options
)
self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
if self._init_script: # pragma: no cover
await self.context.add_init_script(path=self._init_script)
if self._config.cdp_url:
browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
else:
self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
**self._launch_options
)
if self._cookies:
await self.context.add_cookies(self._cookies) # pyright: ignore [reportArgumentType]
for script in _compiled_stealth_scripts():
await self.context.add_init_script(script=script)
if self._config.init_script: # pragma: no cover
await self.context.add_init_script(path=self._config.init_script)
if self._config.cookies:
await self.context.add_cookies(self._config.cookies) # pyright: ignore
else:
raise RuntimeError("Session has been already started")
async def _cloudflare_solver(self, page: async_Page): # pragma: no cover
"""Solve the cloudflare challenge displayed on the playwright page passed. The async version
async def _cloudflare_solver(self, page: async_Page) -> None: # pragma: no cover
"""Solve the cloudflare challenge displayed on the playwright page passed
:param page: The async targeted page
:param page: The targeted page
:return:
"""
await self._wait_for_networkidle(page, timeout=5000)
@@ -331,7 +325,7 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
return
else:
log.info(f'The turnstile version discovered is "{challenge_type}"')
if challenge_type == "non-interactive": # pragma: no cover
if challenge_type == "non-interactive":
while "<title>Just a moment...</title>" in (await ResponseFactory._get_async_page_content(page)):
log.info("Waiting for Cloudflare wait page to disappear.")
await page.wait_for_timeout(1000)
@@ -350,22 +344,27 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
outer_box = {}
iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None:
await self._wait_for_page_stability(iframe, True, True)
await self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded":
while not await (await iframe.frame_element()).is_visible():
# Double-checking that the iframe is loaded
await page.wait_for_timeout(500)
outer_box: Any = await (await iframe.frame_element()).bounding_box()
if not iframe or not outer_box:
if "<title>Just a moment...</title>" not in (await ResponseFactory._get_async_page_content(page)):
log.info("Cloudflare captcha is solved")
return
outer_box: Any = await page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button
await page.mouse.click(captcha_x, captcha_y, delay=60, button="left")
await page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
await self._wait_for_networkidle(page)
if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -377,35 +376,33 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
await page.wait_for_timeout(100)
attempts += 1
if challenge_type != "embedded":
await page.locator(box_selector).wait_for(state="detached")
await page.locator(box_selector).last.wait_for(state="detached")
await page.locator(".zone-name-title").wait_for(state="hidden")
await self._wait_for_page_stability(page, True, False)
log.info("Cloudflare captcha is solved")
return
async def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response:
async def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options.
:param url: The Target url.
:param kwargs: Additional keyword arguments including:
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object.
"""
params = _validate(kwargs, self, CamoufoxConfig)
params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed")
@@ -418,10 +415,6 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
final_response = [None]
handle_response = self._create_response_handler(page_info, final_response)
if TYPE_CHECKING:
if not isinstance(page_info.page, async_Page):
raise TypeError
try:
# Navigate to URL and wait for a specified state
page_info.page.on("response", handle_response)
@@ -461,9 +454,8 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
# Close the page to free up resources
await page_info.page.close()
self.page_pool.pages.remove(page_info)
return response
except Exception as e:
except Exception as e: # pragma: no cover
page_info.mark_error()
raise e
scrapling/engines/_browsers/_types.py
+19 -27
View File
@@ -11,9 +11,10 @@ from scrapling.core._types import (
Mapping,
Optional,
Callable,
Iterable,
Sequence,
TypedDict,
TypeAlias,
SetCookieParam,
SelectorWaitStates,
TYPE_CHECKING,
)
@@ -53,7 +54,7 @@ if TYPE_CHECKING: # pragma: no cover
json: Optional[Dict | List]
# Types for browser session
class BrowserSession(TypedDict, total=False):
class PlaywrightSession(TypedDict, total=False):
max_pages: int
headless: bool
disable_resources: bool
@@ -61,9 +62,10 @@ if TYPE_CHECKING: # pragma: no cover
load_dom: bool
wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates
cookies: Optional[Iterable[Dict]]
cookies: Sequence[SetCookieParam] | None
google_search: bool
wait: int | float
timezone_id: str | None
page_action: Optional[Callable]
proxy: Optional[str | Dict[str, str] | Tuple]
extra_headers: Optional[Dict[str, str]]
@@ -72,42 +74,32 @@ if TYPE_CHECKING: # pragma: no cover
user_data_dir: str
selector_config: Optional[Dict]
additional_args: Optional[Dict]
class PlaywrightSession(BrowserSession, total=False):
cdp_url: Optional[str]
hide_canvas: bool
disable_webgl: bool
locale: Optional[str]
real_chrome: bool
stealth: bool
locale: str
cdp_url: Optional[str]
useragent: Optional[str]
extra_flags: Optional[List[str]]
class PlaywrightFetchParams(TypedDict, total=False):
load_dom: bool
wait: int | float
network_idle: bool
google_search: bool
timeout: int | float
wait: int | float
page_action: Optional[Callable]
extra_headers: Optional[Dict[str, str]]
disable_resources: bool
wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates
network_idle: bool
load_dom: bool
page_action: Optional[Callable]
selector_config: Optional[Dict]
extra_headers: Optional[Dict[str, str]]
wait_selector_state: SelectorWaitStates
class CamoufoxSession(BrowserSession, total=False):
block_images: bool
block_webrtc: bool
class StealthSession(PlaywrightSession, total=False):
allow_webgl: bool
humanize: bool | float
hide_canvas: bool
block_webrtc: bool
solve_cloudflare: bool
addons: Optional[List[str]]
os_randomize: bool
disable_ads: bool
geoip: bool
class CamoufoxFetchParams(PlaywrightFetchParams, total=False):
class StealthFetchParams(PlaywrightFetchParams, total=False):
solve_cloudflare: bool
else: # pragma: no cover
@@ -116,5 +108,5 @@ else: # pragma: no cover
DataRequestParams = TypedDict
PlaywrightSession = TypedDict
PlaywrightFetchParams = TypedDict
CamoufoxSession = TypedDict
CamoufoxFetchParams = TypedDict
StealthSession = TypedDict
StealthFetchParams = TypedDict
scrapling/engines/_browsers/_validators.py
+39 -97
View File
@@ -13,12 +13,13 @@ from scrapling.core._types import (
Tuple,
Optional,
Callable,
Iterable,
SelectorWaitStates,
Sequence,
overload,
SetCookieParam,
SelectorWaitStates,
)
from scrapling.engines.toolbelt.navigation import construct_proxy_dict
from scrapling.engines._browsers._types import PlaywrightFetchParams, CamoufoxFetchParams
from scrapling.engines._browsers._types import PlaywrightFetchParams, StealthFetchParams
# Custom validators for msgspec
@@ -35,15 +36,6 @@ def _is_invalid_file_path(value: str) -> bool | str: # pragma: no cover
return False
def _validate_addon_path(value: str) -> None: # pragma: no cover
"""Fast addon path validation"""
path = Path(value)
if not path.exists():
raise FileNotFoundError(f"Addon path not found: {value}")
if not path.is_dir():
raise ValueError(f"Addon path must be a directory of the extracted addon: {value}")
@lru_cache(2)
def _is_invalid_cdp_url(cdp_url: str) -> bool | str:
"""Fast CDP URL validation"""
@@ -65,39 +57,36 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
"""Configuration struct for validation"""
max_pages: PagesCount = 1
cdp_url: Optional[str] = None
headless: bool = True
google_search: bool = True
hide_canvas: bool = False
disable_webgl: bool = False
real_chrome: bool = False
stealth: bool = False
wait: Seconds = 0
page_action: Optional[Callable] = None
proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
locale: str = "en-US"
extra_headers: Optional[Dict[str, str]] = None
useragent: Optional[str] = None
timeout: Seconds = 30000
init_script: Optional[str] = None
disable_resources: bool = False
wait_selector: Optional[str] = None
cookies: Optional[Iterable[Dict]] = None
network_idle: bool = False
load_dom: bool = True
wait_selector: Optional[str] = None
wait_selector_state: SelectorWaitStates = "attached"
cookies: Sequence[SetCookieParam] | None = []
google_search: bool = True
wait: Seconds = 0
timezone_id: str | None = ""
page_action: Optional[Callable] = None
proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
extra_headers: Optional[Dict[str, str]] = None
timeout: Seconds = 30000
init_script: Optional[str] = None
user_data_dir: str = ""
timezone_id: str = ""
extra_flags: Optional[List[str]] = None
selector_config: Optional[Dict] = {}
additional_args: Optional[Dict] = {}
locale: str | None = None
real_chrome: bool = False
cdp_url: Optional[str] = None
useragent: Optional[str] = None
extra_flags: Optional[List[str]] = None
def __post_init__(self): # pragma: no cover
"""Custom validation after msgspec validation"""
if self.page_action and not callable(self.page_action):
raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
if self.proxy:
self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
self.proxy = construct_proxy_dict(self.proxy)
if self.cdp_url:
cdp_msg = _is_invalid_cdp_url(self.cdp_url)
if cdp_msg:
@@ -118,64 +107,18 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
raise ValueError(validation_msg)
class CamoufoxConfig(Struct, kw_only=True, frozen=False, weakref=True):
"""Configuration struct for validation"""
max_pages: PagesCount = 1
headless: bool = True # noqa: F821
block_images: bool = False
disable_resources: bool = False
block_webrtc: bool = False
class StealthConfig(PlaywrightConfig, kw_only=True, frozen=False, weakref=True):
allow_webgl: bool = True
network_idle: bool = False
load_dom: bool = True
humanize: bool | float = True
hide_canvas: bool = False
block_webrtc: bool = False
solve_cloudflare: bool = False
wait: Seconds = 0
timeout: Seconds = 30000
init_script: Optional[str] = None
page_action: Optional[Callable] = None
wait_selector: Optional[str] = None
addons: Optional[List[str]] = None
wait_selector_state: SelectorWaitStates = "attached"
cookies: Optional[Iterable[Dict]] = None
google_search: bool = True
extra_headers: Optional[Dict[str, str]] = None
proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
os_randomize: bool = False
disable_ads: bool = False
geoip: bool = False
user_data_dir: str = ""
selector_config: Optional[Dict] = {}
additional_args: Optional[Dict] = {}
def __post_init__(self):
"""Custom validation after msgspec validation"""
if self.page_action and not callable(self.page_action):
raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
if self.proxy:
self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
if self.addons:
for addon in self.addons:
_validate_addon_path(addon)
else:
self.addons = []
if self.init_script is not None:
validation_msg = _is_invalid_file_path(self.init_script)
if validation_msg:
raise ValueError(validation_msg)
if not self.cookies:
self.cookies = []
super(StealthConfig, self).__post_init__()
# Cloudflare timeout adjustment
if self.solve_cloudflare and self.timeout < 60_000:
self.timeout = 60_000
if not self.selector_config:
self.selector_config = {}
if not self.additional_args:
self.additional_args = {}
@dataclass
@@ -197,9 +140,9 @@ class _fetch_params:
def validate_fetch(
method_kwargs: Dict | PlaywrightFetchParams | CamoufoxFetchParams,
method_kwargs: Dict | PlaywrightFetchParams | StealthFetchParams,
session: Any,
model: type[PlaywrightConfig] | type[CamoufoxConfig],
model: type[PlaywrightConfig] | type[StealthConfig],
) -> _fetch_params: # pragma: no cover
result = {}
overrides = {}
@@ -210,21 +153,20 @@ def validate_fetch(
for key in fetch_param_fields:
if key in method_kwargs:
overrides[key] = method_kwargs[key]
else:
# Check for underscore-prefixed attribute (private)
attr_name = f"_{key}"
if hasattr(session, attr_name):
result[key] = getattr(session, attr_name)
elif hasattr(session, "_config") and hasattr(session._config, key):
result[key] = getattr(session._config, key)
if overrides:
validated_config = validate(overrides, model)
# Extract only the fields that _fetch_params needs from validated_config
# Extract ONLY the fields that were actually overridden (not all fields)
# This prevents validated defaults from overwriting session config values
validated_dict = {
f.name: getattr(validated_config, f.name)
for f in fields(_fetch_params)
if hasattr(validated_config, f.name)
field: getattr(validated_config, field) for field in overrides.keys() if hasattr(validated_config, field)
}
validated_dict.setdefault("solve_cloudflare", False)
# Preserve solve_cloudflare if the user explicitly provided it, even if the model doesn't have it
if "solve_cloudflare" in overrides:
validated_dict["solve_cloudflare"] = overrides["solve_cloudflare"]
# Start with session defaults, then overwrite with validated overrides
result.update(validated_dict)
@@ -238,7 +180,7 @@ def validate_fetch(
# Cache default values for each model to reduce validation overhead
models_default_values = {}
for _model in (CamoufoxConfig, PlaywrightConfig):
for _model in (StealthConfig, PlaywrightConfig):
_defaults = {}
if hasattr(_model, "__struct_defaults__") and hasattr(_model, "__struct_fields__"):
for field_name, default_value in zip(_model.__struct_fields__, _model.__struct_defaults__): # type: ignore
@@ -256,14 +198,14 @@ def _filter_defaults(params: Dict, model: str) -> Dict:
@overload
def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
def validate(params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
@overload
def validate(params: Dict, model: type[CamoufoxConfig]) -> CamoufoxConfig: ...
def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
def validate(params: Dict, model: type[PlaywrightConfig] | type[CamoufoxConfig]) -> PlaywrightConfig | CamoufoxConfig:
def validate(params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]) -> PlaywrightConfig | StealthConfig:
try:
# Filter out params with the default values (no need to validate them) to speed up validation
filtered = _filter_defaults(params, model.__name__)
scrapling/engines/constants.py
-3
View File
@@ -74,7 +74,6 @@ DEFAULT_STEALTH_FLAGS = (
"--disable-domain-reliability",
"--disable-threaded-animation",
"--disable-threaded-scrolling",
# '--disable-reading-from-canvas', # For Firefox
"--enable-simple-cache-backend",
"--disable-background-networking",
"--enable-surface-synchronization",
@@ -94,9 +93,7 @@ DEFAULT_STEALTH_FLAGS = (
"--autoplay-policy=no-user-gesture-required",
"--disable-offer-store-unmasked-wallet-cards",
"--disable-blink-features=AutomationControlled",
"--webrtc-ip-handling-policy=disable_non_proxied_udp",
"--disable-component-extensions-with-background-pages",
"--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
"--enable-features=NetworkService,NetworkServiceInProcess,TrustTokens,TrustTokensAlwaysAllowIssuance",
"--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4",
"--disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process,TranslateUI,BlinkGenPropertyTrees",
scrapling/engines/toolbelt/fingerprints.py
+21 -7
View File
@@ -13,19 +13,32 @@ from scrapling.core._types import Dict, Literal, Tuple
__OS_NAME__ = platform_system()
OSName = Literal["linux", "macos", "windows"]
# Current versions hardcoded for now (Playwright doesn't allow to know the version of a browser without launching it)
chromium_version = 141
chrome_version = 143
@lru_cache(10, typed=True)
def generate_convincing_referer(url: str) -> str:
def generate_convincing_referer(url: str) -> str | None:
"""Takes the domain from the URL without the subdomain/suffix and make it look like you were searching Google for this website
>>> generate_convincing_referer('https://www.somewebsite.com/blah')
'https://www.google.com/search?q=somewebsite'
:param url: The URL you are about to fetch.
:return: Google's search URL of the domain name
:return: Google's search URL of the domain name, or None for localhost/IP addresses
"""
website_name = extract(url).domain
extracted = extract(url)
website_name = extracted.domain
# Skip generating referer for localhost, IP addresses, or when there's no valid domain
if not website_name or not extracted.suffix or website_name in ("localhost", "127.0.0.1", "::1"):
return None
# Check if it's an IP address (simple check for IPv4)
if all(part.isdigit() for part in website_name.split(".") if part):
return None
return f"https://www.google.com/search?q={website_name}"
@@ -46,7 +59,7 @@ def get_os_name() -> OSName | Tuple:
return SUPPORTED_OPERATING_SYSTEMS
def generate_headers(browser_mode: bool = False) -> Dict:
def generate_headers(browser_mode: bool | str = False) -> Dict:
"""Generate real browser-like headers using browserforge's generator
:param browser_mode: If enabled, the headers created are used for playwright, so it has to match everything
@@ -55,13 +68,14 @@ def generate_headers(browser_mode: bool = False) -> Dict:
# In the browser mode, we don't care about anything other than matching the OS and the browser type with the browser we are using,
# So we don't raise any inconsistency red flags while websites fingerprinting us
os_name = get_os_name()
browsers = [Browser(name="chrome", min_version=130)]
ver = chrome_version if browser_mode and browser_mode == "chrome" else chromium_version
browsers = [Browser(name="chrome", min_version=ver, max_version=ver)]
if not browser_mode:
os_name = ("windows", "macos", "linux")
browsers.extend(
[
Browser(name="firefox", min_version=130),
Browser(name="edge", min_version=130),
Browser(name="firefox", min_version=142),
Browser(name="edge", min_version=140),
]
)
return HeaderGenerator(browser=browsers, os=os_name, device="desktop").generate()
scrapling/engines/toolbelt/navigation.py
+4 -13
View File
@@ -11,7 +11,7 @@ from msgspec import Struct, structs, convert, ValidationError
from playwright.sync_api import Route
from scrapling.core.utils import log
from scrapling.core._types import Dict, Tuple, overload, Literal
from scrapling.core._types import Dict, Tuple
from scrapling.engines.constants import DEFAULT_DISABLED_RESOURCES
__BYPASSES_DIR__ = Path(__file__).parent / "bypasses"
@@ -49,20 +49,11 @@ async def async_intercept_route(route: async_Route):
await route.continue_()
@overload
def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[True]) -> Tuple: ...
@overload
def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[False] = False) -> Dict: ...
def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: bool = False) -> Dict | Tuple:
def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple) -> Dict:
"""Validate a proxy and return it in the acceptable format for Playwright
Reference: https://playwright.dev/python/docs/network#http-proxy
:param proxy_string: A string or a dictionary representation of the proxy.
:param as_tuple: Return the proxy dictionary as a tuple to be cachable
:return:
"""
if isinstance(proxy_string, str):
@@ -78,7 +69,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
}
if proxy.port:
result["server"] += f":{proxy.port}"
return tuple(result.items()) if as_tuple else result
return result
except ValueError:
# Urllib will say that one of the parameters above can't be casted to the correct type like `int` for port etc...
raise ValueError("The proxy argument's string is in invalid format!")
@@ -87,7 +78,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
try:
validated = convert(proxy_string, ProxyDict)
result_dict = structs.asdict(validated)
return tuple(result_dict.items()) if as_tuple else result_dict
return result_dict
except ValidationError as e:
raise TypeError(f"Invalid proxy dictionary: {e}")
scrapling/fetchers/__init__.py
+4 -4
View File
@@ -3,7 +3,7 @@ from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from scrapling.fetchers.requests import Fetcher, AsyncFetcher, FetcherSession
from scrapling.fetchers.chrome import DynamicFetcher, DynamicSession, AsyncDynamicSession
from scrapling.fetchers.firefox import StealthyFetcher, StealthySession, AsyncStealthySession
from scrapling.fetchers.stealth_chrome import StealthyFetcher, StealthySession, AsyncStealthySession
# Lazy import mapping
@@ -14,9 +14,9 @@ _LAZY_IMPORTS = {
"DynamicFetcher": ("scrapling.fetchers.chrome", "DynamicFetcher"),
"DynamicSession": ("scrapling.fetchers.chrome", "DynamicSession"),
"AsyncDynamicSession": ("scrapling.fetchers.chrome", "AsyncDynamicSession"),
"StealthyFetcher": ("scrapling.fetchers.firefox", "StealthyFetcher"),
"StealthySession": ("scrapling.fetchers.firefox", "StealthySession"),
"AsyncStealthySession": ("scrapling.fetchers.firefox", "AsyncStealthySession"),
"StealthyFetcher": ("scrapling.fetchers.stealth_chrome", "StealthyFetcher"),
"StealthySession": ("scrapling.fetchers.stealth_chrome", "StealthySession"),
"AsyncStealthySession": ("scrapling.fetchers.stealth_chrome", "AsyncStealthySession"),
}
__all__ = [
scrapling/fetchers/chrome.py
+49 -64
View File
@@ -5,51 +5,37 @@ from scrapling.engines._browsers._controllers import DynamicSession, AsyncDynami
class DynamicFetcher(BaseFetcher):
"""A `Fetcher` class type that provide many options, all of them are based on PlayWright.
Using this Fetcher class, you can do requests with:
- Vanilla Playwright without any modifications other than the ones you chose.
- Stealthy Playwright with the stealth mode I wrote for it. It's still a work in progress, but it bypasses many online tests like bot.sannysoft.com
Some of the things stealth mode does include:
1) Patches the CDP runtime fingerprint.
2) Mimics some of the real browsers' properties by injecting several JS files and using custom options.
3) Using custom flags on launch to hide Playwright even more and make it faster.
4) Generates real browser's headers of the same type and same user OS, then append it to the request.
- Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
> Note that these are the main options with PlayWright, but it can be mixed.
"""
"""A `Fetcher` that provide many options to fetch/load websites' pages through chromium-based browsers."""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[PlaywrightSession]) -> Response:
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param kwargs: Browser session configuration options including:
- headless: Run the browser in headless/hidden (default), or headful/visible mode.
- disable_resources: Drop requests of unnecessary resources for a speed boost.
- useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- cookies: Set cookies for the next request.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- locale: Set the locale for the browser if wanted. The default value is `en-US`.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- extra_headers: A dictionary of extra headers to add to the request.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- extra_flags: A list of additional browser flags to pass to the browser on launch.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request.
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
@@ -68,31 +54,30 @@ class DynamicFetcher(BaseFetcher):
"""Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param kwargs: Browser session configuration options including:
- headless: Run the browser in headless/hidden (default), or headful/visible mode.
- disable_resources: Drop requests of unnecessary resources for a speed boost.
- useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- cookies: Set cookies for the next request.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- locale: Set the locale for the browser if wanted. The default value is `en-US`.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- extra_headers: A dictionary of extra headers to add to the request.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- extra_flags: A list of additional browser flags to pass to the browser on launch.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request.
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
scrapling/fetchers/firefox.py
-102
View File
@@ -1,102 +0,0 @@
from scrapling.core._types import Unpack
from scrapling.engines._browsers._types import CamoufoxSession
from scrapling.engines.toolbelt.custom import BaseFetcher, Response
from scrapling.engines._browsers._camoufox import StealthySession, AsyncStealthySession
class StealthyFetcher(BaseFetcher):
"""A `Fetcher` class type that is a completely stealthy fetcher that uses a modified version of Firefox.
It works as real browsers passing almost all online tests/protections based on Camoufox.
Other added flavors include setting the faked OS fingerprints to match the user's OS, and the referer of every request is set as if this request came from Google's search of this URL's domain.
"""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param kwargs: Browser session configuration options including:
- headless: Run the browser in headless/hidden (default), or headful/visible mode.
- block_images: Prevent the loading of images through Firefox preferences.
- disable_resources: Drop requests of unnecessary resources for a speed boost.
- block_webrtc: Blocks WebRTC entirely.
- allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- addons: List of Firefox addons to use. Must be paths to extracted addons.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- cookies: Set cookies for the next request.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- extra_headers: A dictionary of extra headers to add to the request.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- additional_args: Additional arguments to be passed to Camoufox as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
with StealthySession(**kwargs) as engine:
return engine.fetch(url)
@classmethod
async def async_fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param kwargs: Browser session configuration options including:
- headless: Run the browser in headless/hidden (default), or headful/visible mode.
- block_images: Prevent the loading of images through Firefox preferences.
- disable_resources: Drop requests of unnecessary resources for a speed boost.
- block_webrtc: Blocks WebRTC entirely.
- allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- addons: List of Firefox addons to use. Must be paths to extracted addons.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- cookies: Set cookies for the next request.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- extra_headers: A dictionary of extra headers to add to the request.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- additional_args: Additional arguments to be passed to Camoufox as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
async with AsyncStealthySession(**kwargs) as engine:
return await engine.fetch(url)
scrapling/fetchers/stealth_chrome.py
+107
View File
@@ -0,0 +1,107 @@
from scrapling.core._types import Unpack
from scrapling.engines._browsers._types import StealthSession
from scrapling.engines.toolbelt.custom import BaseFetcher, Response
from scrapling.engines._browsers._stealth import StealthySession, AsyncStealthySession
class StealthyFetcher(BaseFetcher):
"""A `Fetcher` class type which is a completely stealthy built on top of Chromium.
It works as real browsers passing almost all online tests/protections with many customization options.
"""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
with StealthySession(**kwargs) as engine:
return engine.fetch(url)
@classmethod
async def async_fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
async with AsyncStealthySession(**kwargs) as engine:
return await engine.fetch(url)
setup.cfg
+1 -1
View File
@@ -1,6 +1,6 @@
[metadata]
name = scrapling
version = 0.3.12
version = 0.3.13
author = Karim Shoair
author_email = karim.shoair@pm.me
description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!
tests/cli/test_cli.py
-1
View File
@@ -156,7 +156,6 @@ class TestCLI:
html_url,
str(output_file),
'--headless',
'--stealth',
'--timeout', '60000'
]
)
tests/fetchers/async/test_dynamic.py
+1 -4
View File
@@ -53,10 +53,7 @@ class TestDynamicFetcherAsync:
@pytest.mark.parametrize(
"kwargs",
[
{"disable_webgl": True, "hide_canvas": False},
{"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
{"stealth": True}, # causes issues with GitHub Actions
{"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
{"real_chrome": True, "disable_resources": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
tests/fetchers/async/test_dynamic_session.py
-1
View File
@@ -71,7 +71,6 @@ class TestAsyncDynamicSession:
"""Test AsyncDynamicSession with various options"""
async with AsyncDynamicSession(
headless=False,
stealth=True,
disable_resources=True,
extra_headers={"X-Test": "value"}
) as session:
tests/fetchers/async/test_requests_session.py
-1
View File
@@ -1,4 +1,3 @@
import pytest
from scrapling.engines.static import AsyncFetcherClient
tests/fetchers/async/test_camoufox.py → tests/fetchers/async/test_stealth.py
+5 -10
View File
@@ -1,4 +1,3 @@
from playwright._impl._errors import TimeoutError
import pytest
import pytest_httpbin
@@ -30,8 +29,8 @@ class TestStealthyFetcher:
async def test_basic_fetch(self, fetcher, urls):
"""Test doing a basic fetch request with multiple statuses"""
assert (await fetcher.async_fetch(urls["status_200"])).status == 200
assert (await fetcher.async_fetch(urls["status_404"])).status == 404
assert (await fetcher.async_fetch(urls["status_501"])).status == 501
# assert (await fetcher.async_fetch(urls["status_404"])).status == 404
# assert (await fetcher.async_fetch(urls["status_501"])).status == 501
async def test_cookies_loading(self, fetcher, urls):
"""Test if cookies are set after the request"""
@@ -55,10 +54,9 @@ class TestStealthyFetcher:
@pytest.mark.parametrize(
"kwargs",
[
{"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
{"block_webrtc": False, "allow_webgl": True, "block_images": True},
{"block_webrtc": True, "allow_webgl": True},
{"block_webrtc": False, "allow_webgl": True},
{"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
{"block_images": True, "disable_resources": True, },
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
@@ -67,11 +65,8 @@ class TestStealthyFetcher:
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"google_search": True,
"extra_headers": {"ayo": ""},
"os_randomize": True,
"disable_ads": True,
# "geoip": True,
"selector_config": {"keep_comments": False, "keep_cdata": False},
"additional_args": {"window": (1920, 1080)},
"additional_args": {},
},
],
)
tests/fetchers/async/test_camoufox_session.py → tests/fetchers/async/test_stealth_session.py
+2 -3
View File
@@ -72,9 +72,8 @@ class TestAsyncStealthySession:
"""Test AsyncStealthySession with various options"""
async with AsyncStealthySession(
max_pages=1,
block_images=True,
disable_ads=True,
humanize=True
block_webrtc=True,
allow_webgl=True
) as session:
response = await session.fetch(urls["html"])
assert response.status == 200
tests/fetchers/sync/test_camoufox.py
-79
View File
@@ -1,79 +0,0 @@
import pytest
import pytest_httpbin
from scrapling import StealthyFetcher
StealthyFetcher.adaptive = True
@pytest_httpbin.use_class_based_httpbin
class TestStealthyFetcher:
@pytest.fixture(scope="class")
def fetcher(self):
"""Fixture to create a StealthyFetcher instance for the entire test class"""
return StealthyFetcher
@pytest.fixture(autouse=True)
def setup_urls(self, httpbin):
"""Fixture to set up URLs for testing"""
self.status_200 = f"{httpbin.url}/status/200"
self.status_404 = f"{httpbin.url}/status/404"
self.status_501 = f"{httpbin.url}/status/501"
self.basic_url = f"{httpbin.url}/get"
self.html_url = f"{httpbin.url}/html"
self.delayed_url = f"{httpbin.url}/delay/10" # 10 Seconds delay response
self.cookies_url = f"{httpbin.url}/cookies/set/test/value"
def test_basic_fetch(self, fetcher):
"""Test doing a basic fetch request with multiple statuses"""
assert fetcher.fetch(self.status_200).status == 200
assert fetcher.fetch(self.status_404).status == 404
assert fetcher.fetch(self.status_501).status == 501
def test_cookies_loading(self, fetcher):
"""Test if cookies are set after the request"""
response = fetcher.fetch(self.cookies_url)
cookies = {response.cookies[0]['name']: response.cookies[0]['value']}
assert cookies == {"test": "value"}
def test_automation(self, fetcher):
"""Test if automation breaks the code or not"""
def scroll_page(page):
page.mouse.wheel(10, 0)
page.mouse.move(100, 400)
page.mouse.up()
return page
assert fetcher.fetch(self.html_url, page_action=scroll_page).status == 200
@pytest.mark.parametrize(
"kwargs",
[
{"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
{"block_webrtc": False, "allow_webgl": True, "block_images": True},
{"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
{"block_images": True, "disable_resources": True, },
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
"network_idle": True,
"wait": 10,
"timeout": 30_000,
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"google_search": True,
"extra_headers": {"ayo": ""},
"os_randomize": True,
"disable_ads": True,
# "geoip": True,
"selector_config": {"keep_comments": False, "keep_cdata": False},
"additional_args": {"window": (1920, 1080)},
},
],
)
def test_properties(self, fetcher, kwargs):
"""Test if different arguments break the code or not"""
response = fetcher.fetch(
self.html_url,
**kwargs
)
assert response.status == 200
tests/fetchers/sync/test_dynamic.py
+1 -4
View File
@@ -51,10 +51,7 @@ class TestDynamicFetcher:
@pytest.mark.parametrize(
"kwargs",
[
{"disable_webgl": True, "hide_canvas": False},
{"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
{"stealth": True}, # causes issues with GitHub Actions
{"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
{"disable_resources": True, "real_chrome": True},
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
tests/fetchers/sync/test_camoufox_session.py → tests/fetchers/sync/test_stealth_session.py
+8 -10
View File
@@ -2,11 +2,11 @@ import re
import pytest
import pytest_httpbin
from scrapling.engines._browsers._camoufox import StealthySession, __CF_PATTERN__
from scrapling.engines._browsers._stealth import StealthySession, __CF_PATTERN__
class TestCamoufoxConstants:
"""Test Camoufox constants and patterns"""
class TestStealthConstants:
"""Test Stealth constants and patterns"""
def test_cf_pattern_regex(self):
"""Test __CF_PATTERN__ regex compilation"""
@@ -54,7 +54,6 @@ class TestStealthySession:
with StealthySession(
headless=True,
block_images=True,
disable_resources=True,
solve_cloudflare=True,
wait=1000,
@@ -63,12 +62,11 @@ class TestStealthySession:
) as session:
assert session.max_pages == 1
assert session._headless is True
assert session._block_images is True
assert session._disable_resources is True
assert session._solve_cloudflare is True
assert session._wait == 1000
assert session._timeout == 60000
assert session._config.headless is True
assert session._config.disable_resources is True
assert session._config.solve_cloudflare is True
assert session._config.wait == 1000
assert session._config.timeout == 60000
assert session.context is not None
# Test Cloudflare detection
tests/fetchers/test_utils.py
-8
View File
@@ -190,14 +190,6 @@ class TestConstructProxyDict:
}
assert result == expected
def test_proxy_as_tuple(self):
"""Test returning proxy as a tuple"""
result = construct_proxy_dict("http://proxy.example.com:8080", as_tuple=True)
assert isinstance(result, tuple)
result_dict = dict(result)
assert result_dict["server"] == "http://proxy.example.com:8080"
def test_invalid_proxy_string(self):
"""Test invalid proxy string"""
with pytest.raises(ValueError):
tests/fetchers/test_validator.py
+8 -8
View File
@@ -1,8 +1,8 @@
import pytest
from scrapling.engines._browsers._validators import (
validate,
StealthConfig,
PlaywrightConfig,
CamoufoxConfig
)
@@ -23,7 +23,7 @@ class TestValidators:
assert config.max_pages == 2
assert config.headless is True
assert config.timeout == 30000
assert isinstance(config.proxy, tuple) # Should be converted to tuple
assert isinstance(config.proxy, dict)
def test_playwright_config_invalid_max_pages(self):
"""Test PlaywrightConfig with invalid max_pages"""
@@ -51,8 +51,8 @@ class TestValidators:
with pytest.raises(TypeError):
validate(params, PlaywrightConfig)
def test_camoufox_config_valid(self):
"""Test valid CamoufoxConfig"""
def test_stealth_config_valid(self):
"""Test valid StealthConfig"""
params = {
"max_pages": 1,
"headless": True,
@@ -60,20 +60,20 @@ class TestValidators:
"timeout": 30000
}
config = validate(params, CamoufoxConfig)
config = validate(params, StealthConfig)
assert config.max_pages == 1
assert config.headless is True
assert config.solve_cloudflare is False
assert config.timeout == 30000
def test_camoufox_config_cloudflare_timeout(self):
"""Test CamoufoxConfig timeout adjustment for Cloudflare"""
def test_stealth_config_cloudflare_timeout(self):
"""Test StealthConfig timeout adjustment for Cloudflare"""
params = {
"solve_cloudflare": True,
"timeout": 10000 # Less than the required 60,000
}
config = validate(params, CamoufoxConfig)
config = validate(params, StealthConfig)
assert config.timeout == 60000 # Should be increased
tests/requirements.txt
-1
View File
@@ -1,7 +1,6 @@
pytest>=2.8.0,<9
pytest-cov
playwright
camoufox
werkzeug<3.0.0
pytest-httpbin==2.1.0
pytest-asyncio
tox.ini
+2 -3
View File
@@ -10,9 +10,8 @@ envlist = pre-commit,py{310,311,312,313}
usedevelop = True
changedir = tests
deps =
playwright>=1.56.0
patchright>=1.56.0
camoufox>=0.4.11
playwright==1.56.0
patchright==1.56.0
-r{toxinidir}/tests/requirements.txt
extras = ai,shell
commands =