This commit is contained in:
Karim shoair
2026-01-01 22:07:00 +02:00
committed by GitHub
66 changed files with 1257 additions and 1686 deletions
+2 -30
View File
@@ -58,7 +58,7 @@ jobs:
- name: Install all browsers dependencies - name: Install all browsers dependencies
run: | run: |
python3 -m pip install --upgrade pip python3 -m pip install --upgrade pip
python3 -m pip install playwright==1.56.0 patchright==1.56.0 camoufox>=0.4.11 python3 -m pip install playwright==1.56.0 patchright==1.56.0
- name: Get Playwright version - name: Get Playwright version
id: playwright-version id: playwright-version
@@ -88,35 +88,7 @@ jobs:
else else
echo "Skipping install - using cached Playwright browsers" echo "Skipping install - using cached Playwright browsers"
fi fi
python3 -m playwright install-deps chromium firefox python3 -m playwright install-deps chromium
- name: Get Camoufox version
id: camoufox-version
run: |
CAMOUFOX_VERSION=$(python3 -c "import importlib.metadata; print(importlib.metadata.version('camoufox'))")
echo "version=$CAMOUFOX_VERSION" >> $GITHUB_OUTPUT
echo "Camoufox version: $CAMOUFOX_VERSION"
- name: Retrieve Camoufox browser from cache if any
id: camoufox-cache
uses: actions/cache@v4
with:
path: |
~/.cache/camoufox
~/Library/Caches/camoufox
key: ${{ runner.os }}-camoufox-${{ steps.camoufox-version.outputs.version }}-v1
restore-keys: |
${{ runner.os }}-camoufox-${{ steps.camoufox-version.outputs.version }}-
${{ runner.os }}-camoufox-
- name: Install Camoufox browser
run: |
echo "Cache hit: ${{ steps.camoufox-cache.outputs.cache-hit }}"
if [ "${{ steps.camoufox-cache.outputs.cache-hit }}" != "true" ]; then
python3 -m camoufox fetch --browserforge
else
echo "Skipping fetch - using cached Camoufox browser"
fi
# Cache tox environments # Cache tox environments
- name: Cache tox environments - name: Cache tox environments
+2
View File
@@ -7,6 +7,8 @@ version: 2
# Set the OS, Python version, and other tools you might need # Set the OS, Python version, and other tools you might need
build: build:
os: ubuntu-24.04 os: ubuntu-24.04
apt_packages:
- pngquant
tools: tools:
python: "3.13" python: "3.13"
+4 -4
View File
@@ -4,7 +4,7 @@ Thank you for your interest in contributing to Scrapling!
Everybody is invited and welcome to contribute to Scrapling. Everybody is invited and welcome to contribute to Scrapling.
Minor changes have a better chance of being included promptly. Adding unit tests for new features or test cases for bugs you've fixed helps us ensure that the Pull Request (PR) is acceptable. Minor changes are more likely to be included promptly. Adding unit tests for new features or test cases for bugs you've fixed helps us ensure that the Pull Request (PR) is acceptable.
There are many ways to contribute to Scrapling. Here are some of them: There are many ways to contribute to Scrapling. Here are some of them:
@@ -18,7 +18,7 @@ There are many ways to contribute to Scrapling. Here are some of them:
If you have decided to make a contribution to Scrapling, but you do not know what to contribute, here are some ways to find pending work: If you have decided to make a contribution to Scrapling, but you do not know what to contribute, here are some ways to find pending work:
- Check out the [contribution](https://github.com/D4Vinci/Scrapling/contribute) GitHub page, which lists open issues tagged as good first issue. These issues provide a good starting point. - Check out the [contribution](https://github.com/D4Vinci/Scrapling/contribute) GitHub page, which lists open issues tagged as `good first issue`. These issues provide a good starting point.
- There are also the [help wanted](https://github.com/D4Vinci/Scrapling/issues?q=is%3Aissue%20label%3A%22help%20wanted%22%20state%3Aopen) issues, but know that some may require familiarity with the Scrapling code base first. You can also target any other issue, provided it is not tagged as `invalid`, `wontfix`, or similar tags. - There are also the [help wanted](https://github.com/D4Vinci/Scrapling/issues?q=is%3Aissue%20label%3A%22help%20wanted%22%20state%3Aopen) issues, but know that some may require familiarity with the Scrapling code base first. You can also target any other issue, provided it is not tagged as `invalid`, `wontfix`, or similar tags.
- If you enjoy writing automated tests, you can work on increasing our test coverage. Currently, the test coverage is around 9092%. - If you enjoy writing automated tests, you can work on increasing our test coverage. Currently, the test coverage is around 9092%.
- Join the [Discord community](https://discord.gg/EMgGbDceNQ) and ask questions in the `#help` channel. - Join the [Discord community](https://discord.gg/EMgGbDceNQ) and ask questions in the `#help` channel.
@@ -38,7 +38,7 @@ Please follow these coding conventions as we do when writing code for Scrapling:
| `refactor:` | Code refactoring | | `refactor:` | Code refactoring |
| `chore:` | Maintenance tasks | | `chore:` | Maintenance tasks |
Then include the details of the change in the body/description of the commit message. Then include the details of the change in the commit message body/description.
Example: Example:
``` ```
@@ -99,7 +99,7 @@ pytest --cov=scrapling tests/
To ensure that your PR gets accepted, please make sure that your PR is based on the latest changes from the dev branch and that it satisfies the following requirements: To ensure that your PR gets accepted, please make sure that your PR is based on the latest changes from the dev branch and that it satisfies the following requirements:
- The PR should be made against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling. Any PR made against the main branch will be rejected. - The PR should be made against the [**dev**](https://github.com/D4Vinci/Scrapling/tree/dev) branch of Scrapling. Any PR made against the main branch will be rejected.
- The code should be passing all available tests. We are using tox with GitHub's CI to run the current tests on all supported Python versions with every commit. - The code should be passing all available tests. We use tox with GitHub's CI to run the current tests on all supported Python versions for every code-related commit.
- The code should be passing all code quality checks we mentioned above. We are using GitHub's CI to enforce the code style checks performed by pre-commit. If you were using the pre-commit hooks we discussed above, you should not see any issues when committing your changes. - The code should be passing all code quality checks we mentioned above. We are using GitHub's CI to enforce the code style checks performed by pre-commit. If you were using the pre-commit hooks we discussed above, you should not see any issues when committing your changes.
- Make your changes, keep the code clean with an explanation of any part that might be vague, and remember to create a separate virtual environment for this project. - Make your changes, keep the code clean with an explanation of any part that might be vague, and remember to create a separate virtual environment for this project.
- If you are adding a new feature, please add tests for it. - If you are adding a new feature, please add tests for it.
+1 -2
View File
@@ -23,9 +23,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
--mount=type=cache,target=/var/cache/apt \ --mount=type=cache,target=/var/cache/apt \
--mount=type=cache,target=/var/lib/apt \ --mount=type=cache,target=/var/lib/apt \
apt-get update && \ apt-get update && \
uv run playwright install-deps chromium firefox && \ uv run playwright install-deps chromium && \
uv run playwright install chromium && \ uv run playwright install chromium && \
uv run camoufox fetch --browserforge && \
uv sync --all-extras --compile-bytecode && \ uv sync --all-extras --compile-bytecode && \
apt-get clean && \ apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+12 -13
View File
@@ -90,8 +90,8 @@ Built for the modern Web, Scrapling features **its own rapid parsing engine** an
### Advanced Websites Fetching with Session Support ### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP3. - **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP3.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, real Chrome, and custom stealth mode. - **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium and Google's Chrome.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` using a modified version of Firefox and fingerprint spoofing. Can bypass all types of Cloudflare's Turnstile and Interstitial with automation easily. - **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests. - **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes. - **Async Support**: Complete async support across all fetchers and dedicated async session classes.
@@ -99,7 +99,7 @@ Built for the modern Web, Scrapling features **its own rapid parsing engine** an
- 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms. - 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
- 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more. - 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements. - 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE)) - 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### High-Performance & battle-tested Architecture ### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries. - 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
@@ -153,7 +153,7 @@ data = page.css('.quote .text::text')
``` ```
> [!NOTE] > [!NOTE]
> There's a wonderful guide to get you started quickly with Scraping [here](https://substack.thewebscraping.club/p/scrapling-hands-on-guide) written by The Web Scraping Club. In case you find it easier to get you started than the [documentation website](https://scrapling.readthedocs.io/en/latest/). > There's a wonderful guide to get you started quickly with Scrapling [here](https://substack.thewebscraping.club/p/scrapling-hands-on-guide) written by The Web Scraping Club. In case you find it easier to get you started than the [documentation website](https://scrapling.readthedocs.io/en/latest/).
### Advanced Parsing & Navigation ### Advanced Parsing & Navigation
```python ```python
@@ -222,13 +222,12 @@ Scrapling v0.3 includes a powerful command-line interface:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339) [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Launch the interactive Web Scraping shell
```bash ```bash
# Launch interactive Web Scraping shell
scrapling shell scrapling shell
```
# Extract pages to a file directly without programming (Extracts the content inside `body` tag by default) Extract pages to a file directly without programming (Extracts the content inside the `body` tag by default). If the output file ends with `.txt`, then the text content of the target will be extracted. If it ends in `.md`, it will be a Markdown representation of the HTML content; if it ends in `.html`, it will be the HTML content itself.
# If the output file ends with `.txt`, then the text content of the target will be extracted. ```bash
# If ended with `.md`, it will be a markdown representation of the HTML content, and `.html` will be the HTML content right away.
scrapling extract get 'https://example.com' content.md scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts' scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -240,7 +239,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Performance Benchmarks ## Performance Benchmarks
Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 have delivered exceptional performance improvements across all operations. Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 have delivered exceptional performance improvements across all operations. The following benchmarks compare Scrapling's parser with other popular libraries.
### Text Extraction Speed Test (5000 nested elements) ### Text Extraction Speed Test (5000 nested elements)
@@ -280,14 +279,14 @@ Starting with v0.3.2, this installation only includes the parser engine and its
### Optional Dependencies ### Optional Dependencies
1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies and then install their browser dependencies with 1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies. This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
2. Extra features: 2. Extra features:
- Install the MCP server feature: - Install the MCP server feature:
@@ -313,7 +312,7 @@ Or download it from the GitHub registry:
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
This image is automatically built and pushed through GitHub actions on the repository's main branch. This image is automatically built and pushed using GitHub Actions and the repository's main branch.
## Contributing ## Contributing
+11 -12
View File
@@ -87,8 +87,8 @@ Scrapling ليست مجرد مكتبة أخرى لاستخراج بيانات ا
### جلب متقدم للمواقع مع دعم الجلسات ### جلب متقدم للمواقع مع دعم الجلسات
- **طلبات HTTP**: طلبات HTTP سريعة وخفية مع فئة `Fetcher`. يمكنها تقليد بصمة TLS للمتصفح والرؤوس واستخدام HTTP3. - **طلبات HTTP**: طلبات HTTP سريعة وخفية مع فئة `Fetcher`. يمكنها تقليد بصمة TLS للمتصفح والرؤوس واستخدام HTTP3.
- **التحميل الديناميكي**: جلب المواقع الديناميكية مع أتمتة كاملة للمتصفح من خلال فئة `DynamicFetcher` التي تدعم Chromium من Playwright، وChrome الحقيقي، ووضع التخفي المخصص. - **التحميل الديناميكي**: جلب المواقع الديناميكية مع أتمتة كاملة للمتصفح من خلال فئة `DynamicFetcher` التي تدعم Chromium من Playwright و Google Chrome.
- **تجاوز مكافحة الروبوتات**: قدرات تخفي متقدمة مع `StealthyFetcher` باستخدام نسخة معدلة من Firefox وانتحال البصمات. يمكنه تجاوز جميع أنواع Turnstile وInterstitial من Cloudflare بسهولة بالأتمتة. - **تجاوز مكافحة الروبوتات**: قدرات تخفي متقدمة مع `StealthyFetcher` وانتحال البصمات. يمكنه تجاوز جميع أنواع Turnstile/Interstitial من Cloudflare بسهولة بالأتمتة.
- **إدارة الجلسات**: دعم الجلسات المستمرة مع فئات `FetcherSession` و`StealthySession` و`DynamicSession` لإدارة ملفات تعريف الارتباط والحالة عبر الطلبات. - **إدارة الجلسات**: دعم الجلسات المستمرة مع فئات `FetcherSession` و`StealthySession` و`DynamicSession` لإدارة ملفات تعريف الارتباط والحالة عبر الطلبات.
- **دعم Async**: دعم async كامل عبر جميع الجوالب وفئات الجلسات async المخصصة. - **دعم Async**: دعم async كامل عبر جميع الجوالب وفئات الجلسات async المخصصة.
@@ -96,7 +96,7 @@ Scrapling ليست مجرد مكتبة أخرى لاستخراج بيانات ا
- 🔄 **تتبع العناصر الذكي**: إعادة تحديد موقع العناصر بعد تغييرات الموقع باستخدام خوارزميات التشابه الذكية. - 🔄 **تتبع العناصر الذكي**: إعادة تحديد موقع العناصر بعد تغييرات الموقع باستخدام خوارزميات التشابه الذكية.
- 🎯 **الاختيار المرن الذكي**: محددات CSS، محددات XPath، البحث القائم على الفلاتر، البحث النصي، البحث بالتعبيرات العادية والمزيد. - 🎯 **الاختيار المرن الذكي**: محددات CSS، محددات XPath، البحث القائم على الفلاتر، البحث النصي، البحث بالتعبيرات العادية والمزيد.
- 🔍 **البحث عن عناصر مشابهة**: تحديد العناصر المشابهة للعناصر الموجودة تلقائياً. - 🔍 **البحث عن عناصر مشابهة**: تحديد العناصر المشابهة للعناصر الموجودة تلقائياً.
- 🤖 **خادم MCP للاستخدام مع الذكاء الاصطناعي**: خادم MCP مدمج لاستخراج بيانات الويب بمساعدة الذكاء الاصطناعي واستخراج البيانات. يتميز خادم MCP بقدرات مخصصة قوية تستخدم Scrapling لاستخراج المحتوى المستهدف قبل تمريره إلى الذكاء الاصطناعي (Claude/Cursor/إلخ)، وبالتالي تسريع العمليات وتقليل التكاليف عن طريق تقليل استخدام الرموز. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE)) - 🤖 **خادم MCP للاستخدام مع الذكاء الاصطناعي**: خادم MCP مدمج لاستخراج بيانات الويب بمساعدة الذكاء الاصطناعي واستخراج البيانات. يتميز خادم MCP بقدرات قوية مخصصة تستفيد من Scrapling لاستخراج المحتوى المستهدف قبل تمريره إلى الذكاء الاصطناعي (Claude/Cursor/إلخ)، وبالتالي تسريع العمليات وتقليل التكاليف عن طريق تقليل استخدام الرموز. ([فيديو توضيحي](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### بنية عالية الأداء ومختبرة في المعارك ### بنية عالية الأداء ومختبرة في المعارك
- 🚀 **سريع كالبرق**: أداء محسّن يتفوق على معظم مكتبات استخراج Python. - 🚀 **سريع كالبرق**: أداء محسّن يتفوق على معظم مكتبات استخراج Python.
@@ -212,13 +212,12 @@ async with AsyncStealthySession(max_pages=2) as session:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339) [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
تشغيل غلاف استخراج الويب التفاعلي
```bash ```bash
# تشغيل غلاف استخراج الويب التفاعلي
scrapling shell scrapling shell
```
# استخراج الصفحات إلى ملف مباشرة دون برمجة (يستخرج المحتوى داخل وسم `body` افتراضياً) استخراج الصفحات إلى ملف مباشرة دون برمجة (يستخرج المحتوى داخل وسم `body` افتراضياً). إذا انتهى ملف الإخراج بـ `.txt`، فسيتم استخراج محتوى النص للهدف. إذا انتهى بـ `.md`، فسيكون تمثيل Markdown لمحتوى HTML؛ إذا انتهى بـ `.html`، فسيكون محتوى HTML نفسه.
# إذا انتهى ملف الإخراج بـ `.txt`، فسيتم استخراج محتوى النص للهدف. ```bash
# إذا انتهى بـ `.md`، فسيكون تمثيل markdown لمحتوى HTML، و`.html` سيكون محتوى HTML مباشرة.
scrapling extract get 'https://example.com' content.md scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # جميع العناصر المطابقة لمحدد CSS '#fromSkipToProducts' scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # جميع العناصر المطابقة لمحدد CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## معايير الأداء ## معايير الأداء
Scrapling ليس قوياً فقط - إنه أيضاً سريع بشكل مذهل، والتحديثات منذ الإصدار 0.3 قدمت تحسينات أداء استثنائية عبر جميع العمليات. Scrapling ليس قوياً فقط - إنه أيضاً سريع بشكل مذهل، والتحديثات منذ الإصدار 0.3 قدمت تحسينات أداء استثنائية عبر جميع العمليات. تقارن المعايير التالية محلل Scrapling مع المكتبات الشائعة الأخرى.
### اختبار سرعة استخراج النص (5000 عنصر متداخل) ### اختبار سرعة استخراج النص (5000 عنصر متداخل)
@@ -270,14 +269,14 @@ pip install scrapling
### التبعيات الاختيارية ### التبعيات الاختيارية
1. إذا كنت ستستخدم أياً من الميزات الإضافية أدناه، أو الجوالب، أو فئاتها، فأنت بحاجة إلى تثبيت تبعيات الجوالب ثم تثبيت تبعيات المتصفح الخاصة بها بـ 1. إذا كنت ستستخدم أياً من الميزات الإضافية أدناه، أو الجوالب، أو فئاتها، فستحتاج إلى تثبيت تبعيات الجوالب وتبعيات المتصفح الخاصة بها على النحو التالي:
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
يقوم هذا بتنزيل جميع المتصفحات مع تبعيات النظام وتبعيات معالجة البصمات الخاصة بها. يقوم هذا بتنزيل جميع المتصفحات، إلى جانب تبعيات النظام وتبعيات معالجة البصمات الخاصة بها.
2. ميزات إضافية: 2. ميزات إضافية:
- تثبيت ميزة خادم MCP: - تثبيت ميزة خادم MCP:
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
يتم بناء هذه الصورة ودفعها تلقائياً من خلال إجراءات GitHub على الفرع الرئيسي للمستودع. يتم بناء هذه الصورة ودفعها تلقائياً باستخدام GitHub Actions والفرع الرئيسي للمستودع.
## المساهمة ## المساهمة
+11 -12
View File
@@ -87,8 +87,8 @@ Scrapling不仅仅是另一个网页抓取库。它是第一个**自适应**抓
### 支持会话的高级网站获取 ### 支持会话的高级网站获取
- **HTTP请求**:使用`Fetcher`类进行快速和隐秘的HTTP请求。可以模拟浏览器的TLS指纹、标头并使用HTTP3。 - **HTTP请求**:使用`Fetcher`类进行快速和隐秘的HTTP请求。可以模拟浏览器的TLS指纹、标头并使用HTTP3。
- **动态加载**:通过`DynamicFetcher`类使用完整的浏览器自动化获取动态网站,支持Playwright的Chromium、真实Chrome和自定义隐秘模式 - **动态加载**:通过`DynamicFetcher`类使用完整的浏览器自动化获取动态网站,支持Playwright的Chromium和Google Chrome
- **反机器人绕过**:使用`StealthyFetcher`的高级隐秘功能,使用修改版Firefox和指纹伪装。可以轻松自动绕过所有类型的Cloudflare的TurnstileInterstitial。 - **反机器人绕过**:使用`StealthyFetcher`的高级隐秘功能和指纹伪装。可以轻松自动绕过所有类型的Cloudflare的Turnstile/Interstitial。
- **会话管理**:使用`FetcherSession``StealthySession``DynamicSession`类持久化会话支持,用于跨请求的cookie和状态管理。 - **会话管理**:使用`FetcherSession``StealthySession``DynamicSession`类持久化会话支持,用于跨请求的cookie和状态管理。
- **异步支持**:所有获取器和专用异步会话类的完整异步支持。 - **异步支持**:所有获取器和专用异步会话类的完整异步支持。
@@ -96,7 +96,7 @@ Scrapling不仅仅是另一个网页抓取库。它是第一个**自适应**抓
- 🔄 **智能元素跟踪**:使用智能相似性算法在网站更改后重新定位元素。 - 🔄 **智能元素跟踪**:使用智能相似性算法在网站更改后重新定位元素。
- 🎯 **智能灵活选择**:CSS选择器、XPath选择器、基于过滤器的搜索、文本搜索、正则表达式搜索等。 - 🎯 **智能灵活选择**:CSS选择器、XPath选择器、基于过滤器的搜索、文本搜索、正则表达式搜索等。
- 🔍 **查找相似元素**:自动定位与找到的元素相似的元素。 - 🔍 **查找相似元素**:自动定位与找到的元素相似的元素。
- 🤖 **与AI一起使用的MCP服务器**:内置MCP服务器用于AI辅助网页抓取和数据提取。MCP服务器具有自定义的强大功能,利用Scrapling在将内容传递给AIClaude/Cursor等)之前提取目标内容,从而加快操作并通过最小化令牌使用来降低成本。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE) - 🤖 **与AI一起使用的MCP服务器**:内置MCP服务器用于AI辅助网页抓取和数据提取。MCP服务器具有强大的自定义功能,利用Scrapling在将内容传递给AIClaude/Cursor等)之前提取目标内容,从而加快操作并通过最小化令牌使用来降低成本。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE)
### 高性能和经过实战测试的架构 ### 高性能和经过实战测试的架构
- 🚀 **闪电般快速**:优化性能超越大多数Python抓取库。 - 🚀 **闪电般快速**:优化性能超越大多数Python抓取库。
@@ -212,13 +212,12 @@ Scrapling v0.3包含强大的命令行界面:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339) [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
启动交互式网页抓取shell
```bash ```bash
# 启动交互式网页抓取shell
scrapling shell scrapling shell
```
# 直接将页面提取到文件而无需编程(默认提取`body`标签内的内容) 直接将页面提取到文件而无需编程(默认提取`body`标签内的内容)。如果输出文件以`.txt`结尾,则将提取目标的文本内容。如果以`.md`结尾,它将是HTML内容的Markdown表示;如果以`.html`结尾,它将是HTML内容本身。
# 如果输出文件以`.txt`结尾,则将提取目标的文本内容。 ```bash
# 如果以`.md`结尾,它将是HTML内容的markdown表示,`.html`将直接是HTML内容。
scrapling extract get 'https://example.com' content.md scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # 所有匹配CSS选择器'#fromSkipToProducts'的元素 scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # 所有匹配CSS选择器'#fromSkipToProducts'的元素
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## 性能基准 ## 性能基准
Scrapling不仅功能强大——它还速度极快,自0.3版本以来的更新在所有操作中都提供了卓越的性能改进。 Scrapling不仅功能强大——它还速度极快,自0.3版本以来的更新在所有操作中都提供了卓越的性能改进。以下基准测试将Scrapling的解析器与其他流行库进行了比较。
### 文本提取速度测试(5000个嵌套元素) ### 文本提取速度测试(5000个嵌套元素)
@@ -270,14 +269,14 @@ pip install scrapling
### 可选依赖项 ### 可选依赖项
1. 如果您要使用以下任何额外功能、获取器或它们的类,那么您需要安装获取器的依赖项,然后使用以下命令安装它们的浏览器依赖项 1. 如果您要使用以下任何额外功能、获取器或它们的类,您需要安装获取器的依赖项它们的浏览器依赖项,如下所示:
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
这会下载所有浏览器及其系统依赖项和指纹操作依赖项。 这会下载所有浏览器,以及它们的系统依赖项和指纹操作依赖项。
2. 额外功能: 2. 额外功能:
- 安装MCP服务器功能: - 安装MCP服务器功能:
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
此镜像通过仓库主分支上的GitHub actions自动构建和推送。 此镜像使用GitHub Actions和仓库主分支自动构建和推送。
## 贡献 ## 贡献
+11 -12
View File
@@ -87,8 +87,8 @@ Für das moderne Web entwickelt, bietet Scrapling **seine eigene schnelle Parsin
### Erweiterte Website-Abruf mit Sitzungsunterstützung ### Erweiterte Website-Abruf mit Sitzungsunterstützung
- **HTTP-Anfragen**: Schnelle und heimliche HTTP-Anfragen mit der `Fetcher`-Klasse. Kann Browser-TLS-Fingerabdrücke, Header imitieren und HTTP3 verwenden. - **HTTP-Anfragen**: Schnelle und heimliche HTTP-Anfragen mit der `Fetcher`-Klasse. Kann Browser-TLS-Fingerabdrücke, Header imitieren und HTTP3 verwenden.
- **Dynamisches Laden**: Abrufen dynamischer Websites mit vollständiger Browser-Automatisierung über die `DynamicFetcher`-Klasse, die Playwrights Chromium, echtes Chrome und benutzerdefinierten Stealth-Modus unterstützt. - **Dynamisches Laden**: Abrufen dynamischer Websites mit vollständiger Browser-Automatisierung über die `DynamicFetcher`-Klasse, die Playwrights Chromium und Google Chrome unterstützt.
- **Anti-Bot-Umgehung**: Erweiterte Stealth-Fähigkeiten mit `StealthyFetcher` unter Verwendung einer modifizierten Firefox-Version und Fingerabdruck-Spoofing. Kann alle Arten von Cloudflares Turnstile und Interstitial einfach mit Automatisierung umgehen. - **Anti-Bot-Umgehung**: Erweiterte Stealth-Fähigkeiten mit `StealthyFetcher` und Fingerabdruck-Spoofing. Kann alle Arten von Cloudflares Turnstile/Interstitial einfach mit Automatisierung umgehen.
- **Sitzungsverwaltung**: Persistente Sitzungsunterstützung mit den Klassen `FetcherSession`, `StealthySession` und `DynamicSession` für Cookie- und Zustandsverwaltung über Anfragen hinweg. - **Sitzungsverwaltung**: Persistente Sitzungsunterstützung mit den Klassen `FetcherSession`, `StealthySession` und `DynamicSession` für Cookie- und Zustandsverwaltung über Anfragen hinweg.
- **Async-Unterstützung**: Vollständige Async-Unterstützung über alle Fetcher und dedizierte Async-Sitzungsklassen hinweg. - **Async-Unterstützung**: Vollständige Async-Unterstützung über alle Fetcher und dedizierte Async-Sitzungsklassen hinweg.
@@ -96,7 +96,7 @@ Für das moderne Web entwickelt, bietet Scrapling **seine eigene schnelle Parsin
- 🔄 **Intelligente Element-Verfolgung**: Elemente nach Website-Änderungen mit intelligenten Ähnlichkeitsalgorithmen neu lokalisieren. - 🔄 **Intelligente Element-Verfolgung**: Elemente nach Website-Änderungen mit intelligenten Ähnlichkeitsalgorithmen neu lokalisieren.
- 🎯 **Intelligente flexible Auswahl**: CSS-Selektoren, XPath-Selektoren, filterbasierte Suche, Textsuche, Regex-Suche und mehr. - 🎯 **Intelligente flexible Auswahl**: CSS-Selektoren, XPath-Selektoren, filterbasierte Suche, Textsuche, Regex-Suche und mehr.
- 🔍 **Ähnliche Elemente finden**: Elemente, die gefundenen Elementen ähnlich sind, automatisch lokalisieren. - 🔍 **Ähnliche Elemente finden**: Elemente, die gefundenen Elementen ähnlich sind, automatisch lokalisieren.
- 🤖 **MCP-Server für die Verwendung mit KI**: Integrierter MCP-Server für KI-unterstütztes Web Scraping und Datenextraktion. Der MCP-Server verfügt über benutzerdefinierte, leistungsstarke Funktionen, die Scrapling nutzen, um gezielten Inhalt zu extrahieren, bevor er an die KI (Claude/Cursor/etc.) übergeben wird, wodurch Vorgänge beschleunigt und Kosten durch Minimierung der Token-Nutzung gesenkt werden. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE)) - 🤖 **MCP-Server für die Verwendung mit KI**: Integrierter MCP-Server für KI-unterstütztes Web Scraping und Datenextraktion. Der MCP-Server verfügt über leistungsstarke, benutzerdefinierte Funktionen, die Scrapling nutzen, um gezielten Inhalt zu extrahieren, bevor er an die KI (Claude/Cursor/etc.) übergeben wird, wodurch Vorgänge beschleunigt und Kosten durch Minimierung der Token-Nutzung gesenkt werden. ([Demo-Video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### Hochleistungs- und praxiserprobte Architektur ### Hochleistungs- und praxiserprobte Architektur
- 🚀 **Blitzschnell**: Optimierte Leistung, die die meisten Python-Scraping-Bibliotheken übertrifft. - 🚀 **Blitzschnell**: Optimierte Leistung, die die meisten Python-Scraping-Bibliotheken übertrifft.
@@ -212,13 +212,12 @@ Scrapling v0.3 enthält eine leistungsstarke Befehlszeilenschnittstelle:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339) [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Interaktive Web-Scraping-Shell starten
```bash ```bash
# Interaktive Web-Scraping-Shell starten
scrapling shell scrapling shell
```
# Seiten direkt ohne Programmierung in eine Datei extrahieren (Extrahiert standardmäßig den Inhalt im `body`-Tag) Seiten direkt ohne Programmierung in eine Datei extrahieren (Extrahiert standardmäßig den Inhalt im `body`-Tag). Wenn die Ausgabedatei mit `.txt` endet, wird der Textinhalt des Ziels extrahiert. Wenn sie mit `.md` endet, ist es eine Markdown-Darstellung des HTML-Inhalts; wenn sie mit `.html` endet, ist es der HTML-Inhalt selbst.
# Wenn die Ausgabedatei mit `.txt` endet, wird der Textinhalt des Ziels extrahiert. ```bash
# Wenn sie mit `.md` endet, ist es eine Markdown-Darstellung des HTML-Inhalts, und `.html` ist direkt der HTML-Inhalt.
scrapling extract get 'https://example.com' content.md scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Alle Elemente, die dem CSS-Selektor '#fromSkipToProducts' entsprechen scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Alle Elemente, die dem CSS-Selektor '#fromSkipToProducts' entsprechen
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Leistungsbenchmarks ## Leistungsbenchmarks
Scrapling ist nicht nur leistungsstark es ist auch blitzschnell, und die Updates seit Version 0.3 haben außergewöhnliche Leistungsverbesserungen bei allen Operationen gebracht. Scrapling ist nicht nur leistungsstark es ist auch blitzschnell, und die Updates seit Version 0.3 haben außergewöhnliche Leistungsverbesserungen bei allen Operationen gebracht. Die folgenden Benchmarks vergleichen den Parser von Scrapling mit anderen beliebten Bibliotheken.
### Textextraktions-Geschwindigkeitstest (5000 verschachtelte Elemente) ### Textextraktions-Geschwindigkeitstest (5000 verschachtelte Elemente)
@@ -270,14 +269,14 @@ Ab v0.3.2 enthält diese Installation nur die Parser-Engine und ihre Abhängigke
### Optionale Abhängigkeiten ### Optionale Abhängigkeiten
1. Wenn Sie eine der folgenden zusätzlichen Funktionen, die Fetcher oder ihre Klassen verwenden möchten, müssen Sie die Abhängigkeiten der Fetcher installieren und dann ihre Browser-Abhängigkeiten mit 1. Wenn Sie eine der folgenden zusätzlichen Funktionen, die Fetcher oder ihre Klassen verwenden möchten, müssen Sie die Abhängigkeiten der Fetcher und ihre Browser-Abhängigkeiten wie folgt installieren:
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
Dies lädt alle Browser mit ihren Systemabhängigkeiten und Fingerabdruck-Manipulationsabhängigkeiten herunter. Dies lädt alle Browser zusammen mit ihren Systemabhängigkeiten und Fingerabdruck-Manipulationsabhängigkeiten herunter.
2. Zusätzliche Funktionen: 2. Zusätzliche Funktionen:
- MCP-Server-Funktion installieren: - MCP-Server-Funktion installieren:
@@ -303,7 +302,7 @@ Oder laden Sie es aus der GitHub-Registry herunter:
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
Dieses Image wird automatisch über GitHub Actions im Hauptzweig des Repositorys erstellt und gepusht. Dieses Image wird automatisch mit GitHub Actions und dem Hauptzweig des Repositorys erstellt und gepusht.
## Beitragen ## Beitragen
+11 -12
View File
@@ -87,8 +87,8 @@ Construido para la Web moderna, Scrapling presenta **su propio motor de análisi
### Obtención Avanzada de Sitios Web con Soporte de Sesión ### Obtención Avanzada de Sitios Web con Soporte de Sesión
- **Solicitudes HTTP**: Solicitudes HTTP rápidas y sigilosas con la clase `Fetcher`. Puede imitar la huella TLS de los navegadores, encabezados y usar HTTP3. - **Solicitudes HTTP**: Solicitudes HTTP rápidas y sigilosas con la clase `Fetcher`. Puede imitar la huella TLS de los navegadores, encabezados y usar HTTP3.
- **Carga Dinámica**: Obtén sitios web dinámicos con automatización completa del navegador a través de la clase `DynamicFetcher` compatible con Chromium de Playwright, Chrome real y modo sigiloso personalizado. - **Carga Dinámica**: Obtén sitios web dinámicos con automatización completa del navegador a través de la clase `DynamicFetcher` compatible con Chromium de Playwright y Google Chrome.
- **Evasión Anti-bot**: Capacidades de sigilo avanzadas con `StealthyFetcher` usando una versión modificada de Firefox y falsificación de huellas digitales. Puede evadir todos los tipos de Turnstile e Interstitial de Cloudflare con automatización fácilmente. - **Evasión Anti-bot**: Capacidades de sigilo avanzadas con `StealthyFetcher` y falsificación de huellas digitales. Puede evadir fácilmente todos los tipos de Turnstile/Interstitial de Cloudflare con automatización.
- **Gestión de Sesión**: Soporte de sesión persistente con las clases `FetcherSession`, `StealthySession` y `DynamicSession` para la gestión de cookies y estado entre solicitudes. - **Gestión de Sesión**: Soporte de sesión persistente con las clases `FetcherSession`, `StealthySession` y `DynamicSession` para la gestión de cookies y estado entre solicitudes.
- **Soporte Async**: Soporte async completo en todos los fetchers y clases de sesión async dedicadas. - **Soporte Async**: Soporte async completo en todos los fetchers y clases de sesión async dedicadas.
@@ -96,7 +96,7 @@ Construido para la Web moderna, Scrapling presenta **su propio motor de análisi
- 🔄 **Seguimiento Inteligente de Elementos**: Relocaliza elementos después de cambios en el sitio web usando algoritmos inteligentes de similitud. - 🔄 **Seguimiento Inteligente de Elementos**: Relocaliza elementos después de cambios en el sitio web usando algoritmos inteligentes de similitud.
- 🎯 **Selección Flexible Inteligente**: Selectores CSS, selectores XPath, búsqueda basada en filtros, búsqueda de texto, búsqueda regex y más. - 🎯 **Selección Flexible Inteligente**: Selectores CSS, selectores XPath, búsqueda basada en filtros, búsqueda de texto, búsqueda regex y más.
- 🔍 **Encontrar Elementos Similares**: Localiza automáticamente elementos similares a los elementos encontrados. - 🔍 **Encontrar Elementos Similares**: Localiza automáticamente elementos similares a los elementos encontrados.
- 🤖 **Servidor MCP para usar con IA**: Servidor MCP integrado para Web Scraping asistido por IA y extracción de datos. El servidor MCP presenta capacidades personalizadas y poderosas que utilizan Scrapling para extraer contenido específico antes de pasarlo a la IA (Claude/Cursor/etc), acelerando así las operaciones y reduciendo costos al minimizar el uso de tokens. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE)) - 🤖 **Servidor MCP para usar con IA**: Servidor MCP integrado para Web Scraping asistido por IA y extracción de datos. El servidor MCP presenta capacidades poderosas y personalizadas que aprovechan Scrapling para extraer contenido específico antes de pasarlo a la IA (Claude/Cursor/etc), acelerando así las operaciones y reduciendo costos al minimizar el uso de tokens. ([video demo](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### Arquitectura de Alto Rendimiento y Probada en Batalla ### Arquitectura de Alto Rendimiento y Probada en Batalla
- 🚀 **Ultrarrápido**: Rendimiento optimizado que supera a la mayoría de las bibliotecas de scraping de Python. - 🚀 **Ultrarrápido**: Rendimiento optimizado que supera a la mayoría de las bibliotecas de scraping de Python.
@@ -212,13 +212,12 @@ Scrapling v0.3 incluye una poderosa interfaz de línea de comandos:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339) [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Lanzar shell interactivo de Web Scraping
```bash ```bash
# Lanzar shell interactivo de Web Scraping
scrapling shell scrapling shell
```
# Extraer páginas a un archivo directamente sin programar (Extrae el contenido dentro de la etiqueta `body` por defecto) Extraer páginas a un archivo directamente sin programar (Extrae el contenido dentro de la etiqueta `body` por defecto). Si el archivo de salida termina con `.txt`, entonces se extraerá el contenido de texto del objetivo. Si termina con `.md`, será una representación Markdown del contenido HTML; si termina con `.html`, será el contenido HTML en sí mismo.
# Si el archivo de salida termina con `.txt`, entonces se extraerá el contenido de texto del objetivo. ```bash
# Si termina con `.md`, será una representación markdown del contenido HTML, y `.html` será el contenido HTML directamente.
scrapling extract get 'https://example.com' content.md scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Todos los elementos que coinciden con el selector CSS '#fromSkipToProducts' scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Todos los elementos que coinciden con el selector CSS '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Benchmarks de Rendimiento ## Benchmarks de Rendimiento
Scrapling no solo es poderoso, también es increíblemente rápido, y las actualizaciones desde la versión 0.3 han brindado mejoras de rendimiento excepcionales en todas las operaciones. Scrapling no solo es poderoso, también es increíblemente rápido, y las actualizaciones desde la versión 0.3 han brindado mejoras de rendimiento excepcionales en todas las operaciones. Los siguientes benchmarks comparan el analizador de Scrapling con otras bibliotecas populares.
### Prueba de Velocidad de Extracción de Texto (5000 elementos anidados) ### Prueba de Velocidad de Extracción de Texto (5000 elementos anidados)
@@ -270,14 +269,14 @@ A partir de v0.3.2, esta instalación solo incluye el motor de análisis y sus d
### Dependencias Opcionales ### Dependencias Opcionales
1. Si vas a usar alguna de las características adicionales a continuación, los fetchers, o sus clases, entonces necesitas instalar las dependencias de los fetchers y luego instalar sus dependencias del navegador con 1. Si vas a usar alguna de las características adicionales a continuación, los fetchers, o sus clases, necesitas instalar las dependencias de los fetchers y sus dependencias del navegador de la siguiente manera:
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
Esto descarga todos los navegadores con sus dependencias del sistema y dependencias de manipulación de huellas digitales. Esto descarga todos los navegadores, junto con sus dependencias del sistema y dependencias de manipulación de huellas digitales.
2. Características adicionales: 2. Características adicionales:
- Instalar la característica del servidor MCP: - Instalar la característica del servidor MCP:
@@ -303,7 +302,7 @@ O descárgala desde el registro de GitHub:
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
Esta imagen se construye y publica automáticamente a través de GitHub actions en la rama principal del repositorio. Esta imagen se construye y publica automáticamente usando GitHub Actions y la rama principal del repositorio.
## Contribuir ## Contribuir
+11 -12
View File
@@ -87,8 +87,8 @@ Scraplingは単なるウェブスクレイピングライブラリではあり
### セッションサポート付き高度なウェブサイト取得 ### セッションサポート付き高度なウェブサイト取得
- **HTTPリクエスト**`Fetcher`クラスで高速でステルスなHTTPリクエスト。ブラウザのTLSフィンガープリント、ヘッダーを模倣し、HTTP3を使用できます。 - **HTTPリクエスト**`Fetcher`クラスで高速でステルスなHTTPリクエスト。ブラウザのTLSフィンガープリント、ヘッダーを模倣し、HTTP3を使用できます。
- **動的読み込み**Playwright's Chromium、実際のChrome、カスタムステルスモードをサポートする`DynamicFetcher`クラスを通じた完全なブラウザ自動化で動的ウェブサイトを取得。 - **動的読み込み**Playwright's ChromiumとGoogle Chromeをサポートする`DynamicFetcher`クラスを通じた完全なブラウザ自動化で動的ウェブサイトを取得。
- **アンチボット回避**修正されたFirefoxとフィンガープリント偽装を使用した`StealthyFetcher`による高度なステルス機能。自動化でCloudflareのTurnstileInterstitialのすべてのタイプを簡単に回避できます。 - **アンチボット回避**`StealthyFetcher`とフィンガープリント偽装による高度なステルス機能。自動化でCloudflareのTurnstile/Interstitialのすべてのタイプを簡単に回避できます。
- **セッション管理**:リクエスト間でCookieと状態を管理するための`FetcherSession``StealthySession``DynamicSession`クラスによる永続的なセッションサポート。 - **セッション管理**:リクエスト間でCookieと状態を管理するための`FetcherSession``StealthySession``DynamicSession`クラスによる永続的なセッションサポート。
- **非同期サポート**:すべてのフェッチャーと専用非同期セッションクラス全体での完全な非同期サポート。 - **非同期サポート**:すべてのフェッチャーと専用非同期セッションクラス全体での完全な非同期サポート。
@@ -96,7 +96,7 @@ Scraplingは単なるウェブスクレイピングライブラリではあり
- 🔄 **スマート要素追跡**:インテリジェントな類似性アルゴリズムを使用してウェブサイトの変更後に要素を再配置。 - 🔄 **スマート要素追跡**:インテリジェントな類似性アルゴリズムを使用してウェブサイトの変更後に要素を再配置。
- 🎯 **スマート柔軟選択**:CSSセレクタ、XPathセレクタ、フィルタベース検索、テキスト検索、正規表現検索など。 - 🎯 **スマート柔軟選択**:CSSセレクタ、XPathセレクタ、フィルタベース検索、テキスト検索、正規表現検索など。
- 🔍 **類似要素を見つける**:見つかった要素に類似した要素を自動的に特定。 - 🔍 **類似要素を見つける**:見つかった要素に類似した要素を自動的に特定。
- 🤖 **AIと使用するMCPサーバー**:AI支援ウェブスクレイピングとデータ抽出のための組み込みMCPサーバー。MCPサーバーは、AIClaude/Cursorなど)に渡す前にScraplingを用してターゲットコンテンツを抽出するカスタムで強力な機能を備えており、操作を高速化し、トークン使用量を最小限に抑えることでコストを削減します。([デモビデオ](https://www.youtube.com/watch?v=qyFk3ZNwOxE) - 🤖 **AIと使用するMCPサーバー**:AI支援ウェブスクレイピングとデータ抽出のための組み込みMCPサーバー。MCPサーバーは、AIClaude/Cursorなど)に渡す前にScraplingを用してターゲットコンテンツを抽出する強力でカスタムな機能を備えており、操作を高速化し、トークン使用量を最小限に抑えることでコストを削減します。([デモビデオ](https://www.youtube.com/watch?v=qyFk3ZNwOxE)
### 高性能で実戦テスト済みのアーキテクチャ ### 高性能で実戦テスト済みのアーキテクチャ
- 🚀 **高速**:ほとんどのPythonスクレイピングライブラリを上回る最適化されたパフォーマンス。 - 🚀 **高速**:ほとんどのPythonスクレイピングライブラリを上回る最適化されたパフォーマンス。
@@ -212,13 +212,12 @@ Scrapling v0.3には強力なコマンドラインインターフェースが含
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339) [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
インタラクティブウェブスクレイピングシェルを起動
```bash ```bash
# インタラクティブウェブスクレイピングシェルを起動
scrapling shell scrapling shell
```
# プログラミングせずに直接ページをファイルに抽出(デフォルトで`body`タグ内のコンテンツを抽出) プログラミングせずに直接ページをファイルに抽出(デフォルトで`body`タグ内のコンテンツを抽出)。出力ファイルが`.txt`で終わる場合、ターゲットのテキストコンテンツが抽出されます。`.md`で終わる場合、HTMLコンテンツのMarkdown表現になります;`.html`で終わる場合、HTMLコンテンツそのものになります。
# 出力ファイルが`.txt`で終わる場合、ターゲットのテキストコンテンツが抽出されます。 ```bash
# `.md`で終わる場合、HTMLコンテンツのMarkdown表現になり、`.html`は直接HTMLコンテンツになります。
scrapling extract get 'https://example.com' content.md scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # CSSセレクタ'#fromSkipToProducts'に一致するすべての要素 scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # CSSセレクタ'#fromSkipToProducts'に一致するすべての要素
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## パフォーマンスベンチマーク ## パフォーマンスベンチマーク
Scraplingは強力であるだけでなく、驚くほど高速で、バージョン0.3以降のアップデートはすべての操作で優れたパフォーマンス向上を実現しています。 Scraplingは強力であるだけでなく、驚くほど高速で、バージョン0.3以降のアップデートはすべての操作で優れたパフォーマンス向上を実現しています。以下のベンチマークは、Scraplingのパーサーを他の人気のあるライブラリと比較しています。
### テキスト抽出速度テスト(5000個のネストされた要素) ### テキスト抽出速度テスト(5000個のネストされた要素)
@@ -270,14 +269,14 @@ v0.3.2以降、このインストールにはパーサーエンジンとその
### オプションの依存関係 ### オプションの依存関係
1. 以下の追加機能、フェッチャー、またはそれらのクラスのいずれかを使用する場合は、フェッチャーの依存関係をインストールしてから、次のコマンドでブラウザの依存関係をインストールする必要があります 1. 以下の追加機能、フェッチャー、またはそれらのクラスのいずれかを使用する場合は、フェッチャーの依存関係ブラウザの依存関係を次のようにインストールする必要があります
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
これにより、すべてのブラウザとそのシステム依存関係およびフィンガープリント操作依存関係がダウンロードされます。 これにより、すべてのブラウザ、およびそれらのシステム依存関係フィンガープリント操作依存関係がダウンロードされます。
2. 追加機能: 2. 追加機能:
- MCPサーバー機能をインストール: - MCPサーバー機能をインストール:
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
このイメージは、リポジトリのメインブランチでGitHub actionsを通じて自動的にビルドおよびプッシュされます。 このイメージは、GitHub Actionsとリポジトリのメインブランチを使用して自動的にビルドおよびプッシュされます。
## 貢献 ## 貢献
+11 -12
View File
@@ -87,8 +87,8 @@ Scrapling - это не просто очередная библиотека д
### Продвинутая загрузка сайтов с поддержкой сессий ### Продвинутая загрузка сайтов с поддержкой сессий
- **HTTP-запросы**: Быстрые и скрытные HTTP-запросы с классом `Fetcher`. Может имитировать TLS-отпечаток браузера, заголовки и использовать HTTP3. - **HTTP-запросы**: Быстрые и скрытные HTTP-запросы с классом `Fetcher`. Может имитировать TLS-отпечаток браузера, заголовки и использовать HTTP3.
- **Динамическая загрузка**: Загрузка динамических сайтов с полной автоматизацией браузера через класс `DynamicFetcher`, поддерживающий Chromium от Playwright, настоящий Chrome и пользовательский режим скрытности. - **Динамическая загрузка**: Загрузка динамических сайтов с полной автоматизацией браузера через класс `DynamicFetcher`, поддерживающий Chromium от Playwright и Google Chrome.
- **Обход анти-ботов**: Расширенные возможности скрытности с `StealthyFetcher`, использующим модифицированную версию Firefox и подмену отпечатков. Может легко обойти все типы Turnstile и Interstitial от Cloudflare с помощью автоматизации. - **Обход анти-ботов**: Расширенные возможности скрытности с `StealthyFetcher` и подмену отпечатков. Может легко обойти все типы Turnstile/Interstitial от Cloudflare с помощью автоматизации.
- **Управление сессиями**: Поддержка постоянных сессий с классами `FetcherSession`, `StealthySession` и `DynamicSession` для управления cookie и состоянием между запросами. - **Управление сессиями**: Поддержка постоянных сессий с классами `FetcherSession`, `StealthySession` и `DynamicSession` для управления cookie и состоянием между запросами.
- **Поддержка асинхронности**: Полная асинхронная поддержка во всех фетчерах и выделенных асинхронных классах сессий. - **Поддержка асинхронности**: Полная асинхронная поддержка во всех фетчерах и выделенных асинхронных классах сессий.
@@ -96,7 +96,7 @@ Scrapling - это не просто очередная библиотека д
- 🔄 **Умное отслеживание элементов**: Перемещайте элементы после изменений сайта с помощью интеллектуальных алгоритмов подобия. - 🔄 **Умное отслеживание элементов**: Перемещайте элементы после изменений сайта с помощью интеллектуальных алгоритмов подобия.
- 🎯 **Умный гибкий выбор**: CSS-селекторы, XPath-селекторы, поиск на основе фильтров, текстовый поиск, поиск по регулярным выражениям и многое другое. - 🎯 **Умный гибкий выбор**: CSS-селекторы, XPath-селекторы, поиск на основе фильтров, текстовый поиск, поиск по регулярным выражениям и многое другое.
- 🔍 **Поиск похожих элементов**: Автоматически находите элементы, похожие на найденные элементы. - 🔍 **Поиск похожих элементов**: Автоматически находите элементы, похожие на найденные элементы.
- 🤖 **MCP-сервер для использования с ИИ**: Встроенный MCP-сервер для веб-скрапинга с помощью ИИ и извлечения данных. MCP-сервер обладает пользовательскими, мощными возможностями, которые используют Scrapling для извлечения целевого контента перед передачей его ИИ (Claude/Cursor/и т.д.), тем самым ускоряя операции и снижая затраты за счет минимизации использования токенов. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE)) - 🤖 **MCP-сервер для использования с ИИ**: Встроенный MCP-сервер для веб-скрапинга с помощью ИИ и извлечения данных. MCP-сервер обладает мощными, пользовательскими возможностями, которые используют Scrapling для извлечения целевого контента перед передачей его ИИ (Claude/Cursor/и т.д.), тем самым ускоряя операции и снижая затраты за счет минимизации использования токенов. ([демо-видео](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### Высокопроизводительная и проверенная в боях архитектура ### Высокопроизводительная и проверенная в боях архитектура
- 🚀 **Молниеносно быстро**: Оптимизированная производительность превосходит большинство библиотек скрапинга Python. - 🚀 **Молниеносно быстро**: Оптимизированная производительность превосходит большинство библиотек скрапинга Python.
@@ -212,13 +212,12 @@ Scrapling v0.3 включает мощный интерфейс командно
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339) [![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Запустить интерактивную оболочку веб-скрапинга
```bash ```bash
# Запустить интерактивную оболочку веб-скрапинга
scrapling shell scrapling shell
```
# Извлечь страницы в файл напрямую без программирования (Извлекает содержимое внутри тега `body` по умолчанию) Извлечь страницы в файл напрямую без программирования (Извлекает содержимое внутри тега `body` по умолчанию). Если выходной файл заканчивается на `.txt`, то будет извлечено текстовое содержимое цели. Если заканчивается на `.md`, это будет Markdown-представление HTML-содержимого; если заканчивается на `.html`, это будет само HTML-содержимое.
# Если выходной файл заканчивается на `.txt`, то будет извлечено текстовое содержимое цели. ```bash
# Если заканчивается на `.md`, это будет markdown-представление HTML-содержимого, а `.html` будет непосредственно HTML-содержимым.
scrapling extract get 'https://example.com' content.md scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Все элементы, соответствующие CSS-селектору '#fromSkipToProducts' scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # Все элементы, соответствующие CSS-селектору '#fromSkipToProducts'
scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless scrapling extract fetch 'https://example.com' content.md --css-selector '#fromSkipToProducts' --no-headless
@@ -230,7 +229,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
## Тесты производительности ## Тесты производительности
Scrapling не только мощный - он также невероятно быстрый, и обновления с версии 0.3 обеспечили исключительные улучшения производительности во всех операциях. Scrapling не только мощный - он также невероятно быстрый, и обновления с версии 0.3 обеспечили исключительные улучшения производительности во всех операциях. Следующие тесты производительности сравнивают парсер Scrapling с другими популярными библиотеками.
### Тест скорости извлечения текста (5000 вложенных элементов) ### Тест скорости извлечения текста (5000 вложенных элементов)
@@ -270,14 +269,14 @@ pip install scrapling
### Опциональные зависимости ### Опциональные зависимости
1. Если вы собираетесь использовать какие-либо из дополнительных функций ниже, фетчеры или их классы, то вам нужно установить зависимости фетчеров, а затем установить их зависимости браузера с помощью 1. Если вы собираетесь использовать какие-либо из дополнительных функций ниже, фетчеры или их классы, вам необходимо установить зависимости фетчеров и их зависимости браузера следующим образом:
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
Это загрузит все браузеры с их системными зависимостями и зависимостями манипуляции отпечатками. Это загрузит все браузеры вместе с их системными зависимостями и зависимостями манипуляции отпечатками.
2. Дополнительные функции: 2. Дополнительные функции:
- Установить функцию MCP-сервера: - Установить функцию MCP-сервера:
@@ -303,7 +302,7 @@ docker pull pyd4vinci/scrapling
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
Этот образ автоматически создается и отправляется через GitHub actions в основной ветке репозитория. Этот образ автоматически создается и отправляется с использованием GitHub Actions и основной ветки репозитория.
## Вклад ## Вклад
+20 -16
View File
@@ -2,7 +2,7 @@
<iframe width="560" height="315" src="https://www.youtube.com/embed/qyFk3ZNwOxE?si=3FHzgcYCb66iJ6e3" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe> <iframe width="560" height="315" src="https://www.youtube.com/embed/qyFk3ZNwOxE?si=3FHzgcYCb66iJ6e3" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful Web Scraping capabilities directly to your favorite AI chatbot or AI agent. This integration allows you to scrape websites, extract data, and bypass anti-bot protections conversationally through Claude's AI interface or any other chatbot that supports MCP. The **Scrapling MCP Server** is a new feature that brings Scrapling's powerful Web Scraping capabilities directly to your favorite AI chatbot or AI agent. This integration allows you to scrape websites, extract data, and bypass anti-bot protections conversationally through Claude's AI interface or any interface that supports MCP.
## Features ## Features
@@ -13,11 +13,11 @@ The Scrapling MCP Server provides six powerful tools for web scraping:
- **`bulk_get`**: An async version of the above tool that allows scraping of multiple URLs at the same time! - **`bulk_get`**: An async version of the above tool that allows scraping of multiple URLs at the same time!
### 🌐 Dynamic Content Scraping ### 🌐 Dynamic Content Scraping
- **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, stealth mode, and more! - **`fetch`**: Rapidly fetch dynamic content with Chromium/Chrome browser with complete control over the request/browser, and more!
- **`bulk_fetch`**: An async version of the above tool that allows scraping of multiple URLs in different browser tabs at the same time! - **`bulk_fetch`**: An async version of the above tool that allows scraping of multiple URLs in different browser tabs at the same time!
### 🔒 Stealth Scraping ### 🔒 Stealth Scraping
- **`stealthy_fetch`**: Uses our modified version of Camoufox browser to bypass Cloudflare Turnstile/Interstitial and other anti-bot systems with complete control over the request/browser! - **`stealthy_fetch`**: Uses our Stealthy browser to bypass Cloudflare Turnstile/Interstitial and other anti-bot systems with complete control over the request/browser!
- **`bulk_stealthy_fetch`**: An async version of the above tool that allows stealth scraping of multiple URLs in different browser tabs at the same time! - **`bulk_stealthy_fetch`**: An async version of the above tool that allows stealth scraping of multiple URLs in different browser tabs at the same time!
### Key Capabilities ### Key Capabilities
@@ -30,9 +30,9 @@ The Scrapling MCP Server provides six powerful tools for web scraping:
#### But why use Scrapling MCP Server instead of other available tools? #### But why use Scrapling MCP Server instead of other available tools?
Aside from its stealth capabilities and ability to bypass Cloudflare Turnstile/Interstitial, Scrapling's server is the only one that allows you to pass a CSS selector in the prompt to extract specific elements before handing the content to the AI. Aside from its stealth capabilities and ability to bypass Cloudflare Turnstile/Interstitial, Scrapling's server is the only one that lets you select specific elements to pass to the AI, saving a lot of time and tokens!
The way other servers work is that they extract the content, then pass it all to the AI to extract the fields you want. This causes the AI to consume a lot more tokens that are not needed (from irrelevant content). Scrapling solves this problem by allowing you to pass a CSS selector to narrow down the content you want before passing it to the AI, which makes the whole process much faster and more efficient. The way other servers work is that they extract the content, then pass it all to the AI to extract the fields you want. This causes the AI to consume far more tokens than needed (from irrelevant content). Scrapling solves this problem by allowing you to pass a CSS selector to narrow down the content you want before passing it to the AI, which makes the whole process much faster and more efficient.
If you don't know how to write/use CSS selectors, don't worry. You can tell the AI in the prompt to write selectors to match possible fields for you and watch it try different combinations until it finds the right one, as we will show in the examples section. If you don't know how to write/use CSS selectors, don't worry. You can tell the AI in the prompt to write selectors to match possible fields for you and watch it try different combinations until it finds the right one, as we will show in the examples section.
@@ -48,10 +48,14 @@ pip install "scrapling[ai]"
scrapling install scrapling install
``` ```
Or use the Docker image directly: Or use the Docker image directly from the Docker registry:
```bash ```bash
docker pull pyd4vinci/scrapling docker pull pyd4vinci/scrapling
``` ```
Or download it from the GitHub registry:
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
## Setting up the MCP Server ## Setting up the MCP Server
@@ -83,12 +87,12 @@ If that's the first MCP server you're adding, set the content of the file to thi
} }
} }
``` ```
As per the [official article](https://modelcontextprotocol.io/quickstart/user), this action creates a new configuration file if one doesnt exist or opens your existing configuration. The file is located at As per the [official article](https://modelcontextprotocol.io/quickstart/user), this action either creates a new configuration file if none exists or opens your existing configuration. The file is located at
1. **MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json` 1. **MacOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
2. **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` 2. **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
To ensure it's working, it's best to use the full path to the `scrapling` executable. Open the terminal and execute the following command: To ensure it's working, use the full path to the `scrapling` executable. Open the terminal and execute the following command:
1. **MacOS**: `which scrapling` 1. **MacOS**: `which scrapling`
2. **Windows**: `where scrapling` 2. **Windows**: `where scrapling`
@@ -150,7 +154,7 @@ Use the following to enable 'Streamable HTTP' transport mode:
```bash ```bash
scrapling mcp --http scrapling mcp --http
``` ```
Hence, the default value for the host the server is listening on is '0.0.0.0' and the port is 8000, which both can be configured as below: Hence, the default value for the host the server is listening to is '0.0.0.0' and the port is 8000, which both can be configured as below:
```bash ```bash
scrapling mcp --http --host '127.0.0.1' --port 8000 scrapling mcp --http --host '127.0.0.1' --port 8000
``` ```
@@ -169,13 +173,13 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Scrape the main content from https://example.com and convert it to markdown format. Scrape the main content from https://example.com and convert it to markdown format.
``` ```
Claude will use the `get` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for three attempts, unless you instruct it to do otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt. Claude will use the `get` tool to fetch the page and return clean, readable content. If it fails, it will continue retrying every second for 3 attempts, unless you instruct it otherwise. If it fails to retrieve content for any reason, such as protection or if it's a dynamic website, it will automatically try the other tools. If Claude didn't do that automatically for some reason, you can add that to the prompt.
A more optimized version of the same prompt would be: A more optimized version of the same prompt would be:
``` ```
Use regular requests to scrape the main content from https://example.com and convert it to markdown format. Use regular requests to scrape the main content from https://example.com and convert it to markdown format.
``` ```
This tells Claude about the right tool to use here, so it doesn't have to guess. Sometimes it will start using normal requests on its own, and at other times, it will assume browsers are better suited for this website without any apparent reason. As a general rule of thumb, you should always tell Claude what tool to use if you want to save time, money, and get consistent results. This tells Claude which tool to use here, so it doesn't have to guess. Sometimes it will start using normal requests on its own, and at other times, it will assume browsers are better suited for this website without any apparent reason. As a general rule of thumb, you should always tell Claude which tool to use if you want to save time and money and get consistent results.
2. **Targeted Data Extraction** 2. **Targeted Data Extraction**
@@ -185,7 +189,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Get all product titles from https://shop.example.com using the CSS selector '.product-title'. If the request fails, retry up to 5 times every 10 seconds. Get all product titles from https://shop.example.com using the CSS selector '.product-title'. If the request fails, retry up to 5 times every 10 seconds.
``` ```
The server will extract only the elements matching your selector and return them as a structured list. Notice I told it to set the tool to only try three times in case the website has connection issues, but the default setting should be fine for most cases. The server will extract only the elements matching your selector and return them as a structured list. Notice I told it to set the tool to try only 3 times in case the website has connection issues, but the default setting should be fine for most cases.
3. **E-commerce Data Collection** 3. **E-commerce Data Collection**
@@ -199,7 +203,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
Get the product names, prices, and descriptions from each page. Get the product names, prices, and descriptions from each page.
``` ```
Claude will use `bulk_fetch` to scrape all URLs concurrently, then analyze the extracted data. Claude will use `bulk_fetch` to concurrently scrape all URLs, then analyze the extracted data.
4. **More advanced workflow** 4. **More advanced workflow**
@@ -216,14 +220,14 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
And if you know how to write CSS selectors, you can instruct Claude to apply the selectors to the elements you want, and it will nearly complete the task immediately. And if you know how to write CSS selectors, you can instruct Claude to apply the selectors to the elements you want, and it will nearly complete the task immediately.
``` ```
Use normal requests to extract the URLs of all games on the page below, then perform a bulk request to them and return a list of all action games. Use normal requests to extract the URLs of all games on the page below, then perform a bulk request to them and return a list of all action games.
The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in the second request is `[data-qa="gameInfo#releaseInformation#genre-value"]` The selector for games in the first page is `[href*="/concept/"]` and the selector for the genre in the second request is `[data-qa="gameInfo#releaseInformation#genre-value"]`.
URL: https://store.playstation.com/en-us/pages/browse URL: https://store.playstation.com/en-us/pages/browse
``` ```
5. **Get data from a website with Cloudflare protection** 5. **Get data from a website with Cloudflare protection**
If you think the website you are targeting has Cloudflare protection, you should tell Claude instead of letting it discover that on its own. If you think the website you are targeting has Cloudflare protection, tell Claude instead of letting it discover it on its own.
``` ```
What's the price of this product? Be cautious, as it utilizes Cloudflare's Turnstile protection. Make the browser visible while you work. What's the price of this product? Be cautious, as it utilizes Cloudflare's Turnstile protection. Make the browser visible while you work.
@@ -234,7 +238,7 @@ We will gradually go from simple prompts to more complex ones. We will use Claud
You can, for example, use a prompt like this: You can, for example, use a prompt like this:
``` ```
Extract all the product URLs in the following category, then return the prices and the details of the first three products. Extract all product URLs for the following category, then return the prices and details for the first 3 products.
https://www.arnotts.ie/furniture/bedroom/bed-frames/ https://www.arnotts.ie/furniture/bedroom/bed-frames/
``` ```
+1 -1
View File
@@ -1,6 +1,6 @@
# Performance Benchmarks # Performance Benchmarks
Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 deliver exceptional performance improvements across all operations! Scrapling isn't just powerful—it's also blazing fast, and the updates since version 0.3 have delivered exceptional performance improvements across all operations. The following benchmarks compare Scrapling's parser with other popular libraries.
## Benchmark Results ## Benchmark Results
+17 -20
View File
@@ -2,7 +2,7 @@
**Web Scraping through the terminal without requiring any programming!** **Web Scraping through the terminal without requiring any programming!**
The `scrapling extract` Command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction. The `scrapling extract` command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction.
> 💡 **Prerequisites:** > 💡 **Prerequisites:**
> >
@@ -30,7 +30,7 @@ The extract command is a set of simple terminal tools that:
```bash ```bash
scrapling extract get "https://example.com" page_content.txt scrapling extract get "https://example.com" page_content.txt
``` ```
This does an HTTP GET request and saves the text content of the webpage to `page_content.txt`. This makes an HTTP GET request and saves the webpage's text content to `page_content.txt`.
- **Save as Different Formats** - **Save as Different Formats**
@@ -73,13 +73,13 @@ Commands:
stealthy-fetch Use StealthyFetcher to fetch content with advanced... stealthy-fetch Use StealthyFetcher to fetch content with advanced...
``` ```
We will go through each Command in detail below. We will go through each command in detail below.
### HTTP Requests ### HTTP Requests
1. **GET Request** 1. **GET Request**
The most common Command for downloading website content: The most common command for downloading website content:
```bash ```bash
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS] scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]
@@ -105,7 +105,7 @@ We will go through each Command in detail below.
# Add multiple headers # Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US" scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
``` ```
Get the available options for the Command with `scrapling extract get --help` as follows: Get the available options for the command with `scrapling extract get --help` as follows:
```bash ```bash
Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE
@@ -143,7 +143,7 @@ We will go through each Command in detail below.
# Send JSON data # Send JSON data
scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}' scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'
``` ```
Get the available options for the Command with `scrapling extract post --help` as follows: Get the available options for the command with `scrapling extract post --help` as follows:
```bash ```bash
Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE
@@ -182,7 +182,7 @@ We will go through each Command in detail below.
# Send JSON data # Send JSON data
scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}' scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'
``` ```
Get the available options for the Command with `scrapling extract put --help` as follows: Get the available options for the command with `scrapling extract put --help` as follows:
```bash ```bash
Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE
@@ -220,7 +220,7 @@ We will go through each Command in detail below.
# Send JSON data # Send JSON data
scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome" scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"
``` ```
Get the available options for the Command with `scrapling extract delete --help` as follows: Get the available options for the command with `scrapling extract delete --help` as follows:
```bash ```bash
Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE
@@ -263,7 +263,7 @@ We will go through each Command in detail below.
# Run in visible browser mode (helpful for debugging) # Run in visible browser mode (helpful for debugging)
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
``` ```
Get the available options for the Command with `scrapling extract fetch --help` as follows: Get the available options for the command with `scrapling extract fetch --help` as follows:
```bash ```bash
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
@@ -279,10 +279,8 @@ We will go through each Command in detail below.
--wait INTEGER Additional wait time in milliseconds after page load (default: 0) --wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding --wait-selector TEXT CSS selector to wait for before proceeding
--locale TEXT Browser locale (default: en-US) --locale TEXT Specify user locale. Defaults to the system default locale.
--stealth / --no-stealth Enable stealth mode (default: False) ---real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--disable-webgl / --enable-webgl Disable WebGL support (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port" --proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times) -H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit. --help Show this message and exit.
@@ -304,10 +302,10 @@ We will go through each Command in detail below.
# Solve Cloudflare challenges # Solve Cloudflare challenges
scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a" scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
# Use proxy for anonymity # Use a proxy for anonymity.
scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080" scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
``` ```
Get the available options for the Command with `scrapling extract stealthy-fetch --help` as follows: Get the available options for the command with `scrapling extract stealthy-fetch --help` as follows:
```bash ```bash
Usage: scrapling extract stealthy-fetch [OPTIONS] URL OUTPUT_FILE Usage: scrapling extract stealthy-fetch [OPTIONS] URL OUTPUT_FILE
@@ -317,25 +315,24 @@ We will go through each Command in detail below.
Options: Options:
--headless / --no-headless Run browser in headless mode (default: True) --headless / --no-headless Run browser in headless mode (default: True)
--block-images / --allow-images Block image loading (default: False)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False) --disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--block-webrtc / --allow-webrtc Block WebRTC entirely (default: False) --block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
--humanize / --no-humanize Humanize cursor movement (default: False)
--solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False) --solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
--allow-webgl / --block-webgl Allow WebGL (default: True) --allow-webgl / --block-webgl Allow WebGL (default: True)
--network-idle / --no-network-idle Wait for network idle (default: False) --network-idle / --no-network-idle Wait for network idle (default: False)
--disable-ads / --allow-ads Install uBlock Origin addon (default: False) ---real-chrome/--no-real-chrom If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--hide-canvas/--show-canvas Add noise to canvas operations (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000) --timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0) --wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches. -s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding --wait-selector TEXT CSS selector to wait for before proceeding
--geoip / --no-geoip Use IP/Proxy geolocation for timezone/locale (default: False) --hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port" --proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times) -H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--help Show this message and exit. --help Show this message and exit.
``` ```
## When to use each Command ## When to use each command
If you are not a Web Scraping expert and can't decide what to choose, you can use the following formula to help you decide: If you are not a Web Scraping expert and can't decide what to choose, you can use the following formula to help you decide:
+4 -2
View File
@@ -4,7 +4,7 @@
**Powerful Web Scraping REPL for Developers and Data Scientists** **Powerful Web Scraping REPL for Developers and Data Scientists**
The Scrapling Interactive Shell is an enhanced IPython-based environment designed specifically for Web Scraping tasks. It provides instant access to all Scrapling features, clever shortcuts, automatic page management, and advanced tools like curl command conversion. The Scrapling Interactive Shell is an enhanced IPython-based environment designed specifically for Web Scraping tasks. It provides instant access to all Scrapling features, clever shortcuts, automatic page management, and advanced tools, such as conversion of the curl command.
> 💡 **Prerequisites:** > 💡 **Prerequisites:**
> >
@@ -129,7 +129,9 @@ View scraped pages in your browser:
### Curl Command Integration ### Curl Command Integration
The shell provides a few functions to help you convert curl commands from the browser DevTools to `Fetcher` requests, which are `uncurl` and `curl2fetcher`. First, you need to copy a request as a curl command like the following: The shell provides a few functions to help you convert curl commands from the browser DevTools to `Fetcher` requests: `uncurl` and `curl2fetcher`.
First, you need to copy a request as a curl command like the following:
<img src="../../assets/scrapling_shell_curl.png" title="Copying a request as a curl command from Chrome" alt="Copying a request as a curl command from Chrome" style="width: 70%;"/> <img src="../../assets/scrapling_shell_curl.png" title="Copying a request as a curl command from Chrome" alt="Copying a request as a curl command from Chrome" style="width: 70%;"/>
+1 -1
View File
@@ -27,4 +27,4 @@ and the installation of the fetchers' dependencies with the following command
```bash ```bash
scrapling install scrapling install
``` ```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies. This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
+3 -3
View File
@@ -1,6 +1,6 @@
Scrapling uses SQLite by default, but this tutorial covers writing your storage system to store element properties there for `adaptive` feature. Scrapling uses SQLite by default, but this tutorial shows how to write your own storage system to store element properties for the `adaptive` feature.
You might want to use FireBase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because the spiders will share with each other. You might want to use Firebase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because spiders can share adaptive data with each other.
So first, to make your storage class work, it must do the big 3: So first, to make your storage class work, it must do the big 3:
@@ -8,7 +8,7 @@ So first, to make your storage class work, it must do the big 3:
2. Use the decorator `functools.lru_cache` on top of the class to follow the Singleton design pattern as other classes. 2. Use the decorator `functools.lru_cache` on top of the class to follow the Singleton design pattern as other classes.
3. Implement methods `save` and `retrieve`, as you see from the type hints: 3. Implement methods `save` and `retrieve`, as you see from the type hints:
- The method `save` returns nothing and will get two arguments from the library - The method `save` returns nothing and will get two arguments from the library
* The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the function `element_to_dict` in submodule `scrapling.core.utils._StorageTools` to keep the same format and save it to your database as you wish. * The first one is of type `lxml.html.HtmlElement`, which is the element itself. It must be converted to a dictionary using the `element_to_dict` function in the submodule `scrapling.core.utils._StorageTools` to maintain the same format, and then saved to your database as you wish.
* The second one is a string, the identifier used for retrieval. The combination result of this identifier and the `url` argument from initialization must be unique for each row, or the `adaptive` data will be messed up. * The second one is a string, the identifier used for retrieval. The combination result of this identifier and the `url` argument from initialization must be unique for each row, or the `adaptive` data will be messed up.
- The method `retrieve` takes a string, which is the identifier; using it with the `url` passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns `None`. - The method `retrieve` takes a string, which is the identifier; using it with the `url` passed on initialization, the element's dictionary is retrieved from the database and returned if it exists; otherwise, it returns `None`.
+5 -5
View File
@@ -1,6 +1,6 @@
> You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :) > You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :)
### All current types can be imported alone like below ### All current types can be imported alone, like below
```python ```python
>>> from scrapling.core.custom_types import TextHandler, AttributesHandler >>> from scrapling.core.custom_types import TextHandler, AttributesHandler
@@ -11,11 +11,11 @@
>>> somedict_2 = AttributesHandler(a=1) >>> somedict_2 = AttributesHandler(a=1)
``` ```
Note that `TextHandler` is a subclass of Python's `str`, so all normal operations/methods that work with Python strings will work. Note that `TextHandler` is a subclass of Python's `str`, so all standard operations/methods that work with Python strings will work.
If you want to check for the type in your code, it's better to depend on Python's built-in function `issubclass`. If you want to check the type in your code, it's better to use Python's built-in `issubclass` function.
The class `AttributesHandler` is a subclass of `collections.abc.Mapping`, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the `_data` property, but be careful; it's of type `types.MappingProxyType`, so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds). The class `AttributesHandler` is a subclass of `collections.abc.Mapping`, so it's immutable (read-only), and all operations are inherited from it. The data passed can be accessed later through the `_data` property, but be careful; it's of type `types.MappingProxyType`, so it's immutable (read-only) as well (faster than `collections.abc.Mapping` by fractions of seconds).
So, to make it simple for you if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with class `AttributesHandler` except the ones that try to modify the actual data. So, to make it simple for you, if you are new to Python, the same operations and methods from the Python standard `dict` type will all work with the class `AttributesHandler` except for the ones that try to modify the actual data.
If you want to modify the data inside `AttributesHandler,` you have to convert it to a dictionary first, like using the `dict` function, and then modify it outside. If you want to modify the data inside `AttributesHandler`, you have to convert it to a dictionary first, e.g., using the `dict` function, and then change it outside.
+2 -3
View File
@@ -1,6 +1,6 @@
I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to provide these projects for free to the community. By becoming a sponsor, you would directly fund my coffee reserves, helping me continuously update existing projects and create new ones. I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to make them available to the community for free. By becoming a sponsor, you would directly fund my coffee reserves, helping me continuously update existing projects and create new ones.
You can sponsor me directly through [GitHub sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me A Coffe](https://buymeacoffee.com/d4vinci). You can sponsor me directly through the [GitHub Sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me a Coffee](https://buymeacoffee.com/d4vinci).
Thank you, stay curious, and hack the planet! ❤️ Thank you, stay curious, and hack the planet! ❤️
@@ -22,4 +22,3 @@ Perks:
- The same logo will be featured at [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling). - The same logo will be featured at [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling).
- Your logo will be featured as a top sponsor on [Scrapling's website](https://scrapling.readthedocs.io/en/latest/) main page. - Your logo will be featured as a top sponsor on [Scrapling's website](https://scrapling.readthedocs.io/en/latest/) main page.
- A Shoutout with each [Release note](https://github.com/D4Vinci/Scrapling/releases). - A Shoutout with each [Release note](https://github.com/D4Vinci/Scrapling/releases).
+12 -12
View File
@@ -3,7 +3,7 @@ Fetchers are classes that can do requests or fetch pages for you easily in a sin
This feature was introduced because, before v0.2, Scrapling was only a parsing engine. The target here is to gradually become the one-stop shop for all Web Scraping needs. This feature was introduced because, before v0.2, Scrapling was only a parsing engine. The target here is to gradually become the one-stop shop for all Web Scraping needs.
> Fetchers are not wrappers built on top of other libraries. However, they utilize these libraries as an engine to request/fetch pages easily for you, while fully leveraging that engine and adding features for you. Some fetchers don't even use the official library for requests; instead, they use their own custom version. For example, `StealthyFetcher` utilizes `Camoufox` browser directly, without relying on its Python library for anything except launch options. This last part might change soon as well. > Fetchers are not wrappers built on top of other libraries. However, they only use these libraries as an engine to request/fetch pages. To further clarify this, all fetchers have features that the underlying engines don't, while still fully leveraging those engines and optimizing them for Web Scraping.
## Fetchers Overview ## Fetchers Overview
@@ -12,17 +12,17 @@ Scrapling provides three different fetcher classes with their session classes; e
The following table compares them and can be quickly used for guidance. The following table compares them and can be quickly used for guidance.
| Feature | Fetcher | DynamicFetcher | StealthyFetcher | | Feature | Fetcher | DynamicFetcher | StealthyFetcher |
|--------------------|---------------------------------------------------|--------------------------------------------------------------------------------|--------------------------------------------------------------------------------------| |--------------------|---------------------------------------------------|-----------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------|
| Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇 | | Relative speed | 🐇🐇🐇🐇🐇 | 🐇🐇🐇 | 🐇🐇🐇 |
| Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | | Stealth | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ | | Anti-Bot options | ⭐⭐ | ⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| JavaScript loading | ❌ | ✅ | ✅ | | JavaScript loading | ❌ | ✅ | ✅ |
| Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ | | Memory Usage | ⭐ | ⭐⭐⭐ | ⭐⭐⭐ |
| Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Slight protections | - Dynamically loaded websites <br/>- Small automation <br/>- Complicated protections | | Best used for | Basic scraping when HTTP requests alone can do it | - Dynamically loaded websites <br/>- Small automation<br/>- Small-Mid protections | - Dynamically loaded websites <br/>- Small automation <br/>- Small-Complicated protections |
| Browser(s) | ❌ | Chromium and Google Chrome | Modified Firefox | | Browser(s) | ❌ | Chromium and Google Chrome | Chromium and Google Chrome |
| Browser API used | ❌ | PlayWright | PlayWright | | Browser API used | ❌ | PlayWright | PlayWright |
| Setup Complexity | Simple | Simple | Simple | | Setup Complexity | Simple | Simple | Simple |
In the following pages, we will talk about each one in detail. In the following pages, we will talk about each one in detail.
+56 -94
View File
@@ -1,6 +1,6 @@
# Introduction # Introduction
Here, we will discuss the `DynamicFetcher` class (previously known as `PlayWrightFetcher`). This class provides flexible browser automation with multiple configuration options and some stealth capabilities. Here, we will discuss the `DynamicFetcher` class (formerly `PlayWrightFetcher`). This class provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page). As we will explain later, to automate the page, you need some knowledge of [Playwright's Page API](https://playwright.dev/python/docs/api/class-page).
@@ -23,7 +23,7 @@ Now, we will review most of the arguments one by one, using examples. If you wan
> Note: The async version of the `fetch` method is the `async_fetch` method, of course. > Note: The async version of the `fetch` method is the `async_fetch` method, of course.
This fetcher currently provides four main run options that can be combined as desired. This fetcher currently provides three main run options that can be combined as desired.
Which are: Which are:
@@ -31,77 +31,70 @@ Which are:
```python ```python
DynamicFetcher.fetch('https://example.com') DynamicFetcher.fetch('https://example.com')
``` ```
Using it in that manner will open a Chromium browser and load the page. There are no tricks or extra features unless you enable some; it's just a plain PlayWright API. Using it in that manner will open a Chromium browser and load the page. There are optimizations for speed, and some stealth goes automatically under the hood, but other than that, there are no tricks or extra features unless you enable some; it's just a plain PlayWright API.
### 2. Stealth Mode ### 2. Real Chrome
```python
DynamicFetcher.fetch('https://example.com', stealth=True)
```
It's the same as the vanilla Playwright option, but it provides a simple stealth mode suitable for websites with a small to medium protection layer(s).
Some of the things this fetcher's stealth mode does include:
* Patching the CDP runtime fingerprint by using PatchRight.
* Mimics some of the real browsers' properties by injecting several JS files and using custom options.
* Custom flags are used on launch to hide Playwright even more and make it faster.
* Generates real browser headers of the same type and user OS, then appends them to the request's headers.
### 3. Real Chrome
```python ```python
DynamicFetcher.fetch('https://example.com', real_chrome=True) DynamicFetcher.fetch('https://example.com', real_chrome=True)
``` ```
If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium. If you have a Google Chrome browser installed, use this option. It's the same as the first option, but it will use the Google Chrome browser you installed on your device instead of Chromium. This will make your requests look more authentic, so they're less detectable for better results.
This will make your requests look more authentic, so it's less detectable, and you can even use the `stealth=True` mode with it for better results, like below:
```python
DynamicFetcher.fetch('https://example.com', real_chrome=True, stealth=True)
```
If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually: If you don't have Google Chrome installed and want to use this option, you can use the command below in the terminal to install it for the library instead of installing it manually:
```commandline ```commandline
playwright install chrome playwright install chrome
``` ```
### 4. CDP Connection ### 3. CDP Connection
```python ```python
DynamicFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222') DynamicFetcher.fetch('https://example.com', cdp_url='ws://localhost:9222')
``` ```
Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). Instead of launching a browser locally (Chromium/Google Chrome), you can connect to a remote browser through the [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/).
> Notes:
>
> * There was a `stealth` option here, but it was moved to the `StealthyFetcher` class, as explained on the next page, with additional features since version 0.3.13.<br/>
> * This makes it less confusing for new users, easier to maintain, and provides other benefits, as explained on the [StealthyFetcher page](../fetching/stealthy.md).
## Full list of arguments ## Full list of arguments
Scrapling provides many options with this fetcher and its session classes. To make it as simple as possible, we will list the options here and give examples of how to use most of them. Scrapling provides many options with this fetcher and its session classes. To make it as simple as possible, we will list the options here and give examples of how to use most of them.
| Argument | Description | Optional | | Argument | Description | Optional |
|:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| |:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
| url | Target url | ❌ | | url | Target url | ❌ |
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ | | headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
| disable_resources | Drop requests for unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ | | disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
| cookies | Set cookies for the next request. | ✔️ | | cookies | Set cookies for the next request. | ✔️ |
| useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser.** | ✔️ | | useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ | | network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ | | load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
| timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ | | timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ | | wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ | | page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ |
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ | | wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ | | init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ | | wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ | | google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ | | extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ | | proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ | | real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
| disable_webgl | Disables WebGL and WebGL 2.0 support entirely. | ✔️ | | locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
| stealth | Enables stealth mode; you should always check the documentation to see what the stealth mode does currently. | ✔️ | | timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
| real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ | | cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
| locale | Set the locale for the browser if wanted. The default value is `en-US`. | ✔️ | | user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| timezone_id | Set the timezone for the browser if wanted. | ✔️ | | extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
| cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ | | additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ | | selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
| extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
| additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, and `selector_config`. In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, and `selector_config`.
> 🔍 Notes:
>
> 1. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
> 2. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
> 3. Since version 0.3.13, the `stealth` option has been removed here in favor of the `StealthyFetcher` class, and the `hide_canvas` option has been moved to it. The `disable_webgl` argument has been moved to the `StealthyFetcher` class and renamed as `allow_webgl`.
> 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
## Examples ## Examples
It's easier to understand with examples, so let's take a look. It's easier to understand with examples, so let's take a look.
@@ -110,10 +103,7 @@ It's easier to understand with examples, so let's take a look.
```python ```python
# Disable unnecessary resources # Disable unnecessary resources
page = DynamicFetcher.fetch( page = DynamicFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
'https://example.com',
disable_resources=True # Blocks fonts, images, media, etc...
)
``` ```
### Network Control ### Network Control
@@ -125,11 +115,8 @@ page = DynamicFetcher.fetch('https://example.com', network_idle=True)
# Custom timeout (in milliseconds) # Custom timeout (in milliseconds)
page = DynamicFetcher.fetch('https://example.com', timeout=30000) # 30 seconds page = DynamicFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
# Proxy support # Proxy support (It can also be a dictionary with only the keys 'server', 'username', and 'password'.)
page = DynamicFetcher.fetch( page = DynamicFetcher.fetch('https://example.com', proxy='http://username:password@host:port')
'https://example.com',
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
)
``` ```
### Downloading Files ### Downloading Files
@@ -141,7 +128,7 @@ with open(file='poster.png', mode='wb') as f:
f.write(page.body) f.write(page.body)
``` ```
The `body` attribute of the `Response` object is a `bytes` object containing the response body in case of Non-HTML responses. The `body` attribute of the `Response` object is a `bytes` object containing the response body in case of non-HTML responses.
### Browser Automation ### Browser Automation
This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues. This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues.
@@ -157,10 +144,7 @@ def scroll_page(page: Page):
page.mouse.move(100, 400) page.mouse.move(100, 400)
page.mouse.up() page.mouse.up()
page = DynamicFetcher.fetch( page = DynamicFetcher.fetch('https://example.com', page_action=scroll_page)
'https://example.com',
page_action=scroll_page
)
``` ```
Of course, if you use the async fetch version, the function must also be async. Of course, if you use the async fetch version, the function must also be async.
```python ```python
@@ -171,10 +155,7 @@ async def scroll_page(page: Page):
await page.mouse.move(100, 400) await page.mouse.move(100, 400)
await page.mouse.up() await page.mouse.up()
page = await DynamicFetcher.async_fetch( page = await DynamicFetcher.async_fetch('https://example.com', page_action=scroll_page)
'https://example.com',
page_action=scroll_page
)
``` ```
### Wait Conditions ### Wait Conditions
@@ -201,30 +182,13 @@ The states the fetcher can wait for can be any of the following ([source](https:
### Some Stealth Features ### Some Stealth Features
```python ```python
# Full stealth mode
page = DynamicFetcher.fetch( page = DynamicFetcher.fetch(
'https://example.com', 'https://example.com',
stealth=True, google_search=True,
hide_canvas=True, useragent='Mozilla/5.0...', # Custom user agent
disable_webgl=True, locale='en-US', # Set browser locale
google_search=True
)
# Custom user agent
page = DynamicFetcher.fetch(
'https://example.com',
useragent='Mozilla/5.0...'
)
# Set browser locale
page = DynamicFetcher.fetch(
'https://example.com',
locale='en-US'
) )
``` ```
Hence, the `hide_canvas` argument doesn't disable the canvas; instead, it hides it by adding random noise to canvas operations, preventing fingerprinting. Also, if you didn't set a user agent (preferred), the fetcher will generate a real User Agent of the same browser and use it.
The `google_search` argument is enabled by default, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
### General example ### General example
```python ```python
@@ -259,7 +223,6 @@ from scrapling.fetchers import DynamicSession
# Create a session with default configuration # Create a session with default configuration
with DynamicSession( with DynamicSession(
headless=True, headless=True,
stealth=True,
disable_resources=True, disable_resources=True,
real_chrome=True real_chrome=True
) as session: ) as session:
@@ -279,7 +242,6 @@ from scrapling.fetchers import AsyncDynamicSession
async def scrape_multiple_sites(): async def scrape_multiple_sites():
async with AsyncDynamicSession( async with AsyncDynamicSession(
stealth=True,
network_idle=True, network_idle=True,
timeout=30000, timeout=30000,
max_pages=3 max_pages=3
@@ -298,7 +260,7 @@ You may have noticed the `max_pages` argument. This is a new argument that enabl
1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal. 1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive. 2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
This logic allows for multiple websites to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :) This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one. In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
@@ -317,6 +279,6 @@ Use DynamicFetcher when:
- Want multiple browser options - Want multiple browser options
- Using a real Chrome browser - Using a real Chrome browser
- Need custom browser config - Need custom browser config
- Want flexible stealth options - Want a few stealth options
If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md). If you want more stealth and control without much config, check out the [StealthyFetcher](stealthy.md).
+6 -5
View File
@@ -25,7 +25,7 @@ All methods for making requests here share some arguments, so let's discuss them
- **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**. - **timeout**: The number of seconds to wait for each request to be finished. **Defaults to 30 seconds**.
- **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**. - **retries**: The number of retries that the fetcher will do for failed requests. **Defaults to three retries**.
- **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**. - **retry_delay**: Number of seconds to wait between retry attempts. **Defaults to 1 second**.
- **impersonate**: Impersonate specific browsers' TLS fingerprints. Accepts browser strings like `"chrome110"`, `"firefox102"`, `"safari15_5"` to use specific versions or `"chrome"`, `"firefox"`, `"safari"`, `"edge"` to automatically use the latest version available. This makes your requests appear as if they're coming from real browsers at the TLS level. **Defaults to the latest available Chrome version.** - **impersonate**: Impersonate specific browsers' TLS fingerprints. Accepts browser strings or a list of them like `"chrome110"`, `"firefox102"`, `"safari15_5"` to use specific versions or `"chrome"`, `"firefox"`, `"safari"`, `"edge"` to automatically use the latest version available. This makes your requests appear to come from real browsers at the TLS level. If you pass it a list of strings, it will choose a random one with each request. **Defaults to the latest available Chrome version.**
- **http3**: Use HTTP/3 protocol for requests. **Defaults to False**. It might be problematic if used with `impersonate`. - **http3**: Use HTTP/3 protocol for requests. **Defaults to False**. It might be problematic if used with `impersonate`.
- **cookies**: Cookies to use in the request. Can be a dictionary of `name→value` or a list of dictionaries. - **cookies**: Cookies to use in the request. Can be a dictionary of `name→value` or a list of dictionaries.
- **proxy**: As the name implies, the proxy for this request is used to route all traffic (HTTP and HTTPS). The format accepted here is `http://username:password@localhost:8030`. - **proxy**: As the name implies, the proxy for this request is used to route all traffic (HTTP and HTTPS). The format accepted here is `http://username:password@localhost:8030`.
@@ -39,14 +39,15 @@ All methods for making requests here share some arguments, so let's discuss them
> Note: <br/> > Note: <br/>
> 1. The currently available browsers to impersonate are (`"edge"`, `"chrome"`, `"chrome_android"`, `"safari"`, `"safari_beta"`, `"safari_ios"`, `"safari_ios_beta"`, `"firefox"`, `"tor"`)<br/> > 1. The currently available browsers to impersonate are (`"edge"`, `"chrome"`, `"chrome_android"`, `"safari"`, `"safari_beta"`, `"safari_ios"`, `"safari_ios_beta"`, `"firefox"`, `"tor"`)<br/>
> 2. The available browsers to impersonate and their corresponding versions are automatically displayed in the argument autocompletion and updated automatically with each `curl_cffi` update. > 2. The available browsers to impersonate, along with their corresponding versions, are automatically displayed in the argument autocompletion and updated with each `curl_cffi` update.<br/>
> 3. If any of the arguments `impersonate` or `stealthy_headers` are enabled, the fetchers will automatically generate real browser headers that match the browser version used.
Other than this, for further customization, you can pass any arguments that `curl_cffi` supports for any method if that method doesn't already support it. Other than this, for further customization, you can pass any arguments that `curl_cffi` supports for any method if that method doesn't already support it.
### HTTP Methods ### HTTP Methods
There are additional arguments for each method, depending on the method, such as `params` for GET requests and `data`/`json` for POST/PUT/DELETE requests. There are additional arguments for each method, depending on the method, such as `params` for GET requests and `data`/`json` for POST/PUT/DELETE requests.
Examples are the best way to explain this, as follows. Examples are the best way to explain this:
> Hence: `OPTIONS` and `HEAD` methods are not supported. > Hence: `OPTIONS` and `HEAD` methods are not supported.
#### GET #### GET
@@ -166,7 +167,7 @@ And for asynchronous requests, it's a small adjustment
## Session Management ## Session Management
For making multiple requests with the same configuration, use the `FetcherSession` class. It can be used in both synchronous and asynchronous code without issue; the class detects and changes the session type automatically without requiring a different import. For making multiple requests with the same configuration, use the `FetcherSession` class. It can be used in both synchronous and asynchronous code without issue; the class automatically detects and changes the session type, without requiring a different import.
The `FetcherSession` class can accept nearly all the arguments that the methods can take, which enables you to specify a config for the entire session and later choose a different config for one of the requests effortlessly, as you will see in the following examples. The `FetcherSession` class can accept nearly all the arguments that the methods can take, which enables you to specify a config for the entire session and later choose a different config for one of the requests effortlessly, as you will see in the following examples.
@@ -181,7 +182,7 @@ with FetcherSession(
timeout=30, timeout=30,
retries=3 retries=3
) as session: ) as session:
# Make multiple requests with the same settings # Make multiple requests with the same settings and the same cookies
page1 = session.get('https://scrapling.requestcatcher.com/get') page1 = session.get('https://scrapling.requestcatcher.com/get')
page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}) page2 = session.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'})
page3 = session.get('https://api.github.com/events') page3 = session.get('https://api.github.com/events')
+160 -144
View File
@@ -1,14 +1,17 @@
# Introduction # Introduction
Here, we will discuss the `StealthyFetcher` class. This class is similar to [DynamicFetcher](dynamic.md#introduction) in many ways, including browser automation and the use of [Playwright's API](https://playwright.dev/python/docs/intro). The main difference is that this class provides advanced anti-bot protection bypass capabilities and a custom version of a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), from which most stealth comes. Here, we will discuss the `StealthyFetcher` class. This class is very similar to the [DynamicFetcher](dynamic.md#introduction) class, including the browsers, the automation, and the use of [Playwright's API](https://playwright.dev/python/docs/intro). The main difference is that this class provides advanced anti-bot protection bypass capabilities; most of them are handled automatically under the hood, and the rest is up to you to enable.
As with [DynamicFetcher](dynamic.md#introduction), you will need some knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) to automate the page, as we will explain later. As with [DynamicFetcher](dynamic.md#introduction), you will need some knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) to automate the page, as we will explain later.
**Note:** _This fetcher was using a custom version of [Camoufox](https://github.com/daijro/camoufox) as an engine before version 0.3.13, which was replaced now with [patchright](https://github.com/Kaliiiiiiiiii-Vinyzu/patchright) for many reasons. See [this section](#using-camoufox-as-an-engine) for information if you still need to use [Camoufox](https://github.com/daijro/camoufox). We might switch back to [Camoufox](https://github.com/daijro/camoufox) in the future if its development continues._
> 💡 **Prerequisites:** > 💡 **Prerequisites:**
> >
> 1. Youve completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use. > 1. You've completed or read the [DynamicFetcher](dynamic.md#introduction) page since this class builds upon it, and we won't repeat the same information here for that reason.
> 2. Youve completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object. > 2. Youve completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
> 3. Youve completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class. > 3. Youve completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
> 4. Youve completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
## Basic Usage ## Basic Usage
You have one primary way to import this Fetcher, which is the same for all fetchers. You have one primary way to import this Fetcher, which is the same for all fetchers.
@@ -20,81 +23,80 @@ Check out how to configure the parsing options [here](choosing.md#parser-configu
> Note: The async version of the `fetch` method is the `async_fetch` method, of course. > Note: The async version of the `fetch` method is the `async_fetch` method, of course.
## What does it do?
The `StealthyFetcher` class is a stealthy version of the [DynamicFetcher](dynamic.md#introduction) class, and here are some of the things it does:
1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
6. It makes requests look as if they came from Google's search page of the requested website.
7. and other anti-protection options...
## Full list of arguments ## Full list of arguments
Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments Scrapling provides many options with this fetcher and its session classes. Before jumping to the [examples](#examples), here's the full list of arguments
| Argument | Description | Optional | | Argument | Description | Optional |
|:-------------------:|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:| |:-------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------:|
| url | Target url | ❌ | | url | Target url | ❌ |
| headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ | | headless | Pass `True` to run the browser in headless/hidden (**default**) or `False` for headful/visible mode. | ✔️ |
| block_images | Prevent the loading of images through Firefox preferences. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ | | disable_resources | Drop requests for unnecessary resources for a speed boost. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. | ✔️ |
| disable_resources | Drop requests for unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.<br/>Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. _This can help save your proxy usage, but be cautious with this option, as it may cause some websites to never finish loading._ | ✔️ | | cookies | Set cookies for the next request. | ✔️ |
| cookies | Set cookies for the next request. | ✔️ | | useragent | Pass a useragent string to be used. **Otherwise, the fetcher will generate and use a real Useragent of the same browser and version.** | ✔️ |
| google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ | | network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ |
| extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ | | load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ |
| block_webrtc | Blocks WebRTC entirely. | ✔️ | | timeout | The timeout (milliseconds) used in all operations and waits through the page. The default is 30,000 ms (30 seconds). | ✔️ |
| page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ | | wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ |
| addons | List of Firefox addons to use. **Must be paths to extracted addons.** | ✔️ | | page_action | Added for automation. Pass a function that takes the `page` object and does the necessary automation. | ✔️ |
| humanize | Humanize the cursor movement. The cursor movement takes either True or the maximum duration in seconds. The cursor typically takes up to 1.5 seconds to move across the window. | ✔️ | | wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ |
| allow_webgl | Enabled by default. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ | | init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ |
| geoip | Recommended to use with proxies; Automatically use IPs' longitude, latitude, timezone, country, locale, & spoof the WebRTC IP address. It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. | ✔️ | | wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ |
| os_randomize | If enabled, Scrapling will randomize the OS fingerprints used. The default is matching the fingerprints with the current OS. | ✔️ | | google_search | Enabled by default, Scrapling will set the referer header as if this request came from a Google search of this website's domain name. | ✔️ |
| disable_ads | Disabled by default; this installs the `uBlock Origin` addon on the browser if enabled. | ✔️ | | extra_headers | A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ | ✔️ |
| solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ | | proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ |
| network_idle | Wait for the page until there are no network connections for at least 500 ms. | ✔️ | | real_chrome | If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch and use an instance of your browser. | ✔️ |
| load_dom | Enabled by default, wait for all JavaScript on page(s) to fully load and execute (wait for the `domcontentloaded` state). | ✔️ | | locale | Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect `navigator.language` value, `Accept-Language` request header value, as well as number and date formatting rules. Defaults to the system default locale. | ✔️ |
| timeout | The timeout used in all operations and waits through the page. It's in milliseconds, and the default is 30000. | ✔️ | | timezone_id | Changes the timezone of the browser. Defaults to the system timezone. | ✔️ |
| wait | The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the `Response` object. | ✔️ | | cdp_url | Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. | ✔️ |
| wait_selector | Wait for a specific css selector to be in a specific state. | ✔️ | | user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ |
| init_script | An absolute path to a JavaScript file to be executed on page creation for all pages in this session. | ✔️ | | extra_flags | A list of additional browser flags to pass to the browser on launch. | ✔️ |
| wait_selector_state | Scrapling will wait for the given state to be fulfilled for the selector given with `wait_selector`. _Default state is `attached`._ | ✔️ | | solve_cloudflare | When enabled, fetcher solves all types of Cloudflare's Turnstile/Interstitial challenges before returning the response to you. | ✔️ |
| proxy | The proxy to be used with requests. It can be a string or a dictionary with only the keys 'server', 'username', and 'password'. | ✔️ | | block_webrtc | Forces WebRTC to respect proxy settings to prevent local IP address leak. | ✔️ |
| user_data_dir | Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. **Only Works with sessions** | ✔️ | | hide_canvas | Add random noise to canvas operations to prevent fingerprinting. | ✔️ |
| additional_args | Additional arguments to be passed to Camoufox as additional settings, and they take higher priority than Scrapling's settings. | ✔️ | | allow_webgl | Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended, as many WAFs now check if WebGL is enabled. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ | | additional_args | Additional arguments to be passed to Playwright's context as additional settings, and they take higher priority than Scrapling's settings. | ✔️ |
| selector_config | A dictionary of custom parsing arguments to be used when creating the final `Selector`/`Response` class. | ✔️ |
In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, and `selector_config`. In session classes, all these arguments can be set globally for the session. Still, you can configure each request individually by passing some of the arguments here that can be configured on the browser tab level like: `google_search`, `timeout`, `wait`, `page_action`, `extra_headers`, `disable_resources`, `wait_selector`, `wait_selector_state`, `network_idle`, `load_dom`, `solve_cloudflare`, and `selector_config`.
> 🔍 Notes:
>
> 1. It's basically the same arguments as [DynamicFetcher](dynamic.md#introduction) class but with these additional arguments `solve_cloudflare`, `block_webrtc`, `hide_canvas`, and `allow_webgl`.
> 2. The `disable_resources` option made requests ~25% faster in my tests for some websites and can help save your proxy usage, but be careful with it, as it can cause some websites to never finish loading.
> 3. The `google_search` argument is enabled by default for all requests, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
> 4. If you didn't set a user agent and enabled headless mode, the fetcher will generate a real user agent for the same browser version and use it. If you didn't set a user agent and didn't enable headless mode, the fetcher will use the browser's default user agent, which is the same as in standard browsers in the latest versions.
## Examples ## Examples
It's easier to understand with examples, so we will now review most of the arguments individually. It's easier to understand with examples, so we will now review most of the arguments individually. Since it's the same class as the [DynamicFetcher](dynamic.md#introduction), you can refer to that page for more examples, as we won't repeat all the examples from there.
### Browser Modes ### Cloudflare and stealth options
```python
# Headless/hidden mode (default)
page = StealthyFetcher.fetch('https://example.com', headless=True)
# Visible browser mode
page = StealthyFetcher.fetch('https://example.com', headless=False)
```
### Resource Control
```python
# Block images
page = StealthyFetcher.fetch('https://example.com', block_images=True)
# Disable unnecessary resources
page = StealthyFetcher.fetch('https://example.com', disable_resources=True) # Blocks fonts, images, media, etc.
```
### Cloudflare Protection Bypass
```python ```python
# Automatic Cloudflare solver # Automatic Cloudflare solver
page = StealthyFetcher.fetch( page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True)
'https://nopecha.com/demo/cloudflare',
solve_cloudflare=True # Automatically solve Cloudflare challenges
)
# Works with other stealth options # Works with other stealth options
page = StealthyFetcher.fetch( page = StealthyFetcher.fetch(
'https://protected-site.com', 'https://protected-site.com',
solve_cloudflare=True, solve_cloudflare=True,
humanize=True, block_webrtc=True,
geoip=True, real_chrome=True,
os_randomize=True hide_canvas=True,
google_search=True,
proxy='http://username:password@host:port', # It can also be a dictionary with only the keys 'server', 'username', and 'password'.
) )
``` ```
@@ -104,64 +106,13 @@ The `solve_cloudflare` parameter enables automatic detection and solving all typ
- Interactive challenges (clicking verification boxes) - Interactive challenges (clicking verification boxes)
- Invisible challenges (automatic background verification) - Invisible challenges (automatic background verification)
And even solves the custom pages. And even solves the custom pages with embedded captcha.
**Important notes:** > 🔍 **Important notes:**
>
- Sometimes, with websites that use custom implementations, you will need to use `wait_selector` to make sure Scrapling waits for the real website content to be loaded after solving the captcha. Some websites can be the real definition of an edge case while we are trying to make the solver as generic as possible. > 1. Sometimes, with websites that use custom implementations, you will need to use `wait_selector` to make sure Scrapling waits for the real website content to be loaded after solving the captcha. Some websites can be the real definition of an edge case while we are trying to make the solver as generic as possible.
- When `solve_cloudflare=True` is enabled, `humanize=True` is automatically activated for more realistic behavior > 2. The timeout should be at least 60 seconds when using the Cloudflare solver for sufficient challenge-solving time.
- The timeout should be at least 60 seconds when using the Cloudflare solver for sufficient challenge-solving time > 3. This feature works seamlessly with proxies and other stealth options.
- This feature works seamlessly with proxies and other stealth options
### Additional stealth options
```python
page = StealthyFetcher.fetch(
'https://example.com',
block_webrtc=True, # Block WebRTC
allow_webgl=False, # Disable WebGL
humanize=True, # Make the mouse move as a human would move it
geoip=True, # Use IP's longitude, latitude, timezone, country, and locale, then spoof the WebRTC IP address...
os_randomize=True, # Randomize the OS fingerprints used. The default is matching the fingerprints with the current OS.
disable_ads=True, # Block ads with uBlock Origin addon (enabled by default)
google_search=True
)
# Custom humanization duration
page = StealthyFetcher.fetch(
'https://example.com',
humanize=1.5 # Max 1.5 seconds for cursor movement
)
```
The `google_search` argument is enabled by default, making the request appear to come from a Google search page. So, a request for `https://example.com` will set the referer to `https://www.google.com/search?q=example`. Also, if used together, it takes priority over the referer set by the `extra_headers` argument.
### Network Control
```python
# Wait for network idle (Consider fetch to be finished when there are no network connections for at least 500 ms)
page = StealthyFetcher.fetch('https://example.com', network_idle=True)
# Custom timeout (in milliseconds)
page = StealthyFetcher.fetch('https://example.com', timeout=30000) # 30 seconds
# Proxy support
page = StealthyFetcher.fetch(
'https://example.com',
proxy='http://username:password@host:port' # Or it can be a dictionary with the keys 'server', 'username', and 'password' only
)
```
### Downloading Files
```python
page = StealthyFetcher.fetch('https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/poster.png')
with open(file='poster.png', mode='wb') as f:
f.write(page.body)
```
The `body` attribute of the `Response` object is a `bytes` object containing the response body in case of Non-HTML responses.
### Browser Automation ### Browser Automation
This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues. This is where your knowledge about [Playwright's Page API](https://playwright.dev/python/docs/api/class-page) comes into play. The function you pass here takes the page object from Playwright's API, performs the desired action, and then the fetcher continues.
@@ -177,10 +128,7 @@ def scroll_page(page: Page):
page.mouse.move(100, 400) page.mouse.move(100, 400)
page.mouse.up() page.mouse.up()
page = StealthyFetcher.fetch( page = StealthyFetcher.fetch('https://example.com', page_action=scroll_page)
'https://example.com',
page_action=scroll_page
)
``` ```
Of course, if you use the async fetch version, the function must also be async. Of course, if you use the async fetch version, the function must also be async.
```python ```python
@@ -191,10 +139,7 @@ async def scroll_page(page: Page):
await page.mouse.move(100, 400) await page.mouse.move(100, 400)
await page.mouse.up() await page.mouse.up()
page = await StealthyFetcher.async_fetch( page = await StealthyFetcher.async_fetch('https://example.com', page_action=scroll_page)
'https://example.com',
page_action=scroll_page
)
``` ```
### Wait Conditions ### Wait Conditions
@@ -217,19 +162,9 @@ The states the fetcher can wait for can be any of the following ([source](https:
- `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible. - `visible`: wait for an element to have a non-empty bounding box and no `visibility:hidden`. Note that an element without any content or with `display:none` has an empty bounding box and is not considered visible.
- `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option. - `hidden`: wait for an element to be either detached from the DOM, or have an empty bounding box, or `visibility:hidden`. This is opposite to the `'visible'` option.
### Firefox Addons
```python
# Custom Firefox addons
page = StealthyFetcher.fetch(
'https://example.com',
addons=['/path/to/addon1', '/path/to/addon2']
)
```
The paths here must point to extracted addons that will be installed automatically upon browser launch.
### Real-world example (Amazon) ### Real-world example (Amazon)
This is for educational purposes only; this example was generated by AI, which shows how easy it is to work with Scrapling through AI This is for educational purposes only; this example was generated by AI, which also shows how easy it is to work with Scrapling through AI
```python ```python
def scrape_amazon_product(url): def scrape_amazon_product(url):
# Use StealthyFetcher to bypass protection # Use StealthyFetcher to bypass protection
@@ -261,8 +196,8 @@ from scrapling.fetchers import StealthySession
# Create a session with default configuration # Create a session with default configuration
with StealthySession( with StealthySession(
headless=True, headless=True,
geoip=True, real_chrome=True,
humanize=True, block_webrtc=True,
solve_cloudflare=True solve_cloudflare=True
) as session: ) as session:
# Make multiple requests with the same browser instance # Make multiple requests with the same browser instance
@@ -281,8 +216,8 @@ from scrapling.fetchers import AsyncStealthySession
async def scrape_multiple_sites(): async def scrape_multiple_sites():
async with AsyncStealthySession( async with AsyncStealthySession(
geoip=True, real_chrome=True,
os_randomize=True, block_webrtc=True,
solve_cloudflare=True, solve_cloudflare=True,
timeout=60000, # 60 seconds for Cloudflare challenges timeout=60000, # 60 seconds for Cloudflare challenges
max_pages=3 max_pages=3
@@ -296,12 +231,12 @@ async def scrape_multiple_sites():
return pages return pages
``` ```
You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then: You may have noticed the `max_pages` argument. This is a new argument that enables the fetcher to create a **rotating pool of Browser tabs**. Instead of using a single tab for all your requests, you set a limit on the maximum number of pages that can be displayed at once. With each request, the library will close all tabs that have finished their task and check if the number of the current tabs is lower than the maximum allowed number of pages/tabs, then:
1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal. 1. If you are within the allowed range, the fetcher will create a new tab for you, and then all is as normal.
2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive. 2. Otherwise, it will keep checking every subsecond if creating a new tab is allowed or not for 60 seconds, then raise `TimeoutError`. This can happen when the website you are fetching becomes unresponsive.
This logic allows for multiple websites to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :) This logic allows for multiple URLs to be fetched at the same time in the same browser, which saves a lot of resources, but most importantly, is so fast :)
In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one. In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resources/time. That logic proved flawed, as it's nearly impossible to protect pages/tabs from contamination by the previous configuration used in the request before this one.
@@ -312,6 +247,87 @@ In versions 0.3 and 0.3.1, the pool was reusing finished tabs to save more resou
- **Consistent fingerprint**: Same browser fingerprint across all requests. - **Consistent fingerprint**: Same browser fingerprint across all requests.
- **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch. - **Memory efficiency**: Better resource usage compared to launching new browsers with each fetch.
## Using Camoufox as an engine
If you see that Camoufox is stable on your device, has no high memory issues, and want to continue using Camoufox as before v0.3.13. This section is for you.
First, you will need to install the Camoufox library, browser, and Firefox system dependencies if you didn't already:
```commandline
pip install camoufox
playwright install-deps firefox
camoufox fetch
```
Then you will inherit from `StealthySession` and set it as below:
```python
from scrapling.fetchers import StealthySession
from playwright.sync_api import sync_playwright
from camoufox.utils import launch_options as generate_launch_options
class StealthySession(StealthySession):
def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
self.playwright = sync_playwright().start()
# Configure camoufox run options here
launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})
# Here's an example, part of what we have been doing before v0.3.13
launch_options = generate_launch_options(**{
"geoip": False,
"proxy": self._config.proxy,
"headless": self._config.headless,
"humanize": True if self._config.solve_cloudflare else False, # Better enable humanize for Cloudflare, otherwise it's up to you
"i_know_what_im_doing": True, # To turn warnings off with the user configurations
"allow_webgl": self._config.allow_webgl,
"block_webrtc": self._config.block_webrtc,
"os": None,
"user_data_dir": self._config.user_data_dir,
"firefox_user_prefs": {
# This is what enabling `enable_cache` does internally, so we do it from here instead
"browser.sessionhistory.max_entries": 10,
"browser.sessionhistory.max_total_viewers": -1,
"browser.cache.memory.enable": True,
"browser.cache.disk_cache_ssl": True,
"browser.cache.disk.smart_size.enabled": True,
},
# etc...
})
self.context = self.playwright.firefox.launch_persistent_context(**launch_options)
else:
raise RuntimeError("Session has been already started")
```
After that, you can use it normally as before, even for solving Cloudflare challenges:
```python
with StealthySession(solve_cloudflare=True, headless=True) as session:
page = session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')
if page.css('#page-not-found-404'):
print('Cloudflare challenge solved successfully!')
```
The same logic applies to the `AsyncStealthySession` class with a few differences:
```python
from scrapling.fetchers import AsyncStealthySession
from playwright.async_api import async_playwright
from camoufox.utils import launch_options as generate_launch_options
class AsyncStealthySession(AsyncStealthySession):
async def start(self):
"""Create a browser for this instance and context."""
if not self.playwright:
self.playwright = await async_playwright().start()
# Configure camoufox run options here
launch_options = generate_launch_options(**{"headless": True, "user_data_dir": ''})
# or set the launch options as in the above example
self.context = await self.playwright.firefox.launch_persistent_context(**launch_options)
else:
raise RuntimeError("Session has been already started")
async with AsyncStealthySession(solve_cloudflare=True, headless=True) as session:
page = await session.fetch('https://sergiodemo.com/security/challenge/legacy-challenge')
if page.css('#page-not-found-404'):
print('Cloudflare challenge solved successfully!')
```
Enjoy! :)
## When to Use ## When to Use
Use StealthyFetcher when: Use StealthyFetcher when:
+8 -8
View File
@@ -18,7 +18,7 @@
Scrapling isn't just another Web Scraping library. It's the first **adaptive** scraping library that learns from website changes and evolves with them. While other libraries break when websites update their structure, Scrapling automatically relocates your elements and keeps your scrapers running. Scrapling isn't just another Web Scraping library. It's the first **adaptive** scraping library that learns from website changes and evolves with them. While other libraries break when websites update their structure, Scrapling automatically relocates your elements and keeps your scrapers running.
Built for the modern Web, Scrapling features its own rapid parsing engine and fetchers to handle all Web Scraping challenges you face or will face. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone. Built for the modern Web, Scrapling features **its own rapid parsing engine** and fetchers to handle all Web Scraping challenges you face or will face. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
```python ```python
>> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher >> from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
@@ -51,8 +51,8 @@ Built for the modern Web, Scrapling features its own rapid parsing engine and fe
### Advanced Websites Fetching with Session Support ### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3. - **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, real Chrome, and custom stealth mode. - **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium, and Google's Chrome.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` using a modified version of Firefox and fingerprint spoofing. Can bypass all types of Cloudflare's Turnstile/Interstitial with automation easily. - **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can bypass all types of Cloudflare's Turnstile/Interstitial with automation easily.
- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests. - **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes. - **Async Support**: Complete async support across all fetchers and dedicated async session classes.
@@ -60,7 +60,7 @@ Built for the modern Web, Scrapling features its own rapid parsing engine and fe
- 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms. - 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
- 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more. - 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements. - 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features custom, powerful capabilities that utilize Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. - 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### High-Performance & battle-tested Architecture ### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries. - 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
@@ -100,14 +100,14 @@ Starting with v0.3.2, this installation only includes the parser engine and its
### Optional Dependencies ### Optional Dependencies
1. If you are going to use any of the extra features below, the fetchers, or their classes, then you need to install fetchers' dependencies, and then install their browser dependencies with 1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:
```bash ```bash
pip install "scrapling[fetchers]" pip install "scrapling[fetchers]"
scrapling install scrapling install
``` ```
This downloads all browsers with their system dependencies and fingerprint manipulation dependencies. This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
2. Extra features: 2. Extra features:
@@ -135,10 +135,10 @@ Or download it from the GitHub registry:
```bash ```bash
docker pull ghcr.io/d4vinci/scrapling:latest docker pull ghcr.io/d4vinci/scrapling:latest
``` ```
This image is automatically built and pushed through GitHub actions on the repository's main branch. This image is automatically built and pushed using GitHub Actions and the repository's main branch.
## How the documentation is organized ## How the documentation is organized
Scrapling has a lot of documentation, so we try to follow a guideline called the [Diátaxis documentation framework](https://diataxis.fr/). Scrapling has extensive documentation, so we try to follow the [Diátaxis documentation framework](https://diataxis.fr/).
## Support ## Support
+22 -12
View File
@@ -1,4 +1,4 @@
We will start by quickly reviewing the parsing capabilities. Then, we will fetch websites with custom browsers, make requests, and parse the response. We will start by quickly reviewing the parsing capabilities. Then we will fetch websites using custom browsers, make requests, and parse the responses.
Here's an HTML document generated by ChatGPT that we will be using as an example throughout this page: Here's an HTML document generated by ChatGPT that we will be using as an example throughout this page:
```html ```html
@@ -82,7 +82,7 @@ page.get_all_text(ignore_tags=('script', 'style'))
``` ```
## Finding elements ## Finding elements
If there's an element you want to find on the page, you will! Your creativity level is the only limitation! If there's an element you want to find on the page, you will find it! Your creativity level is the only limitation!
Finding the first HTML `section` element Finding the first HTML `section` element
```python ```python
@@ -94,14 +94,14 @@ Find all `section` elements
section_elements = page.find_all('section') section_elements = page.find_all('section')
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>] # [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
``` ```
Find all `section` elements whose `id` attribute value is `products` Find all `section` elements whose `id` attribute value is `products`.
```python ```python
section_elements = page.find_all('section', {'id':"products"}) section_elements = page.find_all('section', {'id':"products"})
# Same as # Same as
section_elements = page.find_all('section', id="products") section_elements = page.find_all('section', id="products")
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>] # [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
``` ```
Find all `section` elements whose `id` attribute value contains `product` Find all `section` elements whose `id` attribute value contains `product`.
```python ```python
section_elements = page.find_all('section', {'id*':"product"}) section_elements = page.find_all('section', {'id*':"product"})
``` ```
@@ -218,7 +218,7 @@ Using the elements we found above
<data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>] <data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
>>> section_element.siblings >>> section_element.siblings
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>] [<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
>>> section_element.next # gets the next element, the same logic applies to `quote.previous` >>> section_element.next # gets the next element, the same logic applies to `quote.previous`.
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'> <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
>>> section_element.children.css('h2::text') >>> section_element.children.css('h2::text')
['Products'] ['Products']
@@ -237,7 +237,7 @@ You can search for a specific ancestor of an element that satisfies a function;
``` ```
## Fetching websites ## Fetching websites
Instead of passing the raw HTML to Scrapling, you can get a website's response directly through HTTP requests or by fetching it from browsers. Instead of passing the raw HTML to Scrapling, you can retrieve a website's response directly via HTTP requests or by fetching it in a browser.
A fetcher is made for every use case. A fetcher is made for every use case.
@@ -267,7 +267,7 @@ For Async requests, you will replace the import like below:
> Notes: > Notes:
> >
> 1. You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from a Google search of this domain. It's enabled by default. > 1. You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a referer header, as if this request came from a Google search of this domain. It's enabled by default.
> 2. The `impersonate` argument allows you to fake the TLS fingerprint for a specific version of a browser. > 2. The `impersonate` argument lets you fake the TLS fingerprint for a specific browser version.
> 3. There's also the `http3` argument, which, when enabled, makes the fetcher use HTTP/3 for requests, which makes your requests more authentic > 3. There's also the `http3` argument, which, when enabled, makes the fetcher use HTTP/3 for requests, which makes your requests more authentic
This is just the tip of the iceberg with this fetcher; check out the rest from [here](fetching/static.md) This is just the tip of the iceberg with this fetcher; check out the rest from [here](fetching/static.md)
@@ -275,7 +275,7 @@ This is just the tip of the iceberg with this fetcher; check out the rest from [
### Dynamic loading ### Dynamic loading
We have you covered if you deal with dynamic websites like most today! We have you covered if you deal with dynamic websites like most today!
The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides many options to fetch/load websites' pages through browsers. The `DynamicFetcher` class (formerly `PlayWrightFetcher`) offers many options for fetching and loading web pages using Chromium-based browsers.
```python ```python
>>> from scrapling.fetchers import DynamicFetcher >>> from scrapling.fetchers import DynamicFetcher
>>> page = DynamicFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option >>> page = DynamicFetcher.fetch('https://www.google.com/search?q=%22Scrapling%22', disable_resources=True) # Vanilla Playwright option
@@ -286,10 +286,9 @@ The `DynamicFetcher` class (previously known as `PlayWrightFetcher`) provides ma
>>> page.css_first("#search a::attr(href)") >>> page.css_first("#search a::attr(href)")
'https://github.com/D4Vinci/Scrapling' 'https://github.com/D4Vinci/Scrapling'
``` ```
It's built on top of [Playwright](https://playwright.dev/python/) and it's currently providing three main run options that can be mixed as you want: It's built on top of [Playwright](https://playwright.dev/python/), and it's currently providing two main run options that can be mixed as you want:
- Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser. - Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser.
- Stealthy Playwright with custom stealth mode explicitly written for it. It's not top-tier stealth mode, but it bypasses many online tests like [Sannysoft's](https://bot.sannysoft.com/). Check out the `StealthyFetcher` class below for more advanced stealth mode. It uses the Chromium browser.
- Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it. - Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
@@ -298,7 +297,18 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
### Dynamic anti-protection loading ### Dynamic anti-protection loading
We also have you covered if you deal with dynamic websites with annoying anti-protections! We also have you covered if you deal with dynamic websites with annoying anti-protections!
The `StealthyFetcher` class uses a custom version of a modified Firefox browser called [Camoufox](https://github.com/daijro/camoufox), bypassing most bot detections by default. Scrapling offers a faster custom version, includes extra tools, and features easy configurations to further increase undetectability. The `StealthyFetcher` class uses a stealthy version of the `DynamicFetcher` explained above.
Some of the things it does:
1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
2. It bypasses CDP runtime leaks and WebRTC leaks.
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
4. It generates canvas noise to prevent fingerprinting through canvas.
5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
6. It makes requests look as if they came from Google's search page of the requested website.
7. and other anti-protection options...
```python ```python
>>> from scrapling.fetchers import StealthyFetcher >>> from scrapling.fetchers import StealthyFetcher
>>> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default >>> page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default
@@ -318,4 +328,4 @@ Again, this is just the tip of the iceberg with this fetcher. Check out the rest
--- ---
That's Scrapling at a glance. If you want to learn more about it, continue to the next section. That's Scrapling at a glance. If you want to learn more, continue to the next section.
+18 -16
View File
@@ -50,7 +50,7 @@ When website owners implement structural changes like
``` ```
The selector will no longer function, and your code needs maintenance. That's where Scrapling's `adaptive` feature comes into play. The selector will no longer function, and your code needs maintenance. That's where Scrapling's `adaptive` feature comes into play.
With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element and without AI :) With Scrapling, you can enable the `adaptive` feature the first time you select an element, and the next time you select that element and it doesn't exist, Scrapling will remember its properties and search on the website for the element with the highest percentage of similarity to that element, and without AI :)
```python ```python
from scrapling import Selector, Fetcher from scrapling import Selector, Fetcher
@@ -65,7 +65,7 @@ if not element: # One day website changes?
element = page.css('#p1', adaptive=True) # Scrapling still finds it! element = page.css('#p1', adaptive=True) # Scrapling still finds it!
# the rest of your code... # the rest of your code...
``` ```
Below, I will show you one usage example for this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection. Below, I will show you an example of how to use this feature. Then, we will dive deep into how to use it and provide details about this feature. Note that it works with all selection methods, not just CSS/XPATH selection.
## Real-World Scenario ## Real-World Scenario
Let's use a real website as an example and use one of the fetchers to fetch its source. To achieve this, we need to identify a website that is about to update its design/structure, copy its source, and then wait for the website to change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha. Let's use a real website as an example and use one of the fetchers to fetch its source. To achieve this, we need to identify a website that is about to update its design/structure, copy its source, and then wait for the website to change. Of course, that's nearly impossible to know unless I know the website's owner, but that will make it a staged test, haha.
@@ -98,7 +98,9 @@ Note that I introduced a new argument called `adaptive_domain`. This is because,
The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `adaptive_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :) The code will be the same in a real-world scenario, except it will use the same URL for both requests, so you won't need to use the `adaptive_domain` argument. This is the closest example I can give to real-world cases, so I hope it didn't confuse you :)
Hence, in the two examples above, I used both the `Selector` class and the `Fetcher` class to show you that the logic for adaptive is the same. Hence, in the two examples above, I used both the `Selector` and `Fetcher` classes to show that the adaptive logic is the same.
> Note: the main reason for creating the `adaptive_domain` argument was to handle if the website changed its URL while changing the design/structure. In that case, you can use it to continue using the previously stored adaptive data for the new URL. Otherwise, scrapling will consider it a new website and discard the old data.
## How the adaptive scraping feature works ## How the adaptive scraping feature works
Adaptive scraping works in two phases: Adaptive scraping works in two phases:
@@ -106,19 +108,19 @@ Adaptive scraping works in two phases:
1. **Save Phase**: Store unique properties of elements 1. **Save Phase**: Store unique properties of elements
2. **Match Phase**: Find elements with similar properties later 2. **Match Phase**: Find elements with similar properties later
Let's say you've got an element through selection or any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes. Let's say you've selected an element through any method and want the library to find it the next time you scrape this website, even if it undergoes structural/design changes.
With as few technical details as possible, the general logic goes as follows: With as few technical details as possible, the general logic goes as follows:
1. You tell Scrapling to save that element's unique properties in one of the ways we will show below. 1. You tell Scrapling to save that element's unique properties in one of the ways we will show below.
2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties. 2. Scrapling uses its configured database (SQLite by default) and saves each element's unique properties.
3. Now, because everything about the element can be changed or removed from the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things: 3. Now, because everything about the element can be changed or removed by the website's owner(s), nothing from the element can be used as a unique identifier for the database. To solve this issue, I made the storage system rely on two things:
1. The domain of the current website. If you are using the `Selector` class, you should pass it while initializing the class, or if you are using one of the fetchers, the domain will be taken from the URL automatically. 1. The domain of the current website. If you are using the `Selector` class, pass it when initializing; if you are using a fetcher, the domain will be automatically taken from the URL.
2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself, as you will see later when we discuss this. 2. An `identifier` to query that element's properties from the database. You don't always have to set the identifier yourself; we'll discuss this later.
Together, they will be used to retrieve the element's unique properties from the database later. Together, they will later be used to retrieve the element's unique properties from the database.
4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated for their similarity with the desired element. In that comparison, everything is taken into consideration, as you will see later 4. Later, when the website's structure changes, you tell Scrapling to find the element by enabling `adaptive`. Scrapling retrieves the element's unique properties and matches all elements on the page against the unique properties we already have for this element. A score is calculated based on their similarity to the desired element. In that comparison, everything is taken into consideration, as you will see later
5. The element(s) with the highest similarity score to the wanted element are returned. 5. The element(s) with the highest similarity score to the wanted element are returned.
### The unique properties ### The unique properties
@@ -129,7 +131,7 @@ For Scrapling, the unique elements we are relying on are:
- Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only). - Element tag name, text, attributes (names and values), siblings (tag names only), and path (tag names only).
- Element's parent tag name, attributes (names and values), and text. - Element's parent tag name, attributes (names and values), and text.
But you need to understand that the comparison between elements is not exact; it's more about finding how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now. But you need to understand that the comparison between elements isn't exact; it's more about how similar these values are. So everything is considered, even the values' order, like the order in which the element class names were written before and the order in which the same element class names are written now.
## How to use adaptive feature ## How to use adaptive feature
The adaptive feature can be applied to any found element, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later. The adaptive feature can be applied to any found element, and it's added as arguments to CSS/XPath Selection methods, as you saw above, but we will get back to that later.
@@ -146,11 +148,11 @@ Examples:
``` ```
If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain. If you are using the [Selector](main_classes.md#selector) class, you need to pass the url of the website you are using with the argument `url` so Scrapling can separate the properties saved for each element by domain.
If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you used the same identifier later for a different website and didn't pass the URL parameter while initializing it. The save process will overwrite the previous data, and the `adaptive` feature only uses the latest saved properties. If you didn't pass a URL, the word `default` will be used in place of the URL field while saving the element's unique properties. So, this will only be an issue if you use the same identifier later for a different website and don't pass the URL parameter when initializing it. The save process overwrites previous data, and the `adaptive` feature uses only the latest saved properties.
Besides those arguments, we have `storage` and `storage_args`. Both are for the class to be used to connect to the database; by default, it's set to the SQLite class that the library is using. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md). Besides those arguments, we have `storage` and `storage_args`. Both are for the class to connect to the database; by default, it uses the SQLite class provided by the library. Those arguments shouldn't matter unless you want to write your own storage system, which we will cover on a [separate page in the development section](../development/adaptive_storage_system.md).
Now, after enabling the `adaptive` feature globally, you have two main ways to use it. Now that you've enabled the `adaptive` feature globally, you have two main ways to use it.
### The CSS/XPath Selection way ### The CSS/XPath Selection way
As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page, like below As you have seen in the example above, first, you have to use the `auto_save` argument while selecting an element that exists on the page, like below
@@ -163,7 +165,7 @@ element = page.css('#p1', adaptive=True)
``` ```
Pretty simple, eh? Pretty simple, eh?
Well, a lot happened under the hood here. Remember the identifier part we mentioned before that you need to set so you can retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :) Well, a lot happened under the hood here. Remember the identifier we mentioned before that you need to set to retrieve the element you want? Here, with the `css`/`css_first`/`xpath`/`xpath_first` methods, the identifier is set automatically as the selector you passed here to make things easier :)
Additionally, for all these methods, you can pass the `identifier` argument to set it yourself. This is useful in some instances, or you can use it to save properties with the `auto_save` argument. Additionally, for all these methods, you can pass the `identifier` argument to set it yourself. This is useful in some instances, or you can use it to save properties with the `auto_save` argument.
@@ -174,7 +176,7 @@ First, let's say you got an element like this by text:
```python ```python
>>> element = page.find_by_text('Tipping the Velvet', first_match=True) >>> element = page.find_by_text('Tipping the Velvet', first_match=True)
``` ```
You can save its unique properties with the `save` method, like below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :) You can save its unique properties using the `save` method, as shown below, but you must set the identifier yourself. For this example, I chose `my_special_element` as an identifier, but it's best to use a meaningful identifier in your code for the same reason you use meaningful variable names :)
```python ```python
>>> page.save(element, 'my_special_element') >>> page.save(element, 'my_special_element')
``` ```
@@ -221,7 +223,7 @@ page.save(product, 'specific_product')
``` ```
## Known Issues ## Known Issues
In the `adaptive` save process, the unique properties of the first element from the selection results are the only ones that get saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors get separated, and each selector gets executed alone. In the `adaptive` save process, only the unique properties of the first element in the selection results are saved. So if the selector you are using selects different elements on the page in other locations, `adaptive` will return the first element to you only when you relocate it later. This doesn't include combined CSS selectors (Using commas to combine more than one selector, for example), as these selectors are separated and each is executed alone.
## Final thoughts ## Final thoughts
Explaining this feature in detail without complications turned out to be challenging. However, still, if there's something left unclear, you can head out to the [discussions section](https://github.com/D4Vinci/Scrapling/discussions), and I will reply to you ASAP, or the Discord server, or reach out to me privately and have a chat :) Explaining this feature in detail without complications turned out to be challenging. However, still, if there's something left unclear, you can head out to the [discussions section](https://github.com/D4Vinci/Scrapling/discussions), and I will reply to you ASAP, or the Discord server, or reach out to me privately and have a chat :)
+21 -21
View File
@@ -41,7 +41,7 @@ Then you have the arguments for parsing adjustments or adjusting/manipulating th
I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed. I have intended to ignore the arguments `huge_tree` and `root` to avoid making this page more complicated than needed.
You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested. You may notice that I'm doing that a lot because it involves advanced features that you don't need to know to use the library. The development section will cover these missing parts if you are very invested.
After that, for the main page and elements within, most properties are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :) After that, most properties on the main page and its elements are lazily loaded. This means they don't get initialized until you use them like the text content of a page/element, and this is one of the reasons for Scrapling speed :)
### Properties ### Properties
You have already seen much of this on the [overview](../overview.md) page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the [traversal](#traversal) section. You have already seen much of this on the [overview](../overview.md) page, but don't worry if you didn't. We will review it more thoroughly using more advanced methods/usages. For clarity, the properties for traversal are separated below in the [traversal](#traversal) section.
@@ -111,7 +111,7 @@ But if you try to get the direct text content, it will be empty because it doesn
``` ```
The `get_all_text` method has the following optional arguments: The `get_all_text` method has the following optional arguments:
1. **separator**: All strings collected will be concatenated using this separator. The default is '\n' 1. **separator**: All strings collected will be concatenated using this separator. The default is '\n'.
2. **strip**: If enabled, strings will be stripped before concatenation. Disabled by default. 2. **strip**: If enabled, strings will be stripped before concatenation. Disabled by default.
3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results and ignore any elements nested within them. The default is `('script', 'style',)`. 3. **ignore_tags**: A tuple of all tag names you want to ignore in the final results and ignore any elements nested within them. The default is `('script', 'style',)`.
4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default 4. **valid_values**: If enabled, the method will only collect elements with real values, so all elements with empty text content or only whitespaces will be ignored. It's enabled by default
@@ -132,7 +132,7 @@ If you use it on the page directly, you will find that you are operating on the
>>> page.tag >>> page.tag
'html' 'html'
``` ```
Now, I think I hammered the (`page`/`element`) idea, so I won't return to it again. Now, I think I've hammered the (`page`/`element`) idea, so I won't return to it.
Getting the attributes of the element Getting the attributes of the element
```python ```python
@@ -196,7 +196,7 @@ Same case with XPath
### Traversal ### Traversal
Using the elements we found above, we will go over the properties/methods for moving on the page in detail. Using the elements we found above, we will go over the properties/methods for moving on the page in detail.
If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online for a better understanding. If you are unfamiliar with the DOM tree or the tree data structure in general, the following traversal part can be confusing. I recommend you look up these concepts online to better understand them.
If you are too lazy to search about it, here's a quick explanation to give you a good idea.<br/> If you are too lazy to search about it, here's a quick explanation to give you a good idea.<br/>
In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.<br/> In simple words, the `html` element is the root of the website's tree, as every page starts with an `html` element.<br/>
@@ -310,7 +310,7 @@ In the [Selector](#selector) class, all methods/properties that should return a
Let's see what [Selectors](#selectors) class adds to the table with that out of the way. Let's see what [Selectors](#selectors) class adds to the table with that out of the way.
### Properties ### Properties
Apart from the normal operations on Python lists, such as iteration and slicing, etc. Apart from the standard operations on Python lists, such as iteration and slicing.
You can do the following: You can do the following:
@@ -334,7 +334,7 @@ Execute CSS and XPath selectors directly on the [Selector](#selector) instances
<data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>, <data='<a href="catalogue/soumission_998/index....' parent='<h3><a href="catalogue/soumission_998/in...'>,
...] ...]
``` ```
Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will still leave these methods to be explained in the [TextHandler](#texthandler) section below. Run the `re` and `re_first` methods directly. They take the same arguments passed to the [Selector](#selector) class. I will leave the explanation of these methods to the [TextHandler](#texthandler) section below.
However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance. However, in this class, the `re_first` behaves differently as it runs `re` on each [Selector](#selector) within and returns the first one with a result. The `re` method will return a [TextHandlers](#texthandlers) object as normal, which combines all the [TextHandler](#texthandler) instances into one [TextHandlers](#texthandlers) instance.
```python ```python
@@ -355,7 +355,7 @@ However, in this class, the `re_first` behaves differently as it runs `re` on ea
``` ```
With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`. With the `search` method, you can search quickly in the available [Selector](#selector) instances. The function you pass must accept a [Selector](#selector) instance as the first argument and return True/False. The method will return the first [Selector](#selector) instance that satisfies the function; otherwise, it will return `None`.
```python ```python
# Find all the products with price '53.23' # Find all the products with price '53.23'.
>>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23 >>> search_function = lambda p: float(p.css('.price_color').re_first(r'[\d\.]+')) == 54.23
>>> page.css('.product_pod').search(search_function) >>> page.css('.product_pod').search(search_function)
<data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'> <data='<article class="product_pod"><div class=...' parent='<li class="col-xs-6 col-sm-4 col-md-3 co...'>
@@ -374,7 +374,7 @@ If you are too lazy like me and want to know the number of [Selector](#selector)
```python ```python
page.css('.product_pod').length page.css('.product_pod').length
``` ```
instead of this which is equivalent to
```python ```python
len(page.css('.product_pod')) len(page.css('.product_pod'))
``` ```
@@ -389,15 +389,15 @@ Of course, TextHandler provides extra methods and properties that standard Pytho
### Usage ### Usage
First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well. First, before discussing the added methods, you need to know that all operations on it, like slicing, accessing by index, etc., and methods like `split`, `replace`, `strip`, etc., all return a `TextHandler` again, so you can chain them as you want. If you find a method or property that returns a standard string instead of `TextHandler`, please open an issue, and we will override it as well.
First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they will accept the same arguments as well. First, we start with the `re` and `re_first` methods. These are the same methods that exist in the other classes ([Selector](#selector), [Selectors](#selectors), and [TextHandlers](#texthandlers)), so they accept the same arguments.
- The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but as you probably figured out from the naming, it returns the first result only as a `TextHandler` instance. - The `re` method takes a string/compiled regex pattern as the first argument. It searches the data for all strings matching the regex and returns them as a [TextHandlers](#texthandlers) instance. The `re_first` method takes the same arguments and behaves similarly, but, as you probably figured out from the name, it returns only the first result as a `TextHandler` instance.
Also, it takes other helpful arguments, which are: Also, it takes other helpful arguments, which are:
- **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters. - **replace_entities**: This is enabled by default. It replaces character entity references with their corresponding characters.
- **clean_match**: It's disabled by default. This causes the method to ignore all whitespace and consecutive spaces while matching. - **clean_match**: It's disabled by default. This causes the method to ignore all whitespace, including consecutive spaces, while matching.
- **case_sensitive**: It's enabled by default. As the name implies, disabling it will cause the regex to ignore the case of letters while compiling. - **case_sensitive**: It's enabled by default. As the name implies, disabling it causes the regex to ignore letter case during compilation.
You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method. You have seen these examples before; the return result is [TextHandlers](#texthandlers) because we used the `re` method.
```python ```python
@@ -471,14 +471,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
```python ```python
>>> page.css_first('div::text').json() >>> page.css_first('div::text').json()
``` ```
You will get an error because the `div` tag doesn't have direct text content that can be serialized to JSON; it actually doesn't have direct text content at all.<br/><br/> You will get an error because the `div` tag doesn't have any direct text content that can be serialized to JSON; it doesn't have any direct text content at all.<br/><br/>
In this case, the `get_all_text` method comes to the rescue, so you can do something like that In this case, the `get_all_text` method comes to the rescue, so you can do something like that
```python ```python
>>> page.css_first('div').get_all_text(ignore_tags=[]).json() >>> page.css_first('div').get_all_text(ignore_tags=[]).json()
{'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3} {'lastUpdated': '2024-09-22T10:30:00Z', 'totalProducts': 3}
``` ```
I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.<br/><br/> I used the `ignore_tags` argument here because the default value of it is `('script', 'style',)`, as you are aware.<br/><br/>
Another related behavior to be aware of occurs when using any of the fetchers, which we will explain later. If you have a JSON response like this example: Another related behavior to be aware of occurs when using any fetcher, which we will explain later. If you have a JSON response like this example:
```python ```python
>>> page = Selector("""{"some_key": "some_value"}""") >>> page = Selector("""{"some_key": "some_value"}""")
``` ```
@@ -493,14 +493,14 @@ First, we start with the `re` and `re_first` methods. These are the same methods
{'some_key': 'some_value'} {'some_key': 'some_value'}
``` ```
You might wonder how this happened, given that the `html` tag doesn't contain direct text.<br/> You might wonder how this happened, given that the `html` tag doesn't contain direct text.<br/>
Well, for cases like JSON responses, I made the [Selector](#selector) class maintain a raw copy of the content passed to it. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it used the `get_all_text` method directly.<br/><br/>This might sound hacky a bit but remember, Scrapling is currently optimized to work with HTML pages only so that's the best way till now to handle JSON responses currently without sacrificing speed. This will be changed in the upcoming versions. Well, for cases like JSON responses, I made the [Selector](#selector) class keep a raw copy of the content it receives. This way, when you use the `.json()` method, it checks for that raw copy and then converts it to JSON. If the raw copy is not available like the case with the elements, it checks for the current element text content, or otherwise it uses the `get_all_text` method directly.<br/>
- Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance - Another handy method is `.clean()`, which will remove all white spaces and consecutive spaces for you and return a new `TextHandler` instance
```python ```python
>>> TextHandler('\n wonderful idea, \reh?').clean() >>> TextHandler('\n wonderful idea, \reh?').clean()
'wonderful idea, eh?' 'wonderful idea, eh?'
``` ```
Also, you can pass `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters. Also, you can pass the `remove_entities` argument to make `clean` replace HTML entities with their corresponding characters.
- Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists - Another method that might be helpful in some cases is the `.sort()` method to sort the string for you, as you do with lists
```python ```python
@@ -518,10 +518,10 @@ Other methods and properties will be added over time, but remember that this cla
## TextHandlers ## TextHandlers
You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods. You probably guessed it: This class is similar to [Selectors](#selectors) and [Selector](#selector), but here it inherits the same logic and method as standard lists, with only `re` and `re_first` as new methods.
The only difference is that the `re_first` method logic here does `re` on each [TextHandler](#texthandler) within and returns the first result it has or `None`. Nothing new needs to be explained here, but new methods will be added over time. The only difference is that the `re_first` method logic here runs `re` on each [TextHandler](#texthandler) and returns the first result, or `None`. Nothing new needs to be explained here, but new methods will be added over time.
## AttributesHandler ## AttributesHandler
This is a read-only version of Python's standard dictionary, or `dict`, that is used solely to store the attributes of each element or each [Selector](#selector) instance. This is a read-only version of Python's standard dictionary, or `dict`, used solely to store the attributes of each element or [Selector](#selector) instance.
```python ```python
>>> print(page.find('script').attrib) >>> print(page.find('script').attrib)
{'id': 'page-data', 'type': 'application/json'} {'id': 'page-data', 'type': 'application/json'}
@@ -534,7 +534,7 @@ It currently adds two extra simple methods:
- The `search_values` method - The `search_values` method
In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values instead of keys, it will require some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item. In standard dictionaries, you can do `dict.get("key_name")` to check if a key exists. However, if you want to search by values rather than keys, you will need some additional code lines. This method does that for you. It allows you to search the current attributes by values and returns a dictionary of each matching item.
A simple example would be A simple example would be
```python ```python
@@ -555,13 +555,13 @@ It currently adds two extra simple methods:
<data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>,
<data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>] <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
``` ```
All these elements have 'product' as a value for the attribute `class`. All these elements have 'product' as the value for the `class` attribute.
Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements. Hence, I used the `list` function here because `search_values` returns a generator, so it would be `True` for all elements.
- The `json_string` property - The `json_string` property
This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error This property converts current attributes to a JSON string if the attributes are JSON serializable; otherwise, it throws an error.
```python ```python
>>>page.find('script').attrib.json_string >>>page.find('script').attrib.json_string
+17 -17
View File
@@ -22,7 +22,7 @@ Scrapling implements CSS3 selectors as described in the [W3C specification](http
Also, Scrapling implements some non-standard pseudo-elements like: Also, Scrapling implements some non-standard pseudo-elements like:
* To select text nodes, use ``::text`` * To select text nodes, use ``::text``.
* To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of * To select attribute values, use ``::attr(name)`` where name is the name of the attribute that you want the value of
In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :) In short, if you come from Scrapy/Parsel, you will find the same logic for selectors here to make it easier. No need to implement a stranger logic to the one that most of us are used to :)
@@ -36,19 +36,19 @@ In short, it is the same situation as CSS Selectors; if you come from Scrapy/Par
To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above, and `xpath_first` is faster. To select elements with XPath selectors, you have the `xpath` and `xpath_first` methods. Again, these methods follow the same logic as the CSS selectors methods above, and `xpath_first` is faster.
> Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail. > Note that each method of `css`, `css_first`, `xpath`, and `xpath_first` has additional arguments, but we didn't explain them here, as they are all about the adaptive feature. The adaptive feature will have its own page later to be described in detail.
### Selectors examples ### Selectors examples
Let's see some shared examples of using CSS and XPath Selectors. Let's see some shared examples of using CSS and XPath Selectors.
Select all elements with the class `product` Select all elements with the class `product`.
```python ```python
products = page.css('.product') products = page.css('.product')
products = page.xpath('//*[@class="product"]') products = page.xpath('//*[@class="product"]')
``` ```
Note: The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class** Note: The XPath one won't be accurate if there's another class; **it's always better to rely on CSS for selecting by class**
Select the first element with the class `product` Select the first element with the class `product`.
```python ```python
product = page.css_first('.product') product = page.css_first('.product')
product = page.xpath_first('//*[@class="product"]') product = page.xpath_first('//*[@class="product"]')
@@ -73,12 +73,12 @@ Get the `href` attribute of the first element with the `a` tag name
link = page.css_first('a::attr(href)') link = page.css_first('a::attr(href)')
link = page.xpath_first('//a/@href') link = page.xpath_first('//a/@href')
``` ```
Select the text of the first element with the `h1` tag name, which contains 'Phone', and under an element with class 'product' Select the text of the first element with the `h1` tag name, which contains `Phone`, and under an element with class `product`.
```python ```python
title = page.css_first('.product h1:contains("Phone")::text') title = page.css_first('.product h1:contains("Phone")::text')
title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()') title = page.page.xpath_first('//*[@class="product"]//h1[contains(text(),"Phone")]/text()')
``` ```
You can nest and chain selectors as you want, given that it returns results You can nest and chain selectors as you want, given that they return results
```python ```python
page.css_first('.product').css_first('h1:contains("Phone")::text') page.css_first('.product').css_first('h1:contains("Phone")::text')
page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()') page.xpath_first('//*[@class="product"]').xpath_first('//h1[contains(text(),"Phone")]/text()')
@@ -104,7 +104,7 @@ Scrapling provides the ability to select elements based on their direct text con
What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access. What you can do with `find_by_text` can be done with `find_by_regex` if you are good enough with regular expressions (regex), but we are providing more options to make them easier for all users to access.
With `find_by_text`, you will pass the text as the first argument; with the `find_by_regex` method, the regex pattern is the first argument. Both methods share the following arguments: With `find_by_text`, you pass the text as the first argument; with `find_by_regex`, the regex pattern is the first argument. Both methods share the following arguments:
* **first_match**: If `True` (the default), the method used will return the first result it finds. * **first_match**: If `True` (the default), the method used will return the first result it finds.
* **case_sensitive**: If `True`, the case of the letters will be considered. * **case_sensitive**: If `True`, the case of the letters will be considered.
@@ -117,7 +117,7 @@ By default, Scrapling searches for the exact matching of the text/pattern you pa
Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples. Note: The method `find_by_regex` can accept both regular strings and a compiled regex pattern as its first argument, as you will see in the upcoming examples.
### Finding Similar Elements ### Finding Similar Elements
One of the most remarkable new features that Scrapling puts on the table is the feature that allows the user to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here. One of the most remarkable new features Scrapling puts on the table is the ability to tell Scrapling to find elements similar to the element at hand. This feature's inspiration came from the AutoScraper library, but in Scrapling, it can be used on elements found by any method. Most of its usage would likely occur after finding elements through text content, similar to how AutoScraper works, making it convenient to explain here.
So, how does it work? So, how does it work?
@@ -129,8 +129,8 @@ Imagine a scenario where you found a product by its title, for example, and you
That's a lot of talking, I know, but I had to go deep. I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method: That's a lot of talking, I know, but I had to go deep. I will give examples of using this method in the next section, but first, these are the arguments that can be passed to this method:
* **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the values of the attributes of both elements should be at least 20% similar. If you want to turn off this check (Step 3, basically), you can set this attribute to 0, but I recommend you read what the other arguments do first. * **similarity_threshold**: This is the percentage we discussed in step 3 for comparing elements' attributes. The default value is 0.2. In Simpler words, the tag attributes of both elements should be at least 20% similar. If you want to turn off this check (basically Step 3), you can set this attribute to 0, but I recommend you read what the other arguments do first.
* **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change a lot between elements, making them unreliable. * **ignore_attributes**: The attribute names passed will be ignored while matching the attributes in the last step. The default value is `('href', 'src',)` because URLs can change significantly across elements, making them unreliable.
* **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends. * **match_text**: If `True`, the element's text content will be considered when matching (Step 3). Using this argument in typical cases is not recommended, but it depends.
Now, let's check out the examples below. Now, let's check out the examples below.
@@ -148,7 +148,7 @@ Find the first element whose text fully matches this text
>>> page.find_by_text('Tipping the Velvet') >>> page.find_by_text('Tipping the Velvet')
<data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'> <data='<a href="catalogue/tipping-the-velvet_99...' parent='<h3><a href="catalogue/tipping-the-velve...'>
``` ```
Combining it with `page.urljoin` to return the full URL from the relative `href` Combining it with `page.urljoin` to return the full URL from the relative `href`.
```python ```python
>>> page.find_by_text('Tipping the Velvet').attrib['href'] >>> page.find_by_text('Tipping the Velvet').attrib['href']
'catalogue/tipping-the-velvet_999/index.html' 'catalogue/tipping-the-velvet_999/index.html'
@@ -174,7 +174,7 @@ Get all elements that contain the word `the` (Partial matching)
'Mesaerion: The Best Science ...', 'Mesaerion: The Best Science ...',
"It's Only the Himalayas"] "It's Only the Himalayas"]
``` ```
The search is case-insensitive, so those results have `The`, not only the lowercase one `the`; let's limit the search to the elements with `the` only. The search is case-insensitive, so those results include `The`, not just the lowercase `the`; let's limit the search to elements with `the` only.
```python ```python
>>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True) >>> results = page.find_by_text('the', partial=True, first_match=False, case_sensitive=True)
>>> [i.text for i in results] >>> [i.text for i in results]
@@ -332,14 +332,14 @@ def extract_reviews(page):
] ]
``` ```
## Filters-based searching ## Filters-based searching
This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier to learn for newcomers to Web Scraping than writing selectors. This search method is arguably the best way to find elements in Scrapling, as it is powerful and easier for newcomers to Web Scraping to learn than writing selectors.
Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can take multiple types of filters and return all elements in the pages that all these filters apply to. Inspired by BeautifulSoup's `find_all` function, you can find elements using the `find_all` and `find` methods. Both methods can accept multiple filters and return all elements on the pages where all these filters apply.
To be more specific: To be more specific:
* Any string passed is considered a tag name. * Any string passed is considered a tag name.
* Any iterable passed, like List/Tuple/Set, is considered an iterable of tag names. * Any iterable passed, like List/Tuple/Set, will be considered as an iterable of tag names.
* Any dictionary is considered a mapping of HTML element(s), attribute names, and attribute values. * Any dictionary is considered a mapping of HTML element(s), attribute names, and attribute values.
* Any regex patterns passed are used to filter elements by content, like the `find_by_regex` method * Any regex patterns passed are used to filter elements by content, like the `find_by_regex` method
* Any functions passed are used to filter elements * Any functions passed are used to filter elements
@@ -357,7 +357,7 @@ It filters all elements in the current page/element in the following order:
Notes: Notes:
1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on. 1. As you probably understood, the filtering process always starts from the first filter it finds in the filtering order above. So, if no tag name(s) are passed but attributes are passed, the process starts from that step (number 2), and so on.
2. The order in which you pass the arguments doesn't matter. The only order taken into consideration is the order explained above. 2. The order in which you pass the arguments doesn't matter. The only order considered is the one explained above.
Check examples to clear any confusion :) Check examples to clear any confusion :)
@@ -509,4 +509,4 @@ We will have a deep look at it while explaining the [TextHandler](main_classes.m
>>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html') >>> page.find_by_text('Tipping the Velvet').attrib['href'].re(r'catalogue/(.*)/index.html')
['tipping-the-velvet_999'] ['tipping-the-velvet_999']
``` ```
And so on. You get the idea. We will explain this in more detail on the next page while explaining the [TextHandler](main_classes.md#texthandler) class. And so on. You get the idea. We will explain this in more detail on the next page, along with the [TextHandler](main_classes.md#texthandler) class.
+8 -5
View File
@@ -1,5 +1,8 @@
mkdocs-material mkdocstrings>=1.0.0
mkdocstrings mkdocstrings-python>=2.0.1
mkdocstrings-python griffe-inherited-docstrings
mkdocs-material[imaging] griffe-runtime-objects
black griffe-sphinx
mkdocs-material[imaging]>=9.7.1
black>=25.12.0
pngquant
@@ -1,10 +1,10 @@
# Migrating from BeautifulSoup to Scrapling # Migrating from BeautifulSoup to Scrapling
If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is incredibly faster, provides the same parsing capabilities, adds more parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities. If you're already familiar with BeautifulSoup, you're in for a treat. Scrapling is much faster, provides the same parsing capabilities as BS, adds additional parsing capabilities not found in BS, and introduces powerful new features for fetching and handling modern web pages. This guide will help you quickly adapt your existing BeautifulSoup code to leverage Scrapling's capabilities.
Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to accomplish a specific task using BeautifulSoup and the corresponding method in Scrapling. Below is a table that covers the most common operations you'll perform when scraping web pages. Each row illustrates how to achieve a specific task using BeautifulSoup and the corresponding method in Scrapling.
You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, but that's one of the reasons why BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short oneliner, there is no need to sacrifice performance to shorten that short line :) You will notice that some shortcuts in BeautifulSoup are missing in Scrapling, which is one of the reasons BeautifulSoup is slower than Scrapling. The point is: If the same feature can be used in a short one-liner, there is no need to sacrifice performance to shorten that short line :)
| Task | BeautifulSoup Code | Scrapling Code | | Task | BeautifulSoup Code | Scrapling Code |
@@ -56,7 +56,7 @@ Here's a simple example of scraping a web page to extract all the links using Be
import requests import requests
from bs4 import BeautifulSoup from bs4 import BeautifulSoup
url = 'http://example.com' url = 'https://example.com'
response = requests.get(url) response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser') soup = BeautifulSoup(response.text, 'html.parser')
@@ -70,7 +70,7 @@ for link in links:
```python ```python
from scrapling import Fetcher from scrapling import Fetcher
url = 'http://example.com' url = 'https://example.com'
page = Fetcher.get(url) page = Fetcher.get(url)
links = page.css('a::attr(href)') links = page.css('a::attr(href)')
@@ -78,12 +78,12 @@ for link in links:
print(link) print(link)
``` ```
As you can see, Scrapling simplifies the process by handling the fetching and parsing in a single step, making your code cleaner and more efficient. As you can see, Scrapling simplifies the process by combining fetching and parsing into a single step, making your code cleaner and more efficient.
**Additional Notes:** **Additional Notes:**
- **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons. - **Different parsers**: BeautifulSoup allows you to set the parser engine to use, and one of them is `lxml`. Scrapling doesn't do that and uses the `lxml` library by default for performance reasons.
- **Element Types**: In BeautifulSoup, elements are `Tag` objects, while in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction. - **Element Types**: In BeautifulSoup, elements are `Tag` objects; in Scrapling, they are `Selector` objects. However, they provide similar methods and properties for navigation and data extraction.
- **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.css_first()`). To avoid errors, check for `None` before accessing properties. - **Error Handling**: Both libraries return `None` when an element is not found (e.g., `soup.find()` or `page.css_first()`). To avoid errors, check for `None` before accessing properties.
- **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can help remove extra whitespace, consecutive spaces, or unwanted characters. Please check out the documentation for the complete list. - **Text Extraction**: Scrapling provides additional methods for handling text through `TextHandler`, such as `clean()`, which can help remove extra whitespace, consecutive spaces, or unwanted characters. Please check out the documentation for the complete list.
+11 -11
View File
@@ -26,13 +26,13 @@ How will you solve that manually? I'm referring to generic web scraping of vario
## AI to the rescue, but at a high cost ## AI to the rescue, but at a high cost
Of course, the AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :) Of course, AI can easily solve most of these issues because it can understand the page source and identify the fields you want or create selectors for them. That's, of course, if you already solved the anti-bot measures through other tools :)
This approach is, of course, beautiful. I love AI and find it very fascinating, especially Generative AI. You will probably spend a lot of time on prompt engineering and tweaking the prompts, but if that's cool with you, you will soon hit the real issue with using AI here. This approach is, of course, beautiful. I love AI and find it very fascinating, especially Generative AI. You will probably spend a lot of time on prompt engineering and tweaking the prompts, but if that's cool with you, you will soon hit the real issue with using AI here.
Most websites have vast amounts of content per page, which you will need to pass to the AI somehow so it can do its magic. This will burn through tokens like fire in a haystack, quickly accumulating high costs. Most websites have vast amounts of content per page, which you will need to pass to the AI somehow so it can do its magic. This will burn through tokens like fire in a haystack, quickly accumulating high costs.
Unless money is irrelevant to you, you will try to find less expensive approaches, and that's why I made Scrapling :smile: Unless money is irrelevant to you, you will try to find less expensive approaches, and that's where Scrapling comes into play :smile:
## Scrapling got you covered ## Scrapling got you covered
@@ -41,11 +41,11 @@ Scrapling can handle almost all issues you will face during Web Scraping, and th
### Solving issue T1: Rapidly changing website structures ### Solving issue T1: Rapidly changing website structures
That's why the [adaptive](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/) feature was made. You knew I would talk about it, and here we are :) That's why the [adaptive](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/) feature was made. You knew I would talk about it, and here we are :)
While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties to find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on. While Web Scraping, if you have the `adaptive` feature enabled, you can save any element's unique properties so you can find it again later when the website's structure changes. The most frustrating thing about changes is that anything about an element can change, so there's nothing to rely on.
That's how the adaptive feature works: it stores everything unique about an element. When the website structure changes, it returns the element with the highest similarity score of the previous element. That's how the adaptive feature works: it stores everything unique about an element. When the website structure changes, it returns the element with the highest similarity score of the previous element.
I have already explained that in more detail and with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works). I have already explained this in more detail, with many examples. Read more from [here](https://scrapling.readthedocs.io/en/latest/parsing/adaptive/#how-the-adaptive-feature-works).
### Solving issue T2: Unstable selectors ### Solving issue T2: Unstable selectors
If you have been doing Web scraping for a long enough time, you have likely experienced this once. I'm referring to a website that employs poor design patterns, built on raw HTML without any IDs/classes, or uses random class names with nothing else to rely on, etc... If you have been doing Web scraping for a long enough time, you have likely experienced this once. I'm referring to a website that employs poor design patterns, built on raw HTML without any IDs/classes, or uses random class names with nothing else to rely on, etc...
@@ -59,16 +59,16 @@ In these cases, standard selection methods with CSS/XPath selectors won't be opt
There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this. There is no need to explain any of these; click on the links, and it will be clear how Scrapling solves this.
### Solving issue T3: Increasingly complex anti-bot measures ### Solving issue T3: Increasingly complex anti-bot measures
It's known that making an undetectable spider takes more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve: It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides many flexible options, like stealth mode suitable for small to medium protections and using your real browser. 1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) — This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher utilizes our version of a modified Firefox browser, called [Camoufox](https://camoufox.com/stealth/), which nearly passes all known tests and incorporates additional tricks. **With v0.3, this fetcher can bypass Cloudflare for you automatically as well!** 2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy/) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic/) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
These two will be improved a lot with the upcoming updates, so stay tuned :) We keep improving these two with each update, so stay tuned :)
### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data ### Solving issues B1 & B2: Extreme Website Diversity / Identifying Relevant Data
This one is tough to handle, but it's possible with Scrapling's flexibility. This one is tough to handle, but Scrapling's flexibility makes it possible.
I talked with someone who uses AI to extract prices from different websites. He is only interested in prices and titles, so he uses AI to find the price for him. I talked with someone who uses AI to extract prices from different websites. He is only interested in prices and titles, so he uses AI to find the price for him.
@@ -94,7 +94,7 @@ It will be a bit boring, but it's definitely less expensive than AI.
This example illustrates the point I aim to convey here. Not every challenge will need AI to be solved, but sometimes you need to be creative, and that might save you a lot of money. This example illustrates the point I aim to convey here. Not every challenge will need AI to be solved, but sometimes you need to be creative, and that might save you a lot of money.
### Solving issue B3: Pagination variations ### Solving issue B3: Pagination variations
This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the following updates :) This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :)
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea. But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea.
@@ -112,6 +112,6 @@ For a quick comparison.
This table is based on pricing from [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing) This table is based on pricing from [Browse AI Pricing](https://www.browse.ai/pricing) and [Oxylabs Web Scraper API Pricing](https://oxylabs.io/products/scraper-api/web/pricing)
## Conclusion ## Conclusion
While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit designed to tackle the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers. While AI offers powerful capabilities, its cost can be prohibitive for many Web scraping tasks. Scrapling provides a robust, flexible, and cost-effective toolkit for tackling the real-world challenges of both targeted and broad scraping, often eliminating the need for expensive AI solutions. You can build resilient scrapers more efficiently by leveraging features like `adaptive`, diverse selection methods, and advanced fetchers.
Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects! Explore the documentation further and see how Scrapling can simplify your future Web Scraping projects!
+17 -4
View File
@@ -29,6 +29,7 @@ theme:
- navigation.sections - navigation.sections
- navigation.tracking - navigation.tracking
- navigation.instant - navigation.instant
- navigation.instant.prefetch
- navigation.instant.progress - navigation.instant.progress
# - navigation.tabs # - navigation.tabs
# - navigation.expand # - navigation.expand
@@ -41,6 +42,7 @@ theme:
- content.action.view - content.action.view
- content.action.edit - content.action.edit
- content.code.copy - content.code.copy
- content.code.select
- content.code.annotate - content.code.annotate
- content.code.annotation - content.code.annotation
@@ -103,10 +105,13 @@ markdown_extensions:
plugins: plugins:
- search - search
# - social: - privacy:
# cards_layout_options: links: false
# background_color: "#1f1f1f" - optimize
# font_family: Roboto - social:
cards_layout_options:
background_color: "#1f1f1f"
font_family: Roboto
- mkdocstrings: - mkdocstrings:
handlers: handlers:
python: python:
@@ -122,11 +127,19 @@ plugins:
unwrap_annotated: true unwrap_annotated: true
filters: filters:
- '!^_' - '!^_'
- "^__"
merge_init_into_class: true merge_init_into_class: true
docstring_section_style: spacy docstring_section_style: spacy
signature_crossrefs: true signature_crossrefs: true
show_symbol_type_heading: true show_symbol_type_heading: true
show_symbol_type_toc: true show_symbol_type_toc: true
show_inheritance_diagram: true
modernize_annotations: true
extensions:
- griffe_runtime_objects
- griffe_sphinx
- griffe_inherited_docstrings:
merge: true
extra: extra:
homepage: https://scrapling.readthedocs.io/en/latest/ homepage: https://scrapling.readthedocs.io/en/latest/
+3 -4
View File
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "scrapling" name = "scrapling"
# Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand # Static version instead of a dynamic version so we can get better layer caching while building docker, check the docker file to understand
version = "0.3.12" version = "0.3.13"
description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!" description = "Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!"
readme = {file = "docs/README.md", content-type = "text/markdown"} readme = {file = "docs/README.md", content-type = "text/markdown"}
license = {file = "LICENSE"} license = {file = "LICENSE"}
@@ -60,7 +60,7 @@ dependencies = [
"lxml>=6.0.2", "lxml>=6.0.2",
"cssselect>=1.3.0", "cssselect>=1.3.0",
"orjson>=3.11.5", "orjson>=3.11.5",
"tldextract>=5.3.0", "tldextract>=5.3.1",
] ]
[project.optional-dependencies] [project.optional-dependencies]
@@ -69,8 +69,7 @@ fetchers = [
"curl_cffi>=0.14.0", "curl_cffi>=0.14.0",
"playwright==1.56.0", "playwright==1.56.0",
"patchright==1.56.0", "patchright==1.56.0",
"camoufox>=0.4.11", "browserforge>=1.2.1",
"geoip2>=5.2.0",
"msgspec>=0.20.0", "msgspec>=0.20.0",
] ]
ai = [ ai = [
+1 -1
View File
@@ -1,5 +1,5 @@
__author__ = "Karim Shoair (karim.shoair@pm.me)" __author__ = "Karim Shoair (karim.shoair@pm.me)"
__version__ = "0.3.12" __version__ = "0.3.13"
__copyright__ = "Copyright (c) 2024 Karim Shoair" __copyright__ = "Copyright (c) 2024 Karim Shoair"
from typing import Any, TYPE_CHECKING from typing import Any, TYPE_CHECKING
+19 -52
View File
@@ -125,14 +125,9 @@ def install(force): # pragma: no cover
"playwright", "playwright",
"install-deps", "install-deps",
"chromium", "chromium",
"firefox",
], ],
"Playwright dependencies", "Playwright dependencies",
) )
__Execute(
[python_executable, "-m", "camoufox", "fetch", "--browserforge"],
"Camoufox browser and databases",
)
# if no errors raised by the above commands, then we add the below file # if no errors raised by the above commands, then we add the below file
__PACKAGE_DIR__.joinpath(".scrapling_dependencies_installed").touch() __PACKAGE_DIR__.joinpath(".scrapling_dependencies_installed").touch()
else: else:
@@ -610,17 +605,11 @@ def delete(
help="CSS selector to extract specific content from the page. It returns all matches.", help="CSS selector to extract specific content from the page. It returns all matches.",
) )
@option("--wait-selector", help="CSS selector to wait for before proceeding") @option("--wait-selector", help="CSS selector to wait for before proceeding")
@option("--locale", default="en-US", help="Browser locale (default: en-US)") @option("--locale", default=None, help="Specify user locale. Defaults to the system default locale.")
@option("--stealth/--no-stealth", default=False, help="Enable stealth mode (default: False)")
@option( @option(
"--hide-canvas/--show-canvas", "--real-chrome/--no-real-chrome",
default=False, default=False,
help="Add noise to canvas operations (default: False)", help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
)
@option(
"--disable-webgl/--enable-webgl",
default=False,
help="Disable WebGL support (default: False)",
) )
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"') @option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option( @option(
@@ -640,9 +629,7 @@ def fetch(
css_selector, css_selector,
wait_selector, wait_selector,
locale, locale,
stealth, real_chrome,
hide_canvas,
disable_webgl,
proxy, proxy,
extra_headers, extra_headers,
): ):
@@ -659,9 +646,7 @@ def fetch(
:param css_selector: CSS selector to extract specific content. :param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser. :param locale: Set the locale for the browser.
:param stealth: Enables stealth mode. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param proxy: The proxy to be used with requests. :param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request. :param extra_headers: Extra headers to add to the request.
""" """
@@ -676,9 +661,7 @@ def fetch(
"network_idle": network_idle, "network_idle": network_idle,
"timeout": timeout, "timeout": timeout,
"locale": locale, "locale": locale,
"stealth": stealth, "real_chrome": real_chrome,
"hide_canvas": hide_canvas,
"disable_webgl": disable_webgl,
} }
if wait > 0: if wait > 0:
@@ -703,11 +686,6 @@ def fetch(
default=True, default=True,
help="Run browser in headless mode (default: True)", help="Run browser in headless mode (default: True)",
) )
@option(
"--block-images/--allow-images",
default=False,
help="Block image loading (default: False)",
)
@option( @option(
"--disable-resources/--enable-resources", "--disable-resources/--enable-resources",
default=False, default=False,
@@ -718,11 +696,6 @@ def fetch(
default=False, default=False,
help="Block WebRTC entirely (default: False)", help="Block WebRTC entirely (default: False)",
) )
@option(
"--humanize/--no-humanize",
default=False,
help="Humanize cursor movement (default: False)",
)
@option( @option(
"--solve-cloudflare/--no-solve-cloudflare", "--solve-cloudflare/--no-solve-cloudflare",
default=False, default=False,
@@ -735,9 +708,14 @@ def fetch(
help="Wait for network idle (default: False)", help="Wait for network idle (default: False)",
) )
@option( @option(
"--disable-ads/--allow-ads", "--real-chrome/--no-real-chrome",
default=False, default=False,
help="Install uBlock Origin addon (default: False)", help="If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)",
)
@option(
"--hide-canvas/--show-canvas",
default=False,
help="Add noise to canvas operations (default: False)",
) )
@option( @option(
"--timeout", "--timeout",
@@ -757,11 +735,6 @@ def fetch(
help="CSS selector to extract specific content from the page. It returns all matches.", help="CSS selector to extract specific content from the page. It returns all matches.",
) )
@option("--wait-selector", help="CSS selector to wait for before proceeding") @option("--wait-selector", help="CSS selector to wait for before proceeding")
@option(
"--geoip/--no-geoip",
default=False,
help="Use IP geolocation for timezone/locale (default: False)",
)
@option("--proxy", help='Proxy URL in format "http://username:password@host:port"') @option("--proxy", help='Proxy URL in format "http://username:password@host:port"')
@option( @option(
"--extra-headers", "--extra-headers",
@@ -773,19 +746,17 @@ def stealthy_fetch(
url, url,
output_file, output_file,
headless, headless,
block_images,
disable_resources, disable_resources,
block_webrtc, block_webrtc,
humanize,
solve_cloudflare, solve_cloudflare,
allow_webgl, allow_webgl,
network_idle, network_idle,
disable_ads, real_chrome,
hide_canvas,
timeout, timeout,
wait, wait,
css_selector, css_selector,
wait_selector, wait_selector,
geoip,
proxy, proxy,
extra_headers, extra_headers,
): ):
@@ -795,19 +766,17 @@ def stealthy_fetch(
:param url: Target url. :param url: Target url.
:param output_file: Output file path (.md for Markdown, .html for HTML). :param output_file: Output file path (.md for Markdown, .html for HTML).
:param headless: Run the browser in headless/hidden, or headful/visible mode. :param headless: Run the browser in headless/hidden, or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. :param disable_resources: Drop requests of unnecessary resources for a speed boost.
:param block_webrtc: Blocks WebRTC entirely. :param block_webrtc: Blocks WebRTC entirely.
:param humanize: Humanize the cursor movement.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges. :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges.
:param allow_webgl: Allow WebGL (recommended to keep enabled). :param allow_webgl: Allow WebGL (recommended to keep enabled).
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param disable_ads: Install the uBlock Origin addon on the browser. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. :param timeout: The timeout in milliseconds that is used in all operations and waits through the page.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before returning. :param wait: The time (milliseconds) the fetcher will wait after everything finishes before returning.
:param css_selector: CSS selector to extract specific content. :param css_selector: CSS selector to extract specific content.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param geoip: Automatically use IP's longitude, latitude, timezone, country, locale.
:param proxy: The proxy to be used with requests. :param proxy: The proxy to be used with requests.
:param extra_headers: Extra headers to add to the request. :param extra_headers: Extra headers to add to the request.
""" """
@@ -818,16 +787,14 @@ def stealthy_fetch(
# Build request arguments # Build request arguments
kwargs = { kwargs = {
"headless": headless, "headless": headless,
"block_images": block_images,
"disable_resources": disable_resources, "disable_resources": disable_resources,
"block_webrtc": block_webrtc, "block_webrtc": block_webrtc,
"humanize": humanize,
"solve_cloudflare": solve_cloudflare, "solve_cloudflare": solve_cloudflare,
"allow_webgl": allow_webgl, "allow_webgl": allow_webgl,
"network_idle": network_idle, "network_idle": network_idle,
"disable_ads": disable_ads, "real_chrome": real_chrome,
"hide_canvas": hide_canvas,
"timeout": timeout, "timeout": timeout,
"geoip": geoip,
} }
if wait > 0: if wait > 0:
+41 -36
View File
@@ -1,9 +1,13 @@
from scrapling.core._types import ( from scrapling.core._types import (
Dict,
Any, Any,
Dict,
List, List,
Tuple, Tuple,
Sequence,
Callable,
Optional, Optional,
SetCookieParam,
SelectorWaitStates,
) )
# Parameter definitions for shell function signatures (defined once at module level) # Parameter definitions for shell function signatures (defined once at module level)
@@ -31,57 +35,58 @@ _REQUESTS_PARAMS = {
_FETCH_PARAMS = { _FETCH_PARAMS = {
"headless": bool, "headless": bool,
"google_search": bool,
"hide_canvas": bool,
"disable_webgl": bool,
"real_chrome": bool,
"stealth": bool,
"wait": int | float,
"page_action": Optional[Any],
"proxy": Optional[str | Dict],
"locale": str,
"extra_headers": Optional[Dict[str, str]],
"useragent": Optional[str],
"cdp_url": Optional[str],
"timeout": int | float,
"disable_resources": bool, "disable_resources": bool,
"wait_selector": Optional[str],
"init_script": Optional[str],
"cookies": Optional[List[Dict]],
"network_idle": bool, "network_idle": bool,
"load_dom": bool, "load_dom": bool,
"wait_selector_state": Any, "wait_selector": Optional[str],
"extra_flags": Optional[List[str]], "wait_selector_state": SelectorWaitStates,
"additional_args": Optional[Dict], "cookies": Sequence[SetCookieParam],
"google_search": bool,
"wait": int | float,
"timezone_id": str | None,
"page_action": Optional[Callable],
"proxy": Optional[str | Dict[str, str] | Tuple],
"extra_headers": Optional[Dict[str, str]],
"timeout": int | float,
"init_script": Optional[str],
"user_data_dir": str,
"selector_config": Optional[Dict], "selector_config": Optional[Dict],
"additional_args": Optional[Dict],
"locale": Optional[str],
"real_chrome": bool,
"cdp_url": Optional[str],
"useragent": Optional[str],
"extra_flags": Optional[List[str]],
} }
_STEALTHY_FETCH_PARAMS = { _STEALTHY_FETCH_PARAMS = {
"headless": bool, "headless": bool,
"block_images": bool,
"disable_resources": bool, "disable_resources": bool,
"block_webrtc": bool,
"allow_webgl": bool,
"network_idle": bool, "network_idle": bool,
"load_dom": bool, "load_dom": bool,
"humanize": bool | float,
"solve_cloudflare": bool,
"wait": int | float,
"timeout": int | float,
"page_action": Optional[Any],
"wait_selector": Optional[str], "wait_selector": Optional[str],
"init_script": Optional[str], "wait_selector_state": SelectorWaitStates,
"addons": Optional[List[str]], "cookies": Sequence[SetCookieParam],
"wait_selector_state": Any,
"cookies": Optional[List[Dict]],
"google_search": bool, "google_search": bool,
"wait": int | float,
"timezone_id": str | None,
"page_action": Optional[Callable],
"proxy": Optional[str | Dict[str, str] | Tuple],
"extra_headers": Optional[Dict[str, str]], "extra_headers": Optional[Dict[str, str]],
"proxy": Optional[str | Dict], "timeout": int | float,
"os_randomize": bool, "init_script": Optional[str],
"disable_ads": bool, "user_data_dir": str,
"geoip": bool,
"selector_config": Optional[Dict], "selector_config": Optional[Dict],
"additional_args": Optional[Dict], "additional_args": Optional[Dict],
"locale": Optional[str],
"real_chrome": bool,
"cdp_url": Optional[str],
"useragent": Optional[str],
"extra_flags": Optional[List[str]],
"allow_webgl": bool,
"hide_canvas": bool,
"block_webrtc": bool,
"solve_cloudflare": bool,
} }
# Mapping of function names to their parameter definitions # Mapping of function names to their parameter definitions
+14
View File
@@ -57,3 +57,17 @@ except ImportError: # pragma: no cover
from typing_extensions import Self # Backport from typing_extensions import Self # Backport
except ImportError: except ImportError:
Self = object Self = object
# Copied from `playwright._impl._api_structures.SetCookieParam`
class SetCookieParam(TypedDict, total=False):
name: str
value: str
url: Optional[str]
domain: Optional[str]
path: Optional[str]
expires: Optional[float]
httpOnly: Optional[bool]
secure: Optional[bool]
sameSite: Optional[Literal["Lax", "None", "Strict"]]
partitionKey: Optional[str]
+89 -104
View File
@@ -17,13 +17,15 @@ from scrapling.fetchers import (
from scrapling.core._types import ( from scrapling.core._types import (
Optional, Optional,
Tuple, Tuple,
extraction_types,
Mapping, Mapping,
Dict, Dict,
List, List,
Any, Any,
SelectorWaitStates,
Generator, Generator,
Sequence,
SetCookieParam,
extraction_types,
SelectorWaitStates,
) )
@@ -213,20 +215,18 @@ class ScraplingMCPServer:
main_content_only: bool = True, main_content_only: bool = True,
headless: bool = False, headless: bool = False,
google_search: bool = True, google_search: bool = True,
hide_canvas: bool = False,
disable_webgl: bool = False,
real_chrome: bool = False, real_chrome: bool = False,
stealth: bool = False,
wait: int | float = 0, wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None, proxy: Optional[str | Dict[str, str]] = None,
locale: str = "en-US", timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None, extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None, useragent: Optional[str] = None,
cdp_url: Optional[str] = None, cdp_url: Optional[str] = None,
timeout: int | float = 30000, timeout: int | float = 30000,
disable_resources: bool = False, disable_resources: bool = False,
wait_selector: Optional[str] = None, wait_selector: Optional[str] = None,
cookies: Optional[List[Dict]] = None, cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False, network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached", wait_selector_state: SelectorWaitStates = "attached",
) -> ResponseModel: ) -> ResponseModel:
@@ -242,21 +242,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None. :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag. :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts. :param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`. :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -269,15 +267,13 @@ class ScraplingMCPServer:
locale=locale, locale=locale,
timeout=timeout, timeout=timeout,
cookies=cookies, cookies=cookies,
stealth=stealth,
cdp_url=cdp_url, cdp_url=cdp_url,
headless=headless, headless=headless,
useragent=useragent, useragent=useragent,
hide_canvas=hide_canvas, timezone_id=timezone_id,
real_chrome=real_chrome, real_chrome=real_chrome,
network_idle=network_idle, network_idle=network_idle,
wait_selector=wait_selector, wait_selector=wait_selector,
disable_webgl=disable_webgl,
extra_headers=extra_headers, extra_headers=extra_headers,
google_search=google_search, google_search=google_search,
disable_resources=disable_resources, disable_resources=disable_resources,
@@ -301,20 +297,18 @@ class ScraplingMCPServer:
main_content_only: bool = True, main_content_only: bool = True,
headless: bool = False, headless: bool = False,
google_search: bool = True, google_search: bool = True,
hide_canvas: bool = False,
disable_webgl: bool = False,
real_chrome: bool = False, real_chrome: bool = False,
stealth: bool = False,
wait: int | float = 0, wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None, proxy: Optional[str | Dict[str, str]] = None,
locale: str = "en-US", timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None, extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None, useragent: Optional[str] = None,
cdp_url: Optional[str] = None, cdp_url: Optional[str] = None,
timeout: int | float = 30000, timeout: int | float = 30000,
disable_resources: bool = False, disable_resources: bool = False,
wait_selector: Optional[str] = None, wait_selector: Optional[str] = None,
cookies: Optional[List[Dict]] = None, cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False, network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached", wait_selector_state: SelectorWaitStates = "attached",
) -> List[ResponseModel]: ) -> List[ResponseModel]:
@@ -330,21 +324,19 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None. :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag. :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts. :param cookies: Set cookies for the next request. It should be in a dictionary format that Playwright accepts.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`. :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -356,17 +348,15 @@ class ScraplingMCPServer:
locale=locale, locale=locale,
timeout=timeout, timeout=timeout,
cookies=cookies, cookies=cookies,
stealth=stealth,
cdp_url=cdp_url, cdp_url=cdp_url,
headless=headless, headless=headless,
max_pages=len(urls), max_pages=len(urls),
useragent=useragent, useragent=useragent,
hide_canvas=hide_canvas, timezone_id=timezone_id,
real_chrome=real_chrome, real_chrome=real_chrome,
network_idle=network_idle, network_idle=network_idle,
wait_selector=wait_selector, wait_selector=wait_selector,
google_search=google_search, google_search=google_search,
disable_webgl=disable_webgl,
extra_headers=extra_headers, extra_headers=extra_headers,
disable_resources=disable_resources, disable_resources=disable_resources,
wait_selector_state=wait_selector_state, wait_selector_state=wait_selector_state,
@@ -393,29 +383,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None, css_selector: Optional[str] = None,
main_content_only: bool = True, main_content_only: bool = True,
headless: bool = True, # noqa: F821 headless: bool = True, # noqa: F821
block_images: bool = False, google_search: bool = True,
real_chrome: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
hide_canvas: bool = False,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False, disable_resources: bool = False,
wait_selector: Optional[str] = None,
cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False, block_webrtc: bool = False,
allow_webgl: bool = True, allow_webgl: bool = True,
network_idle: bool = False,
humanize: bool | float = True,
solve_cloudflare: bool = False, solve_cloudflare: bool = False,
wait: int | float = 0,
timeout: int | float = 30000,
wait_selector: Optional[str] = None,
addons: Optional[List[str]] = None,
wait_selector_state: SelectorWaitStates = "attached",
cookies: Optional[List[Dict]] = None,
google_search: bool = True,
extra_headers: Optional[Dict[str, str]] = None,
proxy: Optional[str | Dict[str, str]] = None,
os_randomize: bool = False,
disable_ads: bool = False,
geoip: bool = False,
additional_args: Optional[Dict] = None, additional_args: Optional[Dict] = None,
) -> ResponseModel: ) -> ResponseModel:
"""Use Scrapling's version of the Camoufox browser to fetch a URL and return a structured output of the result. """Use the stealthy fetcher to fetch a URL and return a structured output of the result.
Note: This is best suitable for high protection levels. It's slower than the other tools. Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned. Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param url: The URL to request. :param url: The URL to request.
@@ -426,54 +416,52 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None. :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag. :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param block_webrtc: Blocks WebRTC entirely.
:param cookies: Set cookies for the next request. :param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you. :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled. :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address. :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
""" """
page = await StealthyFetcher.async_fetch( page = await StealthyFetcher.async_fetch(
url, url,
wait=wait, wait=wait,
proxy=proxy, proxy=proxy,
geoip=geoip, locale=locale,
addons=addons, cdp_url=cdp_url,
timeout=timeout, timeout=timeout,
cookies=cookies, cookies=cookies,
headless=headless, headless=headless,
humanize=humanize, useragent=useragent,
timezone_id=timezone_id,
real_chrome=real_chrome,
hide_canvas=hide_canvas,
allow_webgl=allow_webgl, allow_webgl=allow_webgl,
disable_ads=disable_ads,
network_idle=network_idle, network_idle=network_idle,
block_images=block_images,
block_webrtc=block_webrtc, block_webrtc=block_webrtc,
os_randomize=os_randomize,
wait_selector=wait_selector, wait_selector=wait_selector,
google_search=google_search, google_search=google_search,
extra_headers=extra_headers, extra_headers=extra_headers,
additional_args=additional_args,
solve_cloudflare=solve_cloudflare, solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources, disable_resources=disable_resources,
wait_selector_state=wait_selector_state, wait_selector_state=wait_selector_state,
additional_args=additional_args,
) )
return _ContentTranslator( return _ContentTranslator(
Convertor._extract_content( Convertor._extract_content(
@@ -492,29 +480,29 @@ class ScraplingMCPServer:
css_selector: Optional[str] = None, css_selector: Optional[str] = None,
main_content_only: bool = True, main_content_only: bool = True,
headless: bool = True, # noqa: F821 headless: bool = True, # noqa: F821
block_images: bool = False, google_search: bool = True,
real_chrome: bool = False,
wait: int | float = 0,
proxy: Optional[str | Dict[str, str]] = None,
timezone_id: str | None = None,
locale: str | None = None,
extra_headers: Optional[Dict[str, str]] = None,
useragent: Optional[str] = None,
hide_canvas: bool = False,
cdp_url: Optional[str] = None,
timeout: int | float = 30000,
disable_resources: bool = False, disable_resources: bool = False,
wait_selector: Optional[str] = None,
cookies: Sequence[SetCookieParam] | None = None,
network_idle: bool = False,
wait_selector_state: SelectorWaitStates = "attached",
block_webrtc: bool = False, block_webrtc: bool = False,
allow_webgl: bool = True, allow_webgl: bool = True,
network_idle: bool = False,
humanize: bool | float = True,
solve_cloudflare: bool = False, solve_cloudflare: bool = False,
wait: int | float = 0,
timeout: int | float = 30000,
wait_selector: Optional[str] = None,
addons: Optional[List[str]] = None,
wait_selector_state: SelectorWaitStates = "attached",
cookies: Optional[List[Dict]] = None,
google_search: bool = True,
extra_headers: Optional[Dict[str, str]] = None,
proxy: Optional[str | Dict[str, str]] = None,
os_randomize: bool = False,
disable_ads: bool = False,
geoip: bool = False,
additional_args: Optional[Dict] = None, additional_args: Optional[Dict] = None,
) -> List[ResponseModel]: ) -> List[ResponseModel]:
"""Use Scrapling's version of the Camoufox browser to fetch a group of URLs at the same time, and for each page return a structured output of the result. """Use the stealthy fetcher to fetch a group of URLs at the same time, and for each page return a structured output of the result.
Note: This is best suitable for high protection levels. It's slower than the other tools. Note: This is the only suitable fetcher for high protection levels.
Note: If the `css_selector` resolves to more than one element, all the elements will be returned. Note: If the `css_selector` resolves to more than one element, all the elements will be returned.
:param urls: A tuple of the URLs to request. :param urls: A tuple of the URLs to request.
@@ -525,54 +513,51 @@ class ScraplingMCPServer:
:param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None. :param css_selector: CSS selector to extract the content from the page. If main_content_only is True, then it will be executed on the main content of the page. Defaults to None.
:param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag. :param main_content_only: Whether to extract only the main content of the page. Defaults to True. The main content here is the data inside the `<body>` tag.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param block_webrtc: Blocks WebRTC entirely.
:param cookies: Set cookies for the next request. :param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you. :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled. :param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address. :param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
""" """
async with AsyncStealthySession( async with AsyncStealthySession(
wait=wait, wait=wait,
proxy=proxy, proxy=proxy,
geoip=geoip, locale=locale,
addons=addons, cdp_url=cdp_url,
timeout=timeout, timeout=timeout,
cookies=cookies, cookies=cookies,
headless=headless, headless=headless,
humanize=humanize, useragent=useragent,
max_pages=len(urls), timezone_id=timezone_id,
real_chrome=real_chrome,
hide_canvas=hide_canvas,
allow_webgl=allow_webgl, allow_webgl=allow_webgl,
disable_ads=disable_ads,
block_images=block_images,
block_webrtc=block_webrtc,
network_idle=network_idle, network_idle=network_idle,
os_randomize=os_randomize, block_webrtc=block_webrtc,
wait_selector=wait_selector, wait_selector=wait_selector,
google_search=google_search, google_search=google_search,
extra_headers=extra_headers, extra_headers=extra_headers,
additional_args=additional_args,
solve_cloudflare=solve_cloudflare, solve_cloudflare=solve_cloudflare,
disable_resources=disable_resources, disable_resources=disable_resources,
wait_selector_state=wait_selector_state, wait_selector_state=wait_selector_state,
additional_args=additional_args,
) as session: ) as session:
tasks = [session.fetch(url) for url in urls] tasks = [session.fetch(url) for url in urls]
responses = await gather(*tasks) responses = await gather(*tasks)
+1 -1
View File
@@ -130,5 +130,5 @@ translator = HTMLTranslator()
@lru_cache(maxsize=256) @lru_cache(maxsize=256)
def css_to_xpath(query: str) -> str: def css_to_xpath(query: str) -> str:
"""Return translated XPath version of a given CSS query""" """Return the translated XPath version of a given CSS query"""
return translator.css_to_xpath(query) return translator.css_to_xpath(query)
+113 -155
View File
@@ -1,7 +1,6 @@
from time import time from time import time
from asyncio import sleep as asyncio_sleep, Lock from asyncio import sleep as asyncio_sleep, Lock
from camoufox import DefaultAddons
from playwright.sync_api._generated import Page from playwright.sync_api._generated import Page
from playwright.sync_api import ( from playwright.sync_api import (
Frame, Frame,
@@ -17,18 +16,18 @@ from playwright.async_api import (
BrowserContext as AsyncBrowserContext, BrowserContext as AsyncBrowserContext,
) )
from playwright._impl._errors import Error as PlaywrightError from playwright._impl._errors import Error as PlaywrightError
from camoufox.pkgman import installed_verstr as camoufox_version
from camoufox.utils import launch_options as generate_launch_options
from ._page import PageInfo, PagePool from ._page import PageInfo, PagePool
from scrapling.parser import Selector from scrapling.parser import Selector
from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING from ._validators import validate, PlaywrightConfig, StealthConfig
from scrapling.engines.toolbelt.fingerprints import get_os_name from ._config_tools import __default_chrome_useragent__, __default_useragent__
from ._validators import validate, PlaywrightConfig, CamoufoxConfig
from ._config_tools import _compiled_stealth_scripts, _launch_kwargs, _context_kwargs
from scrapling.engines.toolbelt.navigation import intercept_route, async_intercept_route from scrapling.engines.toolbelt.navigation import intercept_route, async_intercept_route
from scrapling.core._types import Any, cast, Dict, List, Optional, Callable, TYPE_CHECKING, overload, Tuple
__ff_version_str__ = camoufox_version().split(".", 1)[0] from scrapling.engines.constants import (
DEFAULT_STEALTH_FLAGS,
HARMFUL_DEFAULT_ARGS,
DEFAULT_FLAGS,
)
class SyncSession: class SyncSession:
@@ -84,10 +83,6 @@ class SyncSession:
if disable_resources: if disable_resources:
page.route("**/*", intercept_route) page.route("**/*", intercept_route)
if getattr(self, "stealth", False):
for script in _compiled_stealth_scripts():
page.add_init_script(script=script)
page_info = self.page_pool.add_page(page) page_info = self.page_pool.add_page(page)
page_info.mark_busy() page_info.mark_busy()
return page_info return page_info
@@ -202,10 +197,6 @@ class AsyncSession:
if disable_resources: if disable_resources:
await page.route("**/*", async_intercept_route) await page.route("**/*", async_intercept_route)
if getattr(self, "stealth", False):
for script in _compiled_stealth_scripts():
await page.add_init_script(script=script)
return self.page_pool.add_page(page) return self.page_pool.add_page(page)
def get_pool_stats(self) -> Dict[str, int]: def get_pool_stats(self) -> Dict[str, int]:
@@ -251,151 +242,118 @@ class AsyncSession:
return handle_response return handle_response
class DynamicSessionMixin: class BaseSessionMixin:
def __validate__(self, **params): @overload
def __validate_routine__(self, params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
@overload
def __validate_routine__(self, params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
def __validate_routine__(
self, params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]
) -> PlaywrightConfig | StealthConfig:
# Dark color scheme bypasses the 'prefersLightColor' check in creepjs
self._context_options: Dict[str, Any] = {"color_scheme": "dark", "device_scale_factor": 2}
self._launch_options: Dict[str, Any] = self._context_options | {
"args": DEFAULT_FLAGS,
"ignore_default_args": HARMFUL_DEFAULT_ARGS,
}
if "__max_pages" in params: if "__max_pages" in params:
params["max_pages"] = params.pop("__max_pages") params["max_pages"] = params.pop("__max_pages")
config = validate(params, model=PlaywrightConfig) config = validate(params, model=model)
self._headers_keys = (
{header.lower() for header in config.extra_headers.keys()} if config.extra_headers else set()
)
self._max_pages = config.max_pages return config
self._headless = config.headless
self._hide_canvas = config.hide_canvas
self._disable_webgl = config.disable_webgl
self._real_chrome = config.real_chrome
self._stealth = config.stealth
self._google_search = config.google_search
self._wait = config.wait
self._proxy = config.proxy
self._locale = config.locale
self._extra_headers = config.extra_headers
self._useragent = config.useragent
self._timeout = config.timeout
self._cookies = config.cookies
self._disable_resources = config.disable_resources
self._cdp_url = config.cdp_url
self._network_idle = config.network_idle
self._load_dom = config.load_dom
self._wait_selector = config.wait_selector
self._init_script = config.init_script
self._wait_selector_state = config.wait_selector_state
self._extra_flags = config.extra_flags
self._selector_config = config.selector_config
self._timezone_id = config.timezone_id
self._additional_args = config.additional_args
self._page_action = config.page_action
self._user_data_dir = config.user_data_dir
self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
self.__initiate_browser_options__()
def __initiate_browser_options__(self): def __generate_options__(self, extra_flags: Tuple | None = None) -> None:
if TYPE_CHECKING: config = cast(PlaywrightConfig, getattr(self, "_config", None))
assert isinstance(self._proxy, tuple) self._context_options.update(
{
if not self._cdp_url: "proxy": config.proxy,
# `launch_options` is used with persistent context "locale": config.locale,
self.launch_options = dict( "timezone_id": config.timezone_id,
_launch_kwargs( "extra_http_headers": config.extra_headers,
self._headless,
self._proxy,
self._locale,
tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
self._useragent,
self._real_chrome,
self._stealth,
self._hide_canvas,
self._disable_webgl,
self._timezone_id,
tuple(self._extra_flags) if self._extra_flags else tuple(),
)
)
self.launch_options["extra_http_headers"] = dict(self.launch_options["extra_http_headers"])
self.launch_options["proxy"] = dict(self.launch_options["proxy"]) or None
self.launch_options["user_data_dir"] = self._user_data_dir
self.launch_options.update(cast(Dict, self._additional_args))
self.context_options = dict()
else:
# while `context_options` is left to be used when cdp mode is enabled
self.launch_options = dict()
self.context_options = dict(
_context_kwargs(
self._proxy,
self._locale,
tuple(self._extra_headers.items()) if self._extra_headers else tuple(),
self._useragent,
self._stealth,
)
)
self.context_options["extra_http_headers"] = dict(self.context_options["extra_http_headers"])
self.context_options["proxy"] = dict(self.context_options["proxy"]) or None
self.context_options.update(cast(Dict, self._additional_args))
class StealthySessionMixin:
def __validate__(self, **params):
if "__max_pages" in params:
params["max_pages"] = params.pop("__max_pages")
config: CamoufoxConfig = validate(params, model=CamoufoxConfig)
self._max_pages = config.max_pages
self._headless = config.headless
self._block_images = config.block_images
self._disable_resources = config.disable_resources
self._block_webrtc = config.block_webrtc
self._allow_webgl = config.allow_webgl
self._network_idle = config.network_idle
self._load_dom = config.load_dom
self._humanize = config.humanize
self._solve_cloudflare = config.solve_cloudflare
self._wait = config.wait
self._timeout = config.timeout
self._page_action = config.page_action
self._wait_selector = config.wait_selector
self._init_script = config.init_script
self._addons = config.addons
self._wait_selector_state = config.wait_selector_state
self._cookies = config.cookies
self._google_search = config.google_search
self._extra_headers = config.extra_headers
self._proxy = config.proxy
self._os_randomize = config.os_randomize
self._disable_ads = config.disable_ads
self._geoip = config.geoip
self._selector_config = config.selector_config
self._additional_args = config.additional_args
self._user_data_dir = config.user_data_dir
self._headers_keys = {header.lower() for header in self._extra_headers.keys()} if self._extra_headers else set()
self.__initiate_browser_options__()
def __initiate_browser_options__(self):
"""Initiate browser options."""
self.launch_options: Dict[str, Any] = generate_launch_options(
**{
"geoip": self._geoip,
"proxy": dict(self._proxy) if self._proxy and isinstance(self._proxy, tuple) else self._proxy,
"addons": self._addons,
"exclude_addons": [] if self._disable_ads else [DefaultAddons.UBO],
"headless": self._headless,
"humanize": True if self._solve_cloudflare else self._humanize,
"i_know_what_im_doing": True, # To turn warnings off with the user configurations
"allow_webgl": self._allow_webgl,
"block_webrtc": self._block_webrtc,
"block_images": self._block_images, # Careful! it makes some websites don't finish loading at all like stackoverflow even in headful mode.
"os": None if self._os_randomize else get_os_name(),
"user_data_dir": self._user_data_dir,
"ff_version": __ff_version_str__,
"firefox_user_prefs": {
# This is what enabling `enable_cache` does internally, so we do it from here instead
"browser.sessionhistory.max_entries": 10,
"browser.sessionhistory.max_total_viewers": -1,
"browser.cache.memory.enable": True,
"browser.cache.disk_cache_ssl": True,
"browser.cache.disk.smart_size.enabled": True,
},
**cast(Dict, self._additional_args),
} }
) )
# The default useragent in the headful is always correct now in the current versions of Playwright
if config.useragent:
self._context_options["user_agent"] = config.useragent
elif not config.useragent and config.headless:
self._context_options["user_agent"] = (
__default_chrome_useragent__ if config.real_chrome else __default_useragent__
)
if not config.cdp_url:
self._launch_options |= self._context_options
self._context_options = {}
flags = self._launch_options["args"]
if config.extra_flags or extra_flags:
flags = list(set(flags + (config.extra_flags or extra_flags)))
self._launch_options.update(
{
"args": flags,
"headless": config.headless,
"user_data_dir": config.user_data_dir,
"channel": "chrome" if config.real_chrome else "chromium",
}
)
if config.additional_args:
self._launch_options.update(config.additional_args)
else:
# while `context_options` is left to be used when cdp mode is enabled
self._launch_options = dict()
if config.additional_args:
self._context_options.update(config.additional_args)
class DynamicSessionMixin(BaseSessionMixin):
def __validate__(self, **params):
self._config = self.__validate_routine__(params, model=PlaywrightConfig)
self.__generate_options__()
class StealthySessionMixin(BaseSessionMixin):
def __validate__(self, **params):
self._config: StealthConfig = self.__validate_routine__(params, model=StealthConfig)
self._context_options.update(
{
"is_mobile": False,
"has_touch": False,
# I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
"service_workers": "allow",
"ignore_https_errors": True,
"screen": {"width": 1920, "height": 1080},
"viewport": {"width": 1920, "height": 1080},
"permissions": ["geolocation", "notifications"],
}
)
self.__generate_stealth_options()
def __generate_stealth_options(self) -> None:
flags = tuple()
if not self._config.cdp_url:
flags = DEFAULT_FLAGS + DEFAULT_STEALTH_FLAGS
if self._config.block_webrtc:
flags += (
"--webrtc-ip-handling-policy=disable_non_proxied_udp",
"--force-webrtc-ip-handling-policy", # Ensures the policy is enforced
)
if not self._config.allow_webgl:
flags += (
"--disable-webgl",
"--disable-webgl-image-chromium",
"--disable-webgl2",
)
if self._config.hide_canvas:
flags += ("--fingerprinting-canvas-image-data-noise",)
super(StealthySessionMixin, self).__generate_options__(flags)
@staticmethod @staticmethod
def _detect_cloudflare(page_content: str) -> str | None: def _detect_cloudflare(page_content: str) -> str | None:
+1 -102
View File
@@ -1,15 +1,10 @@
from functools import lru_cache from functools import lru_cache
from scrapling.core._types import Tuple
from scrapling.engines.constants import (
DEFAULT_STEALTH_FLAGS,
HARMFUL_DEFAULT_ARGS,
DEFAULT_FLAGS,
)
from scrapling.engines.toolbelt.navigation import js_bypass_path from scrapling.engines.toolbelt.navigation import js_bypass_path
from scrapling.engines.toolbelt.fingerprints import generate_headers from scrapling.engines.toolbelt.fingerprints import generate_headers
__default_useragent__ = generate_headers(browser_mode=True).get("User-Agent") __default_useragent__ = generate_headers(browser_mode=True).get("User-Agent")
__default_chrome_useragent__ = generate_headers(browser_mode="chrome").get("User-Agent")
@lru_cache(1) @lru_cache(1)
@@ -41,99 +36,3 @@ def _compiled_stealth_scripts():
with open(script_path, "r") as f: with open(script_path, "r") as f:
scripts.append(f.read()) scripts.append(f.read())
return tuple(scripts) return tuple(scripts)
@lru_cache(2, typed=True)
def _set_flags(hide_canvas, disable_webgl): # pragma: no cover
"""Returns the flags that will be used while launching the browser if stealth mode is enabled"""
flags = DEFAULT_STEALTH_FLAGS
if hide_canvas:
flags += ("--fingerprinting-canvas-image-data-noise",)
if disable_webgl:
flags += (
"--disable-webgl",
"--disable-webgl-image-chromium",
"--disable-webgl2",
)
return flags
@lru_cache(2, typed=True)
def _launch_kwargs(
headless,
proxy: Tuple,
locale,
extra_headers,
useragent,
real_chrome,
stealth,
hide_canvas,
disable_webgl,
timezone_id,
extra_flags: Tuple,
) -> Tuple:
"""Creates the arguments we will use while launching playwright's browser"""
base_args = DEFAULT_FLAGS
if extra_flags:
base_args = base_args + extra_flags
launch_kwargs = {
"locale": locale,
"timezone_id": timezone_id or None,
"headless": headless,
"args": base_args,
"color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
"proxy": proxy or tuple(),
"device_scale_factor": 2,
"ignore_default_args": HARMFUL_DEFAULT_ARGS,
"channel": "chrome" if real_chrome else "chromium",
"extra_http_headers": extra_headers or tuple(),
"user_agent": useragent or __default_useragent__,
}
if stealth:
stealth_args = base_args + _set_flags(hide_canvas, disable_webgl)
launch_kwargs.update(
{
"args": stealth_args,
"chromium_sandbox": True,
"is_mobile": False,
"has_touch": False,
# I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
"service_workers": "allow",
"ignore_https_errors": True,
"screen": {"width": 1920, "height": 1080},
"viewport": {"width": 1920, "height": 1080},
"permissions": ["geolocation", "notifications"],
}
)
return tuple(launch_kwargs.items())
@lru_cache(2, typed=True)
def _context_kwargs(proxy, locale, extra_headers, useragent, stealth) -> Tuple:
"""Creates the arguments for the browser context"""
context_kwargs = {
"proxy": proxy or tuple(),
"locale": locale,
"color_scheme": "dark", # Bypasses the 'prefersLightColor' check in creepjs
"device_scale_factor": 2,
"extra_http_headers": extra_headers or tuple(),
"user_agent": useragent or __default_useragent__,
}
if stealth:
context_kwargs.update(
{
"is_mobile": False,
"has_touch": False,
# I'm thinking about disabling it to rest from all Service Workers' headache, but let's keep it as it is for now
"service_workers": "allow",
"ignore_https_errors": True,
"screen": {"width": 1920, "height": 1080},
"viewport": {"width": 1920, "height": 1080},
"permissions": ["geolocation", "notifications"],
}
)
return tuple(context_kwargs.items())
+61 -104
View File
@@ -9,8 +9,6 @@ from playwright.async_api import (
Playwright as AsyncPlaywright, Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext, BrowserContext as AsyncBrowserContext,
) )
from patchright.sync_api import sync_playwright as sync_patchright
from patchright.async_api import async_playwright as async_patchright
from scrapling.core.utils import log from scrapling.core.utils import log
from scrapling.core._types import Unpack, TYPE_CHECKING from scrapling.core._types import Unpack, TYPE_CHECKING
@@ -21,53 +19,27 @@ from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
class DynamicSession(DynamicSessionMixin, SyncSession): class DynamicSession(SyncSession, DynamicSessionMixin):
"""A Browser session manager with page pooling.""" """A Browser session manager with page pooling."""
__slots__ = ( __slots__ = (
"_max_pages", "_config",
"_headless", "_context_options",
"_hide_canvas", "_launch_options",
"_disable_webgl", "max_pages",
"_real_chrome",
"_stealth",
"_google_search",
"_proxy",
"_locale",
"_extra_headers",
"_useragent",
"_timeout",
"_cookies",
"_disable_resources",
"_network_idle",
"_load_dom",
"_wait_selector",
"_init_script",
"_wait_selector_state",
"_wait",
"playwright",
"browser",
"context",
"page_pool", "page_pool",
"_max_wait_for_page",
"playwright",
"context",
"_closed", "_closed",
"_selector_config",
"_page_action",
"launch_options",
"context_options",
"_cdp_url",
"_headers_keys",
"_extra_flags",
"_additional_args",
"_user_data_dir",
) )
def __init__(self, **kwargs: Unpack[PlaywrightSession]): def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory. """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. :param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -76,13 +48,11 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session. :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`. :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
:param timezone_id: Set the timezone for the browser if wanted. rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute. :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
@@ -94,27 +64,24 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
""" """
self.__validate__(**kwargs) self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages) super().__init__()
def start(self): def start(self):
"""Create a browser for this instance and context.""" """Create a browser for this instance and context."""
if not self.playwright: if not self.playwright:
sync_context = sync_patchright if self._stealth else sync_playwright self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
self.playwright: Playwright = sync_context().start() # pyright: ignore [reportAttributeAccessIssue] if self._config.cdp_url: # pragma: no cover
browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
if self._cdp_url: # pragma: no cover self.context = browser.new_context(**self._context_options)
self.context = self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url).new_context(
**self.context_options
)
else: else:
self.context = self.playwright.chromium.launch_persistent_context(**self.launch_options) self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
if self._init_script: # pragma: no cover if self._config.init_script: # pragma: no cover
self.context.add_init_script(path=self._init_script) self.context.add_init_script(path=self._config.init_script)
if self._cookies: # pragma: no cover if self._config.cookies: # pragma: no cover
self.context.add_cookies(self._cookies) self.context.add_cookies(self._config.cookies)
else: else:
raise RuntimeError("Session has been already started") raise RuntimeError("Session has been already started")
@@ -122,24 +89,21 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
"""Opens up the browser and do your request based on your chosen options. """Opens up the browser and do your request based on your chosen options.
:param url: The Target url. :param url: The Target url.
:param kwargs: Additional keyword arguments including: :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- wait_selector: Wait for a specific CSS selector to be in a specific state. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- network_idle: Wait for the page until there are no network connections for at least 500 ms. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object. :return: A `Response` object.
""" """
params = _validate(kwargs, self, PlaywrightConfig) params = _validate(kwargs, self, PlaywrightConfig)
if self._closed: # pragma: no cover if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed") raise RuntimeError("Context manager has been closed")
@@ -193,16 +157,15 @@ class DynamicSession(DynamicSessionMixin, SyncSession):
raise e raise e
class AsyncDynamicSession(DynamicSessionMixin, AsyncSession): class AsyncDynamicSession(AsyncSession, DynamicSessionMixin):
"""An async Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.""" """An async Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory."""
def __init__(self, **kwargs: Unpack[PlaywrightSession]): def __init__(self, **kwargs: Unpack[PlaywrightSession]):
"""A Browser session manager with page pooling """A Browser session manager with page pooling
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request. :param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
@@ -212,13 +175,11 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session. :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Set the locale for the browser if wanted. The default value is `en-US`. :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
:param timezone_id: Set the timezone for the browser if wanted. rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
@@ -230,28 +191,26 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
""" """
self.__validate__(**kwargs) self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages) super().__init__(max_pages=self._config.max_pages)
async def start(self): async def start(self):
"""Create a browser for this instance and context.""" """Create a browser for this instance and context."""
if not self.playwright: if not self.playwright:
async_context = async_patchright if self._stealth else async_playwright self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
self.playwright: AsyncPlaywright = await async_context().start() # pyright: ignore [reportAttributeAccessIssue] if self._config.cdp_url:
browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
if self._cdp_url: self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._cdp_url)
self.context: AsyncBrowserContext = await browser.new_context(**self.context_options)
else: else:
self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context( self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
**self.launch_options **self._launch_options
) )
if self._init_script: # pragma: no cover if self._config.init_script: # pragma: no cover
await self.context.add_init_script(path=self._init_script) await self.context.add_init_script(path=self._config.init_script)
if self._cookies: if self._config.cookies:
await self.context.add_cookies(self._cookies) # pyright: ignore await self.context.add_cookies(self._config.cookies) # pyright: ignore
else: else:
raise RuntimeError("Session has been already started") raise RuntimeError("Session has been already started")
@@ -259,20 +218,18 @@ class AsyncDynamicSession(DynamicSessionMixin, AsyncSession):
"""Opens up the browser and do your request based on your chosen options. """Opens up the browser and do your request based on your chosen options.
:param url: The Target url. :param url: The Target url.
:param kwargs: Additional keyword arguments including: :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- wait_selector: Wait for a specific CSS selector to be in a specific state. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- network_idle: Wait for the page until there are no network connections for at least 500 ms. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object. :return: A `Response` object.
""" """
params = _validate(kwargs, self, PlaywrightConfig) params = _validate(kwargs, self, PlaywrightConfig)
@@ -2,117 +2,101 @@ from random import randint
from re import compile as re_compile from re import compile as re_compile
from playwright.sync_api import ( from playwright.sync_api import (
Page,
Locator, Locator,
sync_playwright, Page,
Playwright,
) )
from playwright.async_api import ( from playwright.async_api import (
async_playwright,
Page as async_Page, Page as async_Page,
Locator as AsyncLocator, Locator as AsyncLocator,
Playwright as AsyncPlaywright, Playwright as AsyncPlaywright,
BrowserContext as AsyncBrowserContext, BrowserContext as AsyncBrowserContext,
) )
from patchright.sync_api import sync_playwright
from patchright.async_api import async_playwright
from scrapling.core.utils import log from scrapling.core.utils import log
from ._types import CamoufoxSession, CamoufoxFetchParams from scrapling.core._types import Any, Unpack
from scrapling.core._types import Any, Unpack, TYPE_CHECKING from ._config_tools import _compiled_stealth_scripts
from ._types import StealthSession, StealthFetchParams
from ._base import SyncSession, AsyncSession, StealthySessionMixin from ._base import SyncSession, AsyncSession, StealthySessionMixin
from ._validators import validate_fetch as _validate, CamoufoxConfig from ._validators import validate_fetch as _validate, StealthConfig
from scrapling.engines.toolbelt.convertor import Response, ResponseFactory from scrapling.engines.toolbelt.convertor import Response, ResponseFactory
from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer from scrapling.engines.toolbelt.fingerprints import generate_convincing_referer
__CF_PATTERN__ = re_compile("challenges.cloudflare.com/cdn-cgi/challenge-platform/.*") __CF_PATTERN__ = re_compile("challenges.cloudflare.com/cdn-cgi/challenge-platform/.*")
class StealthySession(StealthySessionMixin, SyncSession): class StealthySession(SyncSession, StealthySessionMixin):
"""A Stealthy session manager with page pooling.""" """A Stealthy Browser session manager with page pooling."""
__slots__ = ( __slots__ = (
"_max_pages", "_config",
"_headless", "_context_options",
"_block_images", "_launch_options",
"_disable_resources", "max_pages",
"_block_webrtc",
"_allow_webgl",
"_network_idle",
"_load_dom",
"_humanize",
"_solve_cloudflare",
"_wait",
"_timeout",
"_page_action",
"_wait_selector",
"_init_script",
"_addons",
"_wait_selector_state",
"_cookies",
"_google_search",
"_extra_headers",
"_proxy",
"_os_randomize",
"_disable_ads",
"_geoip",
"_selector_config",
"_additional_args",
"playwright",
"browser",
"context",
"page_pool", "page_pool",
"_max_wait_for_page",
"playwright",
"context",
"_closed", "_closed",
"launch_options",
"_headers_keys",
"_user_data_dir",
) )
def __init__(self, **kwargs: Unpack[CamoufoxSession]): def __init__(self, **kwargs: Unpack[StealthSession]):
"""A Browser session manager with page pooling """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param block_webrtc: Blocks WebRTC entirely.
:param cookies: Set cookies for the next request. :param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session. :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address. :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
""" """
self.__validate__(**kwargs) self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages) super().__init__()
def start(self): def start(self):
"""Create a browser for this instance and context.""" """Create a browser for this instance and context."""
if not self.playwright: if not self.playwright:
self.playwright = sync_playwright().start() self.playwright: Playwright = sync_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
self.context = self.playwright.firefox.launch_persistent_context(**self.launch_options)
if self._init_script: # pragma: no cover if self._config.cdp_url: # pragma: no cover
self.context.add_init_script(path=self._init_script) browser = self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
self.context = browser.new_context(**self._context_options)
else:
self.context = self.playwright.chromium.launch_persistent_context(**self._launch_options)
if self._cookies: # pragma: no cover for script in _compiled_stealth_scripts():
self.context.add_cookies(self._cookies) self.context.add_init_script(script=script)
if self._config.init_script: # pragma: no cover
self.context.add_init_script(path=self._config.init_script)
if self._config.cookies: # pragma: no cover
self.context.add_cookies(self._config.cookies)
else: else:
raise RuntimeError("Session has been already started") raise RuntimeError("Session has been already started")
@@ -148,22 +132,27 @@ class StealthySession(StealthySessionMixin, SyncSession):
outer_box = {} outer_box = {}
iframe = page.frame(url=__CF_PATTERN__) iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None: if iframe is not None:
self._wait_for_page_stability(iframe, True, True) self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded": if challenge_type != "embedded":
while not iframe.frame_element().is_visible(): while not iframe.frame_element().is_visible():
# Double-checking that the iframe is loaded # Double-checking that the iframe is loaded
page.wait_for_timeout(500) page.wait_for_timeout(500)
outer_box: Any = iframe.frame_element().bounding_box() outer_box: Any = iframe.frame_element().bounding_box()
if not iframe or not outer_box: if not iframe or not outer_box:
if "<title>Just a moment...</title>" not in (ResponseFactory._get_page_content(page)):
log.info("Cloudflare captcha is solved")
return
outer_box: Any = page.locator(box_selector).last.bounding_box() outer_box: Any = page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport # Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27) captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button # Move the mouse to the center of the window, then press and hold the left mouse button
page.mouse.click(captcha_x, captcha_y, delay=60, button="left") page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
self._wait_for_networkidle(page) self._wait_for_networkidle(page)
if iframe is not None: if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms) # Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -182,29 +171,26 @@ class StealthySession(StealthySessionMixin, SyncSession):
log.info("Cloudflare captcha is solved") log.info("Cloudflare captcha is solved")
return return
def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response: def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options. """Opens up the browser and do your request based on your chosen options.
:param url: The Target url. :param url: The Target url.
:param kwargs: Additional keyword arguments including: :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- wait_selector: Wait for a specific CSS selector to be in a specific state. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- network_idle: Wait for the page until there are no network connections for at least 500 ms. :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object. :return: A `Response` object.
""" """
params = _validate(kwargs, self, CamoufoxConfig) params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed") raise RuntimeError("Context manager has been closed")
@@ -233,7 +219,7 @@ class StealthySession(StealthySessionMixin, SyncSession):
if params.page_action: if params.page_action:
try: try:
_ = params.page_action(page_info.page) _ = params.page_action(page_info.page)
except Exception as e: except Exception as e: # pragma: no cover
log.error(f"Error executing page_action: {e}") log.error(f"Error executing page_action: {e}")
if params.wait_selector: if params.wait_selector:
@@ -242,10 +228,12 @@ class StealthySession(StealthySessionMixin, SyncSession):
waiter.first.wait_for(state=params.wait_selector_state) waiter.first.wait_for(state=params.wait_selector_state)
# Wait again after waiting for the selector, helpful with protections like Cloudflare # Wait again after waiting for the selector, helpful with protections like Cloudflare
self._wait_for_page_stability(page_info.page, params.load_dom, params.network_idle) self._wait_for_page_stability(page_info.page, params.load_dom, params.network_idle)
except Exception as e: except Exception as e: # pragma: no cover
log.error(f"Error waiting for selector {params.wait_selector}: {e}") log.error(f"Error waiting for selector {params.wait_selector}: {e}")
page_info.page.wait_for_timeout(params.wait) page_info.page.wait_for_timeout(params.wait)
# Create response object
response = ResponseFactory.from_playwright_response( response = ResponseFactory.from_playwright_response(
page_info.page, first_response, final_response[0], params.selector_config page_info.page, first_response, final_response[0], params.selector_config
) )
@@ -256,72 +244,78 @@ class StealthySession(StealthySessionMixin, SyncSession):
return response return response
except Exception as e: # pragma: no cover except Exception as e:
page_info.mark_error() page_info.mark_error()
raise e raise e
class AsyncStealthySession(StealthySessionMixin, AsyncSession): class AsyncStealthySession(AsyncSession, StealthySessionMixin):
"""A Stealthy session manager with page pooling.""" """An async Stealthy Browser session manager with page pooling."""
def __init__(self, **kwargs: Unpack[CamoufoxSession]): def __init__(self, **kwargs: Unpack[StealthSession]):
"""A Browser session manager with page pooling """A Browser session manager with page pooling, it's using a persistent browser Context by default with a temporary user profile directory.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode. :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param block_images: Prevent the loading of images through Firefox preferences. :param disable_resources: Drop requests for unnecessary resources for a speed boost.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading.
:param disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param block_webrtc: Blocks WebRTC entirely.
:param cookies: Set cookies for the next request. :param cookies: Set cookies for the next request.
:param addons: List of Firefox addons to use. Must be paths to extracted addons.
:param humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement. The cursor typically takes up to 1.5 seconds to move across the window.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
:param os_randomize: If enabled, Scrapling will randomize the OS fingerprints used. The default is Scrapling matching the fingerprints with the current OS.
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session. :param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address. :param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
It will also calculate and spoof the browser's language based on the distribution of language speakers in the target region. rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param max_pages: The maximum number of tabs to be opened at the same time. It will be used in rotation through a PagePool.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory. :param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Camoufox as additional settings, and it takes higher priority than Scrapling's settings. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
""" """
self.__validate__(**kwargs) self.__validate__(**kwargs)
super().__init__(max_pages=self._max_pages) super().__init__(max_pages=self._config.max_pages)
async def start(self): async def start(self):
"""Create a browser for this instance and context.""" """Create a browser for this instance and context."""
if not self.playwright: if not self.playwright:
self.playwright: AsyncPlaywright = await async_playwright().start() self.playwright: AsyncPlaywright = await async_playwright().start() # pyright: ignore [reportAttributeAccessIssue]
self.context: AsyncBrowserContext = await self.playwright.firefox.launch_persistent_context(
**self.launch_options
)
if self._init_script: # pragma: no cover if self._config.cdp_url:
await self.context.add_init_script(path=self._init_script) browser = await self.playwright.chromium.connect_over_cdp(endpoint_url=self._config.cdp_url)
self.context: AsyncBrowserContext = await browser.new_context(**self._context_options)
else:
self.context: AsyncBrowserContext = await self.playwright.chromium.launch_persistent_context(
**self._launch_options
)
if self._cookies: for script in _compiled_stealth_scripts():
await self.context.add_cookies(self._cookies) # pyright: ignore [reportArgumentType] await self.context.add_init_script(script=script)
if self._config.init_script: # pragma: no cover
await self.context.add_init_script(path=self._config.init_script)
if self._config.cookies:
await self.context.add_cookies(self._config.cookies) # pyright: ignore
else: else:
raise RuntimeError("Session has been already started") raise RuntimeError("Session has been already started")
async def _cloudflare_solver(self, page: async_Page): # pragma: no cover async def _cloudflare_solver(self, page: async_Page) -> None: # pragma: no cover
"""Solve the cloudflare challenge displayed on the playwright page passed. The async version """Solve the cloudflare challenge displayed on the playwright page passed
:param page: The async targeted page :param page: The targeted page
:return: :return:
""" """
await self._wait_for_networkidle(page, timeout=5000) await self._wait_for_networkidle(page, timeout=5000)
@@ -331,7 +325,7 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
return return
else: else:
log.info(f'The turnstile version discovered is "{challenge_type}"') log.info(f'The turnstile version discovered is "{challenge_type}"')
if challenge_type == "non-interactive": # pragma: no cover if challenge_type == "non-interactive":
while "<title>Just a moment...</title>" in (await ResponseFactory._get_async_page_content(page)): while "<title>Just a moment...</title>" in (await ResponseFactory._get_async_page_content(page)):
log.info("Waiting for Cloudflare wait page to disappear.") log.info("Waiting for Cloudflare wait page to disappear.")
await page.wait_for_timeout(1000) await page.wait_for_timeout(1000)
@@ -350,22 +344,27 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
outer_box = {} outer_box = {}
iframe = page.frame(url=__CF_PATTERN__) iframe = page.frame(url=__CF_PATTERN__)
if iframe is not None: if iframe is not None:
await self._wait_for_page_stability(iframe, True, True) await self._wait_for_page_stability(iframe, True, False)
if challenge_type != "embedded": if challenge_type != "embedded":
while not await (await iframe.frame_element()).is_visible(): while not await (await iframe.frame_element()).is_visible():
# Double-checking that the iframe is loaded # Double-checking that the iframe is loaded
await page.wait_for_timeout(500) await page.wait_for_timeout(500)
outer_box: Any = await (await iframe.frame_element()).bounding_box() outer_box: Any = await (await iframe.frame_element()).bounding_box()
if not iframe or not outer_box: if not iframe or not outer_box:
if "<title>Just a moment...</title>" not in (await ResponseFactory._get_async_page_content(page)):
log.info("Cloudflare captcha is solved")
return
outer_box: Any = await page.locator(box_selector).last.bounding_box() outer_box: Any = await page.locator(box_selector).last.bounding_box()
# Calculate the Captcha coordinates for any viewport # Calculate the Captcha coordinates for any viewport
captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27) captcha_x, captcha_y = outer_box["x"] + randint(26, 28), outer_box["y"] + randint(25, 27)
# Move the mouse to the center of the window, then press and hold the left mouse button # Move the mouse to the center of the window, then press and hold the left mouse button
await page.mouse.click(captcha_x, captcha_y, delay=60, button="left") await page.mouse.click(captcha_x, captcha_y, delay=randint(100, 200), button="left")
await self._wait_for_networkidle(page) await self._wait_for_networkidle(page)
if iframe is not None: if iframe is not None:
# Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms) # Wait for the frame to be removed from the page (with 30s timeout = 300 iterations * 100 ms)
@@ -377,35 +376,33 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
await page.wait_for_timeout(100) await page.wait_for_timeout(100)
attempts += 1 attempts += 1
if challenge_type != "embedded": if challenge_type != "embedded":
await page.locator(box_selector).wait_for(state="detached") await page.locator(box_selector).last.wait_for(state="detached")
await page.locator(".zone-name-title").wait_for(state="hidden") await page.locator(".zone-name-title").wait_for(state="hidden")
await self._wait_for_page_stability(page, True, False) await self._wait_for_page_stability(page, True, False)
log.info("Cloudflare captcha is solved") log.info("Cloudflare captcha is solved")
return return
async def fetch(self, url: str, **kwargs: Unpack[CamoufoxFetchParams]) -> Response: async def fetch(self, url: str, **kwargs: Unpack[StealthFetchParams]) -> Response:
"""Opens up the browser and do your request based on your chosen options. """Opens up the browser and do your request based on your chosen options.
:param url: The Target url. :param url: The Target url.
:param kwargs: Additional keyword arguments including: :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
- extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._ :param disable_resources: Drop requests for unnecessary resources for a speed boost.
- disable_resources: Drop requests of unnecessary resources for a speed boost. It depends, but it made requests ~25% faster in my tests for some websites. Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
This can help save your proxy usage but be careful with this option as it makes some websites never finish loading. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- wait_selector: Wait for a specific CSS selector to be in a specific state. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- network_idle: Wait for the page until there are no network connections for at least 500 ms. :param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:return: A `Response` object. :return: A `Response` object.
""" """
params = _validate(kwargs, self, CamoufoxConfig) params = _validate(kwargs, self, StealthConfig)
if self._closed: # pragma: no cover if self._closed: # pragma: no cover
raise RuntimeError("Context manager has been closed") raise RuntimeError("Context manager has been closed")
@@ -418,10 +415,6 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
final_response = [None] final_response = [None]
handle_response = self._create_response_handler(page_info, final_response) handle_response = self._create_response_handler(page_info, final_response)
if TYPE_CHECKING:
if not isinstance(page_info.page, async_Page):
raise TypeError
try: try:
# Navigate to URL and wait for a specified state # Navigate to URL and wait for a specified state
page_info.page.on("response", handle_response) page_info.page.on("response", handle_response)
@@ -461,9 +454,8 @@ class AsyncStealthySession(StealthySessionMixin, AsyncSession):
# Close the page to free up resources # Close the page to free up resources
await page_info.page.close() await page_info.page.close()
self.page_pool.pages.remove(page_info) self.page_pool.pages.remove(page_info)
return response return response
except Exception as e: except Exception as e: # pragma: no cover
page_info.mark_error() page_info.mark_error()
raise e raise e
+19 -27
View File
@@ -11,9 +11,10 @@ from scrapling.core._types import (
Mapping, Mapping,
Optional, Optional,
Callable, Callable,
Iterable, Sequence,
TypedDict, TypedDict,
TypeAlias, TypeAlias,
SetCookieParam,
SelectorWaitStates, SelectorWaitStates,
TYPE_CHECKING, TYPE_CHECKING,
) )
@@ -53,7 +54,7 @@ if TYPE_CHECKING: # pragma: no cover
json: Optional[Dict | List] json: Optional[Dict | List]
# Types for browser session # Types for browser session
class BrowserSession(TypedDict, total=False): class PlaywrightSession(TypedDict, total=False):
max_pages: int max_pages: int
headless: bool headless: bool
disable_resources: bool disable_resources: bool
@@ -61,9 +62,10 @@ if TYPE_CHECKING: # pragma: no cover
load_dom: bool load_dom: bool
wait_selector: Optional[str] wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates wait_selector_state: SelectorWaitStates
cookies: Optional[Iterable[Dict]] cookies: Sequence[SetCookieParam] | None
google_search: bool google_search: bool
wait: int | float wait: int | float
timezone_id: str | None
page_action: Optional[Callable] page_action: Optional[Callable]
proxy: Optional[str | Dict[str, str] | Tuple] proxy: Optional[str | Dict[str, str] | Tuple]
extra_headers: Optional[Dict[str, str]] extra_headers: Optional[Dict[str, str]]
@@ -72,42 +74,32 @@ if TYPE_CHECKING: # pragma: no cover
user_data_dir: str user_data_dir: str
selector_config: Optional[Dict] selector_config: Optional[Dict]
additional_args: Optional[Dict] additional_args: Optional[Dict]
locale: Optional[str]
class PlaywrightSession(BrowserSession, total=False):
cdp_url: Optional[str]
hide_canvas: bool
disable_webgl: bool
real_chrome: bool real_chrome: bool
stealth: bool cdp_url: Optional[str]
locale: str
useragent: Optional[str] useragent: Optional[str]
extra_flags: Optional[List[str]] extra_flags: Optional[List[str]]
class PlaywrightFetchParams(TypedDict, total=False): class PlaywrightFetchParams(TypedDict, total=False):
load_dom: bool
wait: int | float
network_idle: bool
google_search: bool google_search: bool
timeout: int | float timeout: int | float
wait: int | float
page_action: Optional[Callable]
extra_headers: Optional[Dict[str, str]]
disable_resources: bool disable_resources: bool
wait_selector: Optional[str] wait_selector: Optional[str]
wait_selector_state: SelectorWaitStates page_action: Optional[Callable]
network_idle: bool
load_dom: bool
selector_config: Optional[Dict] selector_config: Optional[Dict]
extra_headers: Optional[Dict[str, str]]
wait_selector_state: SelectorWaitStates
class CamoufoxSession(BrowserSession, total=False): class StealthSession(PlaywrightSession, total=False):
block_images: bool
block_webrtc: bool
allow_webgl: bool allow_webgl: bool
humanize: bool | float hide_canvas: bool
block_webrtc: bool
solve_cloudflare: bool solve_cloudflare: bool
addons: Optional[List[str]]
os_randomize: bool
disable_ads: bool
geoip: bool
class CamoufoxFetchParams(PlaywrightFetchParams, total=False): class StealthFetchParams(PlaywrightFetchParams, total=False):
solve_cloudflare: bool solve_cloudflare: bool
else: # pragma: no cover else: # pragma: no cover
@@ -116,5 +108,5 @@ else: # pragma: no cover
DataRequestParams = TypedDict DataRequestParams = TypedDict
PlaywrightSession = TypedDict PlaywrightSession = TypedDict
PlaywrightFetchParams = TypedDict PlaywrightFetchParams = TypedDict
CamoufoxSession = TypedDict StealthSession = TypedDict
CamoufoxFetchParams = TypedDict StealthFetchParams = TypedDict
+39 -97
View File
@@ -13,12 +13,13 @@ from scrapling.core._types import (
Tuple, Tuple,
Optional, Optional,
Callable, Callable,
Iterable, Sequence,
SelectorWaitStates,
overload, overload,
SetCookieParam,
SelectorWaitStates,
) )
from scrapling.engines.toolbelt.navigation import construct_proxy_dict from scrapling.engines.toolbelt.navigation import construct_proxy_dict
from scrapling.engines._browsers._types import PlaywrightFetchParams, CamoufoxFetchParams from scrapling.engines._browsers._types import PlaywrightFetchParams, StealthFetchParams
# Custom validators for msgspec # Custom validators for msgspec
@@ -35,15 +36,6 @@ def _is_invalid_file_path(value: str) -> bool | str: # pragma: no cover
return False return False
def _validate_addon_path(value: str) -> None: # pragma: no cover
"""Fast addon path validation"""
path = Path(value)
if not path.exists():
raise FileNotFoundError(f"Addon path not found: {value}")
if not path.is_dir():
raise ValueError(f"Addon path must be a directory of the extracted addon: {value}")
@lru_cache(2) @lru_cache(2)
def _is_invalid_cdp_url(cdp_url: str) -> bool | str: def _is_invalid_cdp_url(cdp_url: str) -> bool | str:
"""Fast CDP URL validation""" """Fast CDP URL validation"""
@@ -65,39 +57,36 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
"""Configuration struct for validation""" """Configuration struct for validation"""
max_pages: PagesCount = 1 max_pages: PagesCount = 1
cdp_url: Optional[str] = None
headless: bool = True headless: bool = True
google_search: bool = True
hide_canvas: bool = False
disable_webgl: bool = False
real_chrome: bool = False
stealth: bool = False
wait: Seconds = 0
page_action: Optional[Callable] = None
proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
locale: str = "en-US"
extra_headers: Optional[Dict[str, str]] = None
useragent: Optional[str] = None
timeout: Seconds = 30000
init_script: Optional[str] = None
disable_resources: bool = False disable_resources: bool = False
wait_selector: Optional[str] = None
cookies: Optional[Iterable[Dict]] = None
network_idle: bool = False network_idle: bool = False
load_dom: bool = True load_dom: bool = True
wait_selector: Optional[str] = None
wait_selector_state: SelectorWaitStates = "attached" wait_selector_state: SelectorWaitStates = "attached"
cookies: Sequence[SetCookieParam] | None = []
google_search: bool = True
wait: Seconds = 0
timezone_id: str | None = ""
page_action: Optional[Callable] = None
proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
extra_headers: Optional[Dict[str, str]] = None
timeout: Seconds = 30000
init_script: Optional[str] = None
user_data_dir: str = "" user_data_dir: str = ""
timezone_id: str = ""
extra_flags: Optional[List[str]] = None
selector_config: Optional[Dict] = {} selector_config: Optional[Dict] = {}
additional_args: Optional[Dict] = {} additional_args: Optional[Dict] = {}
locale: str | None = None
real_chrome: bool = False
cdp_url: Optional[str] = None
useragent: Optional[str] = None
extra_flags: Optional[List[str]] = None
def __post_init__(self): # pragma: no cover def __post_init__(self): # pragma: no cover
"""Custom validation after msgspec validation""" """Custom validation after msgspec validation"""
if self.page_action and not callable(self.page_action): if self.page_action and not callable(self.page_action):
raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}") raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
if self.proxy: if self.proxy:
self.proxy = construct_proxy_dict(self.proxy, as_tuple=True) self.proxy = construct_proxy_dict(self.proxy)
if self.cdp_url: if self.cdp_url:
cdp_msg = _is_invalid_cdp_url(self.cdp_url) cdp_msg = _is_invalid_cdp_url(self.cdp_url)
if cdp_msg: if cdp_msg:
@@ -118,64 +107,18 @@ class PlaywrightConfig(Struct, kw_only=True, frozen=False, weakref=True):
raise ValueError(validation_msg) raise ValueError(validation_msg)
class CamoufoxConfig(Struct, kw_only=True, frozen=False, weakref=True): class StealthConfig(PlaywrightConfig, kw_only=True, frozen=False, weakref=True):
"""Configuration struct for validation"""
max_pages: PagesCount = 1
headless: bool = True # noqa: F821
block_images: bool = False
disable_resources: bool = False
block_webrtc: bool = False
allow_webgl: bool = True allow_webgl: bool = True
network_idle: bool = False hide_canvas: bool = False
load_dom: bool = True block_webrtc: bool = False
humanize: bool | float = True
solve_cloudflare: bool = False solve_cloudflare: bool = False
wait: Seconds = 0
timeout: Seconds = 30000
init_script: Optional[str] = None
page_action: Optional[Callable] = None
wait_selector: Optional[str] = None
addons: Optional[List[str]] = None
wait_selector_state: SelectorWaitStates = "attached"
cookies: Optional[Iterable[Dict]] = None
google_search: bool = True
extra_headers: Optional[Dict[str, str]] = None
proxy: Optional[str | Dict[str, str] | Tuple] = None # The default value for proxy in Playwright's source is `None`
os_randomize: bool = False
disable_ads: bool = False
geoip: bool = False
user_data_dir: str = ""
selector_config: Optional[Dict] = {}
additional_args: Optional[Dict] = {}
def __post_init__(self): def __post_init__(self):
"""Custom validation after msgspec validation""" """Custom validation after msgspec validation"""
if self.page_action and not callable(self.page_action): super(StealthConfig, self).__post_init__()
raise TypeError(f"page_action must be callable, got {type(self.page_action).__name__}")
if self.proxy:
self.proxy = construct_proxy_dict(self.proxy, as_tuple=True)
if self.addons:
for addon in self.addons:
_validate_addon_path(addon)
else:
self.addons = []
if self.init_script is not None:
validation_msg = _is_invalid_file_path(self.init_script)
if validation_msg:
raise ValueError(validation_msg)
if not self.cookies:
self.cookies = []
# Cloudflare timeout adjustment # Cloudflare timeout adjustment
if self.solve_cloudflare and self.timeout < 60_000: if self.solve_cloudflare and self.timeout < 60_000:
self.timeout = 60_000 self.timeout = 60_000
if not self.selector_config:
self.selector_config = {}
if not self.additional_args:
self.additional_args = {}
@dataclass @dataclass
@@ -197,9 +140,9 @@ class _fetch_params:
def validate_fetch( def validate_fetch(
method_kwargs: Dict | PlaywrightFetchParams | CamoufoxFetchParams, method_kwargs: Dict | PlaywrightFetchParams | StealthFetchParams,
session: Any, session: Any,
model: type[PlaywrightConfig] | type[CamoufoxConfig], model: type[PlaywrightConfig] | type[StealthConfig],
) -> _fetch_params: # pragma: no cover ) -> _fetch_params: # pragma: no cover
result = {} result = {}
overrides = {} overrides = {}
@@ -210,21 +153,20 @@ def validate_fetch(
for key in fetch_param_fields: for key in fetch_param_fields:
if key in method_kwargs: if key in method_kwargs:
overrides[key] = method_kwargs[key] overrides[key] = method_kwargs[key]
else: elif hasattr(session, "_config") and hasattr(session._config, key):
# Check for underscore-prefixed attribute (private) result[key] = getattr(session._config, key)
attr_name = f"_{key}"
if hasattr(session, attr_name):
result[key] = getattr(session, attr_name)
if overrides: if overrides:
validated_config = validate(overrides, model) validated_config = validate(overrides, model)
# Extract only the fields that _fetch_params needs from validated_config # Extract ONLY the fields that were actually overridden (not all fields)
# This prevents validated defaults from overwriting session config values
validated_dict = { validated_dict = {
f.name: getattr(validated_config, f.name) field: getattr(validated_config, field) for field in overrides.keys() if hasattr(validated_config, field)
for f in fields(_fetch_params)
if hasattr(validated_config, f.name)
} }
validated_dict.setdefault("solve_cloudflare", False)
# Preserve solve_cloudflare if the user explicitly provided it, even if the model doesn't have it
if "solve_cloudflare" in overrides:
validated_dict["solve_cloudflare"] = overrides["solve_cloudflare"]
# Start with session defaults, then overwrite with validated overrides # Start with session defaults, then overwrite with validated overrides
result.update(validated_dict) result.update(validated_dict)
@@ -238,7 +180,7 @@ def validate_fetch(
# Cache default values for each model to reduce validation overhead # Cache default values for each model to reduce validation overhead
models_default_values = {} models_default_values = {}
for _model in (CamoufoxConfig, PlaywrightConfig): for _model in (StealthConfig, PlaywrightConfig):
_defaults = {} _defaults = {}
if hasattr(_model, "__struct_defaults__") and hasattr(_model, "__struct_fields__"): if hasattr(_model, "__struct_defaults__") and hasattr(_model, "__struct_fields__"):
for field_name, default_value in zip(_model.__struct_fields__, _model.__struct_defaults__): # type: ignore for field_name, default_value in zip(_model.__struct_fields__, _model.__struct_defaults__): # type: ignore
@@ -256,14 +198,14 @@ def _filter_defaults(params: Dict, model: str) -> Dict:
@overload @overload
def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ... def validate(params: Dict, model: type[StealthConfig]) -> StealthConfig: ...
@overload @overload
def validate(params: Dict, model: type[CamoufoxConfig]) -> CamoufoxConfig: ... def validate(params: Dict, model: type[PlaywrightConfig]) -> PlaywrightConfig: ...
def validate(params: Dict, model: type[PlaywrightConfig] | type[CamoufoxConfig]) -> PlaywrightConfig | CamoufoxConfig: def validate(params: Dict, model: type[PlaywrightConfig] | type[StealthConfig]) -> PlaywrightConfig | StealthConfig:
try: try:
# Filter out params with the default values (no need to validate them) to speed up validation # Filter out params with the default values (no need to validate them) to speed up validation
filtered = _filter_defaults(params, model.__name__) filtered = _filter_defaults(params, model.__name__)
-3
View File
@@ -74,7 +74,6 @@ DEFAULT_STEALTH_FLAGS = (
"--disable-domain-reliability", "--disable-domain-reliability",
"--disable-threaded-animation", "--disable-threaded-animation",
"--disable-threaded-scrolling", "--disable-threaded-scrolling",
# '--disable-reading-from-canvas', # For Firefox
"--enable-simple-cache-backend", "--enable-simple-cache-backend",
"--disable-background-networking", "--disable-background-networking",
"--enable-surface-synchronization", "--enable-surface-synchronization",
@@ -94,9 +93,7 @@ DEFAULT_STEALTH_FLAGS = (
"--autoplay-policy=no-user-gesture-required", "--autoplay-policy=no-user-gesture-required",
"--disable-offer-store-unmasked-wallet-cards", "--disable-offer-store-unmasked-wallet-cards",
"--disable-blink-features=AutomationControlled", "--disable-blink-features=AutomationControlled",
"--webrtc-ip-handling-policy=disable_non_proxied_udp",
"--disable-component-extensions-with-background-pages", "--disable-component-extensions-with-background-pages",
"--force-webrtc-ip-handling-policy=disable_non_proxied_udp",
"--enable-features=NetworkService,NetworkServiceInProcess,TrustTokens,TrustTokensAlwaysAllowIssuance", "--enable-features=NetworkService,NetworkServiceInProcess,TrustTokens,TrustTokensAlwaysAllowIssuance",
"--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4", "--blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4",
"--disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process,TranslateUI,BlinkGenPropertyTrees", "--disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process,TranslateUI,BlinkGenPropertyTrees",
+21 -7
View File
@@ -13,19 +13,32 @@ from scrapling.core._types import Dict, Literal, Tuple
__OS_NAME__ = platform_system() __OS_NAME__ = platform_system()
OSName = Literal["linux", "macos", "windows"] OSName = Literal["linux", "macos", "windows"]
# Current versions hardcoded for now (Playwright doesn't allow to know the version of a browser without launching it)
chromium_version = 141
chrome_version = 143
@lru_cache(10, typed=True) @lru_cache(10, typed=True)
def generate_convincing_referer(url: str) -> str: def generate_convincing_referer(url: str) -> str | None:
"""Takes the domain from the URL without the subdomain/suffix and make it look like you were searching Google for this website """Takes the domain from the URL without the subdomain/suffix and make it look like you were searching Google for this website
>>> generate_convincing_referer('https://www.somewebsite.com/blah') >>> generate_convincing_referer('https://www.somewebsite.com/blah')
'https://www.google.com/search?q=somewebsite' 'https://www.google.com/search?q=somewebsite'
:param url: The URL you are about to fetch. :param url: The URL you are about to fetch.
:return: Google's search URL of the domain name :return: Google's search URL of the domain name, or None for localhost/IP addresses
""" """
website_name = extract(url).domain extracted = extract(url)
website_name = extracted.domain
# Skip generating referer for localhost, IP addresses, or when there's no valid domain
if not website_name or not extracted.suffix or website_name in ("localhost", "127.0.0.1", "::1"):
return None
# Check if it's an IP address (simple check for IPv4)
if all(part.isdigit() for part in website_name.split(".") if part):
return None
return f"https://www.google.com/search?q={website_name}" return f"https://www.google.com/search?q={website_name}"
@@ -46,7 +59,7 @@ def get_os_name() -> OSName | Tuple:
return SUPPORTED_OPERATING_SYSTEMS return SUPPORTED_OPERATING_SYSTEMS
def generate_headers(browser_mode: bool = False) -> Dict: def generate_headers(browser_mode: bool | str = False) -> Dict:
"""Generate real browser-like headers using browserforge's generator """Generate real browser-like headers using browserforge's generator
:param browser_mode: If enabled, the headers created are used for playwright, so it has to match everything :param browser_mode: If enabled, the headers created are used for playwright, so it has to match everything
@@ -55,13 +68,14 @@ def generate_headers(browser_mode: bool = False) -> Dict:
# In the browser mode, we don't care about anything other than matching the OS and the browser type with the browser we are using, # In the browser mode, we don't care about anything other than matching the OS and the browser type with the browser we are using,
# So we don't raise any inconsistency red flags while websites fingerprinting us # So we don't raise any inconsistency red flags while websites fingerprinting us
os_name = get_os_name() os_name = get_os_name()
browsers = [Browser(name="chrome", min_version=130)] ver = chrome_version if browser_mode and browser_mode == "chrome" else chromium_version
browsers = [Browser(name="chrome", min_version=ver, max_version=ver)]
if not browser_mode: if not browser_mode:
os_name = ("windows", "macos", "linux") os_name = ("windows", "macos", "linux")
browsers.extend( browsers.extend(
[ [
Browser(name="firefox", min_version=130), Browser(name="firefox", min_version=142),
Browser(name="edge", min_version=130), Browser(name="edge", min_version=140),
] ]
) )
return HeaderGenerator(browser=browsers, os=os_name, device="desktop").generate() return HeaderGenerator(browser=browsers, os=os_name, device="desktop").generate()
+4 -13
View File
@@ -11,7 +11,7 @@ from msgspec import Struct, structs, convert, ValidationError
from playwright.sync_api import Route from playwright.sync_api import Route
from scrapling.core.utils import log from scrapling.core.utils import log
from scrapling.core._types import Dict, Tuple, overload, Literal from scrapling.core._types import Dict, Tuple
from scrapling.engines.constants import DEFAULT_DISABLED_RESOURCES from scrapling.engines.constants import DEFAULT_DISABLED_RESOURCES
__BYPASSES_DIR__ = Path(__file__).parent / "bypasses" __BYPASSES_DIR__ = Path(__file__).parent / "bypasses"
@@ -49,20 +49,11 @@ async def async_intercept_route(route: async_Route):
await route.continue_() await route.continue_()
@overload def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple) -> Dict:
def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[True]) -> Tuple: ...
@overload
def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: Literal[False] = False) -> Dict: ...
def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: bool = False) -> Dict | Tuple:
"""Validate a proxy and return it in the acceptable format for Playwright """Validate a proxy and return it in the acceptable format for Playwright
Reference: https://playwright.dev/python/docs/network#http-proxy Reference: https://playwright.dev/python/docs/network#http-proxy
:param proxy_string: A string or a dictionary representation of the proxy. :param proxy_string: A string or a dictionary representation of the proxy.
:param as_tuple: Return the proxy dictionary as a tuple to be cachable
:return: :return:
""" """
if isinstance(proxy_string, str): if isinstance(proxy_string, str):
@@ -78,7 +69,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
} }
if proxy.port: if proxy.port:
result["server"] += f":{proxy.port}" result["server"] += f":{proxy.port}"
return tuple(result.items()) if as_tuple else result return result
except ValueError: except ValueError:
# Urllib will say that one of the parameters above can't be casted to the correct type like `int` for port etc... # Urllib will say that one of the parameters above can't be casted to the correct type like `int` for port etc...
raise ValueError("The proxy argument's string is in invalid format!") raise ValueError("The proxy argument's string is in invalid format!")
@@ -87,7 +78,7 @@ def construct_proxy_dict(proxy_string: str | Dict[str, str] | Tuple, as_tuple: b
try: try:
validated = convert(proxy_string, ProxyDict) validated = convert(proxy_string, ProxyDict)
result_dict = structs.asdict(validated) result_dict = structs.asdict(validated)
return tuple(result_dict.items()) if as_tuple else result_dict return result_dict
except ValidationError as e: except ValidationError as e:
raise TypeError(f"Invalid proxy dictionary: {e}") raise TypeError(f"Invalid proxy dictionary: {e}")
+4 -4
View File
@@ -3,7 +3,7 @@ from typing import TYPE_CHECKING, Any
if TYPE_CHECKING: if TYPE_CHECKING:
from scrapling.fetchers.requests import Fetcher, AsyncFetcher, FetcherSession from scrapling.fetchers.requests import Fetcher, AsyncFetcher, FetcherSession
from scrapling.fetchers.chrome import DynamicFetcher, DynamicSession, AsyncDynamicSession from scrapling.fetchers.chrome import DynamicFetcher, DynamicSession, AsyncDynamicSession
from scrapling.fetchers.firefox import StealthyFetcher, StealthySession, AsyncStealthySession from scrapling.fetchers.stealth_chrome import StealthyFetcher, StealthySession, AsyncStealthySession
# Lazy import mapping # Lazy import mapping
@@ -14,9 +14,9 @@ _LAZY_IMPORTS = {
"DynamicFetcher": ("scrapling.fetchers.chrome", "DynamicFetcher"), "DynamicFetcher": ("scrapling.fetchers.chrome", "DynamicFetcher"),
"DynamicSession": ("scrapling.fetchers.chrome", "DynamicSession"), "DynamicSession": ("scrapling.fetchers.chrome", "DynamicSession"),
"AsyncDynamicSession": ("scrapling.fetchers.chrome", "AsyncDynamicSession"), "AsyncDynamicSession": ("scrapling.fetchers.chrome", "AsyncDynamicSession"),
"StealthyFetcher": ("scrapling.fetchers.firefox", "StealthyFetcher"), "StealthyFetcher": ("scrapling.fetchers.stealth_chrome", "StealthyFetcher"),
"StealthySession": ("scrapling.fetchers.firefox", "StealthySession"), "StealthySession": ("scrapling.fetchers.stealth_chrome", "StealthySession"),
"AsyncStealthySession": ("scrapling.fetchers.firefox", "AsyncStealthySession"), "AsyncStealthySession": ("scrapling.fetchers.stealth_chrome", "AsyncStealthySession"),
} }
__all__ = [ __all__ = [
+49 -64
View File
@@ -5,51 +5,37 @@ from scrapling.engines._browsers._controllers import DynamicSession, AsyncDynami
class DynamicFetcher(BaseFetcher): class DynamicFetcher(BaseFetcher):
"""A `Fetcher` class type that provide many options, all of them are based on PlayWright. """A `Fetcher` that provide many options to fetch/load websites' pages through chromium-based browsers."""
Using this Fetcher class, you can do requests with:
- Vanilla Playwright without any modifications other than the ones you chose.
- Stealthy Playwright with the stealth mode I wrote for it. It's still a work in progress, but it bypasses many online tests like bot.sannysoft.com
Some of the things stealth mode does include:
1) Patches the CDP runtime fingerprint.
2) Mimics some of the real browsers' properties by injecting several JS files and using custom options.
3) Using custom flags on launch to hide Playwright even more and make it faster.
4) Generates real browser's headers of the same type and same user OS, then append it to the request.
- Real browsers by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
> Note that these are the main options with PlayWright, but it can be mixed.
"""
@classmethod @classmethod
def fetch(cls, url: str, **kwargs: Unpack[PlaywrightSession]) -> Response: def fetch(cls, url: str, **kwargs: Unpack[PlaywrightSession]) -> Response:
"""Opens up a browser and do your request based on your chosen options below. """Opens up a browser and do your request based on your chosen options below.
:param url: Target url. :param url: Target url.
:param kwargs: Browser session configuration options including: :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- headless: Run the browser in headless/hidden (default), or headful/visible mode. :param disable_resources: Drop requests of unnecessary resources for a speed boost.
- disable_resources: Drop requests of unnecessary resources for a speed boost. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. :param cookies: Set cookies for the next request.
- cookies: Set cookies for the next request. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- network_idle: Wait for the page until there are no network connections for at least 500 ms. :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute. :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
- wait_selector: Wait for a specific CSS selector to be in a specific state. :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request. :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- locale: Set the locale for the browser if wanted. The default value is `en-US`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- stealth: Enables stealth mode, check the documentation to see what stealth mode does currently. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- hide_canvas: Add random noise to canvas operations to prevent fingerprinting. :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- disable_webgl: Disables WebGL and WebGL 2.0 support entirely. :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param extra_headers: A dictionary of extra headers to add to the request.
- extra_headers: A dictionary of extra headers to add to the request. :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. :param extra_flags: A list of additional browser flags to pass to the browser on launch.
- extra_flags: A list of additional browser flags to pass to the browser on launch. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
- additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object. :return: A `Response` object.
""" """
selector_config = kwargs.get("selector_config", {}) or kwargs.get( selector_config = kwargs.get("selector_config", {}) or kwargs.get(
@@ -68,31 +54,30 @@ class DynamicFetcher(BaseFetcher):
"""Opens up a browser and do your request based on your chosen options below. """Opens up a browser and do your request based on your chosen options below.
:param url: Target url. :param url: Target url.
:param kwargs: Browser session configuration options including: :param headless: Run the browser in headless/hidden (default), or headful/visible mode.
- headless: Run the browser in headless/hidden (default), or headful/visible mode. :param disable_resources: Drop requests of unnecessary resources for a speed boost.
- disable_resources: Drop requests of unnecessary resources for a speed boost. :param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
- useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it. :param cookies: Set cookies for the next request.
- cookies: Set cookies for the next request. :param network_idle: Wait for the page until there are no network connections for at least 500 ms.
- network_idle: Wait for the page until there are no network connections for at least 500 ms. :param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute. :param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000 :param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object. :param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- page_action: Added for automation. A function that takes the `page` object and does the automation you need. :param wait_selector: Wait for a specific CSS selector to be in a specific state.
- wait_selector: Wait for a specific CSS selector to be in a specific state. :param init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request. :param locale: Set the locale for the browser if wanted. The default value is `en-US`.
- locale: Set the locale for the browser if wanted. The default value is `en-US`. :param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`. :param stealth: Enables stealth mode, check the documentation to see what stealth mode does currently.
- stealth: Enables stealth mode, check the documentation to see what stealth mode does currently. :param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
- real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. :param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
- hide_canvas: Add random noise to canvas operations to prevent fingerprinting. :param disable_webgl: Disables WebGL and WebGL 2.0 support entirely.
- disable_webgl: Disables WebGL and WebGL 2.0 support entirely. :param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
- cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP. :param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name. :param extra_headers: A dictionary of extra headers to add to the request.
- extra_headers: A dictionary of extra headers to add to the request. :param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only. :param extra_flags: A list of additional browser flags to pass to the browser on launch.
- extra_flags: A list of additional browser flags to pass to the browser on launch. :param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class. :param additional_args: Additional arguments to be passed to Playwright's context as additional settings.
- additional_args: Additional arguments to be passed to Playwright's context as additional settings.
:return: A `Response` object. :return: A `Response` object.
""" """
selector_config = kwargs.get("selector_config", {}) or kwargs.get( selector_config = kwargs.get("selector_config", {}) or kwargs.get(
-102
View File
@@ -1,102 +0,0 @@
from scrapling.core._types import Unpack
from scrapling.engines._browsers._types import CamoufoxSession
from scrapling.engines.toolbelt.custom import BaseFetcher, Response
from scrapling.engines._browsers._camoufox import StealthySession, AsyncStealthySession
class StealthyFetcher(BaseFetcher):
"""A `Fetcher` class type that is a completely stealthy fetcher that uses a modified version of Firefox.
It works as real browsers passing almost all online tests/protections based on Camoufox.
Other added flavors include setting the faked OS fingerprints to match the user's OS, and the referer of every request is set as if this request came from Google's search of this URL's domain.
"""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param kwargs: Browser session configuration options including:
- headless: Run the browser in headless/hidden (default), or headful/visible mode.
- block_images: Prevent the loading of images through Firefox preferences.
- disable_resources: Drop requests of unnecessary resources for a speed boost.
- block_webrtc: Blocks WebRTC entirely.
- allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- addons: List of Firefox addons to use. Must be paths to extracted addons.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- cookies: Set cookies for the next request.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- extra_headers: A dictionary of extra headers to add to the request.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- additional_args: Additional arguments to be passed to Camoufox as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
with StealthySession(**kwargs) as engine:
return engine.fetch(url)
@classmethod
async def async_fetch(cls, url: str, **kwargs: Unpack[CamoufoxSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param kwargs: Browser session configuration options including:
- headless: Run the browser in headless/hidden (default), or headful/visible mode.
- block_images: Prevent the loading of images through Firefox preferences.
- disable_resources: Drop requests of unnecessary resources for a speed boost.
- block_webrtc: Blocks WebRTC entirely.
- allow_webgl: Enabled by default. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
- network_idle: Wait for the page until there are no network connections for at least 500 ms.
- load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
- humanize: Humanize the cursor movement. Takes either True or the MAX duration in seconds of the cursor movement.
- solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
- wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the Response object.
- timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
- page_action: Added for automation. A function that takes the `page` object and does the automation you need.
- wait_selector: Wait for a specific CSS selector to be in a specific state.
- init_script: An absolute path to a JavaScript file to be executed on page creation with this request.
- addons: List of Firefox addons to use. Must be paths to extracted addons.
- wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
- cookies: Set cookies for the next request.
- google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
- extra_headers: A dictionary of extra headers to add to the request.
- proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
- os_randomize: If enabled, Scrapling will randomize the OS fingerprints used.
- disable_ads: Disabled by default, this installs the `uBlock Origin` addon on the browser if enabled.
- geoip: Recommended to use with proxies; Automatically use IP's longitude, latitude, timezone, country, locale, and spoof the WebRTC IP address.
- selector_config: The arguments that will be passed in the end while creating the final Selector's class.
- additional_args: Additional arguments to be passed to Camoufox as additional settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
async with AsyncStealthySession(**kwargs) as engine:
return await engine.fetch(url)
+107
View File
@@ -0,0 +1,107 @@
from scrapling.core._types import Unpack
from scrapling.engines._browsers._types import StealthSession
from scrapling.engines.toolbelt.custom import BaseFetcher, Response
from scrapling.engines._browsers._stealth import StealthySession, AsyncStealthySession
class StealthyFetcher(BaseFetcher):
"""A `Fetcher` class type which is a completely stealthy built on top of Chromium.
It works as real browsers passing almost all online tests/protections with many customization options.
"""
@classmethod
def fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
with StealthySession(**kwargs) as engine:
return engine.fetch(url)
@classmethod
async def async_fetch(cls, url: str, **kwargs: Unpack[StealthSession]) -> Response:
"""
Opens up a browser and do your request based on your chosen options below.
:param url: Target url.
:param headless: Run the browser in headless/hidden (default), or headful/visible mode.
:param disable_resources: Drop requests for unnecessary resources for a speed boost.
Requests dropped are of type `font`, `image`, `media`, `beacon`, `object`, `imageset`, `texttrack`, `websocket`, `csp_report`, and `stylesheet`.
:param useragent: Pass a useragent string to be used. Otherwise the fetcher will generate a real Useragent of the same browser and use it.
:param cookies: Set cookies for the next request.
:param network_idle: Wait for the page until there are no network connections for at least 500 ms.
:param timeout: The timeout in milliseconds that is used in all operations and waits through the page. The default is 30,000
:param wait: The time (milliseconds) the fetcher will wait after everything finishes before closing the page and returning the ` Response ` object.
:param page_action: Added for automation. A function that takes the `page` object and does the automation you need.
:param wait_selector: Wait for a specific CSS selector to be in a specific state.
:param init_script: An absolute path to a JavaScript file to be executed on page creation for all pages in this session.
:param locale: Specify user locale, for example, `en-GB`, `de-DE`, etc. Locale will affect navigator.language value, Accept-Language request header value as well as number and date formatting
rules. Defaults to the system default locale.
:param timezone_id: Changes the timezone of the browser. Defaults to the system timezone.
:param wait_selector_state: The state to wait for the selector given with `wait_selector`. The default state is `attached`.
:param solve_cloudflare: Solves all types of the Cloudflare's Turnstile/Interstitial challenges before returning the response to you.
:param real_chrome: If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it.
:param hide_canvas: Add random noise to canvas operations to prevent fingerprinting.
:param block_webrtc: Forces WebRTC to respect proxy settings to prevent local IP address leak.
:param allow_webgl: Enabled by default. Disabling it disables WebGL and WebGL 2.0 support entirely. Disabling WebGL is not recommended as many WAFs now check if WebGL is enabled.
:param load_dom: Enabled by default, wait for all JavaScript on page(s) to fully load and execute.
:param cdp_url: Instead of launching a new browser instance, connect to this CDP URL to control real browsers through CDP.
:param google_search: Enabled by default, Scrapling will set the referer header to be as if this request came from a Google search of this website's domain name.
:param extra_headers: A dictionary of extra headers to add to the request. _The referer set by the `google_search` argument takes priority over the referer set here if used together._
:param proxy: The proxy to be used with requests, it can be a string or a dictionary with the keys 'server', 'username', and 'password' only.
:param user_data_dir: Path to a User Data Directory, which stores browser session data like cookies and local storage. The default is to create a temporary directory.
:param extra_flags: A list of additional browser flags to pass to the browser on launch.
:param selector_config: The arguments that will be passed in the end while creating the final Selector's class.
:param additional_args: Additional arguments to be passed to Playwright's context as additional settings, and it takes higher priority than Scrapling's settings.
:return: A `Response` object.
"""
selector_config = kwargs.get("selector_config", {}) or kwargs.get(
"custom_config", {}
) # Checking `custom_config` for backward compatibility
if not isinstance(selector_config, dict):
raise TypeError("Argument `selector_config` must be a dictionary.")
kwargs["selector_config"] = {**cls._generate_parser_arguments(), **selector_config}
async with AsyncStealthySession(**kwargs) as engine:
return await engine.fetch(url)
+1 -1
View File
@@ -1,6 +1,6 @@
[metadata] [metadata]
name = scrapling name = scrapling
version = 0.3.12 version = 0.3.13
author = Karim Shoair author = Karim Shoair
author_email = karim.shoair@pm.me author_email = karim.shoair@pm.me
description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be! description = Scrapling is an undetectable, powerful, flexible, high-performance Python library that makes Web Scraping easy and effortless as it should be!
-1
View File
@@ -156,7 +156,6 @@ class TestCLI:
html_url, html_url,
str(output_file), str(output_file),
'--headless', '--headless',
'--stealth',
'--timeout', '60000' '--timeout', '60000'
] ]
) )
+1 -4
View File
@@ -53,10 +53,7 @@ class TestDynamicFetcherAsync:
@pytest.mark.parametrize( @pytest.mark.parametrize(
"kwargs", "kwargs",
[ [
{"disable_webgl": True, "hide_canvas": False}, {"real_chrome": True, "disable_resources": True},
{"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
{"stealth": True}, # causes issues with GitHub Actions
{"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
{"wait_selector": "h1", "wait_selector_state": "attached"}, {"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"}, {"wait_selector": "h1", "wait_selector_state": "visible"},
{ {
@@ -71,7 +71,6 @@ class TestAsyncDynamicSession:
"""Test AsyncDynamicSession with various options""" """Test AsyncDynamicSession with various options"""
async with AsyncDynamicSession( async with AsyncDynamicSession(
headless=False, headless=False,
stealth=True,
disable_resources=True, disable_resources=True,
extra_headers={"X-Test": "value"} extra_headers={"X-Test": "value"}
) as session: ) as session:
@@ -1,4 +1,3 @@
import pytest
from scrapling.engines.static import AsyncFetcherClient from scrapling.engines.static import AsyncFetcherClient
@@ -1,4 +1,3 @@
from playwright._impl._errors import TimeoutError
import pytest import pytest
import pytest_httpbin import pytest_httpbin
@@ -30,8 +29,8 @@ class TestStealthyFetcher:
async def test_basic_fetch(self, fetcher, urls): async def test_basic_fetch(self, fetcher, urls):
"""Test doing a basic fetch request with multiple statuses""" """Test doing a basic fetch request with multiple statuses"""
assert (await fetcher.async_fetch(urls["status_200"])).status == 200 assert (await fetcher.async_fetch(urls["status_200"])).status == 200
assert (await fetcher.async_fetch(urls["status_404"])).status == 404 # assert (await fetcher.async_fetch(urls["status_404"])).status == 404
assert (await fetcher.async_fetch(urls["status_501"])).status == 501 # assert (await fetcher.async_fetch(urls["status_501"])).status == 501
async def test_cookies_loading(self, fetcher, urls): async def test_cookies_loading(self, fetcher, urls):
"""Test if cookies are set after the request""" """Test if cookies are set after the request"""
@@ -55,10 +54,9 @@ class TestStealthyFetcher:
@pytest.mark.parametrize( @pytest.mark.parametrize(
"kwargs", "kwargs",
[ [
{"block_webrtc": True, "allow_webgl": True, "disable_ads": False}, {"block_webrtc": True, "allow_webgl": True},
{"block_webrtc": False, "allow_webgl": True, "block_images": True}, {"block_webrtc": False, "allow_webgl": True},
{"block_webrtc": True, "allow_webgl": False, "disable_resources": True}, {"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
{"block_images": True, "disable_resources": True, },
{"wait_selector": "h1", "wait_selector_state": "attached"}, {"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"}, {"wait_selector": "h1", "wait_selector_state": "visible"},
{ {
@@ -67,11 +65,8 @@ class TestStealthyFetcher:
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}], "cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"google_search": True, "google_search": True,
"extra_headers": {"ayo": ""}, "extra_headers": {"ayo": ""},
"os_randomize": True,
"disable_ads": True,
# "geoip": True,
"selector_config": {"keep_comments": False, "keep_cdata": False}, "selector_config": {"keep_comments": False, "keep_cdata": False},
"additional_args": {"window": (1920, 1080)}, "additional_args": {},
}, },
], ],
) )
@@ -72,9 +72,8 @@ class TestAsyncStealthySession:
"""Test AsyncStealthySession with various options""" """Test AsyncStealthySession with various options"""
async with AsyncStealthySession( async with AsyncStealthySession(
max_pages=1, max_pages=1,
block_images=True, block_webrtc=True,
disable_ads=True, allow_webgl=True
humanize=True
) as session: ) as session:
response = await session.fetch(urls["html"]) response = await session.fetch(urls["html"])
assert response.status == 200 assert response.status == 200
-79
View File
@@ -1,79 +0,0 @@
import pytest
import pytest_httpbin
from scrapling import StealthyFetcher
StealthyFetcher.adaptive = True
@pytest_httpbin.use_class_based_httpbin
class TestStealthyFetcher:
@pytest.fixture(scope="class")
def fetcher(self):
"""Fixture to create a StealthyFetcher instance for the entire test class"""
return StealthyFetcher
@pytest.fixture(autouse=True)
def setup_urls(self, httpbin):
"""Fixture to set up URLs for testing"""
self.status_200 = f"{httpbin.url}/status/200"
self.status_404 = f"{httpbin.url}/status/404"
self.status_501 = f"{httpbin.url}/status/501"
self.basic_url = f"{httpbin.url}/get"
self.html_url = f"{httpbin.url}/html"
self.delayed_url = f"{httpbin.url}/delay/10" # 10 Seconds delay response
self.cookies_url = f"{httpbin.url}/cookies/set/test/value"
def test_basic_fetch(self, fetcher):
"""Test doing a basic fetch request with multiple statuses"""
assert fetcher.fetch(self.status_200).status == 200
assert fetcher.fetch(self.status_404).status == 404
assert fetcher.fetch(self.status_501).status == 501
def test_cookies_loading(self, fetcher):
"""Test if cookies are set after the request"""
response = fetcher.fetch(self.cookies_url)
cookies = {response.cookies[0]['name']: response.cookies[0]['value']}
assert cookies == {"test": "value"}
def test_automation(self, fetcher):
"""Test if automation breaks the code or not"""
def scroll_page(page):
page.mouse.wheel(10, 0)
page.mouse.move(100, 400)
page.mouse.up()
return page
assert fetcher.fetch(self.html_url, page_action=scroll_page).status == 200
@pytest.mark.parametrize(
"kwargs",
[
{"block_webrtc": True, "allow_webgl": True, "disable_ads": False},
{"block_webrtc": False, "allow_webgl": True, "block_images": True},
{"block_webrtc": True, "allow_webgl": False, "disable_resources": True},
{"block_images": True, "disable_resources": True, },
{"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"},
{
"network_idle": True,
"wait": 10,
"timeout": 30_000,
"cookies": [{"name": "test", "value": "123", "domain": "example.com", "path": "/"}],
"google_search": True,
"extra_headers": {"ayo": ""},
"os_randomize": True,
"disable_ads": True,
# "geoip": True,
"selector_config": {"keep_comments": False, "keep_cdata": False},
"additional_args": {"window": (1920, 1080)},
},
],
)
def test_properties(self, fetcher, kwargs):
"""Test if different arguments break the code or not"""
response = fetcher.fetch(
self.html_url,
**kwargs
)
assert response.status == 200
+1 -4
View File
@@ -51,10 +51,7 @@ class TestDynamicFetcher:
@pytest.mark.parametrize( @pytest.mark.parametrize(
"kwargs", "kwargs",
[ [
{"disable_webgl": True, "hide_canvas": False}, {"disable_resources": True, "real_chrome": True},
{"disable_webgl": False, "hide_canvas": True, "disable_resources": True},
{"stealth": True}, # causes issues with GitHub Actions
{"stealth": True, "real_chrome": True}, # causes issues with GitHub Actions
{"wait_selector": "h1", "wait_selector_state": "attached"}, {"wait_selector": "h1", "wait_selector_state": "attached"},
{"wait_selector": "h1", "wait_selector_state": "visible"}, {"wait_selector": "h1", "wait_selector_state": "visible"},
{ {
@@ -2,11 +2,11 @@ import re
import pytest import pytest
import pytest_httpbin import pytest_httpbin
from scrapling.engines._browsers._camoufox import StealthySession, __CF_PATTERN__ from scrapling.engines._browsers._stealth import StealthySession, __CF_PATTERN__
class TestCamoufoxConstants: class TestStealthConstants:
"""Test Camoufox constants and patterns""" """Test Stealth constants and patterns"""
def test_cf_pattern_regex(self): def test_cf_pattern_regex(self):
"""Test __CF_PATTERN__ regex compilation""" """Test __CF_PATTERN__ regex compilation"""
@@ -54,7 +54,6 @@ class TestStealthySession:
with StealthySession( with StealthySession(
headless=True, headless=True,
block_images=True,
disable_resources=True, disable_resources=True,
solve_cloudflare=True, solve_cloudflare=True,
wait=1000, wait=1000,
@@ -63,12 +62,11 @@ class TestStealthySession:
) as session: ) as session:
assert session.max_pages == 1 assert session.max_pages == 1
assert session._headless is True assert session._config.headless is True
assert session._block_images is True assert session._config.disable_resources is True
assert session._disable_resources is True assert session._config.solve_cloudflare is True
assert session._solve_cloudflare is True assert session._config.wait == 1000
assert session._wait == 1000 assert session._config.timeout == 60000
assert session._timeout == 60000
assert session.context is not None assert session.context is not None
# Test Cloudflare detection # Test Cloudflare detection
-8
View File
@@ -190,14 +190,6 @@ class TestConstructProxyDict:
} }
assert result == expected assert result == expected
def test_proxy_as_tuple(self):
"""Test returning proxy as a tuple"""
result = construct_proxy_dict("http://proxy.example.com:8080", as_tuple=True)
assert isinstance(result, tuple)
result_dict = dict(result)
assert result_dict["server"] == "http://proxy.example.com:8080"
def test_invalid_proxy_string(self): def test_invalid_proxy_string(self):
"""Test invalid proxy string""" """Test invalid proxy string"""
with pytest.raises(ValueError): with pytest.raises(ValueError):
+8 -8
View File
@@ -1,8 +1,8 @@
import pytest import pytest
from scrapling.engines._browsers._validators import ( from scrapling.engines._browsers._validators import (
validate, validate,
StealthConfig,
PlaywrightConfig, PlaywrightConfig,
CamoufoxConfig
) )
@@ -23,7 +23,7 @@ class TestValidators:
assert config.max_pages == 2 assert config.max_pages == 2
assert config.headless is True assert config.headless is True
assert config.timeout == 30000 assert config.timeout == 30000
assert isinstance(config.proxy, tuple) # Should be converted to tuple assert isinstance(config.proxy, dict)
def test_playwright_config_invalid_max_pages(self): def test_playwright_config_invalid_max_pages(self):
"""Test PlaywrightConfig with invalid max_pages""" """Test PlaywrightConfig with invalid max_pages"""
@@ -51,8 +51,8 @@ class TestValidators:
with pytest.raises(TypeError): with pytest.raises(TypeError):
validate(params, PlaywrightConfig) validate(params, PlaywrightConfig)
def test_camoufox_config_valid(self): def test_stealth_config_valid(self):
"""Test valid CamoufoxConfig""" """Test valid StealthConfig"""
params = { params = {
"max_pages": 1, "max_pages": 1,
"headless": True, "headless": True,
@@ -60,20 +60,20 @@ class TestValidators:
"timeout": 30000 "timeout": 30000
} }
config = validate(params, CamoufoxConfig) config = validate(params, StealthConfig)
assert config.max_pages == 1 assert config.max_pages == 1
assert config.headless is True assert config.headless is True
assert config.solve_cloudflare is False assert config.solve_cloudflare is False
assert config.timeout == 30000 assert config.timeout == 30000
def test_camoufox_config_cloudflare_timeout(self): def test_stealth_config_cloudflare_timeout(self):
"""Test CamoufoxConfig timeout adjustment for Cloudflare""" """Test StealthConfig timeout adjustment for Cloudflare"""
params = { params = {
"solve_cloudflare": True, "solve_cloudflare": True,
"timeout": 10000 # Less than the required 60,000 "timeout": 10000 # Less than the required 60,000
} }
config = validate(params, CamoufoxConfig) config = validate(params, StealthConfig)
assert config.timeout == 60000 # Should be increased assert config.timeout == 60000 # Should be increased
-1
View File
@@ -1,7 +1,6 @@
pytest>=2.8.0,<9 pytest>=2.8.0,<9
pytest-cov pytest-cov
playwright playwright
camoufox
werkzeug<3.0.0 werkzeug<3.0.0
pytest-httpbin==2.1.0 pytest-httpbin==2.1.0
pytest-asyncio pytest-asyncio
+2 -3
View File
@@ -10,9 +10,8 @@ envlist = pre-commit,py{310,311,312,313}
usedevelop = True usedevelop = True
changedir = tests changedir = tests
deps = deps =
playwright>=1.56.0 playwright==1.56.0
patchright>=1.56.0 patchright==1.56.0
camoufox>=0.4.11
-r{toxinidir}/tests/requirements.txt -r{toxinidir}/tests/requirements.txt
extras = ai,shell extras = ai,shell
commands = commands =