docs: style adjustment
This commit is contained in:
Karim shoair
+7
-7
@@ -125,7 +125,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> يوفر أكثر من 900 واجهة API مستقرة عبر أكثر من 16 منصة تشمل TikTok و X و YouTube و Instagram، مع أكثر من 40 مليون مجموعة بيانات. <br /> يقدم أيضاً <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">نماذج ذكاء اصطناعي بأسعار مخفضة</a> — Claude و GPT و GEMINI والمزيد بخصم يصل إلى 71%.
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> يوفر أكثر من 900 واجهة API مستقرة عبر أكثر من 16 منصة تشمل TikTok و X و YouTube و Instagram، مع أكثر من 40 مليون مجموعة بيانات. <br /> يقدم أيضاً <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">نماذج ذكاء اصطناعي بأسعار مخفضة</a> - Claude و GPT و GEMINI والمزيد بخصم يصل إلى 71%.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -191,12 +191,12 @@ MySpider().start()
|
||||
|
||||
## الميزات الرئيسية
|
||||
|
||||
### Spiders — إطار عمل زحف كامل
|
||||
### Spiders - إطار عمل زحف كامل
|
||||
- 🕷️ **واجهة Spider شبيهة بـ Scrapy**: عرّف Spiders مع `start_urls`، و async `parse` callbacks، وكائنات `Request`/`Response`.
|
||||
- ⚡ **زحف متزامن**: حدود تزامن قابلة للتكوين، وتحكم بالسرعة حسب النطاق، وتأخيرات التنزيل.
|
||||
- 🔄 **دعم الجلسات المتعددة**: واجهة موحدة لطلبات HTTP، ومتصفحات خفية بدون واجهة في Spider واحد — وجّه الطلبات إلى جلسات مختلفة بالمعرّف.
|
||||
- 🔄 **دعم الجلسات المتعددة**: واجهة موحدة لطلبات HTTP، ومتصفحات خفية بدون واجهة في Spider واحد - وجّه الطلبات إلى جلسات مختلفة بالمعرّف.
|
||||
- 💾 **إيقاف واستئناف**: استمرارية الزحف القائمة على Checkpoint. اضغط Ctrl+C للإيقاف بسلاسة؛ أعد التشغيل للاستئناف من حيث توقفت.
|
||||
- 📡 **وضع Streaming**: بث العناصر المستخرجة فور وصولها عبر `async for item in spider.stream()` مع إحصائيات فورية — مثالي لواجهات المستخدم وخطوط الأنابيب وعمليات الزحف الطويلة.
|
||||
- 📡 **وضع Streaming**: بث العناصر المستخرجة فور وصولها عبر `async for item in spider.stream()` مع إحصائيات فورية - مثالي لواجهات المستخدم وخطوط الأنابيب وعمليات الزحف الطويلة.
|
||||
- 🛡️ **كشف الطلبات المحظورة**: كشف تلقائي وإعادة محاولة للطلبات المحظورة مع منطق قابل للتخصيص.
|
||||
- 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو JSON/JSONL المدمج مع `result.items.to_json()` / `result.items.to_jsonl()` على التوالي.
|
||||
|
||||
@@ -323,7 +323,7 @@ class MultiSessionSpider(Spider):
|
||||
```python
|
||||
QuotesSpider(crawldir="./crawl_data").start()
|
||||
```
|
||||
اضغط Ctrl+C للإيقاف بسلاسة — يتم حفظ التقدم تلقائياً. لاحقاً، عند تشغيل Spider مرة أخرى، مرر نفس `crawldir`، وسيستأنف من حيث توقف.
|
||||
اضغط Ctrl+C للإيقاف بسلاسة - يتم حفظ التقدم تلقائياً. لاحقاً، عند تشغيل Spider مرة أخرى، مرر نفس `crawldir`، وسيستأنف من حيث توقف.
|
||||
|
||||
### التحليل المتقدم والتنقل
|
||||
```python
|
||||
@@ -408,7 +408,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
|
||||
|
||||
## معايير الأداء
|
||||
|
||||
Scrapling ليس قوياً فحسب — بل هو أيضاً سريع بشكل مذهل. تقارن المعايير التالية محلل Scrapling مع أحدث إصدارات المكتبات الشائعة الأخرى.
|
||||
Scrapling ليس قوياً فحسب - بل هو أيضاً سريع بشكل مذهل. تقارن المعايير التالية محلل Scrapling مع أحدث إصدارات المكتبات الشائعة الأخرى.
|
||||
|
||||
### اختبار سرعة استخراج النص (5000 عنصر متداخل)
|
||||
|
||||
@@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
|
||||
## الشكر والتقدير
|
||||
|
||||
يتضمن هذا المشروع كوداً معدلاً من:
|
||||
- Parsel (ترخيص BSD) — يُستخدم للوحدة الفرعية [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
- Parsel (ترخيص BSD) - يُستخدم للوحدة الفرعية [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
|
||||
---
|
||||
<div align="center"><small>مصمم ومصنوع بـ ❤️ بواسطة كريم شعير.</small></div><br>
|
||||
|
||||
+8
-8
@@ -49,7 +49,7 @@
|
||||
|
||||
Scrapling 是一个自适应 Web Scraping 框架，能处理从单个请求到大规模爬取的一切需求。
|
||||
|
||||
它的解析器能够从网站变化中学习，并在页面更新时自动重新定位您的元素。它的 Fetcher 能够开箱即用地绕过 Cloudflare Turnstile 等反机器人系统。它的 Spider 框架让您可以扩展到并发、多 Session 爬取，支持暂停/恢复和自动 Proxy 轮换——只需几行 Python 代码。一个库，零妥协。
|
||||
它的解析器能够从网站变化中学习，并在页面更新时自动重新定位您的元素。它的 Fetcher 能够开箱即用地绕过 Cloudflare Turnstile 等反机器人系统。它的 Spider 框架让您可以扩展到并发、多 Session 爬取，支持暂停/恢复和自动 Proxy 轮换--只需几行 Python 代码。一个库，零妥协。
|
||||
|
||||
极速爬取，实时统计和 Streaming。由 Web Scraper 为 Web Scraper 和普通用户而构建，每个人都能找到适合自己的功能。
|
||||
|
||||
@@ -125,7 +125,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> 提供覆盖 16+ 平台（包括 TikTok、X、YouTube 和 Instagram）的 900+ 稳定 API，拥有 4000 万+ 数据集。<br /> 还提供<a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">优惠 AI 模型</a> — Claude、GPT、GEMINI 等，最高优惠 71%。
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> 提供覆盖 16+ 平台（包括 TikTok、X、YouTube 和 Instagram）的 900+ 稳定 API，拥有 4000 万+ 数据集。<br /> 还提供<a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">优惠 AI 模型</a> - Claude、GPT、GEMINI 等，最高优惠 71%。
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -191,12 +191,12 @@ MySpider().start()
|
||||
|
||||
## 主要特性
|
||||
|
||||
### Spider — 完整的爬取框架
|
||||
### Spider - 完整的爬取框架
|
||||
- 🕷️ **类 Scrapy 的 Spider API**：使用 `start_urls`、async `parse` callback 和`Request`/`Response` 对象定义 Spider。
|
||||
- ⚡ **并发爬取**：可配置的并发限制、按域名节流和下载延迟。
|
||||
- 🔄 **多 Session 支持**：统一接口，支持 HTTP 请求和隐秘无头浏览器在同一个 Spider 中使用——通过 ID 将请求路由到不同的 Session。
|
||||
- 🔄 **多 Session 支持**：统一接口，支持 HTTP 请求和隐秘无头浏览器在同一个 Spider 中使用--通过 ID 将请求路由到不同的 Session。
|
||||
- 💾 **暂停与恢复**：基于 Checkpoint 的爬取持久化。按 Ctrl+C 优雅关闭；重启后从上次停止的地方继续。
|
||||
- 📡 **Streaming 模式**：通过 `async for item in spider.stream()` 以实时统计 Streaming 抓取的数据——非常适合 UI、管道和长时间运行的爬取。
|
||||
- 📡 **Streaming 模式**：通过 `async for item in spider.stream()` 以实时统计 Streaming 抓取的数据--非常适合 UI、管道和长时间运行的爬取。
|
||||
- 🛡️ **被阻止请求检测**：自动检测并重试被阻止的请求，支持自定义逻辑。
|
||||
- 📦 **内置导出**：通过钩子和您自己的管道导出结果，或使用内置的 JSON/JSONL，分别通过 `result.items.to_json()`/`result.items.to_jsonl()`。
|
||||
|
||||
@@ -323,7 +323,7 @@ class MultiSessionSpider(Spider):
|
||||
```python
|
||||
QuotesSpider(crawldir="./crawl_data").start()
|
||||
```
|
||||
按 Ctrl+C 优雅暂停——进度会自动保存。之后，当您再次启动 Spider 时，传递相同的 `crawldir`，它将从上次停止的地方继续。
|
||||
按 Ctrl+C 优雅暂停--进度会自动保存。之后，当您再次启动 Spider 时，传递相同的 `crawldir`，它将从上次停止的地方继续。
|
||||
|
||||
### 高级解析与导航
|
||||
```python
|
||||
@@ -408,7 +408,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
|
||||
|
||||
## 性能基准
|
||||
|
||||
Scrapling 不仅功能强大——它还速度极快。以下基准测试将 Scrapling 的解析器与其他流行库的最新版本进行了比较。
|
||||
Scrapling 不仅功能强大--它还速度极快。以下基准测试将 Scrapling 的解析器与其他流行库的最新版本进行了比较。
|
||||
|
||||
### 文本提取速度测试（5000 个嵌套元素）
|
||||
|
||||
@@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
|
||||
## 致谢
|
||||
|
||||
此项目包含改编自以下内容的代码：
|
||||
- Parsel（BSD 许可证）——用于 [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)子模块
|
||||
- Parsel（BSD 许可证）--用于 [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)子模块
|
||||
|
||||
---
|
||||
<div align="center"><small>由 Karim Shoair 用❤️设计和制作。</small></div><br>
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> bietet über 900 stabile APIs auf mehr als 16 Plattformen, darunter TikTok, X, YouTube und Instagram, mit über 40 Mio. Datensätzen. <br /> Bietet außerdem <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">vergünstigte KI-Modelle</a> — Claude, GPT, GEMINI und mehr mit bis zu 71% Rabatt.
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> bietet über 900 stabile APIs auf mehr als 16 Plattformen, darunter TikTok, X, YouTube und Instagram, mit über 40 Mio. Datensätzen. <br /> Bietet außerdem <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">vergünstigte KI-Modelle</a> - Claude, GPT, GEMINI und mehr mit bis zu 71% Rabatt.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
||||
+6
-6
@@ -125,7 +125,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> ofrece más de 900 APIs estables en más de 16 plataformas, incluyendo TikTok, X, YouTube e Instagram, con más de 40M de conjuntos de datos. <br /> También ofrece <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">modelos de IA con descuento</a> — Claude, GPT, GEMINI y más con hasta un 71% de descuento.
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> ofrece más de 900 APIs estables en más de 16 plataformas, incluyendo TikTok, X, YouTube e Instagram, con más de 40M de conjuntos de datos. <br /> También ofrece <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">modelos de IA con descuento</a> - Claude, GPT, GEMINI y más con hasta un 71% de descuento.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -191,12 +191,12 @@ MySpider().start()
|
||||
|
||||
## Características Principales
|
||||
|
||||
### Spiders — Un Framework Completo de Rastreo
|
||||
### Spiders - Un Framework Completo de Rastreo
|
||||
- 🕷️ **API de Spider al estilo Scrapy**: Define spiders con `start_urls`, callbacks async `parse`, y objetos `Request`/`Response`.
|
||||
- ⚡ **Rastreo Concurrente**: Límites de concurrencia configurables, limitación por dominio y retrasos de descarga.
|
||||
- 🔄 **Soporte Multi-Session**: Interfaz unificada para solicitudes HTTP y navegadores headless sigilosos en un solo Spider — enruta solicitudes a diferentes sesiones por ID.
|
||||
- 🔄 **Soporte Multi-Session**: Interfaz unificada para solicitudes HTTP y navegadores headless sigilosos en un solo Spider - enruta solicitudes a diferentes sesiones por ID.
|
||||
- 💾 **Pause & Resume**: Persistencia de rastreo basada en Checkpoint. Presiona Ctrl+C para un cierre ordenado; reinicia para continuar desde donde lo dejaste.
|
||||
- 📡 **Modo Streaming**: Transmite elementos extraídos a medida que llegan con `async for item in spider.stream()` con estadísticas en tiempo real — ideal para UI, pipelines y rastreos de larga duración.
|
||||
- 📡 **Modo Streaming**: Transmite elementos extraídos a medida que llegan con `async for item in spider.stream()` con estadísticas en tiempo real - ideal para UI, pipelines y rastreos de larga duración.
|
||||
- 🛡️ **Detección de Solicitudes Bloqueadas**: Detección automática y reintento de solicitudes bloqueadas con lógica personalizable.
|
||||
- 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o el JSON/JSONL integrado con `result.items.to_json()` / `result.items.to_jsonl()` respectivamente.
|
||||
|
||||
@@ -323,7 +323,7 @@ Pausa y reanuda rastreos largos con checkpoints ejecutando el Spider así:
|
||||
```python
|
||||
QuotesSpider(crawldir="./crawl_data").start()
|
||||
```
|
||||
Presiona Ctrl+C para pausar de forma ordenada — el progreso se guarda automáticamente. Después, cuando inicies el Spider de nuevo, pasa el mismo `crawldir`, y continuará desde donde se detuvo.
|
||||
Presiona Ctrl+C para pausar de forma ordenada - el progreso se guarda automáticamente. Después, cuando inicies el Spider de nuevo, pasa el mismo `crawldir`, y continuará desde donde se detuvo.
|
||||
|
||||
### Análisis Avanzado y Navegación
|
||||
```python
|
||||
@@ -520,7 +520,7 @@ Este trabajo está licenciado bajo la Licencia BSD-3-Clause.
|
||||
## Agradecimientos
|
||||
|
||||
Este proyecto incluye código adaptado de:
|
||||
- Parsel (Licencia BSD)—Usado para el submódulo [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
- Parsel (Licencia BSD)-Usado para el submódulo [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
|
||||
---
|
||||
<div align="center"><small>Diseñado y elaborado con ❤️ por Karim Shoair.</small></div><br>
|
||||
|
||||
+8
-8
@@ -49,7 +49,7 @@
|
||||
|
||||
Scrapling est un framework de Web Scraping adaptatif qui gère tout, d'une simple requête à un crawl à grande échelle.
|
||||
|
||||
Son parser apprend des modifications de sites web et relocalise automatiquement vos éléments lorsque les pages sont mises à jour. Ses fetchers contournent les systèmes anti-bot comme Cloudflare Turnstile nativement. Et son framework Spider vous permet de monter en charge vers des crawls concurrents multi-sessions avec pause/reprise et rotation automatique de proxy — le tout en quelques lignes de Python. Une seule bibliothèque, zéro compromis.
|
||||
Son parser apprend des modifications de sites web et relocalise automatiquement vos éléments lorsque les pages sont mises à jour. Ses fetchers contournent les systèmes anti-bot comme Cloudflare Turnstile nativement. Et son framework Spider vous permet de monter en charge vers des crawls concurrents multi-sessions avec pause/reprise et rotation automatique de proxy - le tout en quelques lignes de Python. Une seule bibliothèque, zéro compromis.
|
||||
|
||||
Des crawls ultra-rapides avec des statistiques en temps réel et du streaming. Conçu par des Web Scrapers pour des Web Scrapers et des utilisateurs réguliers, il y en a pour tout le monde.
|
||||
|
||||
@@ -125,7 +125,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> propose plus de 900 APIs stables sur plus de 16 plateformes, dont TikTok, X, YouTube et Instagram, avec plus de 40M de jeux de données. <br /> Propose également des <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">modèles IA à prix réduit</a> — Claude, GPT, GEMINI et plus, jusqu'à 71% de réduction.
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> propose plus de 900 APIs stables sur plus de 16 plateformes, dont TikTok, X, YouTube et Instagram, avec plus de 40M de jeux de données. <br /> Propose également des <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">modèles IA à prix réduit</a> - Claude, GPT, GEMINI et plus, jusqu'à 71% de réduction.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -191,12 +191,12 @@ MySpider().start()
|
||||
|
||||
## Fonctionnalités principales
|
||||
|
||||
### Spiders — Un framework de crawling complet
|
||||
### Spiders - Un framework de crawling complet
|
||||
- 🕷️ **API Spider à la Scrapy** : Définissez des spiders avec `start_urls`, des callbacks async `parse` et des objets `Request`/`Response`.
|
||||
- ⚡ **Crawling concurrent** : Limites de concurrence configurables, throttling par domaine et délais de téléchargement.
|
||||
- 🔄 **Support multi-sessions** : Interface unifiée pour les requêtes HTTP et les navigateurs headless furtifs dans un seul spider — routez les requêtes vers différentes sessions par ID.
|
||||
- 🔄 **Support multi-sessions** : Interface unifiée pour les requêtes HTTP et les navigateurs headless furtifs dans un seul spider - routez les requêtes vers différentes sessions par ID.
|
||||
- 💾 **Pause & Reprise** : Persistance du crawl basée sur des checkpoints. Appuyez sur Ctrl+C pour un arrêt gracieux ; redémarrez pour reprendre là où vous vous étiez arrêté.
|
||||
- 📡 **Mode streaming** : Diffusez les éléments scrapés en temps réel via `async for item in spider.stream()` avec des statistiques en temps réel — idéal pour les UI, pipelines et crawls de longue durée.
|
||||
- 📡 **Mode streaming** : Diffusez les éléments scrapés en temps réel via `async for item in spider.stream()` avec des statistiques en temps réel - idéal pour les UI, pipelines et crawls de longue durée.
|
||||
- 🛡️ **Détection des requêtes bloquées** : Détection automatique et réessai des requêtes bloquées avec une logique personnalisable.
|
||||
- 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou l'export JSON/JSONL intégré avec `result.items.to_json()` / `result.items.to_jsonl()` respectivement.
|
||||
|
||||
@@ -323,7 +323,7 @@ Mettez en pause et reprenez les longs crawls avec des checkpoints en lançant le
|
||||
```python
|
||||
QuotesSpider(crawldir="./crawl_data").start()
|
||||
```
|
||||
Appuyez sur Ctrl+C pour mettre en pause gracieusement — la progression est sauvegardée automatiquement. Plus tard, lorsque vous relancez le spider, passez le même `crawldir`, et il reprendra là où il s'était arrêté.
|
||||
Appuyez sur Ctrl+C pour mettre en pause gracieusement - la progression est sauvegardée automatiquement. Plus tard, lorsque vous relancez le spider, passez le même `crawldir`, et il reprendra là où il s'était arrêté.
|
||||
|
||||
### Parsing avancé & Navigation
|
||||
```python
|
||||
@@ -408,7 +408,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
|
||||
|
||||
## Benchmarks de performance
|
||||
|
||||
Scrapling n'est pas seulement puissant — il est aussi ultra rapide. Les benchmarks suivants comparent le parser de Scrapling avec les dernières versions d'autres bibliothèques populaires.
|
||||
Scrapling n'est pas seulement puissant - il est aussi ultra rapide. Les benchmarks suivants comparent le parser de Scrapling avec les dernières versions d'autres bibliothèques populaires.
|
||||
|
||||
### Test de vitesse d'extraction de texte (5000 éléments imbriqués)
|
||||
|
||||
@@ -520,7 +520,7 @@ Ce travail est sous licence BSD-3-Clause.
|
||||
## Remerciements
|
||||
|
||||
Ce projet inclut du code adapté de :
|
||||
- Parsel (Licence BSD) — Utilisé pour le sous-module [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
- Parsel (Licence BSD) - Utilisé pour le sous-module [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
|
||||
---
|
||||
<div align="center"><small>Conçu et développé avec ❤️ par Karim Shoair.</small></div><br>
|
||||
|
||||
+6
-6
@@ -49,7 +49,7 @@
|
||||
|
||||
Scrapling は、単一のリクエストから本格的なクロールまですべてを処理する適応型 Web Scraping フレームワークです。
|
||||
|
||||
そのパーサーはウェブサイトの変更から学習し、ページが更新されたときに要素を自動的に再配置します。Fetcher はすぐに使える Cloudflare Turnstile などのアンチボットシステムを回避します。そして Spider フレームワークにより、Pause & Resume や自動 Proxy 回転機能を備えた並行マルチ Session クロールへとスケールアップできます — すべてわずか数行の Python で。1 つのライブラリ、妥協なし。
|
||||
そのパーサーはウェブサイトの変更から学習し、ページが更新されたときに要素を自動的に再配置します。Fetcher はすぐに使える Cloudflare Turnstile などのアンチボットシステムを回避します。そして Spider フレームワークにより、Pause & Resume や自動 Proxy 回転機能を備えた並行マルチ Session クロールへとスケールアップできます - すべてわずか数行の Python で。1 つのライブラリ、妥協なし。
|
||||
|
||||
リアルタイム統計と Streaming による超高速クロール。Web Scraper によって、Web Scraper と一般ユーザーのために構築され、誰にでも何かがあります。
|
||||
|
||||
@@ -125,7 +125,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> は TikTok、X、YouTube、Instagram を含む 16 以上のプラットフォームで 900 以上の安定した API を提供し、4,000 万以上のデータセットを保有。<br /> さらに <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">割引 AI モデル</a>も提供 — Claude、GPT、GEMINI など最大 71% オフ。
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> は TikTok、X、YouTube、Instagram を含む 16 以上のプラットフォームで 900 以上の安定した API を提供し、4,000 万以上のデータセットを保有。<br /> さらに <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">割引 AI モデル</a>も提供 - Claude、GPT、GEMINI など最大 71% オフ。
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -191,12 +191,12 @@ MySpider().start()
|
||||
|
||||
## 主な機能
|
||||
|
||||
### Spider — 本格的なクロールフレームワーク
|
||||
### Spider - 本格的なクロールフレームワーク
|
||||
- 🕷️ **Scrapy 風の Spider API**：`start_urls`、async `parse` callback、`Request`/`Response` オブジェクトで Spider を定義。
|
||||
- ⚡ **並行クロール**：設定可能な並行数制限、ドメインごとのスロットリング、ダウンロード遅延。
|
||||
- 🔄 **マルチ Session サポート**：HTTP リクエストとステルスヘッドレスブラウザの統一インターフェース — ID によって異なる Session にリクエストをルーティング。
|
||||
- 🔄 **マルチ Session サポート**：HTTP リクエストとステルスヘッドレスブラウザの統一インターフェース - ID によって異なる Session にリクエストをルーティング。
|
||||
- 💾 **Pause & Resume**：Checkpoint ベースのクロール永続化。Ctrl+C で正常にシャットダウン；再起動すると中断したところから再開。
|
||||
- 📡 **Streaming モード**：`async for item in spider.stream()` でリアルタイム統計とともにスクレイプされたアイテムを Streaming で受信 — UI、パイプライン、長時間実行クロールに最適。
|
||||
- 📡 **Streaming モード**：`async for item in spider.stream()` でリアルタイム統計とともにスクレイプされたアイテムを Streaming で受信 - UI、パイプライン、長時間実行クロールに最適。
|
||||
- 🛡️ **ブロックされたリクエストの検出**：カスタマイズ可能なロジックによるブロックされたリクエストの自動検出とリトライ。
|
||||
- 📦 **組み込みエクスポート**：フックや独自のパイプライン、または組み込みの JSON/JSONL で結果をエクスポート。それぞれ`result.items.to_json()` / `result.items.to_jsonl()`を使用。
|
||||
|
||||
@@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
|
||||
## 謝辞
|
||||
|
||||
このプロジェクトには次から適応されたコードが含まれています：
|
||||
- Parsel（BSD ライセンス）— [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) サブモジュールに使用
|
||||
- Parsel（BSD ライセンス）- [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) サブモジュールに使用
|
||||
|
||||
---
|
||||
<div align="center"><small>Karim Shoair によって❤️でデザインおよび作成されました。</small></div><br>
|
||||
|
||||
+8
-8
@@ -49,7 +49,7 @@
|
||||
|
||||
Scrapling은 단일 요청부터 대규모 크롤링까지 모든 것을 처리하는 적응형 Web Scraping 프레임워크입니다.
|
||||
|
||||
파서는 웹사이트 변경 사항을 학습하고, 페이지가 업데이트되면 요소를 자동으로 재배치합니다. Fetcher는 Cloudflare Turnstile 같은 안티봇 시스템을 별도 설정 없이 우회합니다. Spider 프레임워크를 사용하면 일시정지/재개 및 자동 프록시 로테이션을 갖춘 동시 멀티 세션 크롤링으로 확장할 수 있습니다 — 모두 Python 몇 줄이면 됩니다. 하나의 라이브러리, 타협 없는 성능.
|
||||
파서는 웹사이트 변경 사항을 학습하고, 페이지가 업데이트되면 요소를 자동으로 재배치합니다. Fetcher는 Cloudflare Turnstile 같은 안티봇 시스템을 별도 설정 없이 우회합니다. Spider 프레임워크를 사용하면 일시정지/재개 및 자동 프록시 로테이션을 갖춘 동시 멀티 세션 크롤링으로 확장할 수 있습니다 - 모두 Python 몇 줄이면 됩니다. 하나의 라이브러리, 타협 없는 성능.
|
||||
|
||||
실시간 통계와 스트리밍을 통한 초고속 크롤링. Web Scraper가 만들고, Web Scraper와 일반 사용자 모두를 위해 설계했습니다.
|
||||
|
||||
@@ -125,7 +125,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a>는 TikTok, X, YouTube, Instagram 등 16개 이상 플랫폼에서 900개 이상의 안정적인 API를 제공하며, 4,000만 이상의 데이터셋을 보유하고 있습니다. <br /> <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">할인된 AI 모델</a>도 제공 — Claude, GPT, GEMINI 등 최대 71% 할인.
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a>는 TikTok, X, YouTube, Instagram 등 16개 이상 플랫폼에서 900개 이상의 안정적인 API를 제공하며, 4,000만 이상의 데이터셋을 보유하고 있습니다. <br /> <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">할인된 AI 모델</a>도 제공 - Claude, GPT, GEMINI 등 최대 71% 할인.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -191,12 +191,12 @@ MySpider().start()
|
||||
|
||||
## 주요 기능
|
||||
|
||||
### Spider — 본격적인 크롤링 프레임워크
|
||||
### Spider - 본격적인 크롤링 프레임워크
|
||||
- 🕷️ **Scrapy 스타일 Spider API**: `start_urls`, 비동기 `parse` 콜백, `Request`/`Response` 객체로 Spider를 정의합니다.
|
||||
- ⚡ **동시 크롤링**: 설정 가능한 동시 요청 수 제한, 도메인별 스로틀링, 다운로드 딜레이를 지원합니다.
|
||||
- 🔄 **멀티 세션 지원**: HTTP 요청과 스텔스 헤드리스 브라우저를 하나의 인터페이스로 통합 — ID로 요청을 다른 세션에 라우팅합니다.
|
||||
- 🔄 **멀티 세션 지원**: HTTP 요청과 스텔스 헤드리스 브라우저를 하나의 인터페이스로 통합 - ID로 요청을 다른 세션에 라우팅합니다.
|
||||
- 💾 **일시정지 & 재개**: 체크포인트 기반의 크롤링 영속화. Ctrl+C로 정상 종료하고, 재시작하면 중단된 지점부터 이어갑니다.
|
||||
- 📡 **스트리밍 모드**: `async for item in spider.stream()`으로 스크레이핑된 아이템을 실시간 통계와 함께 스트리밍으로 수신 — UI, 파이프라인, 장시간 크롤링에 적합합니다.
|
||||
- 📡 **스트리밍 모드**: `async for item in spider.stream()`으로 스크레이핑된 아이템을 실시간 통계와 함께 스트리밍으로 수신 - UI, 파이프라인, 장시간 크롤링에 적합합니다.
|
||||
- 🛡️ **차단된 요청 감지**: 커스텀 로직을 통한 차단된 요청의 자동 감지 및 재시도를 지원합니다.
|
||||
- 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL로 결과를 내보냅니다. 각각 `result.items.to_json()` / `result.items.to_jsonl()`을 사용합니다.
|
||||
|
||||
@@ -256,7 +256,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # 작업
|
||||
page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
|
||||
data = page.css('#padded_content a').getall()
|
||||
|
||||
# 또는 일회성 요청 스타일 — 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다
|
||||
# 또는 일회성 요청 스타일 - 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다
|
||||
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
|
||||
data = page.css('#padded_content a').getall()
|
||||
```
|
||||
@@ -268,7 +268,7 @@ with DynamicSession(headless=True, disable_resources=False, network_idle=True) a
|
||||
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
|
||||
data = page.xpath('//span[@class="text"]/text()').getall() # 원하시면 XPath selector도 사용 가능
|
||||
|
||||
# 또는 일회성 요청 스타일 — 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다
|
||||
# 또는 일회성 요청 스타일 - 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다
|
||||
page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
|
||||
data = page.css('.quote .text::text').getall()
|
||||
```
|
||||
@@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
|
||||
## 감사의 말
|
||||
|
||||
이 프로젝트에는 다음에서 차용한 코드가 포함되어 있습니다:
|
||||
- Parsel (BSD 라이선스) — [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) 서브모듈에 사용
|
||||
- Parsel (BSD 라이선스) - [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) 서브모듈에 사용
|
||||
|
||||
---
|
||||
<div align="center"><small>Karim Shoair가 ❤️으로 디자인하고 만들었습니다.</small></div><br>
|
||||
|
||||
+14
-14
@@ -47,11 +47,11 @@
|
||||
<a href="https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html"><strong>Режим MCP</strong></a>
|
||||
</p>
|
||||
|
||||
Scrapling — это адаптивный фреймворк для Web Scraping, который берёт на себя всё: от одного запроса до полномасштабного обхода сайтов.
|
||||
Scrapling - это адаптивный фреймворк для Web Scraping, который берёт на себя всё: от одного запроса до полномасштабного обхода сайтов.
|
||||
|
||||
Его парсер учится на изменениях сайтов и автоматически перемещает ваши элементы при обновлении страниц. Его Fetcher'ы обходят анти-бот системы вроде Cloudflare Turnstile прямо из коробки. А его Spider-фреймворк позволяет масштабироваться до параллельных, многосессионных обходов с Pause & Resume и автоматической ротацией Proxy — и всё это в нескольких строках Python. Одна библиотека, без компромиссов.
|
||||
Его парсер учится на изменениях сайтов и автоматически перемещает ваши элементы при обновлении страниц. Его Fetcher'ы обходят анти-бот системы вроде Cloudflare Turnstile прямо из коробки. А его Spider-фреймворк позволяет масштабироваться до параллельных, многосессионных обходов с Pause & Resume и автоматической ротацией Proxy - и всё это в нескольких строках Python. Одна библиотека, без компромиссов.
|
||||
|
||||
Молниеносно быстрые обходы с отслеживанием статистики в реальном времени и Streaming. Создано веб-скраперами для веб-скраперов и обычных пользователей — здесь есть что-то для каждого.
|
||||
Молниеносно быстрые обходы с отслеживанием статистики в реальном времени и Streaming. Создано веб-скраперами для веб-скраперов и обычных пользователей - здесь есть что-то для каждого.
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
|
||||
@@ -128,7 +128,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> предоставляет более 900 стабильных API на 16+ платформах, включая TikTok, X, YouTube и Instagram, с более чем 40 млн наборов данных. <br /> Также предлагает <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">AI-модели со скидкой</a> — Claude, GPT, GEMINI и другие со скидкой до 71%.
|
||||
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> предоставляет более 900 стабильных API на 16+ платформах, включая TikTok, X, YouTube и Instagram, с более чем 40 млн наборов данных. <br /> Также предлагает <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">AI-модели со скидкой</a> - Claude, GPT, GEMINI и другие со скидкой до 71%.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -159,7 +159,7 @@ MySpider().start()
|
||||
</a>
|
||||
</td>
|
||||
<td>
|
||||
Прочитайте полный обзор <a href="https://substack.thewebscraping.club/p/scrapling-hands-on-guide?utm_source=github&utm_medium=repo&utm_campaign=scrapling" target="_blank">Scrapling на The Web Scraping Club</a> (ноябрь 2025) — рассылка №1, посвящённая веб-скрейпингу.
|
||||
Прочитайте полный обзор <a href="https://substack.thewebscraping.club/p/scrapling-hands-on-guide?utm_source=github&utm_medium=repo&utm_campaign=scrapling" target="_blank">Scrapling на The Web Scraping Club</a> (ноябрь 2025) - рассылка №1, посвящённая веб-скрейпингу.
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
@@ -194,12 +194,12 @@ MySpider().start()
|
||||
|
||||
## Ключевые особенности
|
||||
|
||||
### Spider'ы — полноценный фреймворк для обхода сайтов
|
||||
### Spider'ы - полноценный фреймворк для обхода сайтов
|
||||
- 🕷️ **Scrapy-подобный Spider API**: Определяйте Spider'ов с `start_urls`, async `parse` callback'ами и объектами `Request`/`Response`.
|
||||
- ⚡ **Параллельный обход**: Настраиваемые лимиты параллелизма, ограничение скорости по домену и задержки загрузки.
|
||||
- 🔄 **Поддержка нескольких сессий**: Единый интерфейс для HTTP-запросов и скрытных headless-браузеров в одном Spider — маршрутизируйте запросы к разным сессиям по ID.
|
||||
- 🔄 **Поддержка нескольких сессий**: Единый интерфейс для HTTP-запросов и скрытных headless-браузеров в одном Spider - маршрутизируйте запросы к разным сессиям по ID.
|
||||
- 💾 **Pause & Resume**: Persistence обхода на основе Checkpoint'ов. Нажмите Ctrl+C для мягкой остановки; перезапустите, чтобы продолжить с того места, где вы остановились.
|
||||
- 📡 **Режим Streaming**: Стримьте извлечённые элементы по мере их поступления через `async for item in spider.stream()` со статистикой в реальном времени — идеально для UI, конвейеров и длительных обходов.
|
||||
- 📡 **Режим Streaming**: Стримьте извлечённые элементы по мере их поступления через `async for item in spider.stream()` со статистикой в реальном времени - идеально для UI, конвейеров и длительных обходов.
|
||||
- 🛡️ **Обнаружение заблокированных запросов**: Автоматическое обнаружение и повторная отправка заблокированных запросов с настраиваемой логикой.
|
||||
- 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенный JSON/JSONL с `result.items.to_json()` / `result.items.to_jsonl()` соответственно.
|
||||
|
||||
@@ -259,7 +259,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Дер
|
||||
page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False)
|
||||
data = page.css('#padded_content a').getall()
|
||||
|
||||
# Или используйте стиль одноразового запроса — открывает браузер для этого запроса, затем закрывает его после завершения
|
||||
# Или используйте стиль одноразового запроса - открывает браузер для этого запроса, затем закрывает его после завершения
|
||||
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
|
||||
data = page.css('#padded_content a').getall()
|
||||
```
|
||||
@@ -271,7 +271,7 @@ with DynamicSession(headless=True, disable_resources=False, network_idle=True) a
|
||||
page = session.fetch('https://quotes.toscrape.com/', load_dom=False)
|
||||
data = page.xpath('//span[@class="text"]/text()').getall() # XPath-селектор, если вы предпочитаете его
|
||||
|
||||
# Или используйте стиль одноразового запроса — открывает браузер для этого запроса, затем закрывает его после завершения
|
||||
# Или используйте стиль одноразового запроса - открывает браузер для этого запроса, затем закрывает его после завершения
|
||||
page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
|
||||
data = page.css('.quote .text::text').getall()
|
||||
```
|
||||
@@ -326,7 +326,7 @@ class MultiSessionSpider(Spider):
|
||||
```python
|
||||
QuotesSpider(crawldir="./crawl_data").start()
|
||||
```
|
||||
Нажмите Ctrl+C для мягкой остановки — прогресс сохраняется автоматически. Позже, когда вы снова запустите Spider, передайте тот же `crawldir`, и он продолжит с того места, где остановился.
|
||||
Нажмите Ctrl+C для мягкой остановки - прогресс сохраняется автоматически. Позже, когда вы снова запустите Spider, передайте тот же `crawldir`, и он продолжит с того места, где остановился.
|
||||
|
||||
### Продвинутый парсинг и навигация
|
||||
```python
|
||||
@@ -383,7 +383,7 @@ async with AsyncStealthySession(max_pages=2) as session:
|
||||
task = session.fetch(url)
|
||||
tasks.append(task)
|
||||
|
||||
print(session.get_pool_stats()) # Опционально — статус пула вкладок браузера (занят/свободен/ошибка)
|
||||
print(session.get_pool_stats()) # Опционально - статус пула вкладок браузера (занят/свободен/ошибка)
|
||||
results = await asyncio.gather(*tasks)
|
||||
print(session.get_pool_stats())
|
||||
```
|
||||
@@ -411,7 +411,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
|
||||
|
||||
## Тесты производительности
|
||||
|
||||
Scrapling не только мощный — он ещё и невероятно быстрый. Следующие тесты производительности сравнивают парсер Scrapling с последними версиями других популярных библиотек.
|
||||
Scrapling не только мощный - он ещё и невероятно быстрый. Следующие тесты производительности сравнивают парсер Scrapling с последними версиями других популярных библиотек.
|
||||
|
||||
### Тест скорости извлечения текста (5000 вложенных элементов)
|
||||
|
||||
@@ -523,7 +523,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest
|
||||
## Благодарности
|
||||
|
||||
Этот проект включает код, адаптированный из:
|
||||
- Parsel (лицензия BSD) — Используется для подмодуля [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
- Parsel (лицензия BSD) - Используется для подмодуля [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)
|
||||
|
||||
---
|
||||
<div align="center"><small>Разработано и создано с ❤️ Карим Шоаир.</small></div><br>
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
# Performance Benchmarks
|
||||
|
||||
Scrapling isn't just powerful—it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.
|
||||
Scrapling isn't just powerful - it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.
|
||||
|
||||
### Text Extraction Speed Test (5000 nested elements)
|
||||
|
||||
|
||||
+4
-4
@@ -18,7 +18,7 @@
|
||||
|
||||
Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.
|
||||
|
||||
Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of Python. One library, zero compromises.
|
||||
Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises.
|
||||
|
||||
Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
|
||||
|
||||
@@ -89,12 +89,12 @@ MySpider().start()
|
||||
|
||||
## Key Features
|
||||
|
||||
### Spiders — A Full Crawling Framework
|
||||
### Spiders - A Full Crawling Framework
|
||||
- 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects.
|
||||
- ⚡ **Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays.
|
||||
- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider — route requests to different sessions by ID.
|
||||
- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
|
||||
- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
|
||||
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats — ideal for UI, pipelines, and long-running crawls.
|
||||
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
|
||||
- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
|
||||
- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.
|
||||
|
||||
|
||||
+2
-2
@@ -4,9 +4,9 @@ Not sure where to start? Pick the path that matches what you're trying to do:
|
||||
|
||||
| I want to... | Start here |
|
||||
|:---|:---|
|
||||
| **Parse HTML** I already have | [Querying elements](parsing/selection.md) — CSS, XPath, and text-based selection |
|
||||
| **Parse HTML** I already have | [Querying elements](parsing/selection.md): CSS, XPath, and text-based selection |
|
||||
| **Quickly scrape a page** and prototype | Pick a [fetcher](fetching/choosing.md) and test right away, or launch the [interactive shell](cli/interactive-shell.md) |
|
||||
| **Build a crawler** that scales | [Spiders](spiders/getting-started.md) — concurrent, multi-session crawls with pause/resume |
|
||||
| **Build a crawler** that scales | [Spiders](spiders/getting-started.md): concurrent, multi-session crawls with pause/resume |
|
||||
| **Scrape without writing code** | [CLI extract commands](cli/extract-commands.md) or hook up the [MCP server](ai/mcp-server.md) to your favourite AI tool |
|
||||
| **Migrate** from another library | [From BeautifulSoup](tutorials/migrating_from_beautifulsoup.md) or [Scrapy comparison](spiders/architecture.md#comparison-with-scrapy) |
|
||||
|
||||
|
||||
@@ -306,7 +306,7 @@ Starting with v0.4, [Selector](#selector) and [Selectors](#selectors) both provi
|
||||
|
||||
**On a [Selector](#selector) object:**
|
||||
|
||||
- `get()` returns a `TextHandler` — for text node selectors, it returns the text value; for HTML element selectors, it returns the serialized outer HTML.
|
||||
- `get()` returns a `TextHandler`: for text node selectors, it returns the text value; for HTML element selectors, it returns the serialized outer HTML.
|
||||
- `getall()` returns a `TextHandlers` list containing the single serialized string.
|
||||
- `extract_first` is an alias for `get()`, and `extract` is an alias for `getall()`.
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ class PoliteSpider(Spider):
|
||||
yield {"title": response.css("title::text").get("")}
|
||||
```
|
||||
|
||||
When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain.
|
||||
When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously, as you can allow high global concurrency while being polite to each individual domain.
|
||||
|
||||
!!! tip
|
||||
|
||||
@@ -66,7 +66,7 @@ else:
|
||||
|
||||
1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits.
|
||||
2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks.
|
||||
3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`.
|
||||
3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off, skipping `start_requests()`.
|
||||
4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically.
|
||||
|
||||
**Checkpoints are also saved periodically during the crawl (every 5 minutes by default).**
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
|
||||
Scrapling's spider system is a Scrapy-inspired async crawling framework designed for concurrent, multi-session crawls with built-in pause/resume support. It brings together Scrapling's parsing engine and fetchers into a unified crawling API while adding scheduling, concurrency control, and checkpointing.
|
||||
|
||||
If you're familiar with Scrapy, you'll feel right at home. If not, don't worry — the system is designed to be straightforward.
|
||||
If you're familiar with Scrapy, you'll feel right at home. If not, don't worry - the system is designed to be straightforward.
|
||||
|
||||
## Data Flow
|
||||
|
||||
@@ -23,7 +23,7 @@ Here's what happens step by step when you run a spider without many details:
|
||||
4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized.
|
||||
5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing.
|
||||
6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused.
|
||||
7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state.
|
||||
7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off, skipping `start_requests()` and restoring the scheduler state.
|
||||
|
||||
|
||||
## Components
|
||||
@@ -49,7 +49,7 @@ class MySpider(Spider):
|
||||
|
||||
### Crawler Engine
|
||||
|
||||
The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you.
|
||||
The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly - the `Spider.start()` and `Spider.stream()` methods handle it for you.
|
||||
|
||||
### Scheduler
|
||||
|
||||
|
||||
@@ -31,9 +31,9 @@ class QuotesSpider(Spider):
|
||||
|
||||
Every spider needs three things:
|
||||
|
||||
1. **`name`** — A unique identifier for the spider.
|
||||
2. **`start_urls`** — A list of URLs to start crawling from.
|
||||
3. **`parse()`** — An async generator method that processes each response and yields results.
|
||||
1. **`name`** - A unique identifier for the spider.
|
||||
2. **`start_urls`** - A list of URLs to start crawling from.
|
||||
3. **`parse()`** - An async generator method that processes each response and yields results.
|
||||
|
||||
The `parse()` method is where the magic happens. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items.
|
||||
|
||||
@@ -45,7 +45,7 @@ To run your spider, create an instance and call `start()`:
|
||||
result = QuotesSpider().start()
|
||||
```
|
||||
|
||||
The `start()` method handles all the async machinery internally — no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
|
||||
The `start()` method handles all the async machinery internally, so no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats.
|
||||
|
||||
Those stats are in the returned `CrawlResult` object, which gives you everything you need:
|
||||
|
||||
@@ -90,7 +90,7 @@ class QuotesSpider(Spider):
|
||||
yield response.follow(next_page, callback=self.parse)
|
||||
```
|
||||
|
||||
`response.follow()` handles relative URLs automatically — it joins them with the current page's URL. It also sets the current page as the `Referer` header by default.
|
||||
`response.follow()` handles relative URLs automatically by joining them with the current page's URL. It also sets the current page as the `Referer` header by default.
|
||||
|
||||
You can point follow-up requests at different callback methods for different page types:
|
||||
|
||||
@@ -145,7 +145,7 @@ class MySpider(Spider):
|
||||
yield response.follow(link, callback=self.parse)
|
||||
```
|
||||
|
||||
Subdomains are matched automatically — setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.
|
||||
Subdomains are matched automatically, so setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc.
|
||||
|
||||
When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped.
|
||||
|
||||
@@ -153,7 +153,7 @@ When a request is filtered out, it's counted in `stats.offsite_requests_count` s
|
||||
|
||||
Now that you have the basics, you can explore:
|
||||
|
||||
- [Requests & Responses](requests-responses.md) — learn about request priority, deduplication, metadata, and more.
|
||||
- [Sessions](sessions.md) — use multiple fetcher types (HTTP, browser, stealth) in a single spider.
|
||||
- [Proxy management & blocking](proxy-blocking.md) — rotate proxies across requests and how to handle blocking in the spider.
|
||||
- [Advanced features](advanced.md) — concurrency control, pause/resume, streaming, lifecycle hooks, and logging.
|
||||
- [Requests & Responses](requests-responses.md) - learn about request priority, deduplication, metadata, and more.
|
||||
- [Sessions](sessions.md) - use multiple fetcher types (HTTP, browser, stealth) in a single spider.
|
||||
- [Proxy management & blocking](proxy-blocking.md) - rotate proxies across requests and how to handle blocking in the spider.
|
||||
- [Advanced features](advanced.md) - concurrency control, pause/resume, streaming, lifecycle hooks, and logging.
|
||||
@@ -7,7 +7,7 @@
|
||||
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
|
||||
2. You've read the [Sessions](sessions.md) page and understand how to configure sessions.
|
||||
|
||||
When scraping at scale, you'll often need to rotate through multiple proxies to avoid rate limits and blocks. Scrapling's `ProxyRotator` makes this straightforward — it works with all session types and integrates with the spider's blocked request retry system.
|
||||
When scraping at scale, you'll often need to rotate through multiple proxies to avoid rate limits and blocks. Scrapling's `ProxyRotator` makes this straightforward. It works with all session types and integrates with the spider's blocked request retry system.
|
||||
|
||||
If you don't know what a proxy is or how to choose a good one, [this guide can help](https://substack.thewebscraping.club/p/everything-about-proxies).
|
||||
|
||||
@@ -70,7 +70,7 @@ def configure_sessions(self, manager):
|
||||
|
||||
## Custom Rotation Strategies
|
||||
|
||||
By default, `ProxyRotator` uses cyclic rotation — it iterates through proxies sequentially, wrapping around at the end.
|
||||
By default, `ProxyRotator` uses cyclic rotation, iterating through proxies sequentially and wrapping around at the end.
|
||||
|
||||
You can provide a custom strategy function to change this behavior, but it has to match the below signature:
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
|
||||
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
|
||||
|
||||
This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
|
||||
This page covers the `Request` object in detail: how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following.
|
||||
|
||||
## The Request Object
|
||||
|
||||
@@ -29,7 +29,7 @@ Here are all the arguments you can pass to `Request`:
|
||||
| Argument | Type | Default | Description |
|
||||
|---------------|------------|------------|-------------------------------------------------------------------------------------------------------|
|
||||
| `url` | `str` | *required* | The URL to fetch |
|
||||
| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) |
|
||||
| `sid` | `str` | `""` | Session ID - routes the request to a specific session (see [Sessions](sessions.md)) |
|
||||
| `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` |
|
||||
| `priority` | `int` | `0` | Higher values are processed first |
|
||||
| `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) |
|
||||
@@ -58,7 +58,7 @@ yield Request(
|
||||
|
||||
```python
|
||||
async def parse(self, response: Response):
|
||||
# Minimal — inherits callback, sid, priority from current request
|
||||
# Minimal - inherits callback, sid, priority from current request
|
||||
yield response.follow("/next-page")
|
||||
|
||||
# Override specific fields
|
||||
@@ -99,9 +99,9 @@ yield response.follow("/page", referer_flow=False)
|
||||
|
||||
Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types:
|
||||
|
||||
- **`dict`** — A scraped item, added to the results
|
||||
- **`Request`** — A follow-up request, added to the queue
|
||||
- **`None`** — Silently ignored
|
||||
- **`dict`** - A scraped item, added to the results
|
||||
- **`Request`** - A follow-up request, added to the queue
|
||||
- **`None`** - Silently ignored
|
||||
|
||||
```python
|
||||
class MySpider(Spider):
|
||||
@@ -130,11 +130,11 @@ Requests with higher priority values are processed first. This is useful when so
|
||||
|
||||
```python
|
||||
async def parse(self, response: Response):
|
||||
# High priority — process product pages first
|
||||
# High priority - process product pages first
|
||||
for link in response.css("a.product::attr(href)").getall():
|
||||
yield response.follow(link, callback=self.parse_product, priority=10)
|
||||
|
||||
# Low priority — pagination links processed after products
|
||||
# Low priority - pagination links processed after products
|
||||
next_page = response.css("a.next::attr(href)").get()
|
||||
if next_page:
|
||||
yield response.follow(next_page, callback=self.parse, priority=0)
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider.
|
||||
2. You're familiar with [Fetchers basics](../fetching/choosing.md) and the differences between HTTP, Dynamic, and Stealthy sessions.
|
||||
|
||||
A spider can use multiple fetcher sessions simultaneously — for example, a fast HTTP session for simple pages and a stealth browser session for protected pages. This page shows you how to configure and use sessions.
|
||||
A spider can use multiple fetcher sessions simultaneously. For example, a fast HTTP session for simple pages and a stealth browser session for protected pages. This page shows you how to configure and use sessions.
|
||||
|
||||
## What are Sessions?
|
||||
|
||||
@@ -23,7 +23,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md
|
||||
|
||||
## Configuring Sessions
|
||||
|
||||
Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance — use `manager.add()` to register sessions:
|
||||
Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance. Use `manager.add()` to register sessions:
|
||||
|
||||
```python
|
||||
from scrapling.spiders import Spider, Response
|
||||
@@ -101,7 +101,7 @@ class ProductSpider(Spider):
|
||||
}
|
||||
```
|
||||
|
||||
The key is the `sid` parameter — it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
|
||||
The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited.
|
||||
|
||||
Note that the sessions don't have to be from different classes only, but can be the same session, but different instances with different configurations, for example, like below:
|
||||
|
||||
|
||||
@@ -8,18 +8,18 @@ In this article, we will discuss these common issues, why companies are shifting
|
||||
|
||||
If you have been doing Web Scraping for a long time, you probably noticed that there are repeating problems with Web Scraping, like:
|
||||
|
||||
1. **Rapidly changing website structures** — Sites frequently update their DOM structures, breaking static XPath/CSS selectors.
|
||||
2. **Unstable selectors** — Class names and IDs often change or use randomly generated values that break scrapers or make scraping these websites difficult.
|
||||
3. **Increasingly complex anti-bot measures** — CAPTCHA systems, browser fingerprinting, and behavior analysis make traditional scraping difficult
|
||||
1. **Rapidly changing website structures** - Sites frequently update their DOM structures, breaking static XPath/CSS selectors.
|
||||
2. **Unstable selectors** - Class names and IDs often change or use randomly generated values that break scrapers or make scraping these websites difficult.
|
||||
3. **Increasingly complex anti-bot measures** - CAPTCHA systems, browser fingerprinting, and behavior analysis make traditional scraping difficult
|
||||
and others
|
||||
|
||||
But that's only if you are doing targeted Web Scraping for known websites, in which case you can write specific code for every website.
|
||||
|
||||
If you start thinking about bigger goals like Broad Scraping or Generic Web Scraping, or what you like to call it, then the above issues intensify, and you will face new issues like:
|
||||
|
||||
1. **Extreme Website Diversity** — Generic scraping must handle countless variations in HTML structures, CSS usage, JavaScript frameworks, and backend technologies.
|
||||
2. **Identifying Relevant Data** — How does the scraper know what data is important on a page it has never seen before?
|
||||
3. **Pagination variations** — Infinite scroll, traditional pagination, "load more" buttons, all requiring different approaches
|
||||
1. **Extreme Website Diversity** - Generic scraping must handle countless variations in HTML structures, CSS usage, JavaScript frameworks, and backend technologies.
|
||||
2. **Identifying Relevant Data** - How does the scraper know what data is important on a page it has never seen before?
|
||||
3. **Pagination variations** - Infinite scroll, traditional pagination, "load more" buttons, all requiring different approaches
|
||||
and more
|
||||
|
||||
How will you solve that manually? I'm referring to generic web scraping of various websites that don't share any common technologies.
|
||||
@@ -61,8 +61,8 @@ There is no need to explain any of these; click on the links, and it will be cle
|
||||
### Solving issue T3: Increasingly complex anti-bot measures
|
||||
It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve:
|
||||
|
||||
1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) — This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
|
||||
2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
|
||||
1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) - This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements.
|
||||
2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html) - Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial!
|
||||
|
||||
We keep improving these two with each update, so stay tuned :)
|
||||
|
||||
@@ -96,7 +96,7 @@ This example illustrates the point I aim to convey here. Not every challenge wil
|
||||
### Solving issue B3: Pagination variations
|
||||
This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :)
|
||||
|
||||
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea.
|
||||
But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'` - you get the idea.
|
||||
|
||||
## Cost Comparison and Savings
|
||||
For a quick comparison.
|
||||
|
||||
Reference in New Issue
Block a user