diff --git a/README.md b/README.md index ab5fb8a..408cfef 100644 --- a/README.md +++ b/README.md @@ -215,6 +215,7 @@ MySpider().start() - 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls. - 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic. - 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching. +- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers. - 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively. ### Advanced Websites Fetching with Session Support diff --git a/docs/README_AR.md b/docs/README_AR.md index a66ed18..c0de787 100644 --- a/docs/README_AR.md +++ b/docs/README_AR.md @@ -210,6 +210,7 @@ MySpider().start() - 📡 **وضع Streaming**: بث العناصر المستخرجة فور وصولها عبر `async for item in spider.stream()` مع إحصائيات فورية - مثالي لواجهات المستخدم وخطوط الأنابيب وعمليات الزحف الطويلة. - 🛡️ **كشف الطلبات المحظورة**: كشف تلقائي وإعادة محاولة للطلبات المحظورة مع منطق قابل للتخصيص. - 🤖 **الامتثال لـ robots.txt**: خيار `robots_txt_obey` الاختياري الذي يحترم توجيهات `Disallow` و `Crawl-delay` و `Request-rate` مع التخزين المؤقت لكل نطاق. +- 🧪 **وضع التطوير**: تخزين الاستجابات على القرص في التشغيل الأول وإعادة تشغيلها في التشغيلات اللاحقة - كرّر العمل على منطق `parse()` دون الحاجة لإرسال طلبات جديدة إلى الخوادم المستهدفة. - 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو JSON/JSONL المدمج مع `result.items.to_json()` / `result.items.to_jsonl()` على التوالي. ### جلب متقدم للمواقع مع دعم الجلسات diff --git a/docs/README_CN.md b/docs/README_CN.md index d75f32b..6cb9d4e 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -210,6 +210,7 @@ MySpider().start() - 📡 **Streaming 模式**：通过 `async for item in spider.stream()` 以实时统计 Streaming 抓取的数据--非常适合 UI、管道和长时间运行的爬取。 - 🛡️ **被阻止请求检测**：自动检测并重试被阻止的请求，支持自定义逻辑。 - 🤖 **robots.txt 合规**：可选的 `robots_txt_obey` 标志，支持 `Disallow`、`Crawl-delay` 和 `Request-rate` 指令，并按域名缓存。 +- 🧪 **开发模式**：首次运行时将响应缓存到磁盘，后续运行时直接回放 - 在不重新请求目标服务器的情况下迭代你的 `parse()` 逻辑。 - 📦 **内置导出**：通过钩子和您自己的管道导出结果，或使用内置的 JSON/JSONL，分别通过 `result.items.to_json()`/`result.items.to_jsonl()`。 ### 支持 Session 的高级网站获取 diff --git a/docs/README_DE.md b/docs/README_DE.md index 5acbf15..82d336f 100644 --- a/docs/README_DE.md +++ b/docs/README_DE.md @@ -210,6 +210,7 @@ MySpider().start() - 📡 **Streaming-Modus**: Gescrapte Elemente in Echtzeit streamen über `async for item in spider.stream()` mit Echtzeit-Statistiken -- ideal für UI, Pipelines und lang laufende Crawls. - 🛡️ **Erkennung blockierter Anfragen**: Automatische Erkennung und Wiederholung blockierter Anfragen mit anpassbarer Logik. - 🤖 **robots.txt-Konformität**: Optionales `robots_txt_obey`-Flag, das `Disallow`-, `Crawl-delay`- und `Request-rate`-Direktiven mit domainbasiertem Caching respektiert. +- 🧪 **Entwicklungsmodus**: Antworten beim ersten Lauf auf der Festplatte zwischenspeichern und bei weiteren Läufen erneut abspielen - iterieren Sie an Ihrer `parse()`-Logik, ohne die Zielserver erneut abzufragen. - 📦 **Integrierter Export**: Ergebnisse über Hooks und Ihre eigene Pipeline oder den integrierten JSON/JSONL-Export mit `result.items.to_json()` / `result.items.to_jsonl()` exportieren. ### Erweitertes Website-Abrufen mit Session-Unterstützung diff --git a/docs/README_ES.md b/docs/README_ES.md index f2cd179..58dccb8 100644 --- a/docs/README_ES.md +++ b/docs/README_ES.md @@ -210,6 +210,7 @@ MySpider().start() - 📡 **Modo Streaming**: Transmite elementos extraídos a medida que llegan con `async for item in spider.stream()` con estadísticas en tiempo real - ideal para UI, pipelines y rastreos de larga duración. - 🛡️ **Detección de Solicitudes Bloqueadas**: Detección automática y reintento de solicitudes bloqueadas con lógica personalizable. - 🤖 **Cumplimiento de robots.txt**: Flag opcional `robots_txt_obey` que respeta las directivas `Disallow`, `Crawl-delay` y `Request-rate` con caché por dominio. +- 🧪 **Modo de Desarrollo**: Almacena las respuestas en disco en la primera ejecución y las reproduce en ejecuciones posteriores - itera sobre tu lógica de `parse()` sin volver a consultar los servidores objetivo. - 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o el JSON/JSONL integrado con `result.items.to_json()` / `result.items.to_jsonl()` respectivamente. ### Obtención Avanzada de Sitios Web con Soporte de Session diff --git a/docs/README_FR.md b/docs/README_FR.md index ce5ac11..a452210 100644 --- a/docs/README_FR.md +++ b/docs/README_FR.md @@ -210,6 +210,7 @@ MySpider().start() - 📡 **Mode streaming** : Diffusez les éléments scrapés en temps réel via `async for item in spider.stream()` avec des statistiques en temps réel - idéal pour les UI, pipelines et crawls de longue durée. - 🛡️ **Détection des requêtes bloquées** : Détection automatique et réessai des requêtes bloquées avec une logique personnalisable. - 🤖 **Conformité robots.txt** : Flag optionnel `robots_txt_obey` qui respecte les directives `Disallow`, `Crawl-delay` et `Request-rate` avec mise en cache par domaine. +- 🧪 **Mode développement** : Mettez les réponses en cache sur le disque lors de la première exécution et rejouez-les lors des exécutions suivantes - itérez sur votre logique `parse()` sans solliciter à nouveau les serveurs cibles. - 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou l'export JSON/JSONL intégré avec `result.items.to_json()` / `result.items.to_jsonl()` respectivement. ### Récupération avancée de sites web avec support de sessions diff --git a/docs/README_JP.md b/docs/README_JP.md index 7c89cb3..9a2a29d 100644 --- a/docs/README_JP.md +++ b/docs/README_JP.md @@ -210,6 +210,7 @@ MySpider().start() - 📡 **Streaming モード**：`async for item in spider.stream()` でリアルタイム統計とともにスクレイプされたアイテムを Streaming で受信 - UI、パイプライン、長時間実行クロールに最適。 - 🛡️ **ブロックされたリクエストの検出**：カスタマイズ可能なロジックによるブロックされたリクエストの自動検出とリトライ。 - 🤖 **robots.txt 準拠**：オプションの `robots_txt_obey` フラグで `Disallow`、`Crawl-delay`、`Request-rate` ディレクティブをドメインごとのキャッシュで遵守。 +- 🧪 **開発モード**：初回実行時にレスポンスをディスクにキャッシュし、以降の実行ではそれを再生 - ターゲットサーバーに再リクエストすることなく `parse()` ロジックを反復開発できます。 - 📦 **組み込みエクスポート**：フックや独自のパイプライン、または組み込みの JSON/JSONL で結果をエクスポート。それぞれ`result.items.to_json()` / `result.items.to_jsonl()`を使用。 ### Session サポート付き高度なウェブサイト取得 diff --git a/docs/README_KR.md b/docs/README_KR.md index b74d877..6d79389 100644 --- a/docs/README_KR.md +++ b/docs/README_KR.md @@ -210,6 +210,7 @@ MySpider().start() - 📡 **스트리밍 모드**: `async for item in spider.stream()`으로 스크레이핑된 아이템을 실시간 통계와 함께 스트리밍으로 수신 - UI, 파이프라인, 장시간 크롤링에 적합합니다. - 🛡️ **차단된 요청 감지**: 커스텀 로직을 통한 차단된 요청의 자동 감지 및 재시도를 지원합니다. - 🤖 **robots.txt 준수**: 선택적 `robots_txt_obey` 플래그로 `Disallow`, `Crawl-delay`, `Request-rate` 지시문을 도메인별 캐싱과 함께 준수합니다. +- 🧪 **개발 모드**: 첫 실행 시 응답을 디스크에 캐싱하고 이후 실행에서는 캐시된 응답을 재생합니다 - 대상 서버에 다시 요청하지 않고 `parse()` 로직을 반복 개발할 수 있습니다. - 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL로 결과를 내보냅니다. 각각 `result.items.to_json()` / `result.items.to_jsonl()`을 사용합니다. ### 세션을 지원하는 고급 웹사이트 가져오기 diff --git a/docs/README_RU.md b/docs/README_RU.md index 2d636a4..0fa251f 100644 --- a/docs/README_RU.md +++ b/docs/README_RU.md @@ -213,6 +213,7 @@ MySpider().start() - 📡 **Режим Streaming**: Стримьте извлечённые элементы по мере их поступления через `async for item in spider.stream()` со статистикой в реальном времени - идеально для UI, конвейеров и длительных обходов. - 🛡️ **Обнаружение заблокированных запросов**: Автоматическое обнаружение и повторная отправка заблокированных запросов с настраиваемой логикой. - 🤖 **Соответствие robots.txt**: Опциональный флаг `robots_txt_obey`, который учитывает директивы `Disallow`, `Crawl-delay` и `Request-rate` с кэшированием по доменам. +- 🧪 **Режим разработки**: Кэшируйте ответы на диск при первом запуске и воспроизводите их при последующих запусках - итерируйте над логикой `parse()`, не отправляя повторные запросы к целевым серверам. - 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенный JSON/JSONL с `result.items.to_json()` / `result.items.to_jsonl()` соответственно. ### Продвинутая загрузка сайтов с поддержкой Session diff --git a/docs/index.md b/docs/index.md index 9ff2d04..ae41d6c 100644 --- a/docs/index.md +++ b/docs/index.md @@ -100,6 +100,7 @@ MySpider().start() - 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls. - 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic. - 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching. +- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers. - 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively. ### Advanced Websites Fetching with Session Support diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.md index 9b1f5b9..98e930a 100644 --- a/docs/spiders/advanced.md +++ b/docs/spiders/advanced.md @@ -97,6 +97,51 @@ async def on_start(self, resuming: bool = False): self.logger.info("Starting fresh crawl") ``` +## Development Mode + +When you're iterating on a spider's `parse()` logic, re-hitting the target servers on every run is slow and noisy. Development mode caches every response to disk on the first run and replays them from disk on subsequent runs, so you can tweak your selectors and re-run the spider as many times as you want without making a single network request. + +Enable it by setting `development_mode = True` on your spider: + +```python +class MySpider(Spider): + name = "my_spider" + start_urls = ["https://example.com"] + development_mode = True + + async def parse(self, response: Response): + yield {"title": response.css("title::text").get("")} +``` + +The first run fetches normally and stores each response on disk. Every subsequent run serves the same requests from the cache, skipping the network entirely. + +### Cache Location + +By default, responses are cached in `.scrapling_cache/{spider.name}/` relative to the current working directory (where you ran the spider from, **not** where the spider script lives). You can override the location with `development_cache_dir`: + +```python +class MySpider(Spider): + name = "my_spider" + start_urls = ["https://example.com"] + development_mode = True + development_cache_dir = "/tmp/my_spider_cache" +``` + +### How It Works + +1. **Cache key**: Each response is keyed by the request's fingerprint, so any change to fingerprint-affecting attributes (`fp_include_kwargs`, `fp_include_headers`, `fp_keep_fragments`) will produce a fresh fetch. +2. **Storage format**: One JSON file per response, named `{fingerprint_hex}.json`. The body is base64-encoded so binary content is preserved exactly. Writes are atomic (temp file + rename). +3. **Replay**: On a cache hit, the engine skips the network entirely, including `download_delay`, rate limiting, and the `is_blocked()` retry path. The cached response goes straight to your callback. +4. **Stats**: Cached requests still count toward `requests_count`, `response_bytes`, and the per-status counters, so your stat output looks the same as a normal crawl. Two extra counters, `cache_hits` and `cache_misses`, let you see how the cache performed. + +### Clearing the Cache + +There's no automatic expiration. To force a fresh crawl, delete the cache directory or call the manager's `clear()` method directly. + +!!! warning + + Development mode is meant for development, not production. Cached responses never expire, and replay bypasses rate limiting and blocked-request retries. Don't ship a spider with `development_mode = True`. + ## Streaming For long-running spiders or applications that need real-time access to scraped items, use the `stream()` method instead of `start()`: @@ -236,6 +281,8 @@ print(f"Failed: {stats.failed_requests_count}") print(f"Blocked: {stats.blocked_requests_count}") print(f"Offsite filtered: {stats.offsite_requests_count}") print(f"Robots.txt disallowed: {stats.robots_disallowed_count}") +print(f"Cache hits: {stats.cache_hits}") +print(f"Cache misses: {stats.cache_misses}") print(f"Items scraped: {stats.items_scraped}") print(f"Items dropped: {stats.items_dropped}") print(f"Response bytes: {stats.response_bytes}") diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md index 82d388f..dc360b8 100644 --- a/docs/spiders/architecture.md +++ b/docs/spiders/architecture.md @@ -69,6 +69,10 @@ When a request comes in, the Session Manager routes it to the correct session ba An optional system that, if enabled, saves the crawler's state (pending requests + seen URL fingerprints) to a pickle file on disk. Writes are atomic (temp file + rename) to prevent corruption. Checkpoints are saved periodically at a configurable interval and on graceful shutdown. Upon successful completion (not paused), checkpoint files are automatically cleaned up. +### Response Cache + +An optional cache that, when development mode is enabled, stores every fetched response on disk and replays it on subsequent runs. Each response is keyed by request fingerprint and serialized as JSON (with the body base64-encoded so binary content survives). It's meant for iterating on `parse()` logic without re-hitting the target servers, not for production use. + ### Output Scraped items are collected in an `ItemList` (a list subclass with `to_json()` and `to_jsonl()` export methods). Crawl statistics are tracked in a `CrawlStats` dataclass which contains a lot of useful info.