freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: updating pages with the new feature

Browse Source
This commit is contained in:
Karim shoair
2026-04-05 05:27:07 +02:00
parent c54081ff1a
commit 4c3ff2f0f1
5 changed files with 31 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/index.md
+1
View File
@@ -99,6 +99,7 @@ MySpider().start()
- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.
- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.
### Advanced Websites Fetching with Session Support