diff --git a/README.md b/README.md index cd97172..62bbfd0 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl. -Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of Python. One library, zero compromises. +Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises. Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone. @@ -129,7 +129,7 @@ MySpider().start() - TikHub.io provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets.
Also offers DISCOUNTED AI models — Claude, GPT, GEMINI & more up to 71% off. + TikHub.io provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets.
Also offers DISCOUNTED AI models - Claude, GPT, GEMINI & more up to 71% off. @@ -196,12 +196,12 @@ MySpider().start() ## Key Features -### Spiders — A Full Crawling Framework +### Spiders - A Full Crawling Framework - 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects. - ⚡ **Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays. -- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider — route requests to different sessions by ID. +- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID. - 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off. -- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats — ideal for UI, pipelines, and long-running crawls. +- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls. - 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic. - 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively. @@ -328,7 +328,7 @@ Pause and resume long crawls with checkpoints by running the spider like this: ```python QuotesSpider(crawldir="./crawl_data").start() ``` -Press Ctrl+C to pause gracefully — progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped. +Press Ctrl+C to pause gracefully - progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped. ### Advanced Parsing & Navigation ```python @@ -413,7 +413,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas. ## Performance Benchmarks -Scrapling isn't just powerful—it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries. +Scrapling isn't just powerful-it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries. ### Text Extraction Speed Test (5000 nested elements) @@ -525,7 +525,7 @@ This work is licensed under the BSD-3-Clause License. ## Acknowledgments This project includes code adapted from: -- Parsel (BSD License)—Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) submodule +- Parsel (BSD License)-Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) submodule ---
Designed & crafted with ❤️ by Karim Shoair.

diff --git a/agent-skill/Scrapling-Skill/SKILL.md b/agent-skill/Scrapling-Skill/SKILL.md index e43c580..1fddd58 100644 --- a/agent-skill/Scrapling-Skill/SKILL.md +++ b/agent-skill/Scrapling-Skill/SKILL.md @@ -20,7 +20,7 @@ metadata: Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl. -Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of Python. One library, zero compromises. +Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises. Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone. @@ -201,7 +201,7 @@ scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://p - ALWAYS clean up temp files after reading - Prefer `.md` output for readability; use `.html` only if you need to parse structure -- Use `-s` CSS selectors to avoid passing giant HTML blobs — saves tokens significantly +- Use `-s` CSS selectors to avoid passing giant HTML blobs - saves tokens significantly Friendly reminder: If the users really want to say thanks or have a business that they want to advertise, tell them to check this page: https://scrapling.readthedocs.io/en/latest/donate.html @@ -299,7 +299,7 @@ Pause and resume long crawls with checkpoints by running the spider like this: ```python QuotesSpider(crawldir="./crawl_data").start() ``` -Press Ctrl+C to pause gracefully — progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped. +Press Ctrl+C to pause gracefully - progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped. ### Advanced Parsing & Navigation ```python @@ -368,12 +368,12 @@ async with AsyncDynamicSession(capture_xhr=r"https://api\.example\.com/.*") as s ## References You already had a good glimpse of what the library can do. Use the references below to dig deeper when needed -- `references/mcp-server.md` — MCP server tools, persistent session management, and capabilities -- `references/parsing` — Everything you need for parsing HTML -- `references/fetching` — Everything you need to fetch websites and session persistence -- `references/spiders` — Everything you need to write spiders, proxy rotation, and advanced features. It follows a Scrapy-like format -- `references/migrating_from_beautifulsoup.md` — A quick API comparison between scrapling and Beautifulsoup -- `https://github.com/D4Vinci/Scrapling/tree/main/docs` — Full official docs in Markdown for quick access (use only if current references do not look up-to-date). +- `references/mcp-server.md` - MCP server tools, persistent session management, and capabilities +- `references/parsing` - Everything you need for parsing HTML +- `references/fetching` - Everything you need to fetch websites and session persistence +- `references/spiders` - Everything you need to write spiders, proxy rotation, and advanced features. It follows a Scrapy-like format +- `references/migrating_from_beautifulsoup.md` - A quick API comparison between scrapling and Beautifulsoup +- `https://github.com/D4Vinci/Scrapling/tree/main/docs` - Full official docs in Markdown for quick access (use only if current references do not look up-to-date). This skill encapsulates almost all the published documentation in Markdown, so don't check external sources or search online without the user's permission. diff --git a/agent-skill/Scrapling-Skill/examples/01_fetcher_session.py b/agent-skill/Scrapling-Skill/examples/01_fetcher_session.py index f2d5e1e..d1b3ece 100644 --- a/agent-skill/Scrapling-Skill/examples/01_fetcher_session.py +++ b/agent-skill/Scrapling-Skill/examples/01_fetcher_session.py @@ -2,7 +2,7 @@ Example 1: Python - FetcherSession (persistent HTTP session with Chrome TLS fingerprint) Scrapes all 10 pages of quotes.toscrape.com using a single HTTP session. -No browser launched — fast and lightweight. +No browser launched - fast and lightweight. Best for: static or semi-static sites, APIs, pages that don't require JavaScript. """ diff --git a/agent-skill/Scrapling-Skill/examples/README.md b/agent-skill/Scrapling-Skill/examples/README.md index 340e48f..3f753c5 100644 --- a/agent-skill/Scrapling-Skill/examples/README.md +++ b/agent-skill/Scrapling-Skill/examples/README.md @@ -1,6 +1,6 @@ # Scrapling Examples -These examples scrape [quotes.toscrape.com](https://quotes.toscrape.com) — a safe, purpose-built scraping sandbox — and demonstrate every tool available in Scrapling, from plain HTTP to full browser automation and spiders. +These examples scrape [quotes.toscrape.com](https://quotes.toscrape.com) - a safe, purpose-built scraping sandbox - and demonstrate every tool available in Scrapling, from plain HTTP to full browser automation and spiders. All examples collect **all 100 quotes across 10 pages**. @@ -17,10 +17,10 @@ scrapling install --force | File | Tool | Type | Best For | |--------------------------|-------------------|-----------------------------|---------------------------------------| -| `01_fetcher_session.py` | `FetcherSession` | Python — persistent HTTP | APIs, fast multi-page scraping | -| `02_dynamic_session.py` | `DynamicSession` | Python — browser automation | Dynamic/SPA pages | -| `03_stealthy_session.py` | `StealthySession` | Python — stealth browser | Cloudflare, fingerprint bypass | -| `04_spider.py` | `Spider` | Python — auto-crawling | Multi-page crawls, full-site scraping | +| `01_fetcher_session.py` | `FetcherSession` | Python - persistent HTTP | APIs, fast multi-page scraping | +| `02_dynamic_session.py` | `DynamicSession` | Python - browser automation | Dynamic/SPA pages | +| `03_stealthy_session.py` | `StealthySession` | Python - stealth browser | Cloudflare, fingerprint bypass | +| `04_spider.py` | `Spider` | Python - auto-crawling | Multi-page crawls, full-site scraping | ## Running diff --git a/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md b/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md index 4adf129..367c028 100644 --- a/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md +++ b/agent-skill/Scrapling-Skill/references/migrating_from_beautifulsoup.md @@ -44,7 +44,7 @@ Some BeautifulSoup shortcuts have no direct Scrapling equivalent. Scrapling avoi ¹ **Note:** BS4's `find_previous`/`find_all_previous` searches all preceding elements in document order, while Scrapling's `path` only returns ancestors (the parent chain). These are not exact equivalents, but ancestor search covers the most common use case. -BeautifulSoup supports modifying/manipulating the parsed DOM. Scrapling does not — it is read-only and optimized for extraction. +BeautifulSoup supports modifying/manipulating the parsed DOM. Scrapling does not - it is read-only and optimized for extraction. ### Full Example: Extracting Links diff --git a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md b/agent-skill/Scrapling-Skill/references/parsing/main_classes.md index 0c73b98..3033e7c 100644 --- a/agent-skill/Scrapling-Skill/references/parsing/main_classes.md +++ b/agent-skill/Scrapling-Skill/references/parsing/main_classes.md @@ -293,7 +293,7 @@ Starting with v0.4, [Selector](#selector) and [Selectors](#selectors) both provi **On a [Selector](#selector) object:** -- `get()` returns a `TextHandler` — for text node selectors, it returns the text value; for HTML element selectors, it returns the serialized outer HTML. +- `get()` returns a `TextHandler`: for text node selectors, it returns the text value; for HTML element selectors, it returns the serialized outer HTML. - `getall()` returns a `TextHandlers` list containing the single serialized string. - `extract_first` is an alias for `get()`, and `extract` is an alias for `getall()`. diff --git a/agent-skill/Scrapling-Skill/references/spiders/advanced.md b/agent-skill/Scrapling-Skill/references/spiders/advanced.md index 5fc682c..955706b 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/advanced.md +++ b/agent-skill/Scrapling-Skill/references/spiders/advanced.md @@ -24,7 +24,7 @@ class PoliteSpider(Spider): yield {"title": response.css("title::text").get("")} ``` -When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain. +When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously - you can allow high global concurrency while being polite to each individual domain. **Tip:** The `download_delay` parameter adds a fixed wait before every request, regardless of the domain. Use it for simple rate limiting. @@ -56,7 +56,7 @@ else: 1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits. 2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks. -3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`. +3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off, skipping `start_requests()`. 4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically. **Checkpoints are also saved periodically during the crawl (every 5 minutes by default).** diff --git a/agent-skill/Scrapling-Skill/references/spiders/architecture.md b/agent-skill/Scrapling-Skill/references/spiders/architecture.md index de95d24..93b059b 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/architecture.md +++ b/agent-skill/Scrapling-Skill/references/spiders/architecture.md @@ -14,7 +14,7 @@ Here's what happens step by step when you run a spider: 4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized. 5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing. 6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused. -7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state. +7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off, skipping `start_requests()` and restoring the scheduler state. ## Components @@ -40,7 +40,7 @@ class MySpider(Spider): ### Crawler Engine -The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you. +The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly - the `Spider.start()` and `Spider.stream()` methods handle it for you. ### Scheduler diff --git a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md index 2a9e6c8..7420079 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/getting-started.md +++ b/agent-skill/Scrapling-Skill/references/spiders/getting-started.md @@ -21,9 +21,9 @@ class QuotesSpider(Spider): Every spider needs three things: -1. **`name`** — A unique identifier for the spider. -2. **`start_urls`** — A list of URLs to start crawling from. -3. **`parse()`** — An async generator method that processes each response and yields results. +1. **`name`**: A unique identifier for the spider. +2. **`start_urls`**: A list of URLs to start crawling from. +3. **`parse()`**: An async generator method that processes each response and yields results. The `parse()` method processes each response. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items. @@ -35,7 +35,7 @@ To run your spider, create an instance and call `start()`: result = QuotesSpider().start() ``` -The `start()` method handles all the async machinery internally — no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats. +The `start()` method handles all the async machinery internally, so there is no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats. Those stats are in the returned `CrawlResult` object, which gives you everything you need: @@ -80,7 +80,7 @@ class QuotesSpider(Spider): yield response.follow(next_page, callback=self.parse) ``` -`response.follow()` handles relative URLs automatically — it joins them with the current page's URL. It also sets the current page as the `Referer` header by default. +`response.follow()` handles relative URLs automatically by joining them with the current page's URL. It also sets the current page as the `Referer` header by default. You can point follow-up requests at different callback methods for different page types: @@ -133,7 +133,7 @@ class MySpider(Spider): yield response.follow(link, callback=self.parse) ``` -Subdomains are matched automatically — setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc. +Subdomains are matched automatically, so setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc. When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped. diff --git a/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md b/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md index 733743b..d9aa7d4 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md +++ b/agent-skill/Scrapling-Skill/references/spiders/proxy-blocking.md @@ -61,7 +61,7 @@ def configure_sessions(self, manager): ## Custom Rotation Strategies -By default, `ProxyRotator` uses cyclic rotation — it iterates through proxies sequentially, wrapping around at the end. +By default, `ProxyRotator` uses cyclic rotation - it iterates through proxies sequentially, wrapping around at the end. You can provide a custom strategy function to change this behavior, but it has to match the below signature: diff --git a/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md b/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md index 8c40723..c93a7e2 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md +++ b/agent-skill/Scrapling-Skill/references/spiders/requests-responses.md @@ -1,6 +1,6 @@ # Requests & Responses -This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following. +This page covers the `Request` object in detail: how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following. ## The Request Object @@ -25,7 +25,7 @@ Here are all the arguments you can pass to `Request`: | Argument | Type | Default | Description | |---------------|------------|------------|-------------------------------------------------------------------------------------------------------| | `url` | `str` | *required* | The URL to fetch | -| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) | +| `sid` | `str` | `""` | Session ID - routes the request to a specific session (see [Sessions](sessions.md)) | | `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` | | `priority` | `int` | `0` | Higher values are processed first | | `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) | @@ -54,7 +54,7 @@ yield Request( ```python async def parse(self, response: Response): - # Minimal — inherits callback, sid, priority from current request + # Minimal - inherits callback, sid, priority from current request yield response.follow("/next-page") # Override specific fields @@ -95,9 +95,9 @@ yield response.follow("/page", referer_flow=False) Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types: -- **`dict`** — A scraped item, added to the results -- **`Request`** — A follow-up request, added to the queue -- **`None`** — Silently ignored +- **`dict`**: A scraped item, added to the results +- **`Request`**: A follow-up request, added to the queue +- **`None`**: Silently ignored ```python class MySpider(Spider): @@ -124,11 +124,11 @@ Requests with higher priority values are processed first. This is useful when so ```python async def parse(self, response: Response): - # High priority — process product pages first + # High priority - process product pages first for link in response.css("a.product::attr(href)").getall(): yield response.follow(link, callback=self.parse_product, priority=10) - # Low priority — pagination links processed after products + # Low priority - pagination links processed after products next_page = response.css("a.next::attr(href)").get() if next_page: yield response.follow(next_page, callback=self.parse, priority=0) diff --git a/agent-skill/Scrapling-Skill/references/spiders/sessions.md b/agent-skill/Scrapling-Skill/references/spiders/sessions.md index 761cea6..8d2c344 100644 --- a/agent-skill/Scrapling-Skill/references/spiders/sessions.md +++ b/agent-skill/Scrapling-Skill/references/spiders/sessions.md @@ -1,6 +1,6 @@ # Spiders sessions -A spider can use multiple fetcher sessions simultaneously — for example, a fast HTTP session for simple pages and a stealth browser session for protected pages. +A spider can use multiple fetcher sessions simultaneously. For example, a fast HTTP session for simple pages and a stealth browser session for protected pages. ## What are Sessions? @@ -18,7 +18,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md ## Configuring Sessions -Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance — use `manager.add()` to register sessions: +Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance - use `manager.add()` to register sessions: ```python from scrapling.spiders import Spider, Response @@ -90,7 +90,7 @@ class ProductSpider(Spider): } ``` -The key is the `sid` parameter — it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited. +The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited. Sessions can also be different instances of the same class with different configurations: diff --git a/docs/README_AR.md b/docs/README_AR.md index 7ed8f9b..f15f0f1 100644 --- a/docs/README_AR.md +++ b/docs/README_AR.md @@ -125,7 +125,7 @@ MySpider().start() - TikHub.io يوفر أكثر من 900 واجهة API مستقرة عبر أكثر من 16 منصة تشمل TikTok و X و YouTube و Instagram، مع أكثر من 40 مليون مجموعة بيانات.
يقدم أيضاً نماذج ذكاء اصطناعي بأسعار مخفضة — Claude و GPT و GEMINI والمزيد بخصم يصل إلى 71%. + TikHub.io يوفر أكثر من 900 واجهة API مستقرة عبر أكثر من 16 منصة تشمل TikTok و X و YouTube و Instagram، مع أكثر من 40 مليون مجموعة بيانات.
يقدم أيضاً نماذج ذكاء اصطناعي بأسعار مخفضة - Claude و GPT و GEMINI والمزيد بخصم يصل إلى 71%. @@ -191,12 +191,12 @@ MySpider().start() ## الميزات الرئيسية -### Spiders — إطار عمل زحف كامل +### Spiders - إطار عمل زحف كامل - 🕷️ **واجهة Spider شبيهة بـ Scrapy**: عرّف Spiders مع `start_urls`، و async `parse` callbacks، وكائنات `Request`/`Response`. - ⚡ **زحف متزامن**: حدود تزامن قابلة للتكوين، وتحكم بالسرعة حسب النطاق، وتأخيرات التنزيل. -- 🔄 **دعم الجلسات المتعددة**: واجهة موحدة لطلبات HTTP، ومتصفحات خفية بدون واجهة في Spider واحد — وجّه الطلبات إلى جلسات مختلفة بالمعرّف. +- 🔄 **دعم الجلسات المتعددة**: واجهة موحدة لطلبات HTTP، ومتصفحات خفية بدون واجهة في Spider واحد - وجّه الطلبات إلى جلسات مختلفة بالمعرّف. - 💾 **إيقاف واستئناف**: استمرارية الزحف القائمة على Checkpoint. اضغط Ctrl+C للإيقاف بسلاسة؛ أعد التشغيل للاستئناف من حيث توقفت. -- 📡 **وضع Streaming**: بث العناصر المستخرجة فور وصولها عبر `async for item in spider.stream()` مع إحصائيات فورية — مثالي لواجهات المستخدم وخطوط الأنابيب وعمليات الزحف الطويلة. +- 📡 **وضع Streaming**: بث العناصر المستخرجة فور وصولها عبر `async for item in spider.stream()` مع إحصائيات فورية - مثالي لواجهات المستخدم وخطوط الأنابيب وعمليات الزحف الطويلة. - 🛡️ **كشف الطلبات المحظورة**: كشف تلقائي وإعادة محاولة للطلبات المحظورة مع منطق قابل للتخصيص. - 📦 **تصدير مدمج**: صدّر النتائج عبر الخطافات وخط الأنابيب الخاص بك أو JSON/JSONL المدمج مع `result.items.to_json()` / `result.items.to_jsonl()` على التوالي. @@ -323,7 +323,7 @@ class MultiSessionSpider(Spider): ```python QuotesSpider(crawldir="./crawl_data").start() ``` -اضغط Ctrl+C للإيقاف بسلاسة — يتم حفظ التقدم تلقائياً. لاحقاً، عند تشغيل Spider مرة أخرى، مرر نفس `crawldir`، وسيستأنف من حيث توقف. +اضغط Ctrl+C للإيقاف بسلاسة - يتم حفظ التقدم تلقائياً. لاحقاً، عند تشغيل Spider مرة أخرى، مرر نفس `crawldir`، وسيستأنف من حيث توقف. ### التحليل المتقدم والتنقل ```python @@ -408,7 +408,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas. ## معايير الأداء -Scrapling ليس قوياً فحسب — بل هو أيضاً سريع بشكل مذهل. تقارن المعايير التالية محلل Scrapling مع أحدث إصدارات المكتبات الشائعة الأخرى. +Scrapling ليس قوياً فحسب - بل هو أيضاً سريع بشكل مذهل. تقارن المعايير التالية محلل Scrapling مع أحدث إصدارات المكتبات الشائعة الأخرى. ### اختبار سرعة استخراج النص (5000 عنصر متداخل) @@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest ## الشكر والتقدير يتضمن هذا المشروع كوداً معدلاً من: -- Parsel (ترخيص BSD) — يُستخدم للوحدة الفرعية [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) +- Parsel (ترخيص BSD) - يُستخدم للوحدة الفرعية [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) ---
مصمم ومصنوع بـ ❤️ بواسطة كريم شعير.

diff --git a/docs/README_CN.md b/docs/README_CN.md index c5ac70d..97f1ec1 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -49,7 +49,7 @@ Scrapling 是一个自适应 Web Scraping 框架，能处理从单个请求到大规模爬取的一切需求。 -它的解析器能够从网站变化中学习，并在页面更新时自动重新定位您的元素。它的 Fetcher 能够开箱即用地绕过 Cloudflare Turnstile 等反机器人系统。它的 Spider 框架让您可以扩展到并发、多 Session 爬取，支持暂停/恢复和自动 Proxy 轮换——只需几行 Python 代码。一个库，零妥协。 +它的解析器能够从网站变化中学习，并在页面更新时自动重新定位您的元素。它的 Fetcher 能够开箱即用地绕过 Cloudflare Turnstile 等反机器人系统。它的 Spider 框架让您可以扩展到并发、多 Session 爬取，支持暂停/恢复和自动 Proxy 轮换--只需几行 Python 代码。一个库，零妥协。 极速爬取，实时统计和 Streaming。由 Web Scraper 为 Web Scraper 和普通用户而构建，每个人都能找到适合自己的功能。 @@ -125,7 +125,7 @@ MySpider().start() - TikHub.io 提供覆盖 16+ 平台（包括 TikTok、X、YouTube 和 Instagram）的 900+ 稳定 API，拥有 4000 万+ 数据集。
还提供优惠 AI 模型 — Claude、GPT、GEMINI 等，最高优惠 71%。 + TikHub.io 提供覆盖 16+ 平台（包括 TikTok、X、YouTube 和 Instagram）的 900+ 稳定 API，拥有 4000 万+ 数据集。
还提供优惠 AI 模型 - Claude、GPT、GEMINI 等，最高优惠 71%。 @@ -191,12 +191,12 @@ MySpider().start() ## 主要特性 -### Spider — 完整的爬取框架 +### Spider - 完整的爬取框架 - 🕷️ **类 Scrapy 的 Spider API**：使用 `start_urls`、async `parse` callback 和`Request`/`Response` 对象定义 Spider。 - ⚡ **并发爬取**：可配置的并发限制、按域名节流和下载延迟。 -- 🔄 **多 Session 支持**：统一接口，支持 HTTP 请求和隐秘无头浏览器在同一个 Spider 中使用——通过 ID 将请求路由到不同的 Session。 +- 🔄 **多 Session 支持**：统一接口，支持 HTTP 请求和隐秘无头浏览器在同一个 Spider 中使用--通过 ID 将请求路由到不同的 Session。 - 💾 **暂停与恢复**：基于 Checkpoint 的爬取持久化。按 Ctrl+C 优雅关闭；重启后从上次停止的地方继续。 -- 📡 **Streaming 模式**：通过 `async for item in spider.stream()` 以实时统计 Streaming 抓取的数据——非常适合 UI、管道和长时间运行的爬取。 +- 📡 **Streaming 模式**：通过 `async for item in spider.stream()` 以实时统计 Streaming 抓取的数据--非常适合 UI、管道和长时间运行的爬取。 - 🛡️ **被阻止请求检测**：自动检测并重试被阻止的请求，支持自定义逻辑。 - 📦 **内置导出**：通过钩子和您自己的管道导出结果，或使用内置的 JSON/JSONL，分别通过 `result.items.to_json()`/`result.items.to_jsonl()`。 @@ -323,7 +323,7 @@ class MultiSessionSpider(Spider): ```python QuotesSpider(crawldir="./crawl_data").start() ``` -按 Ctrl+C 优雅暂停——进度会自动保存。之后，当您再次启动 Spider 时，传递相同的 `crawldir`，它将从上次停止的地方继续。 +按 Ctrl+C 优雅暂停--进度会自动保存。之后，当您再次启动 Spider 时，传递相同的 `crawldir`，它将从上次停止的地方继续。 ### 高级解析与导航 ```python @@ -408,7 +408,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas. ## 性能基准 -Scrapling 不仅功能强大——它还速度极快。以下基准测试将 Scrapling 的解析器与其他流行库的最新版本进行了比较。 +Scrapling 不仅功能强大--它还速度极快。以下基准测试将 Scrapling 的解析器与其他流行库的最新版本进行了比较。 ### 文本提取速度测试（5000 个嵌套元素） @@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest ## 致谢 此项目包含改编自以下内容的代码： -- Parsel（BSD 许可证）——用于 [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)子模块 +- Parsel（BSD 许可证）--用于 [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py)子模块 ---
由 Karim Shoair 用❤️设计和制作。

diff --git a/docs/README_DE.md b/docs/README_DE.md index 4829681..04dfbae 100644 --- a/docs/README_DE.md +++ b/docs/README_DE.md @@ -125,7 +125,7 @@ MySpider().start() - TikHub.io bietet über 900 stabile APIs auf mehr als 16 Plattformen, darunter TikTok, X, YouTube und Instagram, mit über 40 Mio. Datensätzen.
Bietet außerdem vergünstigte KI-Modelle — Claude, GPT, GEMINI und mehr mit bis zu 71% Rabatt. + TikHub.io bietet über 900 stabile APIs auf mehr als 16 Plattformen, darunter TikTok, X, YouTube und Instagram, mit über 40 Mio. Datensätzen.
Bietet außerdem vergünstigte KI-Modelle - Claude, GPT, GEMINI und mehr mit bis zu 71% Rabatt. diff --git a/docs/README_ES.md b/docs/README_ES.md index 3d5f3c9..da8c757 100644 --- a/docs/README_ES.md +++ b/docs/README_ES.md @@ -125,7 +125,7 @@ MySpider().start() - TikHub.io ofrece más de 900 APIs estables en más de 16 plataformas, incluyendo TikTok, X, YouTube e Instagram, con más de 40M de conjuntos de datos.
También ofrece modelos de IA con descuento — Claude, GPT, GEMINI y más con hasta un 71% de descuento. + TikHub.io ofrece más de 900 APIs estables en más de 16 plataformas, incluyendo TikTok, X, YouTube e Instagram, con más de 40M de conjuntos de datos.
También ofrece modelos de IA con descuento - Claude, GPT, GEMINI y más con hasta un 71% de descuento. @@ -191,12 +191,12 @@ MySpider().start() ## Características Principales -### Spiders — Un Framework Completo de Rastreo +### Spiders - Un Framework Completo de Rastreo - 🕷️ **API de Spider al estilo Scrapy**: Define spiders con `start_urls`, callbacks async `parse`, y objetos `Request`/`Response`. - ⚡ **Rastreo Concurrente**: Límites de concurrencia configurables, limitación por dominio y retrasos de descarga. -- 🔄 **Soporte Multi-Session**: Interfaz unificada para solicitudes HTTP y navegadores headless sigilosos en un solo Spider — enruta solicitudes a diferentes sesiones por ID. +- 🔄 **Soporte Multi-Session**: Interfaz unificada para solicitudes HTTP y navegadores headless sigilosos en un solo Spider - enruta solicitudes a diferentes sesiones por ID. - 💾 **Pause & Resume**: Persistencia de rastreo basada en Checkpoint. Presiona Ctrl+C para un cierre ordenado; reinicia para continuar desde donde lo dejaste. -- 📡 **Modo Streaming**: Transmite elementos extraídos a medida que llegan con `async for item in spider.stream()` con estadísticas en tiempo real — ideal para UI, pipelines y rastreos de larga duración. +- 📡 **Modo Streaming**: Transmite elementos extraídos a medida que llegan con `async for item in spider.stream()` con estadísticas en tiempo real - ideal para UI, pipelines y rastreos de larga duración. - 🛡️ **Detección de Solicitudes Bloqueadas**: Detección automática y reintento de solicitudes bloqueadas con lógica personalizable. - 📦 **Exportación Integrada**: Exporta resultados a través de hooks y tu propio pipeline o el JSON/JSONL integrado con `result.items.to_json()` / `result.items.to_jsonl()` respectivamente. @@ -323,7 +323,7 @@ Pausa y reanuda rastreos largos con checkpoints ejecutando el Spider así: ```python QuotesSpider(crawldir="./crawl_data").start() ``` -Presiona Ctrl+C para pausar de forma ordenada — el progreso se guarda automáticamente. Después, cuando inicies el Spider de nuevo, pasa el mismo `crawldir`, y continuará desde donde se detuvo. +Presiona Ctrl+C para pausar de forma ordenada - el progreso se guarda automáticamente. Después, cuando inicies el Spider de nuevo, pasa el mismo `crawldir`, y continuará desde donde se detuvo. ### Análisis Avanzado y Navegación ```python @@ -520,7 +520,7 @@ Este trabajo está licenciado bajo la Licencia BSD-3-Clause. ## Agradecimientos Este proyecto incluye código adaptado de: -- Parsel (Licencia BSD)—Usado para el submódulo [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) +- Parsel (Licencia BSD)-Usado para el submódulo [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) ---
Diseñado y elaborado con ❤️ por Karim Shoair.

diff --git a/docs/README_FR.md b/docs/README_FR.md index 2511a65..7103040 100644 --- a/docs/README_FR.md +++ b/docs/README_FR.md @@ -49,7 +49,7 @@ Scrapling est un framework de Web Scraping adaptatif qui gère tout, d'une simple requête à un crawl à grande échelle. -Son parser apprend des modifications de sites web et relocalise automatiquement vos éléments lorsque les pages sont mises à jour. Ses fetchers contournent les systèmes anti-bot comme Cloudflare Turnstile nativement. Et son framework Spider vous permet de monter en charge vers des crawls concurrents multi-sessions avec pause/reprise et rotation automatique de proxy — le tout en quelques lignes de Python. Une seule bibliothèque, zéro compromis. +Son parser apprend des modifications de sites web et relocalise automatiquement vos éléments lorsque les pages sont mises à jour. Ses fetchers contournent les systèmes anti-bot comme Cloudflare Turnstile nativement. Et son framework Spider vous permet de monter en charge vers des crawls concurrents multi-sessions avec pause/reprise et rotation automatique de proxy - le tout en quelques lignes de Python. Une seule bibliothèque, zéro compromis. Des crawls ultra-rapides avec des statistiques en temps réel et du streaming. Conçu par des Web Scrapers pour des Web Scrapers et des utilisateurs réguliers, il y en a pour tout le monde. @@ -125,7 +125,7 @@ MySpider().start() - TikHub.io propose plus de 900 APIs stables sur plus de 16 plateformes, dont TikTok, X, YouTube et Instagram, avec plus de 40M de jeux de données.
Propose également des modèles IA à prix réduit — Claude, GPT, GEMINI et plus, jusqu'à 71% de réduction. + TikHub.io propose plus de 900 APIs stables sur plus de 16 plateformes, dont TikTok, X, YouTube et Instagram, avec plus de 40M de jeux de données.
Propose également des modèles IA à prix réduit - Claude, GPT, GEMINI et plus, jusqu'à 71% de réduction. @@ -191,12 +191,12 @@ MySpider().start() ## Fonctionnalités principales -### Spiders — Un framework de crawling complet +### Spiders - Un framework de crawling complet - 🕷️ **API Spider à la Scrapy** : Définissez des spiders avec `start_urls`, des callbacks async `parse` et des objets `Request`/`Response`. - ⚡ **Crawling concurrent** : Limites de concurrence configurables, throttling par domaine et délais de téléchargement. -- 🔄 **Support multi-sessions** : Interface unifiée pour les requêtes HTTP et les navigateurs headless furtifs dans un seul spider — routez les requêtes vers différentes sessions par ID. +- 🔄 **Support multi-sessions** : Interface unifiée pour les requêtes HTTP et les navigateurs headless furtifs dans un seul spider - routez les requêtes vers différentes sessions par ID. - 💾 **Pause & Reprise** : Persistance du crawl basée sur des checkpoints. Appuyez sur Ctrl+C pour un arrêt gracieux ; redémarrez pour reprendre là où vous vous étiez arrêté. -- 📡 **Mode streaming** : Diffusez les éléments scrapés en temps réel via `async for item in spider.stream()` avec des statistiques en temps réel — idéal pour les UI, pipelines et crawls de longue durée. +- 📡 **Mode streaming** : Diffusez les éléments scrapés en temps réel via `async for item in spider.stream()` avec des statistiques en temps réel - idéal pour les UI, pipelines et crawls de longue durée. - 🛡️ **Détection des requêtes bloquées** : Détection automatique et réessai des requêtes bloquées avec une logique personnalisable. - 📦 **Export intégré** : Exportez les résultats via des hooks et votre propre pipeline ou l'export JSON/JSONL intégré avec `result.items.to_json()` / `result.items.to_jsonl()` respectivement. @@ -323,7 +323,7 @@ Mettez en pause et reprenez les longs crawls avec des checkpoints en lançant le ```python QuotesSpider(crawldir="./crawl_data").start() ``` -Appuyez sur Ctrl+C pour mettre en pause gracieusement — la progression est sauvegardée automatiquement. Plus tard, lorsque vous relancez le spider, passez le même `crawldir`, et il reprendra là où il s'était arrêté. +Appuyez sur Ctrl+C pour mettre en pause gracieusement - la progression est sauvegardée automatiquement. Plus tard, lorsque vous relancez le spider, passez le même `crawldir`, et il reprendra là où il s'était arrêté. ### Parsing avancé & Navigation ```python @@ -408,7 +408,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas. ## Benchmarks de performance -Scrapling n'est pas seulement puissant — il est aussi ultra rapide. Les benchmarks suivants comparent le parser de Scrapling avec les dernières versions d'autres bibliothèques populaires. +Scrapling n'est pas seulement puissant - il est aussi ultra rapide. Les benchmarks suivants comparent le parser de Scrapling avec les dernières versions d'autres bibliothèques populaires. ### Test de vitesse d'extraction de texte (5000 éléments imbriqués) @@ -520,7 +520,7 @@ Ce travail est sous licence BSD-3-Clause. ## Remerciements Ce projet inclut du code adapté de : -- Parsel (Licence BSD) — Utilisé pour le sous-module [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) +- Parsel (Licence BSD) - Utilisé pour le sous-module [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) ---
Conçu et développé avec ❤️ par Karim Shoair.

diff --git a/docs/README_JP.md b/docs/README_JP.md index c5a2171..87cb86e 100644 --- a/docs/README_JP.md +++ b/docs/README_JP.md @@ -49,7 +49,7 @@ Scrapling は、単一のリクエストから本格的なクロールまですべてを処理する適応型 Web Scraping フレームワークです。 -そのパーサーはウェブサイトの変更から学習し、ページが更新されたときに要素を自動的に再配置します。Fetcher はすぐに使える Cloudflare Turnstile などのアンチボットシステムを回避します。そして Spider フレームワークにより、Pause & Resume や自動 Proxy 回転機能を備えた並行マルチ Session クロールへとスケールアップできます — すべてわずか数行の Python で。1 つのライブラリ、妥協なし。 +そのパーサーはウェブサイトの変更から学習し、ページが更新されたときに要素を自動的に再配置します。Fetcher はすぐに使える Cloudflare Turnstile などのアンチボットシステムを回避します。そして Spider フレームワークにより、Pause & Resume や自動 Proxy 回転機能を備えた並行マルチ Session クロールへとスケールアップできます - すべてわずか数行の Python で。1 つのライブラリ、妥協なし。 リアルタイム統計と Streaming による超高速クロール。Web Scraper によって、Web Scraper と一般ユーザーのために構築され、誰にでも何かがあります。 @@ -125,7 +125,7 @@ MySpider().start() - TikHub.io は TikTok、X、YouTube、Instagram を含む 16 以上のプラットフォームで 900 以上の安定した API を提供し、4,000 万以上のデータセットを保有。
さらに 割引 AI モデルも提供 — Claude、GPT、GEMINI など最大 71% オフ。 + TikHub.io は TikTok、X、YouTube、Instagram を含む 16 以上のプラットフォームで 900 以上の安定した API を提供し、4,000 万以上のデータセットを保有。
さらに 割引 AI モデルも提供 - Claude、GPT、GEMINI など最大 71% オフ。 @@ -191,12 +191,12 @@ MySpider().start() ## 主な機能 -### Spider — 本格的なクロールフレームワーク +### Spider - 本格的なクロールフレームワーク - 🕷️ **Scrapy 風の Spider API**：`start_urls`、async `parse` callback、`Request`/`Response` オブジェクトで Spider を定義。 - ⚡ **並行クロール**：設定可能な並行数制限、ドメインごとのスロットリング、ダウンロード遅延。 -- 🔄 **マルチ Session サポート**：HTTP リクエストとステルスヘッドレスブラウザの統一インターフェース — ID によって異なる Session にリクエストをルーティング。 +- 🔄 **マルチ Session サポート**：HTTP リクエストとステルスヘッドレスブラウザの統一インターフェース - ID によって異なる Session にリクエストをルーティング。 - 💾 **Pause & Resume**：Checkpoint ベースのクロール永続化。Ctrl+C で正常にシャットダウン；再起動すると中断したところから再開。 -- 📡 **Streaming モード**：`async for item in spider.stream()` でリアルタイム統計とともにスクレイプされたアイテムを Streaming で受信 — UI、パイプライン、長時間実行クロールに最適。 +- 📡 **Streaming モード**：`async for item in spider.stream()` でリアルタイム統計とともにスクレイプされたアイテムを Streaming で受信 - UI、パイプライン、長時間実行クロールに最適。 - 🛡️ **ブロックされたリクエストの検出**：カスタマイズ可能なロジックによるブロックされたリクエストの自動検出とリトライ。 - 📦 **組み込みエクスポート**：フックや独自のパイプライン、または組み込みの JSON/JSONL で結果をエクスポート。それぞれ`result.items.to_json()` / `result.items.to_jsonl()`を使用。 @@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest ## 謝辞 このプロジェクトには次から適応されたコードが含まれています： -- Parsel（BSD ライセンス）— [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) サブモジュールに使用 +- Parsel（BSD ライセンス）- [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) サブモジュールに使用 ---
Karim Shoair によって❤️でデザインおよび作成されました。

diff --git a/docs/README_KR.md b/docs/README_KR.md index 07a72a3..d556959 100644 --- a/docs/README_KR.md +++ b/docs/README_KR.md @@ -49,7 +49,7 @@ Scrapling은 단일 요청부터 대규모 크롤링까지 모든 것을 처리하는 적응형 Web Scraping 프레임워크입니다. -파서는 웹사이트 변경 사항을 학습하고, 페이지가 업데이트되면 요소를 자동으로 재배치합니다. Fetcher는 Cloudflare Turnstile 같은 안티봇 시스템을 별도 설정 없이 우회합니다. Spider 프레임워크를 사용하면 일시정지/재개 및 자동 프록시 로테이션을 갖춘 동시 멀티 세션 크롤링으로 확장할 수 있습니다 — 모두 Python 몇 줄이면 됩니다. 하나의 라이브러리, 타협 없는 성능. +파서는 웹사이트 변경 사항을 학습하고, 페이지가 업데이트되면 요소를 자동으로 재배치합니다. Fetcher는 Cloudflare Turnstile 같은 안티봇 시스템을 별도 설정 없이 우회합니다. Spider 프레임워크를 사용하면 일시정지/재개 및 자동 프록시 로테이션을 갖춘 동시 멀티 세션 크롤링으로 확장할 수 있습니다 - 모두 Python 몇 줄이면 됩니다. 하나의 라이브러리, 타협 없는 성능. 실시간 통계와 스트리밍을 통한 초고속 크롤링. Web Scraper가 만들고, Web Scraper와 일반 사용자 모두를 위해 설계했습니다. @@ -125,7 +125,7 @@ MySpider().start() - TikHub.io는 TikTok, X, YouTube, Instagram 등 16개 이상 플랫폼에서 900개 이상의 안정적인 API를 제공하며, 4,000만 이상의 데이터셋을 보유하고 있습니다.
할인된 AI 모델도 제공 — Claude, GPT, GEMINI 등 최대 71% 할인. + TikHub.io는 TikTok, X, YouTube, Instagram 등 16개 이상 플랫폼에서 900개 이상의 안정적인 API를 제공하며, 4,000만 이상의 데이터셋을 보유하고 있습니다.
할인된 AI 모델도 제공 - Claude, GPT, GEMINI 등 최대 71% 할인. @@ -191,12 +191,12 @@ MySpider().start() ## 주요 기능 -### Spider — 본격적인 크롤링 프레임워크 +### Spider - 본격적인 크롤링 프레임워크 - 🕷️ **Scrapy 스타일 Spider API**: `start_urls`, 비동기 `parse` 콜백, `Request`/`Response` 객체로 Spider를 정의합니다. - ⚡ **동시 크롤링**: 설정 가능한 동시 요청 수 제한, 도메인별 스로틀링, 다운로드 딜레이를 지원합니다. -- 🔄 **멀티 세션 지원**: HTTP 요청과 스텔스 헤드리스 브라우저를 하나의 인터페이스로 통합 — ID로 요청을 다른 세션에 라우팅합니다. +- 🔄 **멀티 세션 지원**: HTTP 요청과 스텔스 헤드리스 브라우저를 하나의 인터페이스로 통합 - ID로 요청을 다른 세션에 라우팅합니다. - 💾 **일시정지 & 재개**: 체크포인트 기반의 크롤링 영속화. Ctrl+C로 정상 종료하고, 재시작하면 중단된 지점부터 이어갑니다. -- 📡 **스트리밍 모드**: `async for item in spider.stream()`으로 스크레이핑된 아이템을 실시간 통계와 함께 스트리밍으로 수신 — UI, 파이프라인, 장시간 크롤링에 적합합니다. +- 📡 **스트리밍 모드**: `async for item in spider.stream()`으로 스크레이핑된 아이템을 실시간 통계와 함께 스트리밍으로 수신 - UI, 파이프라인, 장시간 크롤링에 적합합니다. - 🛡️ **차단된 요청 감지**: 커스텀 로직을 통한 차단된 요청의 자동 감지 및 재시도를 지원합니다. - 📦 **내장 내보내기**: 훅이나 자체 파이프라인, 또는 내장 JSON/JSONL로 결과를 내보냅니다. 각각 `result.items.to_json()` / `result.items.to_jsonl()`을 사용합니다. @@ -256,7 +256,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # 작업 page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False) data = page.css('#padded_content a').getall() -# 또는 일회성 요청 스타일 — 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다 +# 또는 일회성 요청 스타일 - 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다 page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare') data = page.css('#padded_content a').getall() ``` @@ -268,7 +268,7 @@ with DynamicSession(headless=True, disable_resources=False, network_idle=True) a page = session.fetch('https://quotes.toscrape.com/', load_dom=False) data = page.xpath('//span[@class="text"]/text()').getall() # 원하시면 XPath selector도 사용 가능 -# 또는 일회성 요청 스타일 — 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다 +# 또는 일회성 요청 스타일 - 이 요청을 위해 브라우저를 열고, 완료 후 닫습니다 page = DynamicFetcher.fetch('https://quotes.toscrape.com/') data = page.css('.quote .text::text').getall() ``` @@ -520,7 +520,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest ## 감사의 말 이 프로젝트에는 다음에서 차용한 코드가 포함되어 있습니다: -- Parsel (BSD 라이선스) — [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) 서브모듈에 사용 +- Parsel (BSD 라이선스) - [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) 서브모듈에 사용 ---
Karim Shoair가 ❤️으로 디자인하고 만들었습니다.

diff --git a/docs/README_RU.md b/docs/README_RU.md index f3872ac..d80d573 100644 --- a/docs/README_RU.md +++ b/docs/README_RU.md @@ -47,11 +47,11 @@ Режим MCP

-Scrapling — это адаптивный фреймворк для Web Scraping, который берёт на себя всё: от одного запроса до полномасштабного обхода сайтов. +Scrapling - это адаптивный фреймворк для Web Scraping, который берёт на себя всё: от одного запроса до полномасштабного обхода сайтов. -Его парсер учится на изменениях сайтов и автоматически перемещает ваши элементы при обновлении страниц. Его Fetcher'ы обходят анти-бот системы вроде Cloudflare Turnstile прямо из коробки. А его Spider-фреймворк позволяет масштабироваться до параллельных, многосессионных обходов с Pause & Resume и автоматической ротацией Proxy — и всё это в нескольких строках Python. Одна библиотека, без компромиссов. +Его парсер учится на изменениях сайтов и автоматически перемещает ваши элементы при обновлении страниц. Его Fetcher'ы обходят анти-бот системы вроде Cloudflare Turnstile прямо из коробки. А его Spider-фреймворк позволяет масштабироваться до параллельных, многосессионных обходов с Pause & Resume и автоматической ротацией Proxy - и всё это в нескольких строках Python. Одна библиотека, без компромиссов. -Молниеносно быстрые обходы с отслеживанием статистики в реальном времени и Streaming. Создано веб-скраперами для веб-скраперов и обычных пользователей — здесь есть что-то для каждого. +Молниеносно быстрые обходы с отслеживанием статистики в реальном времени и Streaming. Создано веб-скраперами для веб-скраперов и обычных пользователей - здесь есть что-то для каждого. ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher @@ -128,7 +128,7 @@ MySpider().start() - TikHub.io предоставляет более 900 стабильных API на 16+ платформах, включая TikTok, X, YouTube и Instagram, с более чем 40 млн наборов данных.
Также предлагает AI-модели со скидкой — Claude, GPT, GEMINI и другие со скидкой до 71%. + TikHub.io предоставляет более 900 стабильных API на 16+ платформах, включая TikTok, X, YouTube и Instagram, с более чем 40 млн наборов данных.
Также предлагает AI-модели со скидкой - Claude, GPT, GEMINI и другие со скидкой до 71%. @@ -159,7 +159,7 @@ MySpider().start() - Прочитайте полный обзор Scrapling на The Web Scraping Club (ноябрь 2025) — рассылка №1, посвящённая веб-скрейпингу. + Прочитайте полный обзор Scrapling на The Web Scraping Club (ноябрь 2025) - рассылка №1, посвящённая веб-скрейпингу. @@ -194,12 +194,12 @@ MySpider().start() ## Ключевые особенности -### Spider'ы — полноценный фреймворк для обхода сайтов +### Spider'ы - полноценный фреймворк для обхода сайтов - 🕷️ **Scrapy-подобный Spider API**: Определяйте Spider'ов с `start_urls`, async `parse` callback'ами и объектами `Request`/`Response`. - ⚡ **Параллельный обход**: Настраиваемые лимиты параллелизма, ограничение скорости по домену и задержки загрузки. -- 🔄 **Поддержка нескольких сессий**: Единый интерфейс для HTTP-запросов и скрытных headless-браузеров в одном Spider — маршрутизируйте запросы к разным сессиям по ID. +- 🔄 **Поддержка нескольких сессий**: Единый интерфейс для HTTP-запросов и скрытных headless-браузеров в одном Spider - маршрутизируйте запросы к разным сессиям по ID. - 💾 **Pause & Resume**: Persistence обхода на основе Checkpoint'ов. Нажмите Ctrl+C для мягкой остановки; перезапустите, чтобы продолжить с того места, где вы остановились. -- 📡 **Режим Streaming**: Стримьте извлечённые элементы по мере их поступления через `async for item in spider.stream()` со статистикой в реальном времени — идеально для UI, конвейеров и длительных обходов. +- 📡 **Режим Streaming**: Стримьте извлечённые элементы по мере их поступления через `async for item in spider.stream()` со статистикой в реальном времени - идеально для UI, конвейеров и длительных обходов. - 🛡️ **Обнаружение заблокированных запросов**: Автоматическое обнаружение и повторная отправка заблокированных запросов с настраиваемой логикой. - 📦 **Встроенный экспорт**: Экспортируйте результаты через хуки и собственный конвейер или встроенный JSON/JSONL с `result.items.to_json()` / `result.items.to_jsonl()` соответственно. @@ -259,7 +259,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Дер page = session.fetch('https://nopecha.com/demo/cloudflare', google_search=False) data = page.css('#padded_content a').getall() -# Или используйте стиль одноразового запроса — открывает браузер для этого запроса, затем закрывает его после завершения +# Или используйте стиль одноразового запроса - открывает браузер для этого запроса, затем закрывает его после завершения page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare') data = page.css('#padded_content a').getall() ``` @@ -271,7 +271,7 @@ with DynamicSession(headless=True, disable_resources=False, network_idle=True) a page = session.fetch('https://quotes.toscrape.com/', load_dom=False) data = page.xpath('//span[@class="text"]/text()').getall() # XPath-селектор, если вы предпочитаете его -# Или используйте стиль одноразового запроса — открывает браузер для этого запроса, затем закрывает его после завершения +# Или используйте стиль одноразового запроса - открывает браузер для этого запроса, затем закрывает его после завершения page = DynamicFetcher.fetch('https://quotes.toscrape.com/') data = page.css('.quote .text::text').getall() ``` @@ -326,7 +326,7 @@ class MultiSessionSpider(Spider): ```python QuotesSpider(crawldir="./crawl_data").start() ``` -Нажмите Ctrl+C для мягкой остановки — прогресс сохраняется автоматически. Позже, когда вы снова запустите Spider, передайте тот же `crawldir`, и он продолжит с того места, где остановился. +Нажмите Ctrl+C для мягкой остановки - прогресс сохраняется автоматически. Позже, когда вы снова запустите Spider, передайте тот же `crawldir`, и он продолжит с того места, где остановился. ### Продвинутый парсинг и навигация ```python @@ -383,7 +383,7 @@ async with AsyncStealthySession(max_pages=2) as session: task = session.fetch(url) tasks.append(task) - print(session.get_pool_stats()) # Опционально — статус пула вкладок браузера (занят/свободен/ошибка) + print(session.get_pool_stats()) # Опционально - статус пула вкладок браузера (занят/свободен/ошибка) results = await asyncio.gather(*tasks) print(session.get_pool_stats()) ``` @@ -411,7 +411,7 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas. ## Тесты производительности -Scrapling не только мощный — он ещё и невероятно быстрый. Следующие тесты производительности сравнивают парсер Scrapling с последними версиями других популярных библиотек. +Scrapling не только мощный - он ещё и невероятно быстрый. Следующие тесты производительности сравнивают парсер Scrapling с последними версиями других популярных библиотек. ### Тест скорости извлечения текста (5000 вложенных элементов) @@ -523,7 +523,7 @@ docker pull ghcr.io/d4vinci/scrapling:latest ## Благодарности Этот проект включает код, адаптированный из: -- Parsel (лицензия BSD) — Используется для подмодуля [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) +- Parsel (лицензия BSD) - Используется для подмодуля [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) ---
Разработано и создано с ❤️ Карим Шоаир.

diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 40a0fdf..e5be366 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -1,6 +1,6 @@ # Performance Benchmarks -Scrapling isn't just powerful—it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries. +Scrapling isn't just powerful - it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries. ### Text Extraction Speed Test (5000 nested elements) diff --git a/docs/index.md b/docs/index.md index a768d06..912af95 100644 --- a/docs/index.md +++ b/docs/index.md @@ -18,7 +18,7 @@ Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl. -Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation — all in a few lines of Python. One library, zero compromises. +Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises. Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone. @@ -89,12 +89,12 @@ MySpider().start() ## Key Features -### Spiders — A Full Crawling Framework +### Spiders - A Full Crawling Framework - 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects. - ⚡ **Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays. -- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider — route requests to different sessions by ID. +- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID. - 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off. -- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats — ideal for UI, pipelines, and long-running crawls. +- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls. - 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic. - 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively. diff --git a/docs/overview.md b/docs/overview.md index f9a0529..b0d1a4e 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -4,9 +4,9 @@ Not sure where to start? Pick the path that matches what you're trying to do: | I want to... | Start here | |:---|:---| -| **Parse HTML** I already have | [Querying elements](parsing/selection.md) — CSS, XPath, and text-based selection | +| **Parse HTML** I already have | [Querying elements](parsing/selection.md): CSS, XPath, and text-based selection | | **Quickly scrape a page** and prototype | Pick a [fetcher](fetching/choosing.md) and test right away, or launch the [interactive shell](cli/interactive-shell.md) | -| **Build a crawler** that scales | [Spiders](spiders/getting-started.md) — concurrent, multi-session crawls with pause/resume | +| **Build a crawler** that scales | [Spiders](spiders/getting-started.md): concurrent, multi-session crawls with pause/resume | | **Scrape without writing code** | [CLI extract commands](cli/extract-commands.md) or hook up the [MCP server](ai/mcp-server.md) to your favourite AI tool | | **Migrate** from another library | [From BeautifulSoup](tutorials/migrating_from_beautifulsoup.md) or [Scrapy comparison](spiders/architecture.md#comparison-with-scrapy) | diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md index f310037..6d2cac0 100644 --- a/docs/parsing/main_classes.md +++ b/docs/parsing/main_classes.md @@ -306,7 +306,7 @@ Starting with v0.4, [Selector](#selector) and [Selectors](#selectors) both provi **On a [Selector](#selector) object:** -- `get()` returns a `TextHandler` — for text node selectors, it returns the text value; for HTML element selectors, it returns the serialized outer HTML. +- `get()` returns a `TextHandler`: for text node selectors, it returns the text value; for HTML element selectors, it returns the serialized outer HTML. - `getall()` returns a `TextHandlers` list containing the single serialized string. - `extract_first` is an alias for `get()`, and `extract` is an alias for `getall()`. diff --git a/docs/spiders/advanced.md b/docs/spiders/advanced.md index 1a363b9..c35c2b1 100644 --- a/docs/spiders/advanced.md +++ b/docs/spiders/advanced.md @@ -32,7 +32,7 @@ class PoliteSpider(Spider): yield {"title": response.css("title::text").get("")} ``` -When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously — you can allow high global concurrency while being polite to each individual domain. +When `concurrent_requests_per_domain` is set, each domain gets its own concurrency limiter in addition to the global limit. This is useful when crawling multiple domains simultaneously, as you can allow high global concurrency while being polite to each individual domain. !!! tip @@ -66,7 +66,7 @@ else: 1. **Pausing**: Press `Ctrl+C` during a crawl. The spider waits for all in-flight requests to finish, saves a checkpoint (pending requests + a set of seen request fingerprints), and then exits. 2. **Force stopping**: Press `Ctrl+C` a second time to stop immediately without waiting for active tasks. -3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off — skipping `start_requests()`. +3. **Resuming**: Run the spider again with the same `crawldir`. It detects the checkpoint, restores the queue and seen set, and continues from where it left off, skipping `start_requests()`. 4. **Cleanup**: When a crawl completes normally (not paused), the checkpoint files are deleted automatically. **Checkpoints are also saved periodically during the crawl (every 5 minutes by default).** diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md index 09b61e4..4ccfad2 100644 --- a/docs/spiders/architecture.md +++ b/docs/spiders/architecture.md @@ -7,7 +7,7 @@ Scrapling's spider system is a Scrapy-inspired async crawling framework designed for concurrent, multi-session crawls with built-in pause/resume support. It brings together Scrapling's parsing engine and fetchers into a unified crawling API while adding scheduling, concurrency control, and checkpointing. -If you're familiar with Scrapy, you'll feel right at home. If not, don't worry — the system is designed to be straightforward. +If you're familiar with Scrapy, you'll feel right at home. If not, don't worry - the system is designed to be straightforward. ## Data Flow @@ -23,7 +23,7 @@ Here's what happens step by step when you run a spider without many details: 4. The **session** fetches the page and returns a [Response](../fetching/choosing.md#response-object) object to the **Crawler Engine**. The engine records statistics and checks for blocked responses. If the response is blocked, the engine retries the request up to `max_blocked_retries` times. Of course, the blocking detection and the retry logic for blocked requests can be customized. 5. The **Crawler Engine** passes the [Response](../fetching/choosing.md#response-object) to the request's callback. The callback either yields a dictionary, which gets treated as a scraped item, or a follow-up request, which gets sent to the scheduler for queuing. 6. The cycle repeats from step 2 until the scheduler is empty and no tasks are active, or the spider is paused. -7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off — skipping `start_requests()` and restoring the scheduler state. +7. If `crawldir` is set while starting the spider, the **Crawler Engine** periodically saves a checkpoint (pending requests + seen URLs set) to disk. On graceful shutdown (Ctrl+C), a final checkpoint is saved. The next time the spider runs with the same `crawldir`, it resumes from where it left off, skipping `start_requests()` and restoring the scheduler state. ## Components @@ -49,7 +49,7 @@ class MySpider(Spider): ### Crawler Engine -The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly — the `Spider.start()` and `Spider.stream()` methods handle it for you. +The engine orchestrates the entire crawl. It manages the main loop, enforces concurrency limits, dispatches requests through the Session Manager, and processes results from callbacks. You don't interact with it directly - the `Spider.start()` and `Spider.stream()` methods handle it for you. ### Scheduler diff --git a/docs/spiders/getting-started.md b/docs/spiders/getting-started.md index bb547b4..e619268 100644 --- a/docs/spiders/getting-started.md +++ b/docs/spiders/getting-started.md @@ -31,9 +31,9 @@ class QuotesSpider(Spider): Every spider needs three things: -1. **`name`** — A unique identifier for the spider. -2. **`start_urls`** — A list of URLs to start crawling from. -3. **`parse()`** — An async generator method that processes each response and yields results. +1. **`name`** - A unique identifier for the spider. +2. **`start_urls`** - A list of URLs to start crawling from. +3. **`parse()`** - An async generator method that processes each response and yields results. The `parse()` method is where the magic happens. You use the same selection methods you'd use with Scrapling's [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object), and `yield` dictionaries to output scraped items. @@ -45,7 +45,7 @@ To run your spider, create an instance and call `start()`: result = QuotesSpider().start() ``` -The `start()` method handles all the async machinery internally — no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats. +The `start()` method handles all the async machinery internally, so no need to worry about event loops. While the spider is running, everything that happens is logged to the terminal, and at the end of the crawl, you get very detailed stats. Those stats are in the returned `CrawlResult` object, which gives you everything you need: @@ -90,7 +90,7 @@ class QuotesSpider(Spider): yield response.follow(next_page, callback=self.parse) ``` -`response.follow()` handles relative URLs automatically — it joins them with the current page's URL. It also sets the current page as the `Referer` header by default. +`response.follow()` handles relative URLs automatically by joining them with the current page's URL. It also sets the current page as the `Referer` header by default. You can point follow-up requests at different callback methods for different page types: @@ -145,7 +145,7 @@ class MySpider(Spider): yield response.follow(link, callback=self.parse) ``` -Subdomains are matched automatically — setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc. +Subdomains are matched automatically, so setting `allowed_domains = {"example.com"}` also allows `sub.example.com`, `blog.example.com`, etc. When a request is filtered out, it's counted in `stats.offsite_requests_count` so you can see how many were dropped. @@ -153,7 +153,7 @@ When a request is filtered out, it's counted in `stats.offsite_requests_count` s Now that you have the basics, you can explore: -- [Requests & Responses](requests-responses.md) — learn about request priority, deduplication, metadata, and more. -- [Sessions](sessions.md) — use multiple fetcher types (HTTP, browser, stealth) in a single spider. -- [Proxy management & blocking](proxy-blocking.md) — rotate proxies across requests and how to handle blocking in the spider. -- [Advanced features](advanced.md) — concurrency control, pause/resume, streaming, lifecycle hooks, and logging. \ No newline at end of file +- [Requests & Responses](requests-responses.md) - learn about request priority, deduplication, metadata, and more. +- [Sessions](sessions.md) - use multiple fetcher types (HTTP, browser, stealth) in a single spider. +- [Proxy management & blocking](proxy-blocking.md) - rotate proxies across requests and how to handle blocking in the spider. +- [Advanced features](advanced.md) - concurrency control, pause/resume, streaming, lifecycle hooks, and logging. \ No newline at end of file diff --git a/docs/spiders/proxy-blocking.md b/docs/spiders/proxy-blocking.md index 4c829b9..c637592 100644 --- a/docs/spiders/proxy-blocking.md +++ b/docs/spiders/proxy-blocking.md @@ -7,7 +7,7 @@ 1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider. 2. You've read the [Sessions](sessions.md) page and understand how to configure sessions. -When scraping at scale, you'll often need to rotate through multiple proxies to avoid rate limits and blocks. Scrapling's `ProxyRotator` makes this straightforward — it works with all session types and integrates with the spider's blocked request retry system. +When scraping at scale, you'll often need to rotate through multiple proxies to avoid rate limits and blocks. Scrapling's `ProxyRotator` makes this straightforward. It works with all session types and integrates with the spider's blocked request retry system. If you don't know what a proxy is or how to choose a good one, [this guide can help](https://substack.thewebscraping.club/p/everything-about-proxies). @@ -70,7 +70,7 @@ def configure_sessions(self, manager): ## Custom Rotation Strategies -By default, `ProxyRotator` uses cyclic rotation — it iterates through proxies sequentially, wrapping around at the end. +By default, `ProxyRotator` uses cyclic rotation, iterating through proxies sequentially and wrapping around at the end. You can provide a custom strategy function to change this behavior, but it has to match the below signature: diff --git a/docs/spiders/requests-responses.md b/docs/spiders/requests-responses.md index c587af8..f49fd53 100644 --- a/docs/spiders/requests-responses.md +++ b/docs/spiders/requests-responses.md @@ -4,7 +4,7 @@ 1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider. -This page covers the `Request` object in detail — how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following. +This page covers the `Request` object in detail: how to construct requests, pass data between callbacks, control priority and deduplication, and use `response.follow()` for link-following. ## The Request Object @@ -29,7 +29,7 @@ Here are all the arguments you can pass to `Request`: | Argument | Type | Default | Description | |---------------|------------|------------|-------------------------------------------------------------------------------------------------------| | `url` | `str` | *required* | The URL to fetch | -| `sid` | `str` | `""` | Session ID — routes the request to a specific session (see [Sessions](sessions.md)) | +| `sid` | `str` | `""` | Session ID - routes the request to a specific session (see [Sessions](sessions.md)) | | `callback` | `callable` | `None` | Async generator method to process the response. Defaults to `parse()` | | `priority` | `int` | `0` | Higher values are processed first | | `dont_filter` | `bool` | `False` | If `True`, skip deduplication (allow duplicate requests) | @@ -58,7 +58,7 @@ yield Request( ```python async def parse(self, response: Response): - # Minimal — inherits callback, sid, priority from current request + # Minimal - inherits callback, sid, priority from current request yield response.follow("/next-page") # Override specific fields @@ -99,9 +99,9 @@ yield response.follow("/page", referer_flow=False) Callbacks are async generator methods on your spider that process responses. They must `yield` one of three types: -- **`dict`** — A scraped item, added to the results -- **`Request`** — A follow-up request, added to the queue -- **`None`** — Silently ignored +- **`dict`** - A scraped item, added to the results +- **`Request`** - A follow-up request, added to the queue +- **`None`** - Silently ignored ```python class MySpider(Spider): @@ -130,11 +130,11 @@ Requests with higher priority values are processed first. This is useful when so ```python async def parse(self, response: Response): - # High priority — process product pages first + # High priority - process product pages first for link in response.css("a.product::attr(href)").getall(): yield response.follow(link, callback=self.parse_product, priority=10) - # Low priority — pagination links processed after products + # Low priority - pagination links processed after products next_page = response.css("a.next::attr(href)").get() if next_page: yield response.follow(next_page, callback=self.parse, priority=0) diff --git a/docs/spiders/sessions.md b/docs/spiders/sessions.md index 1d1d759..d1b146e 100644 --- a/docs/spiders/sessions.md +++ b/docs/spiders/sessions.md @@ -5,7 +5,7 @@ 1. You've read the [Getting started](getting-started.md) page and know how to create and run a basic spider. 2. You're familiar with [Fetchers basics](../fetching/choosing.md) and the differences between HTTP, Dynamic, and Stealthy sessions. -A spider can use multiple fetcher sessions simultaneously — for example, a fast HTTP session for simple pages and a stealth browser session for protected pages. This page shows you how to configure and use sessions. +A spider can use multiple fetcher sessions simultaneously. For example, a fast HTTP session for simple pages and a stealth browser session for protected pages. This page shows you how to configure and use sessions. ## What are Sessions? @@ -23,7 +23,7 @@ By default, every spider creates a single [FetcherSession](../fetching/static.md ## Configuring Sessions -Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance — use `manager.add()` to register sessions: +Override `configure_sessions()` on your spider to set up sessions. The `manager` parameter is a `SessionManager` instance. Use `manager.add()` to register sessions: ```python from scrapling.spiders import Spider, Response @@ -101,7 +101,7 @@ class ProductSpider(Spider): } ``` -The key is the `sid` parameter — it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited. +The key is the `sid` parameter - it tells the spider which session to use for each request. When you call `response.follow()` without `sid`, the session ID from the original request is inherited. Note that the sessions don't have to be from different classes only, but can be the same session, but different instances with different configurations, for example, like below: diff --git a/docs/tutorials/replacing_ai.md b/docs/tutorials/replacing_ai.md index f752a2d..fe84dbb 100644 --- a/docs/tutorials/replacing_ai.md +++ b/docs/tutorials/replacing_ai.md @@ -8,18 +8,18 @@ In this article, we will discuss these common issues, why companies are shifting If you have been doing Web Scraping for a long time, you probably noticed that there are repeating problems with Web Scraping, like: -1. **Rapidly changing website structures** — Sites frequently update their DOM structures, breaking static XPath/CSS selectors. -2. **Unstable selectors** — Class names and IDs often change or use randomly generated values that break scrapers or make scraping these websites difficult. -3. **Increasingly complex anti-bot measures** — CAPTCHA systems, browser fingerprinting, and behavior analysis make traditional scraping difficult +1. **Rapidly changing website structures** - Sites frequently update their DOM structures, breaking static XPath/CSS selectors. +2. **Unstable selectors** - Class names and IDs often change or use randomly generated values that break scrapers or make scraping these websites difficult. +3. **Increasingly complex anti-bot measures** - CAPTCHA systems, browser fingerprinting, and behavior analysis make traditional scraping difficult and others But that's only if you are doing targeted Web Scraping for known websites, in which case you can write specific code for every website. If you start thinking about bigger goals like Broad Scraping or Generic Web Scraping, or what you like to call it, then the above issues intensify, and you will face new issues like: -1. **Extreme Website Diversity** — Generic scraping must handle countless variations in HTML structures, CSS usage, JavaScript frameworks, and backend technologies. -2. **Identifying Relevant Data** — How does the scraper know what data is important on a page it has never seen before? -3. **Pagination variations** — Infinite scroll, traditional pagination, "load more" buttons, all requiring different approaches +1. **Extreme Website Diversity** - Generic scraping must handle countless variations in HTML structures, CSS usage, JavaScript frameworks, and backend technologies. +2. **Identifying Relevant Data** - How does the scraper know what data is important on a page it has never seen before? +3. **Pagination variations** - Infinite scroll, traditional pagination, "load more" buttons, all requiring different approaches and more How will you solve that manually? I'm referring to generic web scraping of various websites that don't share any common technologies. @@ -61,8 +61,8 @@ There is no need to explain any of these; click on the links, and it will be cle ### Solving issue T3: Increasingly complex anti-bot measures It's well known that creating an undetectable spider requires more than residential/mobile proxies and human-like behavior. It also needs a hard-to-detect browser, which Scrapling provides two main options to solve: -1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) — This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements. -2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html) — Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial! +1. [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) - This fetcher provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements. +2. [StealthyFetcher](https://scrapling.readthedocs.io/en/latest/fetching/stealthy.html) - Because we live in a harsh world and you need to take [full measure instead of half-measures](https://www.youtube.com/watch?v=7BE4QcwX4dU), `StealthyFetcher` was born. This fetcher uses our stealthy browser -- a version of [DynamicFetcher](https://scrapling.readthedocs.io/en/latest/fetching/dynamic.html) that nearly bypasses all annoying anti-protections, provides tools to handle the rest, and automatically bypasses all types of Cloudflare's Turnstile/Interstitial! We keep improving these two with each update, so stay tuned :) @@ -96,7 +96,7 @@ This example illustrates the point I aim to convey here. Not every challenge wil ### Solving issue B3: Pagination variations This issue, Scrapling currently doesn't have a direct method to automatically extract pagination's URLs for you, but it will be added with the upcoming updates :) -But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'`—you get the idea. +But you can handle most websites if you search for the most common patterns with `page.find_by_text('Next')['href']` or `page.find_by_text('load more')['href']` or selectors like `'a[href*="?page="]'` or `'a[href*="/page/"]'` - you get the idea. ## Cost Comparison and Savings For a quick comparison. diff --git a/scrapling/core/ai.py b/scrapling/core/ai.py index 2fabf43..208f3e7 100644 --- a/scrapling/core/ai.py +++ b/scrapling/core/ai.py @@ -430,7 +430,7 @@ class ScraplingMCPServer: Note: This is only suitable for low-mid protection levels. Note: If the `css_selector` resolves to more than one element, all the elements will be returned. Note: If a `session_id` is provided (from open_session), the browser session will be reused instead of creating a new one. - When using a session, browser-level params (headless, proxy, locale, etc.) are ignored — they were set at session creation time. + When using a session, browser-level params (headless, proxy, locale, etc.) are ignored since they were set at session creation time. :param url: The URL to request. :param extraction_type: The type of content to extract from the page. Defaults to "markdown". Options are: @@ -512,7 +512,7 @@ class ScraplingMCPServer: Note: This is only suitable for low-mid protection levels. Note: If the `css_selector` resolves to more than one element, all the elements will be returned. Note: If a `session_id` is provided (from open_session), the browser session will be reused instead of creating a new one. - When using a session, browser-level params (headless, proxy, locale, etc.) are ignored — they were set at session creation time. + When using a session, browser-level params (headless, proxy, locale, etc.) are ignored since they were set at session creation time. :param urls: A list of the URLs to request. :param extraction_type: The type of content to extract from the page. Defaults to "markdown". Options are: @@ -617,7 +617,7 @@ class ScraplingMCPServer: Note: This is the only suitable fetcher for high protection levels. Note: If the `css_selector` resolves to more than one element, all the elements will be returned. Note: If a `session_id` is provided (from open_session), the browser session will be reused instead of creating a new one. - When using a session, browser-level params (headless, proxy, locale, etc.) are ignored — they were set at session creation time. + When using a session, browser-level params (headless, proxy, locale, etc.) are ignored since they were set at session creation time. :param url: The URL to request. :param extraction_type: The type of content to extract from the page. Defaults to "markdown". Options are: @@ -714,7 +714,7 @@ class ScraplingMCPServer: Note: This is the only suitable fetcher for high protection levels. Note: If the `css_selector` resolves to more than one element, all the elements will be returned. Note: If a `session_id` is provided (from open_session), the browser session will be reused instead of creating a new one. - When using a session, browser-level params (headless, proxy, locale, etc.) are ignored — they were set at session creation time. + When using a session, browser-level params (headless, proxy, locale, etc.) are ignored since they were set at session creation time. :param urls: A list of the URLs to request. :param extraction_type: The type of content to extract from the page. Defaults to "markdown". Options are: diff --git a/scrapling/engines/toolbelt/proxy_rotation.py b/scrapling/engines/toolbelt/proxy_rotation.py index 37cdba1..e2703b8 100644 --- a/scrapling/engines/toolbelt/proxy_rotation.py +++ b/scrapling/engines/toolbelt/proxy_rotation.py @@ -31,7 +31,7 @@ def is_proxy_error(error: Exception) -> bool: def cyclic_rotation(proxies: List[ProxyType], current_index: int) -> Tuple[ProxyType, int]: - """Default cyclic rotation strategy — iterates through proxies sequentially, wrapping around at the end.""" + """Default cyclic rotation strategy - iterates through proxies sequentially, wrapping around at the end.""" idx = current_index % len(proxies) return proxies[idx], (idx + 1) % len(proxies) diff --git a/tests/core/test_storage_core.py b/tests/core/test_storage_core.py index 3d7921a..5fb10a6 100644 --- a/tests/core/test_storage_core.py +++ b/tests/core/test_storage_core.py @@ -99,7 +99,7 @@ class TestSQLiteStorageSystem: class TestSaveRetrieveRoundTrip: - """Test the save/retrieve round-trip — the core of the adaptive feature.""" + """Test the save/retrieve round-trip - the core of the adaptive feature.""" def _make_storage(self, url="https://example.com"): StorageSystemMixin._get_base_url.cache_clear() diff --git a/tests/parser/test_find_similar_advanced.py b/tests/parser/test_find_similar_advanced.py index 480f155..099b0dd 100644 --- a/tests/parser/test_find_similar_advanced.py +++ b/tests/parser/test_find_similar_advanced.py @@ -20,7 +20,7 @@ def product_page():
Carrot
- +
Grape
@@ -50,7 +50,7 @@ class TestFindSimilarAdvanced: """match_text=True should factor in text content during similarity scoring""" first = product_page.css("div.product")[0] # Apple # With match_text=True and a high threshold, "Apple" vs "Banana"/"Carrot" text - # should reduce similarity scores — result count may drop + # should reduce similarity scores - result count may drop with_text = first.find_similar(similarity_threshold=0.8, match_text=True) without_text = first.find_similar(similarity_threshold=0.8, match_text=False) # match_text=True is stricter when text differs, so result should be <= without_text diff --git a/tests/parser/test_selectors_filter.py b/tests/parser/test_selectors_filter.py index 9612abc..dbddf03 100644 --- a/tests/parser/test_selectors_filter.py +++ b/tests/parser/test_selectors_filter.py @@ -46,7 +46,7 @@ class TestSelectorsFilter: assert len(result) == len(items) def test_filter_chained(self, page): - """filter() should be chainable — apply two filters in sequence""" + """filter() should be chainable - apply two filters in sequence""" items = page.css("li.item") # First: value > 0, then: not disabled result = (