diff --git a/README.md b/README.md index 3c5e5ca..aae6045 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,5 @@ + +

@@ -10,6 +12,8 @@

+ D4Vinci%2FScrapling | Trendshift +
العربيه | Español | Deutsch | 简体中文 | 日本語 | Русский
@@ -33,13 +37,15 @@

Selection methods · - Choosing a fetcher + Fetchers + · + Spiders + · + Proxy Rotation · CLI · - MCP mode - · - Migrating from Beautifulsoup + MCP

Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl. @@ -51,9 +57,9 @@ Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers fo ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive = True -page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Fetch website under the radar! -products = page.css('.product', auto_save=True) # Scrape data that survives website design changes! -products = page.css('.product', adaptive=True) # Later, if the website structure changes, pass `adaptive=True` to find them! +p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Fetch website under the radar! +products = p.css('.product', auto_save=True) # Scrape data that survives website design changes! +products = p.css('.product', adaptive=True) # Later, if the website structure changes, pass `adaptive=True` to find them! ``` Or scale up to full crawls ```python @@ -70,6 +76,7 @@ class MySpider(Spider): MySpider().start() ``` +# Platinum Sponsors # Sponsors @@ -82,7 +89,10 @@ MySpider().start() + + + diff --git a/docs/README_AR.md b/docs/README_AR.md index 4ac86da..35a47aa 100644 --- a/docs/README_AR.md +++ b/docs/README_AR.md @@ -1,3 +1,5 @@ + +

@@ -33,11 +35,13 @@ · اختيار Fetcher · + العناكب + · + تدوير البروكسي + · واجهة سطر الأوامر · وضع MCP - · - الانتقال من Beautifulsoup

Scrapling هو إطار عمل تكيفي لـ Web Scraping يتعامل مع كل شيء من طلب واحد إلى زحف كامل النطاق. @@ -49,9 +53,9 @@ Scrapling هو إطار عمل تكيفي لـ Web Scraping يتعامل مع ك ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive = True -page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # احصل على الموقع بشكل خفي! -products = page.css('.product', auto_save=True) # استخرج بيانات تنجو من تغييرات تصميم الموقع! -products = page.css('.product', adaptive=True) # لاحقاً، إذا تغيرت بنية الموقع، مرر `adaptive=True` للعثور عليها! +p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # احصل على الموقع بشكل خفي! +products = p.css('.product', auto_save=True) # استخرج بيانات تنجو من تغييرات تصميم الموقع! +products = p.css('.product', adaptive=True) # لاحقاً، إذا تغيرت بنية الموقع، مرر `adaptive=True` للعثور عليها! ``` أو توسع إلى عمليات زحف كاملة ```python @@ -69,6 +73,8 @@ MySpider().start() ``` +# الرعاة البلاتينيون + # الرعاة @@ -80,7 +86,10 @@ MySpider().start() + + + diff --git a/docs/README_CN.md b/docs/README_CN.md index b33c422..24112b3 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -1,3 +1,5 @@ + +

@@ -33,11 +35,13 @@ · 选择Fetcher · + 爬虫 + · + 代理轮换 + · CLI · MCP模式 - · - 从Beautifulsoup迁移

Scrapling是一个自适应Web Scraping框架，能处理从单个请求到大规模爬取的一切需求。 @@ -49,9 +53,9 @@ Scrapling是一个自适应Web Scraping框架，能处理从单个请求到大 ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive = True -page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # 隐秘地获取网站！ -products = page.css('.product', auto_save=True) # 抓取在网站设计变更后仍能存活的数据！ -products = page.css('.product', adaptive=True) # 之后，如果网站结构改变，传递 `adaptive=True` 来找到它们！ +p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # 隐秘地获取网站！ +products = p.css('.product', auto_save=True) # 抓取在网站设计变更后仍能存活的数据！ +products = p.css('.product', adaptive=True) # 之后，如果网站结构改变，传递 `adaptive=True` 来找到它们！ ``` 或扩展为完整爬取 ```python @@ -69,6 +73,8 @@ MySpider().start() ``` +# 铂金赞助商 + # 赞助商 @@ -80,7 +86,10 @@ MySpider().start() + + + diff --git a/docs/README_DE.md b/docs/README_DE.md index 7124483..9b0e6a8 100644 --- a/docs/README_DE.md +++ b/docs/README_DE.md @@ -1,3 +1,5 @@ + +

@@ -33,11 +35,13 @@ · Einen Fetcher wählen · + Spiders + · + Proxy-Rotation + · CLI · MCP-Modus - · - Migration von Beautifulsoup

Scrapling ist ein adaptives Web-Scraping-Framework, das alles abdeckt -- von einer einzelnen Anfrage bis hin zu einem umfassenden Crawl. @@ -49,9 +53,9 @@ Blitzschnelle Crawls mit Echtzeit-Statistiken und Streaming. Von Web Scrapern f ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive = True -page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Website unbemerkt abrufen! -products = page.css('.product', auto_save=True) # Daten scrapen, die Website-Designänderungen überleben! -products = page.css('.product', adaptive=True) # Später, wenn sich die Website-Struktur ändert, `adaptive=True` übergeben, um sie zu finden! +p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Website unbemerkt abrufen! +products = p.css('.product', auto_save=True) # Daten scrapen, die Website-Designänderungen überleben! +products = p.css('.product', adaptive=True) # Später, wenn sich die Website-Struktur ändert, `adaptive=True` übergeben, um sie zu finden! ``` Oder auf vollständige Crawls hochskalieren ```python @@ -69,6 +73,8 @@ MySpider().start() ``` +# Platin-Sponsoren + # Sponsoren @@ -80,7 +86,10 @@ MySpider().start() + + + diff --git a/docs/README_ES.md b/docs/README_ES.md index 8957c41..5fe6834 100644 --- a/docs/README_ES.md +++ b/docs/README_ES.md @@ -1,3 +1,5 @@ + +

@@ -29,15 +31,17 @@

- Metodos de seleccion + Métodos de selección · Elegir un fetcher · + Spiders + · + Rotación de proxy + · CLI · Modo MCP - · - Migrar desde Beautifulsoup

Scrapling es un framework de Web Scraping adaptativo que se encarga de todo, desde una sola solicitud hasta un rastreo a gran escala. @@ -49,9 +53,9 @@ Rastreos ultrarrápidos con estadísticas en tiempo real y Streaming. Construido ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive = True -page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # ¡Obtén el sitio web bajo el radar! -products = page.css('.product', auto_save=True) # ¡Extrae datos que sobreviven a cambios de diseño del sitio web! -products = page.css('.product', adaptive=True) # Más tarde, si la estructura del sitio web cambia, ¡pasa `adaptive=True` para encontrarlos! +p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # ¡Obtén el sitio web bajo el radar! +products = p.css('.product', auto_save=True) # ¡Extrae datos que sobreviven a cambios de diseño del sitio web! +products = p.css('.product', adaptive=True) # Más tarde, si la estructura del sitio web cambia, ¡pasa `adaptive=True` para encontrarlos! ``` O escala a rastreos completos ```python @@ -69,6 +73,8 @@ MySpider().start() ``` +# Patrocinadores Platino + # Patrocinadores @@ -80,7 +86,10 @@ MySpider().start() + + + diff --git a/docs/README_JP.md b/docs/README_JP.md index 3423f49..d423d9b 100644 --- a/docs/README_JP.md +++ b/docs/README_JP.md @@ -1,3 +1,5 @@ + +

@@ -33,11 +35,13 @@ · Fetcherの選び方 · + スパイダー + · + プロキシローテーション + · CLI · MCPモード - · - Beautifulsoupからの移行

Scraplingは、単一のリクエストから本格的なクロールまですべてを処理する適応型Web Scrapingフレームワークです。 @@ -49,9 +53,9 @@ Scraplingは、単一のリクエストから本格的なクロールまです ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive = True -page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # レーダーの下でウェブサイトを取得！ -products = page.css('.product', auto_save=True) # ウェブサイトのデザイン変更に耐えるデータをスクレイプ！ -products = page.css('.product', adaptive=True) # 後でウェブサイトの構造が変わったら、`adaptive=True`を渡して見つける！ +p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # レーダーの下でウェブサイトを取得！ +products = p.css('.product', auto_save=True) # ウェブサイトのデザイン変更に耐えるデータをスクレイプ！ +products = p.css('.product', adaptive=True) # 後でウェブサイトの構造が変わったら、`adaptive=True`を渡して見つける！ ``` または本格的なクロールへスケールアップ ```python @@ -69,6 +73,8 @@ MySpider().start() ``` +# プラチナスポンサー + # スポンサー @@ -80,7 +86,10 @@ MySpider().start() + + + diff --git a/docs/README_RU.md b/docs/README_RU.md index dc59ba6..0343cc0 100644 --- a/docs/README_RU.md +++ b/docs/README_RU.md @@ -1,3 +1,5 @@ + +

@@ -33,11 +35,13 @@ · Выбор Fetcher · + Пауки + · + Ротация прокси + · CLI · Режим MCP - · - Миграция с Beautifulsoup

Scrapling — это адаптивный фреймворк для Web Scraping, который берёт на себя всё: от одного запроса до полномасштабного обхода сайтов. @@ -49,9 +53,9 @@ Scrapling — это адаптивный фреймворк для Web Scraping ```python from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher StealthyFetcher.adaptive = True -page = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Загрузите сайт незаметно! -products = page.css('.product', auto_save=True) # Скрапьте данные, которые переживут изменения дизайна сайта! -products = page.css('.product', adaptive=True) # Позже, если структура сайта изменится, передайте `adaptive=True`, чтобы найти их! +p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=True) # Загрузите сайт незаметно! +products = p.css('.product', auto_save=True) # Скрапьте данные, которые переживут изменения дизайна сайта! +products = p.css('.product', adaptive=True) # Позже, если структура сайта изменится, передайте `adaptive=True`, чтобы найти их! ``` Или масштабируйте до полного обхода ```python @@ -69,6 +73,8 @@ MySpider().start() ``` +# Платиновые спонсоры + # Спонсоры @@ -80,7 +86,10 @@ MySpider().start() + + + diff --git a/docs/donate.md b/docs/donate.md index f5464a0..9c8349e 100644 --- a/docs/donate.md +++ b/docs/donate.md @@ -1,24 +1,30 @@ -I've been working on Scrapling and other public projects in my spare time and have invested considerable resources and effort to make them available to the community for free. By becoming a sponsor, you would directly fund my coffee reserves, helping me continuously update existing projects and create new ones. +I've been creating all of these projects in my spare time and have invested considerable resources & effort in providing them to the community for free. By becoming a sponsor, you'd be directly funding my coffee reserves, helping me fulfill my responsibilities, and enabling me to continuously update existing projects and potentially create new ones. You can sponsor me directly through the [GitHub Sponsors program](https://github.com/sponsors/D4Vinci) or [Buy Me a Coffee](https://buymeacoffee.com/d4vinci). Thank you, stay curious, and hack the planet! ❤️ ## Advertisement -If you are looking to **advertise** your business through Scrapling and take advantage of our target audience, check out the [available tiers](https://github.com/sponsors/D4Vinci): +If you are looking to **advertise** your business to our target audience, check out the [available tiers](https://github.com/sponsors/D4Vinci): -### [The Silver tier](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=435496) ($50/month) +### 1. [The Silver tier](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=435496) ($50/month) Perks: -- Your logo will be featured at [the top of Scrapling's project page](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#sponsors). -- The same logo will be featured at [the top of Scrapling's PyPI page](https://pypi.org/project/scrapling/). -- The same logo will be featured at [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling). +1. Your logo will be featured at [the top of Scrapling's project page](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#sponsors). +2. The same logo will be featured at [the top of Scrapling's PyPI page](https://pypi.org/project/scrapling/) and [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling), the same way it was placed on the project's page. -### [The Gold tier](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=435495) ($100/month) +### 2. [The Gold tier](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=435495) ($100/month) Perks: -- Your logo will be featured at [the top of Scrapling's project page](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#sponsors). -- The same logo will be featured at [the top of Scrapling's PyPI page](https://pypi.org/project/scrapling/). -- The same logo will be featured at [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling). -- Your logo will be featured as a top sponsor on [Scrapling's website](https://scrapling.readthedocs.io/en/latest/) main page. -- A Shoutout with each [Release note](https://github.com/D4Vinci/Scrapling/releases). +1. Your logo will be featured at [the top of Scrapling's project page](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#sponsors). +2. The same logo will be featured at [the top of Scrapling's PyPI page](https://pypi.org/project/scrapling/) and [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling), the same way it was placed on the project's page. +3. Your logo will be featured as a top sponsor on [Scrapling's website](https://scrapling.readthedocs.io/en/latest/) main page. + +### 3. [The Platinum tier](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646) ($300/month) +Perks: + +1. Your logo will have a special placement at [the very top of Scrapling's project page](https://github.com/D4Vinci/Scrapling?tab=readme-ov-file#platinum-sponsors) with an 80-word paragraph or less. +2. The same logo will be featured at [the top of Scrapling's PyPI page](https://pypi.org/project/scrapling/) and [the top of Docker's image page](https://hub.docker.com/r/pyd4vinci/scrapling), the same way it was placed on the project's page. +3. Your logo will have a special placement as a top sponsor on [Scrapling's website](https://scrapling.readthedocs.io/en/latest/) main page. +4. A partner role at our Discord server. +5. A Shoutout at the end of each [Release notes](https://github.com/D4Vinci/Scrapling/releases). \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index 3e53e33..15f7b5d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -55,6 +55,7 @@ MySpider().start() + diff --git a/docs/tutorials/external.md b/docs/tutorials/external.md deleted file mode 100644 index 15a5f9b..0000000 --- a/docs/tutorials/external.md +++ /dev/null @@ -1,34 +0,0 @@ - -If you have issues with the browser installation, such as resource management, we recommend you try the Cloud Browser from [Scrapeless](https://www.scrapeless.com/en/product/scraping-browser?utm_source=official&utm_term=scrapling) for free! - -The usage is straightforward: create an account and [get your API key](https://docs.scrapeless.com/en/scraping-browser/quickstart/getting-started/?utm_source=official&utm_term=scrapling), then pass it to the `DynamicSession` like this: - -```python -from urllib.parse import urlencode - -from scrapling.fetchers import DynamicSession - -# Configure your browser session -config = { - "token": "YOUR_API_KEY", - "sessionName": "scrapling-session", - "sessionTTL": "300", # 5 minutes - "proxyCountry": "ANY", - "sessionRecording": "false", -} - -# Build WebSocket URL -ws_endpoint = f"wss://browser.scrapeless.com/api/v2/browser?{urlencode(config)}" -print('Connecting to Scrapeless...') - -with DynamicSession(cdp_url=ws_endpoint, disable_resources=True) as s: - print("Connected!") - page = s.fetch("https://httpbin.org/headers", network_idle=True) - print(f"Page loaded, content length: {len(page.body)}") - print(page.json()) -``` -The `DynamicSession` class instance will work as usual, so no further explanation is needed. - -However, the Scrapeless Cloud Browser can be configured with proxy options, like the proxy country in the config above, [custom fingerprint](https://docs.scrapeless.com/en/scraping-browser/features/advanced-privacy-anti-detection/custom-fingerprint/?utm_source=official&utm_term=scrapling) configuration, [captcha solving](https://docs.scrapeless.com/en/scraping-browser/features/advanced-privacy-anti-detection/supported-captchas/?utm_source=official&utm_term=scrapling), and more. - -Check out the [Scrapeless's browser documentation](https://docs.scrapeless.com/en/scraping-browser/quickstart/introduction/?utm_source=official&utm_term=scrapling) for more details. \ No newline at end of file diff --git a/images/ProxyEmpire.png b/images/ProxyEmpire.png new file mode 100644 index 0000000..d407789 Binary files /dev/null and b/images/ProxyEmpire.png differ diff --git a/server.json b/server.json new file mode 100644 index 0000000..c88d9d1 --- /dev/null +++ b/server.json @@ -0,0 +1,50 @@ +{ + "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json", + "name": "io.github.D4Vinci/Scrapling", + "title": "Scrapling MCP Server", + "description": "Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.", + "websiteUrl": "https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html", + "repository": { + "url": "https://github.com/D4Vinci/Scrapling", + "source": "github" + }, + "icons": [ + { + "src": "https://raw.githubusercontent.com/D4Vinci/Scrapling/main/docs/assets/logo.png", + "mimeType": "image/png" + } + ], + "version": "0.4.0", + "packages": [ + { + "registryType": "pypi", + "identifier": "scrapling", + "version": "0.4.0", + "runtimeHint": "uvx", + "packageArguments": [ + { + "type": "positional", + "valueHint": "mcp", + "isFixed": true + } + ], + "transport": { + "type": "stdio" + } + }, + { + "registryType": "oci", + "identifier": "ghcr.io/d4vinci/scrapling", + "packageArguments": [ + { + "type": "positional", + "valueHint": "mcp", + "isFixed": true + } + ], + "transport": { + "type": "stdio" + } + } + ] +} \ No newline at end of file diff --git a/zensical.toml b/zensical.toml index 06b5575..0b52de7 100644 --- a/zensical.toml +++ b/zensical.toml @@ -50,8 +50,7 @@ nav = [ ]}, {Tutorials = [ {"A Free Alternative to AI for Robust Web Scraping" = "tutorials/replacing_ai.md"}, - {"Migrating from BeautifulSoup" = "tutorials/migrating_from_beautifulsoup.md"}, - {"Using Scrapeless browser" = "tutorials/external.md"} + {"Migrating from BeautifulSoup" = "tutorials/migrating_from_beautifulsoup.md"} ]}, {Development = [ {"API Reference" = [