diff --git a/docs/development/adaptive_storage_system.md b/docs/development/adaptive_storage_system.md index e8bb07f..e086dae 100644 --- a/docs/development/adaptive_storage_system.md +++ b/docs/development/adaptive_storage_system.md @@ -1,3 +1,5 @@ +# Writing your retrieval system + Scrapling uses SQLite by default, but this tutorial shows how to write your own storage system to store element properties for the `adaptive` feature. You might want to use Firebase, for example, and share the database between multiple spiders on different machines. It's a great idea to use an online database like that because spiders can share adaptive data with each other. diff --git a/docs/development/scrapling_custom_types.md b/docs/development/scrapling_custom_types.md index d1dd585..2f638a9 100644 --- a/docs/development/scrapling_custom_types.md +++ b/docs/development/scrapling_custom_types.md @@ -1,3 +1,5 @@ +# Using Scrapling's custom types + > You can take advantage of the custom-made types for Scrapling and use them outside the library if you want. It's better than copying their code, after all :) ### All current types can be imported alone, like below diff --git a/docs/fetching/choosing.md b/docs/fetching/choosing.md index 632b22d..5f78c73 100644 --- a/docs/fetching/choosing.md +++ b/docs/fetching/choosing.md @@ -1,3 +1,5 @@ +# Fetchers basics + ## Introduction Fetchers are classes that can do requests or fetch pages for you easily in a single-line fashion with many features and then return a [Response](#response-object) object. Starting with v0.3, all fetchers have separate classes to keep the session running, so for example, a fetcher that uses a browser will keep the browser open till you finish all your requests through it instead of opening multiple browsers. So it depends on your use case. diff --git a/docs/fetching/dynamic.md b/docs/fetching/dynamic.md index d47b909..31574a6 100644 --- a/docs/fetching/dynamic.md +++ b/docs/fetching/dynamic.md @@ -1,4 +1,4 @@ -# Introduction +# Fetching dynamic websites Here, we will discuss the `DynamicFetcher` class (formerly `PlayWrightFetcher`). This class provides flexible browser automation with multiple configuration options and little under-the-hood stealth improvements. diff --git a/docs/fetching/static.md b/docs/fetching/static.md index 479539c..0071785 100644 --- a/docs/fetching/static.md +++ b/docs/fetching/static.md @@ -1,4 +1,4 @@ -# Introduction +# HTTP requests The `Fetcher` class provides rapid and lightweight HTTP requests using the high-performance `curl_cffi` library with a lot of stealth capabilities. diff --git a/docs/fetching/stealthy.md b/docs/fetching/stealthy.md index ea99996..7cd65e8 100644 --- a/docs/fetching/stealthy.md +++ b/docs/fetching/stealthy.md @@ -1,4 +1,4 @@ -# Introduction +# Fetching dynamic websites with hard protections Here, we will discuss the `StealthyFetcher` class. This class is very similar to the [DynamicFetcher](dynamic.md#introduction) class, including the browsers, the automation, and the use of [Playwright's API](https://playwright.dev/python/docs/intro). The main difference is that this class provides advanced anti-bot protection bypass capabilities; most of them are handled automatically under the hood, and the rest is up to you to enable. diff --git a/docs/parsing/adaptive.md b/docs/parsing/adaptive.md index b341751..0fc7255 100644 --- a/docs/parsing/adaptive.md +++ b/docs/parsing/adaptive.md @@ -1,4 +1,4 @@ -## Introduction +# Adaptive scraping !!! success "Prerequisites" diff --git a/docs/parsing/main_classes.md b/docs/parsing/main_classes.md index 795af5c..9f8ec3b 100644 --- a/docs/parsing/main_classes.md +++ b/docs/parsing/main_classes.md @@ -1,4 +1,4 @@ -## Introduction +# Parsing main classes !!! success "Prerequisites" diff --git a/docs/parsing/selection.md b/docs/parsing/selection.md index 7b91155..909cde3 100644 --- a/docs/parsing/selection.md +++ b/docs/parsing/selection.md @@ -1,4 +1,4 @@ -## Introduction +# Querying elements Scrapling currently supports parsing HTML pages exclusively, so it doesn't support XML feeds. This decision was made because the adaptive feature won't work with XML, but that might change soon, so stay tuned :) In Scrapling, there are five main ways to find elements: diff --git a/docs/spiders/architecture.md b/docs/spiders/architecture.md index 249f91c..09b61e4 100644 --- a/docs/spiders/architecture.md +++ b/docs/spiders/architecture.md @@ -1,4 +1,4 @@ -## Introduction +# Spiders architecture !!! success "Prerequisites" diff --git a/docs/spiders/getting-started.md b/docs/spiders/getting-started.md index 04da95d..bb547b4 100644 --- a/docs/spiders/getting-started.md +++ b/docs/spiders/getting-started.md @@ -1,3 +1,5 @@ +# Getting started + ## Introduction !!! success "Prerequisites" diff --git a/docs/spiders/proxy-blocking.md b/docs/spiders/proxy-blocking.md index a17c9f5..4c829b9 100644 --- a/docs/spiders/proxy-blocking.md +++ b/docs/spiders/proxy-blocking.md @@ -203,7 +203,7 @@ class MySpider(Spider): yield {"title": response.css("title::text").get("")} ``` -What happened above is that I left the blocking detection logic unchanged and made the spider mainly use requests until it gets blocked, then it switches to the stealthy browser. +What happened above is that I left the blocking detection logic unchanged and had the spider mainly use requests until it got blocked, then switch to the stealthy browser. Putting it all together: diff --git a/docs/spiders/requests-responses.md b/docs/spiders/requests-responses.md index 6ec78e5..b61e727 100644 --- a/docs/spiders/requests-responses.md +++ b/docs/spiders/requests-responses.md @@ -1,4 +1,4 @@ -## Introduction +# Requests & Responses !!! success "Prerequisites" diff --git a/docs/spiders/sessions.md b/docs/spiders/sessions.md index 50118d7..d922ee1 100644 --- a/docs/spiders/sessions.md +++ b/docs/spiders/sessions.md @@ -1,4 +1,4 @@ -## Introduction +# Spiders sessions !!! success "Prerequisites"