docs: use a better looking architecture image instead of mermaid
This commit is contained in:
Binary file not shown.
|
After Width: | Height: | Size: 127 KiB |
@@ -13,39 +13,7 @@ If you're familiar with Scrapy, you'll feel right at home. If not, don't worry
|
|||||||
|
|
||||||
The diagram below shows how data flows through the spider system when a crawl is running:
|
The diagram below shows how data flows through the spider system when a crawl is running:
|
||||||
|
|
||||||
```mermaid
|
<img src="../assets/spider_architecture.png" title="Spider architecture diagram by @TrueSkills" alt="Spider architecture diagram by @TrueSkills" style="width: 70%;"/>
|
||||||
graph TB
|
|
||||||
Spider["Spider"]
|
|
||||||
Scheduler["Scheduler"]
|
|
||||||
Engine["Crawler Engine"]
|
|
||||||
SessionMgr["Session Manager"]
|
|
||||||
Output["Output"]
|
|
||||||
Checkpoint["Checkpoint"]
|
|
||||||
|
|
||||||
Spider -- "1. Initial Requests" --> Scheduler
|
|
||||||
Scheduler -- "2. Next Request" --> Engine
|
|
||||||
Engine -- "3. Fetch" --> SessionMgr
|
|
||||||
SessionMgr -- "4. Response" --> Engine
|
|
||||||
Engine -- "5. Response to callback" --> Spider
|
|
||||||
Spider -- "6. New Requests" --> Scheduler
|
|
||||||
Spider -- "7. Items" --> Output
|
|
||||||
Engine -. "Save" .-> Checkpoint
|
|
||||||
Checkpoint -. "Resume" .-> Scheduler
|
|
||||||
|
|
||||||
classDef spiderClass fill:#7c4dff,stroke:#333,color:#fff
|
|
||||||
classDef engineClass fill:#00897b,stroke:#333,color:#fff
|
|
||||||
classDef schedulerClass fill:#1565c0,stroke:#333,color:#fff
|
|
||||||
classDef sessionClass fill:#ef6c00,stroke:#333,color:#fff
|
|
||||||
classDef outputClass fill:#2e7d32,stroke:#333,color:#fff
|
|
||||||
classDef checkpointClass fill:#6d4c41,stroke:#333,color:#fff
|
|
||||||
|
|
||||||
class Spider spiderClass
|
|
||||||
class Engine engineClass
|
|
||||||
class Scheduler schedulerClass
|
|
||||||
class SessionMgr sessionClass
|
|
||||||
class Output outputClass
|
|
||||||
class Checkpoint checkpointClass
|
|
||||||
```
|
|
||||||
|
|
||||||
Here's what happens step by step when you run a spider without many details:
|
Here's what happens step by step when you run a spider without many details:
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user