freedak/Scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: simplify the architecture mermaid

Browse Source
This commit is contained in:
Karim shoair
2026-02-12 03:30:14 +02:00
parent 5a599445e2
commit d0b2cec849
1 changed files with 12 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/spiders/architecture.md
+12 -46
View File
@@ -15,57 +15,23 @@ The diagram below shows how data flows through the spider system when a crawl is
```mermaid ```mermaid
graph TB graph TB
subgraph Spider["🕷️ Spider"] Spider["Spider"]
direction TB Scheduler["Scheduler"]
SR["start_requests()"] Engine["Crawler Engine"]
Parse["parse() / callbacks"] SessionMgr["Session Manager"]
Hooks["Lifecycle hooks<br/><small>on_start · on_close · on_error<br/>is_blocked · on_scraped_item</small>"] Output["Output"]
end Checkpoint["Checkpoint"]
subgraph Engine["⚙️ Crawler Engine"] Spider -- "1. Initial Requests" --> Scheduler
direction TB Scheduler -- "2. Next Request" --> Engine
Loop["Crawl loop"]
Conc["Concurrency control<br/><small>Global limiter · Per-domain limiter<br/>Download delay</small>"]
BlockDet["Blocked request<br/>detection & retry"]
end
subgraph Scheduler["📋 Scheduler"]
PQ["Priority queue"]
Dedup["Fingerprint<br/>deduplication"]
end
subgraph SessionMgr["🔌 Session Manager"]
direction TB
Router["Request router<br/><small>Routes by session ID</small>"]
subgraph Sessions["Sessions"]
direction LR
HTTP["FetcherSession<br/><small>HTTP requests</small>"]
Dynamic["AsyncDynamicSession<br/><small>Browser automation</small>"]
Stealth["AsyncStealthySession<br/><small>Anti-bot bypass</small>"]
end
end
subgraph Output["📦 Output"]
Items["ItemList<br/><small>to_json() · to_jsonl()</small>"]
Stream["Streaming<br/><small>async for item in spider.stream()</small>"]
Stats["CrawlStats"]
end
Checkpoint["💾 Checkpoint<br/><small>Pause/Resume</small>"]
%% Data flow
SR -- "1. Initial<br/>Requests" --> Scheduler
Scheduler -- "2. Next<br/>Request" --> Engine
Engine -- "3. Fetch" --> SessionMgr Engine -- "3. Fetch" --> SessionMgr
Router --> Sessions
SessionMgr -- "4. Response" --> Engine SessionMgr -- "4. Response" --> Engine
Engine -- "5. Response" --> Parse Engine -- "5. Response to callback" --> Spider
Parse -- "6a. New Requests" --> Scheduler Spider -- "6. New Requests" --> Scheduler
Parse -- "6b. Items (dict)" --> Output Spider -- "7. Items" --> Output
Engine -. "Periodic save" .-> Checkpoint Engine -. "Save" .-> Checkpoint
Checkpoint -. "Resume" .-> Scheduler Checkpoint -. "Resume" .-> Scheduler
%% Styling
classDef spiderClass fill:#7c4dff,stroke:#333,color:#fff classDef spiderClass fill:#7c4dff,stroke:#333,color:#fff
classDef engineClass fill:#00897b,stroke:#333,color:#fff classDef engineClass fill:#00897b,stroke:#333,color:#fff
classDef schedulerClass fill:#1565c0,stroke:#333,color:#fff classDef schedulerClass fill:#1565c0,stroke:#333,color:#fff