MyAiDesk/nomifun-tauri/docs/architecture/agent-engine.md

Agent Engine

The agent engine lives under crates/agent/ and is consumed by the backend primarily through nomifun-ai-agent. This page is an implementation map for the current workspace, not an extraction plan.

Crate Map

Crate	Responsibility
nomi-types	Provider-neutral messages, tool types, compaction types, file state, skill types, and spawner types.
nomi-protocol	Host/agent command and event protocol plus approval state.
nomi-compact	Context compaction and message-window shaping.
nomi-config	Runtime/provider/profile/auth configuration.
nomi-providers	Anthropic, OpenAI-compatible, Bedrock, Vertex, and shared streaming/retry/provider logic.
nomi-tools	Built-in tools and tool registry primitives.
nomi-mcp	MCP client, manager, transports, and tool proxying.
nomi-skills	Skill discovery, frontmatter, loading, and skill-index support.
nomi-memory	Memory storage and retrieval primitives.
nomi-agent	Core engine loop, sessions, compaction glue, confirmations, output sinks, skill tool, requirement tools, and subagent spawning.
nomi-cli	Standalone nomi CLI consumer of the engine.
nomi-computer	Desktop computer-use tool implementation.
nomi-a11y	Accessibility helpers for computer-use flows.
nomi-browser-engine	Self-hosted browser/CDP automation engine.
nomi-browser	Browser-use tool facade.

The agent crates do not depend on nomifun-* backend crates. Backend-to-agent integration normally flows through nomifun-ai-agent; feature-gated bridge surfaces in nomifun-app and nomifun-gateway directly depend on browser and computer-use crates to expose those capabilities as stdio/public tools.

Runtime Families

NomiFun supports several runtime families:

• Nomi engine: in-tree engine from nomi-agent, with providers, built-in tools, skills, MCP, memory, browser, and computer-use support.
• ACP-style CLI agents: Claude Code, Codex, Gemini CLI, Qwen/OpenCode-style integrations, and related CLIs managed by nomifun-ai-agent.
• Remote/Open capability surfaces: external agents connect through companion-token authenticated /mcp, /mcp-agent, or /v1 fronts.

The implementation source of truth for factory behavior is:

• crates/backend/nomifun-ai-agent/src/factory/nomi.rs
• crates/backend/nomifun-ai-agent/src/factory/acp.rs
• crates/backend/nomifun-ai-agent/src/factory/acp_assembler.rs

MCP And Tool Injection

MCP/tool availability is assembled per runtime and per session. It is not a single flat list.

Common sources include:

• user-configured MCP server rows from nomifun-mcp,
• requirement declaration tools when AutoWork requires them,
• scoped knowledge search when a session has mounted knowledge bases,
• Desktop Gateway tools for sessions flagged with desktop-gateway access,
• Windows/open helper bridge,
• feature-gated computer-use and browser-use stdio bridges,
• runtime-native skills or first-message skill injection,
• Nomi's native tool registry.

When documenting tool availability, cite the factory files above rather than assuming all agents receive the same injected servers.

Skills

Skills are instruction/tool bundles whose materialization depends on runtime capability:

• Nomi has a real Skill tool path in the engine.
• Native CLI runtimes may receive symlinked/copied skill files or lightweight first-message guidance when the runtime supports it.
• Custom workspace or non-native paths can be summarized in a first-message skill index.

Relevant source files:

• crates/backend/nomifun-extension/src/skill_service.rs
• crates/backend/nomifun-ai-agent/src/capability/skill_manager/mod.rs
• crates/backend/nomifun-ai-agent/src/capability/first_message_injector.rs
• crates/agent/nomi-agent/src/skill_tool.rs

Session Flow

UI request
  -> nomifun-conversation route/service
  -> nomifun-ai-agent AgentService / WorkerTaskManager
  -> runtime family factory
  -> Nomi engine or external CLI process
  -> AgentStreamEvent
  -> nomifun-realtime /ws
  -> renderer stream handlers

Nomi-engine sessions run inside the process. ACP-style sessions spawn and manage child CLIs. Public remote capability calls enter through nomifun-public and the Desktop Gateway registry rather than the conversation HTTP route.

Design Notes

Older specs describe the agent layer as mechanically extraction-ready and list only 11 crates. Those files are historical. The current code still keeps a strong boundary, but browser/computer bridge work and public gateway surfaces mean the real rule is “primary seam plus documented feature-gated exceptions.”