freedak/MyAiDesk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update: 将子项目从 submodule 转为完整内容

Browse Source 
- 移除 GovAI, nomifun-tauri, 算力盒子 的 submodule 引用
- 添加所有子项目的完整源代码
- 保留原始 .git 为 .git.bak 备份
This commit is contained in:
freedak
2026-07-04 19:20:46 +08:00
parent 54d6465fa7
commit f7a720204a
3360 changed files with 802660 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
nomifun-tauri/crates/agent/nomi-browser/tests/fixtures/e2e-form.html
Vendored
+55
View File
@@ -0,0 +1,55 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>P2 F3 multi-step e2e form</title>
<style>
html, body { margin: 0; padding: 0; font: 16px sans-serif; }
/* 所有可交互元素固定在视口内 + 大尺寸，让 facade 经真实 getContentQuads 稳定命中
（无 DPR、不依赖滚动）。 */
form#signup { margin: 8px; }
#signup label { display: block; margin: 6px 0; }
#username, #password { display: block; width: 280px; height: 32px; margin: 4px 0; }
#plan { display: block; width: 280px; height: 32px; margin: 4px 0; }
#submit { display: block; width: 240px; height: 40px; margin: 8px 0; }
</style>
</head>
<body>
<h1>P2 F3 e2e form</h1>
<!--
F3 端到端多步 fixture（facade BrowserTool::execute 真 Chrome 跑）：
navigate → observe → type username → type password → select_option plan → click submit。
submit 按钮 accname 含 "Submit"（→ facade redline classify_action 判 Irreversible）：
- yolo/审批旁路会话 click submit → facade redline 门 hard-deny Blocked（红线生效证据）；
- 普通会话 click submit → 门不拦（交 orchestration），表单真提交 → onsubmit 写可见标记。
onsubmit preventDefault（不真导航），把提交时捕获的 username + 所选 plan 写进 #form-status（role=status，
aria 可观测），证明 type/select 真写入 + click submit 真触发。password 值不回显（脱敏精神）。
-->
<form id="signup" action="javascript:void(0)">
<label>Username <input id="username" name="username" type="text" autocomplete="username"></label>
<label>Password <input id="password" name="password" type="password" autocomplete="new-password"></label>
<label>Plan
<select id="plan" name="plan" aria-label="Plan">
<option value="free">Free</option>
<option value="pro">Pro</option>
<option value="enterprise">Enterprise</option>
</select>
</label>
<button id="submit" type="submit" aria-label="Submit order">Submit order</button>
</form>
<!-- 提交后的可见标记（aria 可观测：role=status）。初始 idle。 -->
<div id="form-status" role="status" aria-label="form status">idle</div>
<script>
document.getElementById('signup').addEventListener('submit', function (e) {
e.preventDefault();
var u = document.getElementById('username').value;
var p = document.getElementById('plan').value;
// 把提交时捕获的 username + plan 写进可见标记（证明 type/select 真写入 + submit 真触发）；
// 不回显 password 值（脱敏精神）。
document.getElementById('form-status').textContent = 'submitted:' + u + ':' + p;
});
</script>
</body>
</html>
nomifun-tauri/crates/agent/nomi-browser/tests/fixtures/extract-products.html
Vendored
+17
View File
@@ -0,0 +1,17 @@
<!DOCTYPE html>
<html lang="en">
<head><meta charset="UTF-8"><title>Products Table</title></head>
<body>
<h1>Product Catalog</h1>
<table id="products">
<thead>
<tr><th>Name</th><th>Price</th><th>In Stock</th></tr>
</thead>
<tbody>
<tr><td>Widget A</td><td>$9.99</td><td>Yes</td></tr>
<tr><td>Gadget B</td><td>$19.50</td><td>No</td></tr>
<tr><td>Doohickey C</td><td>$4.25</td><td>Yes</td></tr>
</tbody>
</table>
</body>
</html>
nomifun-tauri/crates/agent/nomi-browser/tests/fixtures/redline_gate.html
Vendored
+23
View File
@@ -0,0 +1,23 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8">
<title>F1-sec redline gate facade test</title>
<style>
html, body { margin: 0; padding: 0; font: 16px sans-serif; }
button { display: block; margin: 12px; padding: 8px 16px; }
</style>
</head>
<body>
<h1>F1-sec redline gate</h1>
<!--
F1-sec facade 集成测试用：一个不可逆（accname="Pay now"）按钮 + 一个良性（"Show more"）按钮。
facade 的 redline 门据 observe 的 accname 分类 click：
- yolo/companion（审批旁路）会话点 "Pay now" → hard-deny Blocked（证 fail-open 已闭）；
- 普通会话点 "Pay now" → 门不拦（交 orchestration）。
用 file:// fixture（非 data: URL），与其它集成测试同接线，避免 data: URL 解析坑。
-->
<button id="pay" aria-label="Pay now">Pay now</button>
<button id="more" aria-label="Show more">Show more</button>
</body>
</html>
nomifun-tauri/crates/agent/nomi-browser/tests/fixtures/site-memory.html
Vendored
+18
View File
@@ -0,0 +1,18 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Site Memory Fixture</title>
</head>
<body>
<h1>Site Memory Test Page</h1>
<button id="action-btn" aria-label="Perform Action">Perform Action</button>
<a href="#help" id="help-link">Help Center</a>
<div id="status">ready</div>
<script>
document.getElementById('action-btn').addEventListener('click', function() {
document.getElementById('status').textContent = 'clicked';
});
</script>
</body>
</html>
nomifun-tauri/crates/agent/nomi-browser/tests/fixtures/som-fallback.html
Vendored
+21
View File
@@ -0,0 +1,21 @@
<!doctype html>
<html>
<head>
<meta charset="utf-8" />
<title>SoM fallback test</title>
<!-- Stack the buttons vertically so the SoM overlay's (y, then x) numbering is
deterministic: the topmost button ("Alpha") is always label 1. -->
<style>
button { display: block; margin: 12px; width: 200px; height: 36px; }
</style>
</head>
<body>
<h1>SoM fallback test</h1>
<!-- Real, accessible buttons → each gets an aria ref + a bounding box during observe.
They set distinct results so the test can prove WHICH one the SoM click landed on. -->
<button id="b-top" onclick="document.getElementById('result').textContent = 'top-clicked'">Alpha</button>
<button id="b-mid" onclick="document.getElementById('result').textContent = 'mid-clicked'">Bravo</button>
<button id="b-bot" onclick="document.getElementById('result').textContent = 'bottom-clicked'">Charlie</button>
<div id="result" role="status">none</div>
</body>
</html>
nomifun-tauri/crates/agent/nomi-browser/tests/fixtures/visual-fallback-canvas.html
Vendored
+47
View File
@@ -0,0 +1,47 @@
<!DOCTYPE html>
<html>
<head>
<title>Visual Fallback Canvas Test</title>
<style>
body { margin: 0; padding: 0; }
canvas { display: block; }
</style>
</head>
<body>
<!-- A canvas-only page with NO accessible button in the DOM.
The "button" is drawn purely as pixels on the canvas.
DOM/aria anchoring WILL FAIL for this element (no accessible tree entry).
This exercises the visual fallback path. -->
<canvas id="c" width="400" height="300"></canvas>
<div id="click-result" style="display:none;"></div>
<script>
const canvas = document.getElementById('c');
const ctx = canvas.getContext('2d');
// Draw a "Submit" button at (150, 120) with size (100, 40).
const btnX = 150, btnY = 120, btnW = 100, btnH = 40;
function drawButton() {
ctx.fillStyle = '#4CAF50';
ctx.fillRect(btnX, btnY, btnW, btnH);
ctx.fillStyle = 'white';
ctx.font = '16px Arial';
ctx.textAlign = 'center';
ctx.textBaseline = 'middle';
ctx.fillText('Submit', btnX + btnW / 2, btnY + btnH / 2);
}
drawButton();
// Listen for clicks on the canvas — if within the button bounds, record it.
canvas.addEventListener('click', function(e) {
const rect = canvas.getBoundingClientRect();
const x = e.clientX - rect.left;
const y = e.clientY - rect.top;
if (x >= btnX && x <= btnX + btnW && y >= btnY && y <= btnY + btnH) {
document.getElementById('click-result').textContent = 'canvas-button-clicked';
document.getElementById('click-result').style.display = 'block';
}
});
</script>
</body>
</html>
nomifun-tauri/crates/agent/nomi-browser/tests/integration_e2e.rs
+761
View File
@@ -0,0 +1,761 @@
//! **P2 F3：多步 e2e（facade 端到端）+ 安全门生效证据**（`#[ignore]`，本机/打包 chrome）。
//!
//! 这是 P2 收官的端到端验证：**经 `BrowserTool` facade（`Tool::execute`）**串起完整真实流程，证明
//! P2 的各组件（navigate settle / observe ref 表 / actionability 五检查 + 三级兜底 / verify-after-act /
//! 不可逆分类器 + facade 独立 fail-closed 门 / secret 域绑定）在真 Chrome 上**协同工作**。
//!
//! 与 engine 层集成测试（`nomi-browser-engine/tests/integration_act.rs` 的 `c1_*`/`c2_*`）的区别：
//! 那些直接驱动 `engine.act(&ActSpec, &Progress)`（引擎契约）；本测试走**更高层**——经 facade 的
//! `execute(json!{...})`（LLM 真正调用的入口），故同时覆盖：①facade 的 dispatch/参数解析；②facade 的
//! redline 独立门（在 dispatch 前拦审批旁路会话的不可逆动作）；③facade 的 `secret:NAME` origin 门。
//!
//! ## 覆盖的 P2 DoD 验收点
//! - **多步协同**：navigate → observe → type username → type password → select_option Pro → click submit
//! （普通会话 submit 真提交 → onsubmit 标记 `submitted:<user>:<plan>`，经再 observe 读回证实）。
//! - **安全门生效（红线）**：
//! 1. **yolo/审批旁路会话** click submit（accname="Submit order" → 分类 Irreversible）→ facade redline
//! 门 **hard-deny Blocked**（设计裁决⑧：不靠被旁路的 orchestration，靠 facade 独立 fail-closed 门）；
//! 2. **普通会话** 同一 submit → 门**不拦**（交 orchestration），动作真执行；
//! 3. **secret 域绑定 fail-closed**：`secret:NAME` 在 file:// 源（无 eTLD+1）→ Blocked，明文不入输出。
//!
//! 手动跑（本机 Windows 有系统 Chrome）：
//! set NOMIFUN_CHROME_BINARY=C:\Program Files\Google\Chrome\Application\chrome.exe
//! cargo nextest run -p nomi-browser --run-ignored all -E 'test(e2e)'
//! 跑完核对任务管理器无残留 chrome（engine 的 Builder kill_on_drop 应自动清；tool Drop 即释放）。
use std::path::PathBuf;
use nomi_browser::BrowserTool;
use nomi_config::config::BrowserConfig;
use nomi_tools::Tool;
use serde_json::json;
/// fixture 的 file:// URL。`CARGO_MANIFEST_DIR` 在 unix 是 `/abs`（已带前导斜杠）、在 windows
/// 是 `C:/abs`（需补一个），故仅缺失时补斜杠——避免 unix 上 `file:///{manifest}` 产生四斜杠
/// （`file:////...`）触发 chrome 归一成三斜杠 → navigate redirect 误判。
fn fixture_url(name: &str) -> String {
let manifest = env!("CARGO_MANIFEST_DIR").replace('\\', "/");
let abs = if manifest.starts_with('/') {
manifest
} else {
format!("/{manifest}")
};
format!("file://{abs}/tests/fixtures/{name}")
}
/// 从 facade observe 的 aria YAML 文本里，按 `role` + accname 子串找到 `[ref=f<seq>e<n>]`。
///
/// observe 输出形如 `- textbox "Username" [ref=f0e1]` / `- button "Submit order" [ref=f0e4]`。
/// 我们找含 role 词 + accname 子串 + `[ref=` 标记的那一行，抽出 ref。facade 不暴露结构化
/// `Observation`（那是 engine 契约），故按 LLM 真正看到的文本解析（与模型同视角）。
fn find_ref(observe_text: &str, role: &str, accname: &str) -> String {
observe_text
.lines()
.find(|line| line.contains(role) && line.contains(accname) && line.contains("[ref="))
.and_then(|line| {
let start = line.find("[ref=")? + 5;
let end = line[start..].find(']')? + start;
Some(line[start..end].to_string())
})
.unwrap_or_else(|| {
panic!("observe output should expose a {role:?} with accname {accname:?}; got:\n{observe_text}")
})
}
/// headless BrowserConfig（本机集成测试默认 headless；不依赖显示）。
fn headless_config() -> BrowserConfig {
BrowserConfig { headless: true, ..Default::default() }
}
/// 本测试专属隔离 data_dir（避免与运行中的 app browser-data 争用同一 profile）。
fn isolated_data_dir(suffix: &str) -> PathBuf {
std::env::temp_dir().join(format!("nomifun-f3-e2e-{suffix}-data"))
}
/// **多步 e2e（普通会话，经 facade）+ 安全门「普通会话 submit 不被门拦」证据。**
///
/// navigate → observe → type username → type password → select_option Pro → click submit →
/// 再 observe 读回 `#form-status == submitted:e2e-user:pro`（证 type/select 真写入 + submit 真触发，
/// 且普通会话的 Irreversible submit **未被 facade 门拦**——门方向正确：只拦审批旁路会话）。
#[tokio::test]
#[ignore = "需本机/打包 chrome：set NOMIFUN_CHROME_BINARY 后 --run-ignored all"]
async fn e2e_multistep_form_flow_through_facade_normal_session() {
// 普通会话（session_bypasses_approval=false）：facade redline 门不拦不可逆动作（交 orchestration）。
let tool = BrowserTool::with_data_dir(isolated_data_dir("normal"), false);
// ── 1. navigate ────────────────────────────────────────────────────────────
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("e2e-form.html")}))
.await;
eprintln!("navigate -> is_error={} content={:?}", nav.is_error, nav.content);
assert!(!nav.is_error, "navigate must succeed: {}", nav.content);
assert!(nav.content.contains("Navigated to"), "navigate message: {}", nav.content);
// ── 2. observe（填 ref 表 + 武装注入侧 elements 缓存，act 反查的前置）────────────
let obs = tool.execute(json!({"action": "observe"})).await;
eprintln!("=== observe output ===\n{}", obs.content);
assert!(!obs.is_error, "observe must succeed: {}", obs.content);
let user_ref = find_ref(&obs.content, "textbox", "Username");
let pass_ref = find_ref(&obs.content, "textbox", "Password");
let plan_ref = find_ref(&obs.content, "combobox", "Plan");
let submit_ref = find_ref(&obs.content, "button", "Submit order");
eprintln!("refs: user={user_ref} pass={pass_ref} plan={plan_ref} submit={submit_ref}");
// ── 3. type username（literal）→ verify changed ──────────────────────────────
let type_user = tool
.execute(json!({"action": "type", "ref": user_ref, "text": "e2e-user"}))
.await;
eprintln!("type username -> is_error={} content={:?}", type_user.is_error, type_user.content);
assert!(!type_user.is_error, "type username must succeed: {}", type_user.content);
assert!(type_user.content.contains("changed=true"), "type username should change value: {}", type_user.content);
// ── 4. type password（literal——secret 路径的 fail-closed 在专门用例验，见下；正向 secret
// 路径需真 http 源 + eTLD+1，离线 file:// 测不到，由 facade/engine 既有测试覆盖）─────
let type_pass = tool
.execute(json!({"action": "type", "ref": pass_ref, "text": "literal-pw-not-secret"}))
.await;
eprintln!("type password -> is_error={} content={:?}", type_pass.is_error, type_pass.content);
assert!(!type_pass.is_error, "type password must succeed: {}", type_pass.content);
assert!(type_pass.content.contains("changed=true"), "type password should change value: {}", type_pass.content);
// ── 5. select_option Pro → verify after-anchor 含 "pro"（C2 修复点：读 .value 非 textContent）─
let select = tool
.execute(json!({"action": "select_option", "ref": plan_ref, "options": ["Pro"]}))
.await;
eprintln!("select_option -> is_error={} content={:?}", select.is_error, select.content);
assert!(!select.is_error, "select_option must succeed: {}", select.content);
assert!(select.content.contains("changed=true"), "select Pro should change value (free→pro): {}", select.content);
assert!(
select.content.contains("pro"),
"select_option verify after-anchor should reflect the chosen value 'pro': {}",
select.content
);
// ── 6. 普通会话 click submit（accname="Submit order" → Irreversible）→ 门不拦 + 真提交 ──
// 先确认 facade 把它分类为 Irreversible（category_for 据 last_snapshot 的 accname 判）。
assert_eq!(
tool.category_for(&json!({"action": "click", "ref": submit_ref})),
nomi_protocol::events::ToolCategory::Irreversible,
"submit-order click must classify as Irreversible (so orchestration prompts in a normal session)"
);
let submit = tool
.execute(json!({"action": "click", "ref": submit_ref}))
.await;
eprintln!("click submit (normal session) -> is_error={} content={:?}", submit.is_error, submit.content);
// 普通会话：facade 门**不**hard-deny（方向正确）。click 真执行（成功或良性失败，但绝不是 Blocked）。
let lower = submit.content.to_lowercase();
assert!(
!(submit.is_error && (lower.contains("blocked") || lower.contains("irreversible"))),
"normal-session irreversible submit must NOT be hard-denied by the facade gate: {}",
submit.content
);
// ── 7. 再 observe 读回 #form-status（role=status）== submitted:e2e-user:pro ─────────
let after = tool.execute(json!({"action": "observe"})).await;
eprintln!("=== observe after submit ===\n{}", after.content);
assert!(!after.is_error, "post-submit observe must succeed: {}", after.content);
assert!(
after.content.contains("submitted:e2e-user:pro"),
"form submit should fire with the typed username + selected plan (onsubmit marker); \
observe output:\n{}",
after.content
);
eprintln!(
"=== F3 E2E READBACK SUMMARY (normal session) ===\n\
navigate = ok\n\
observe refs = user={user_ref} pass={pass_ref} plan={plan_ref} submit={submit_ref}\n\
type user = changed=true\n\
type pass = changed=true\n\
select Pro = changed=true (value 'pro')\n\
submit = NOT blocked in normal session (classified Irreversible → orchestration)\n\
form-status = submitted:e2e-user:pro (onsubmit fired)"
);
}
/// **安全门生效证据（红线）：审批旁路（yolo/companion）会话里的不可逆 submit → facade hard-deny Blocked。**
///
/// 这是设计裁决⑧的端到端证明：不靠被旁路的 orchestration 审批闸，靠 facade 的独立 fail-closed 门。
/// 经 `with_policy(.., session_bypasses_approval=true, ..)` 构造一个审批旁路会话的 tool（= yolo / companion
/// 强制 yolo / --auto-approve 的等价 test seam），navigate + observe 真页拿到真 submit ref（accname
/// "Submit order" → 分类 Irreversible），然后 `execute(click submit)` → **Blocked**（门在 dispatch 之前拦）。
#[tokio::test]
#[ignore = "需本机/打包 chrome：set NOMIFUN_CHROME_BINARY 后 --run-ignored all"]
async fn e2e_security_gate_blocks_irreversible_submit_in_bypassing_session() {
// 审批旁路会话（with_policy 第二参 = config.tools.auto_approve = true）→ redline 门武装。
// 注意：with_policy 用 app_config_dir 的 browser-data；为隔离，先 with_data_dir 再... 但 with_data_dir
// 不带 policy。这里直接用 with_policy（headless）——它的 data_dir 是 app browser-data；本测试只 navigate
// 一个 file:// fixture（不落数据），且 chrome user-data-dir 由 engine 专属管理，争用风险低。
let tool = BrowserTool::with_policy(&headless_config(), /* session_bypasses_approval */ true, false, false, None, None, None);
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("e2e-form.html")}))
.await;
assert!(!nav.is_error, "navigate must succeed: {}", nav.content);
let obs = tool.execute(json!({"action": "observe"})).await;
assert!(!obs.is_error, "observe must succeed: {}", obs.content);
let submit_ref = find_ref(&obs.content, "button", "Submit order");
eprintln!("yolo session submit ref = {submit_ref}");
// 旁路会话 + 不可逆 submit → facade redline 门 hard-deny（dispatch 前拦）。
let blocked = tool
.execute(json!({"action": "click", "ref": submit_ref}))
.await;
eprintln!("yolo click submit -> is_error={} content={:?}", blocked.is_error, blocked.content);
assert!(
blocked.is_error,
"irreversible submit in an approval-bypassing session MUST be hard-denied: {}",
blocked.content
);
let lower = blocked.content.to_lowercase();
assert!(
lower.contains("blocked") || lower.contains("irreversible"),
"block message should explain the redline (blocked/irreversible): {}",
blocked.content
);
eprintln!(
"=== F3 SECURITY GATE EVIDENCE ===\n\
session = approval-bypassing (yolo/companion/auto_approve)\n\
action = click submit [ref={submit_ref}] (accname 'Submit order' → Irreversible)\n\
result = HARD-DENY Blocked (facade fail-closed gate, NOT orchestration)\n\
message = {:?}",
blocked.content
);
}
/// **安全门生效证据（secret 域绑定 fail-closed）：`secret:NAME` 在无 eTLD+1 的 file:// 源 → Blocked，
/// 明文绝不入输出。**
///
/// secret 正向注入路径需真 http 源（eTLD+1 域绑定），离线 file:// 无 registrable domain → 域门 fail-closed。
/// 这正好验**最关键的安全方向**：源不匹配 / 无源 → 拒绝解析，且 `secret:NAME` 字面量绝不当普通文本输入、
/// 也绝不泄漏配置的值。即便 yolo 会话也拦（门是 vault 的属性，非 orchestration 审批）。
#[tokio::test]
#[ignore = "需本机/打包 chrome：set NOMIFUN_CHROME_BINARY 后 --run-ignored all"]
async fn e2e_secret_origin_gate_fails_closed_on_file_origin() {
use nomifun_secret::SecretStore;
// 配一个绑定到 example.com 的 secret（其值绝不应出现在任何输出里）。
let mut store = SecretStore::ephemeral().expect("ephemeral store");
let secret_plaintext = "F3-TOP-SECRET-PLAINTEXT-must-never-leak";
store
.register("login_pw", secret_plaintext, vec!["example.com".to_string()])
.expect("register secret");
let tool = BrowserTool::with_secret_store(isolated_data_dir("secret"), false, store);
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("e2e-form.html")}))
.await;
assert!(!nav.is_error, "navigate must succeed: {}", nav.content);
let obs = tool.execute(json!({"action": "observe"})).await;
assert!(!obs.is_error, "observe must succeed: {}", obs.content);
let pass_ref = find_ref(&obs.content, "textbox", "Password");
// current origin = file://...e2e-form.html → 无 eTLD+1 → secret 域门 fail-closed（即便 secret 存在）。
let res = tool
.execute(json!({"action": "type", "ref": pass_ref, "text": "secret:login_pw"}))
.await;
eprintln!("type secret on file:// origin -> is_error={} content={:?}", res.is_error, res.content);
assert!(
res.is_error,
"a secret bound to example.com must NOT resolve on a file:// origin (fail-closed): {}",
res.content
);
// 安全铁律：明文绝不出现在错误输出里；`secret:login_pw` 字面量也不能被当普通文本输入（值不泄漏）。
assert!(
!res.content.contains(secret_plaintext),
"SECURITY: the secret plaintext must NEVER appear in the tool output: {}",
res.content
);
eprintln!(
"=== F3 SECRET GATE EVIDENCE ===\n\
origin = file:// (no registrable eTLD+1)\n\
secret = bound to example.com (mismatch)\n\
result = fail-closed Blocked; plaintext NOT typed, NOT in output\n\
message = {:?}",
res.content
);
}
// ─── P3 Structured Extract (real Chrome + stub model) ───────────────────────
use std::sync::Arc;
use nomi_browser::extract::ExtractModel;
/// A stub model that "extracts" by returning a hardcoded JSON response.
/// In a real scenario the LLM would parse the aria snapshot; here we simulate
/// a correct extraction to verify the end-to-end facade wiring.
struct StubExtractModel;
#[async_trait::async_trait]
impl ExtractModel for StubExtractModel {
async fn complete(&self, _prompt: &str) -> Result<String, String> {
// Return structured JSON matching the schema we'll request.
Ok(r#"{"products": [{"name": "Widget A", "price": 9.99}, {"name": "Gadget B", "price": 19.50}, {"name": "Doohickey C", "price": 4.25}]}"#.into())
}
}
/// **P3 e2e: structured extract with a stub model on a real Chrome page.**
///
/// navigate fixture table → Extract{schema} with StubExtractModel injected →
/// verify the response is the model's structured JSON (not the raw deterministic payload).
///
/// Run: `NOMIFUN_CHROME_BINARY="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" cargo nextest run -p nomi-browser --run-ignored all -E 'test(e2e_structured_extract)'`
#[tokio::test]
#[ignore = "需本机/打包 chrome：set NOMIFUN_CHROME_BINARY 后 --run-ignored all"]
async fn e2e_structured_extract_with_stub_model() {
let data_dir = isolated_data_dir("extract");
let tool = BrowserTool::with_data_dir(data_dir.clone(), false)
.with_extract_model(Arc::new(StubExtractModel));
// Navigate to the fixture table.
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("extract-products.html")}))
.await;
eprintln!("navigate -> is_error={} content={:?}", nav.is_error, nav.content);
assert!(!nav.is_error, "navigate must succeed: {}", nav.content);
// Run Extract with a schema requesting products.
let extract = tool
.execute(json!({
"action": "extract",
"schema": {
"type": "object",
"required": ["products"],
"properties": {
"products": {
"type": "array"
}
}
}
}))
.await;
eprintln!("extract -> is_error={} content={:?}", extract.is_error, extract.content);
assert!(!extract.is_error, "extract must succeed: {}", extract.content);
// The output should be the model's structured JSON (pretty-printed).
let parsed: serde_json::Value = serde_json::from_str(&extract.content)
.expect("extract output must be valid JSON when model is available");
assert!(parsed.get("products").is_some(), "response must have 'products' field");
let products = parsed["products"].as_array().unwrap();
assert_eq!(products.len(), 3, "expected 3 products");
assert_eq!(products[0]["name"], "Widget A");
assert_eq!(products[1]["price"], 19.50);
eprintln!("=== P3 STRUCTURED EXTRACT EVIDENCE ===\nmodel output parsed as valid JSON with schema fields");
let _ = std::fs::remove_dir_all(&data_dir);
}
/// **P3 e2e: extract WITHOUT model returns deterministic payload (graceful degradation).**
#[tokio::test]
#[ignore = "需本机/打包 chrome：set NOMIFUN_CHROME_BINARY 后 --run-ignored all"]
async fn e2e_extract_without_model_returns_deterministic_payload() {
let data_dir = isolated_data_dir("extract-no-model");
// No model injected → graceful degradation.
let tool = BrowserTool::with_data_dir(data_dir.clone(), false);
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("extract-products.html")}))
.await;
assert!(!nav.is_error, "navigate must succeed: {}", nav.content);
let extract = tool
.execute(json!({
"action": "extract",
"schema": { "type": "object", "required": ["products"] }
}))
.await;
eprintln!("extract (no model) -> is_error={} content length={}", extract.is_error, extract.content.len());
assert!(!extract.is_error, "extract must succeed even without model");
// Without model, the output is the deterministic payload (not JSON-parseable as structured data).
assert!(
extract.content.contains("structured page representation")
|| extract.content.contains("accessibility snapshot")
|| extract.content.contains("[visible text]"),
"without model, output must be the engine's deterministic payload, got: {}",
&extract.content[..extract.content.len().min(200)]
);
let _ = std::fs::remove_dir_all(&data_dir);
}
/// **Task 6 P7C: record→replay e2e** (`#[ignore]`, needs `NOMIFUN_CHROME_BINARY`).
///
/// Records click + type on a fixture form, replays on a fresh page, asserts the
/// same end-state. Proves the full record→replay pipeline end-to-end with a real
/// browser.
///
/// Run: `NOMIFUN_CHROME_BINARY="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
/// cargo nextest run -p nomi-browser --run-ignored all -E 'test(record_replay_e2e_smoke)'`
#[tokio::test]
#[ignore = "需 NOMIFUN_CHROME_BINARY（真 Chrome）：record→replay 端到端冒烟"]
async fn record_replay_e2e_smoke() {
use nomi_browser::recording::{RecordedStep, Recording};
use nomi_browser::replay::ReplayRunner;
let data_dir = isolated_data_dir("record-replay");
let tool = BrowserTool::with_data_dir(data_dir.clone(), false);
// 1) Navigate to the fixture form.
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("e2e-form.html")}))
.await;
assert!(!nav.is_error, "navigate: {}", nav.content);
// 2) Observe to get refs.
let obs = tool.execute(json!({"action": "observe"})).await;
assert!(!obs.is_error, "observe: {}", obs.content);
let user_ref = find_ref(&obs.content, "textbox", "Username");
let pass_ref = find_ref(&obs.content, "textbox", "Password");
eprintln!("record: user_ref={user_ref}, pass_ref={pass_ref}");
// 3) Start recording and type into the username field.
tool.start_recording();
assert!(tool.is_recording());
let type_res = tool
.execute(json!({"action": "type", "ref": &user_ref, "text": "replay-test-user"}))
.await;
assert!(!type_res.is_error, "type: {}", type_res.content);
let type_pass = tool
.execute(json!({"action": "type", "ref": &pass_ref, "text": "replay-pass-123"}))
.await;
assert!(!type_pass.is_error, "type pass: {}", type_pass.content);
// 4) Stop recording.
let recording = tool.stop_recording().expect("should have recording");
assert_eq!(recording.steps.len(), 2, "should have 2 recorded steps");
assert_eq!(recording.steps[0].action, "type");
assert_eq!(recording.steps[1].action, "type");
eprintln!("recorded {} steps", recording.steps.len());
// 5) Navigate to a fresh instance of the same page.
let nav2 = tool
.execute(json!({"action": "navigate", "url": fixture_url("e2e-form.html")}))
.await;
assert!(!nav2.is_error, "navigate fresh: {}", nav2.content);
// 6) Observe on the fresh page to get new refs.
let obs2 = tool.execute(json!({"action": "observe"})).await;
assert!(!obs2.is_error, "observe fresh: {}", obs2.content);
let new_user_ref = find_ref(&obs2.content, "textbox", "Username");
let new_pass_ref = find_ref(&obs2.content, "textbox", "Password");
eprintln!("replay: new_user_ref={new_user_ref}, new_pass_ref={new_pass_ref}");
// 7) Build a replay recording with the fresh refs (simulating selector→ref
// re-resolution that a real replay system would do).
let replay_recording = Recording {
steps: vec![
RecordedStep {
intent: recording.steps[0].intent.clone(),
action: "type".into(),
args: json!({"ref": &new_user_ref, "text": "replay-test-user"}),
selector: recording.steps[0].selector.clone(),
url: recording.steps[0].url.clone(),
},
RecordedStep {
intent: recording.steps[1].intent.clone(),
action: "type".into(),
args: json!({"ref": &new_pass_ref, "text": "replay-pass-123"}),
selector: recording.steps[1].selector.clone(),
url: recording.steps[1].url.clone(),
},
],
created_url: recording.created_url.clone(),
};
// 8) Replay.
let replay_result = ReplayRunner::replay(&replay_recording, &tool).await;
assert_eq!(
replay_result.succeeded, 2,
"both replay steps should succeed; outcomes: {:?}",
replay_result.outcomes.iter().map(|o| (&o.action, o.success, &o.result.content)).collect::<Vec<_>>()
);
assert_eq!(replay_result.failed, 0);
// 9) Verify the page state matches: re-observe and check the inputs have values.
let final_obs = tool.execute(json!({"action": "observe"})).await;
assert!(!final_obs.is_error, "final observe: {}", final_obs.content);
eprintln!(
"=== P7C RECORD→REPLAY E2E ===\n\
recorded = 2 type actions\n\
replayed = 2 steps, all succeeded\n\
pipeline = recording → fresh page → re-resolve refs → replay via act path\n\
gates intact = replay dispatches through execute() (same path as live actions)"
);
let _ = std::fs::remove_dir_all(&data_dir);
}
///
/// Proves the full takeover flow end-to-end against a real browser:
/// 1. Opens a headful window with a bypass session + takeover enabled.
/// 2. Navigates to a form, observes to get refs.
/// 3. Clicks the "Submit order" button (irreversible).
/// 4. With force_resolution=Confirmed, the redline gate releases the action.
/// 5. The submit actually executes (verify via re-observe).
///
/// Manual run:
/// NOMIFUN_CHROME_BINARY="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
/// cargo nextest run -p nomi-browser --run-ignored all -E 'test(takeover_smoke)'
#[tokio::test]
#[ignore = "requires NOMIFUN_CHROME_BINARY + display (headful takeover smoke)"]
async fn takeover_smoke_confirmed_releases_irreversible_through_facade() {
use nomi_browser::takeover::TakeoverResolution;
let data_dir = isolated_data_dir("takeover-smoke");
// Bypass session (yolo) + takeover enabled with forced Confirmed.
let mut tool = BrowserTool::with_policy(
&BrowserConfig { headless: true, ..Default::default() },
true, // session_bypasses_approval
false, // evaluate_full_power
false, // evaluate_persistent_login
None,
None,
None,
);
tool.takeover_controller_mut().enabled = true;
tool.takeover_controller_mut().force_resolution = Some(TakeoverResolution::Confirmed);
// Navigate.
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("e2e-form.html")}))
.await;
eprintln!("takeover smoke: navigate -> {}", nav.content);
assert!(!nav.is_error, "navigate: {}", nav.content);
// Observe.
let obs = tool.execute(json!({"action": "observe"})).await;
assert!(!obs.is_error, "observe: {}", obs.content);
let submit_ref = find_ref(&obs.content, "button", "Submit order");
eprintln!("takeover smoke: submit_ref={submit_ref}");
// Click submit (irreversible in bypass session → takeover → Confirmed → proceeds).
let click = tool
.execute(json!({"action": "click", "ref": submit_ref}))
.await;
eprintln!(
"takeover smoke: click submit -> is_error={} content={}",
click.is_error,
&click.content[..click.content.len().min(200)]
);
// With Confirmed takeover, the action should proceed past the redline gate.
assert!(
!click.content.to_lowercase().contains("blocked"),
"Confirmed takeover must release the submit past the redline gate: {}",
click.content
);
// must_re_observe should be set after the Confirmed takeover.
assert!(
tool.needs_re_observe(),
"must_re_observe should be set after Confirmed takeover"
);
// Re-observe to clear the flag and verify the submit went through.
let obs2 = tool.execute(json!({"action": "observe"})).await;
assert!(!obs2.is_error, "re-observe: {}", obs2.content);
assert!(
!tool.needs_re_observe(),
"must_re_observe should be cleared after observe"
);
let _ = std::fs::remove_dir_all(&data_dir);
}
/// **Task 6 P7B: visual-fallback canvas smoke** (`#[ignore]`, needs `NOMIFUN_CHROME_BINARY`).
///
/// Navigates to a `<canvas>` fixture with NO accessible button in the DOM (the "button"
/// is drawn purely as pixels on the canvas). Asserts:
/// 1. DOM/aria anchoring fails (observe does not expose the canvas "button").
/// 2. With a stub locator returning the known button box coordinates, the visual
/// fallback click lands correctly (verified by checking `#click-result` text).
///
/// This proves the full visual fallback path end-to-end with a real Chrome:
/// navigate → observe (no ref for canvas button) → attempt click with stale/fake ref
/// → NodeStale → visual fallback → locator returns known coords → DPR mapping
/// → click_at_css_point → canvas click handler fires.
///
/// Run:
/// ```sh
/// NOMIFUN_CHROME_BINARY="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
/// cargo nextest run -p nomi-browser --run-ignored all -E 'test(visual_fallback_canvas_smoke)'
/// ```
#[tokio::test]
#[ignore = "需 NOMIFUN_CHROME_BINARY（真 Chrome）：visual-fallback canvas 冒烟"]
async fn visual_fallback_canvas_smoke() {
use nomi_browser::visual_fallback::{PixelBox, VisualLocateResult, VisualLocator};
use std::sync::Arc;
/// Stub locator that returns the known canvas button center coordinates.
/// The button is drawn at (150, 120) with size (100, 40) — center = (200, 140).
/// In headless Chrome (DPR=1.0), pixel coords == CSS coords.
struct CanvasButtonLocator;
#[async_trait::async_trait]
impl VisualLocator for CanvasButtonLocator {
async fn locate(
&self,
_screenshot: &[u8],
_instruction: &str,
) -> Result<VisualLocateResult, String> {
Ok(VisualLocateResult {
pixel_box: PixelBox {
x: 150.0,
y: 120.0,
width: 100.0,
height: 40.0,
},
confidence: 1.0,
})
}
}
let data_dir = isolated_data_dir("visual-fallback-canvas");
let tool = BrowserTool::with_data_dir(data_dir.clone(), false)
.with_visual_fallback_enabled(true)
.with_visual_locator(Arc::new(CanvasButtonLocator));
// 1. Navigate to the canvas fixture.
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("visual-fallback-canvas.html")}))
.await;
eprintln!("navigate -> is_error={} content={:?}", nav.is_error, nav.content);
assert!(!nav.is_error, "navigate must succeed: {}", nav.content);
// 2. Observe — the canvas button should NOT appear in the accessibility tree.
let obs = tool.execute(json!({"action": "observe"})).await;
eprintln!("=== observe output ===\n{}", obs.content);
assert!(!obs.is_error, "observe must succeed: {}", obs.content);
// The canvas is just a generic element — no "Submit" button is exposed.
assert!(
!obs.content.contains("Submit") || obs.content.contains("canvas"),
"observe must NOT expose the canvas-drawn button as an interactive element"
);
// 3. Attempt a click with a deliberately stale ref (from the observe output, there is
// no ref for the canvas button). Use a fake ref that doesn't exist — this will
// trigger NodeStale, which then triggers the visual fallback.
let click_result = tool
.execute(json!({"action": "click", "ref": "f999e999"}))
.await;
eprintln!("click (stale ref) -> is_error={} content={:?}", click_result.is_error, click_result.content);
// The visual fallback should have fired and clicked at (200, 140) CSS pixels
// (center of the button box).
assert!(
click_result.content.contains("via visual fallback"),
"expected visual fallback to fire: {}",
click_result.content
);
// 4. Verify the click actually landed on the canvas button by checking #click-result.
// Wait a moment for the click handler to fire.
tokio::time::sleep(std::time::Duration::from_millis(200)).await;
let obs2 = tool.execute(json!({"action": "observe"})).await;
eprintln!("=== post-click observe ===\n{}", obs2.content);
// The click handler sets #click-result text to "canvas-button-clicked".
assert!(
obs2.content.contains("canvas-button-clicked"),
"the visual fallback click must have landed on the canvas button (expected \
'canvas-button-clicked' in post-click observe): {}",
obs2.content
);
let _ = std::fs::remove_dir_all(&data_dir);
}
/// **P7B SoM (Set-of-Marks) visual-fallback e2e** (`#[ignore]`, needs `NOMIFUN_CHROME_BINARY`).
///
/// Proves the full SoM path on real Chrome: `observe` (with visual fallback on) collects per-ref
/// CSS-pixel boxes → a stale-ref click triggers the fallback → the facade draws a numbered overlay
/// on the screenshot and asks the (stub) locator for a label → the label maps back to the real
/// button's CSS center → the click lands on it. Three stacked real buttons make label numbering
/// deterministic: the topmost ("Alpha") is always label 1, which the stub picks.
///
/// Run: `NOMIFUN_CHROME_BINARY="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
/// cargo nextest run -p nomi-browser --run-ignored all -E 'test(visual_fallback_som_smoke)'`
#[tokio::test]
#[ignore = "需 NOMIFUN_CHROME_BINARY（真 Chrome）：visual-fallback SoM 冒烟"]
async fn visual_fallback_som_smoke() {
use nomi_browser::visual_fallback::{SomLabelResult, VisualLocateResult, VisualLocator};
use std::sync::Arc;
/// Stub SoM locator: always picks label 1 (the topmost button = "Alpha"). Its `locate`
/// (raw bbox) returns Err so that IF the code fell back to raw instead of SoM, the click
/// would fail — making a green test proof that the SoM path actually ran.
struct PickLabelOne;
#[async_trait::async_trait]
impl VisualLocator for PickLabelOne {
async fn locate(
&self,
_screenshot: &[u8],
_instruction: &str,
) -> Result<VisualLocateResult, String> {
Err("raw bbox path must not be used in the SoM smoke".to_string())
}
async fn locate_labeled(
&self,
_annotated_screenshot: &[u8],
_instruction: &str,
_n_labels: usize,
) -> Result<SomLabelResult, String> {
Ok(SomLabelResult { label: 1, confidence: 1.0 })
}
}
let data_dir = isolated_data_dir("visual-fallback-som");
let tool = BrowserTool::with_data_dir(data_dir.clone(), false)
.with_visual_fallback_enabled(true)
.with_visual_locator(Arc::new(PickLabelOne));
// 1. Navigate to the multi-button fixture.
let nav = tool
.execute(json!({"action": "navigate", "url": fixture_url("som-fallback.html")}))
.await;
assert!(!nav.is_error, "navigate must succeed: {}", nav.content);
// 2. Observe — visual_fallback_enabled ⇒ observe collects per-ref boxes (cached for SoM).
let obs = tool.execute(json!({"action": "observe"})).await;
assert!(!obs.is_error, "observe must succeed: {}", obs.content);
// 3. Click with a deliberately stale ref → NodeStale → visual fallback → SoM mode (boxes
// are cached, count is in range). The stub picks label 1 = the topmost button "Alpha".
let click_result = tool
.execute(json!({"action": "click", "ref": "f999e999"}))
.await;
eprintln!(
"click (stale ref) -> is_error={} content={:?}",
click_result.is_error, click_result.content
);
assert!(
click_result.content.contains("via visual fallback (SoM)"),
"expected the SoM path to fire (not raw bbox): {}",
click_result.content
);
// 4. Verify the click landed on the topmost button (label 1) by its distinct result.
tokio::time::sleep(std::time::Duration::from_millis(200)).await;
let obs2 = tool.execute(json!({"action": "observe"})).await;
eprintln!("=== post-click observe ===\n{}", obs2.content);
assert!(
obs2.content.contains("top-clicked"),
"the SoM click must have landed on the topmost button (label 1) — expected \
'top-clicked' in post-click observe: {}",
obs2.content
);
let _ = std::fs::remove_dir_all(&data_dir);
}
nomifun-tauri/crates/agent/nomi-browser/tests/site_memory.rs
+153
View File
@@ -0,0 +1,153 @@
//! Tests for the site-memory module (P7A).
use nomi_browser::site_memory::{key_for, InMemorySink, SiteMemoryEntry, SiteMemoryStore};
use std::collections::HashMap;
#[test]
fn etld1_key_groups_subdomains() {
// mail.google.com and drive.google.com share eTLD+1 "google.com".
let k1 = key_for("https://mail.google.com/x");
let k2 = key_for("https://drive.google.com/y");
assert_eq!(k1, k2);
assert_eq!(k1, Some("google.com".to_string()));
// co.uk multi-level suffix: a.co.uk and b.co.uk are DISTINCT eTLD+1s.
let ka = key_for("https://www.a.co.uk/page");
let kb = key_for("https://www.b.co.uk/page");
assert_ne!(ka, kb);
assert_eq!(ka, Some("a.co.uk".to_string()));
assert_eq!(kb, Some("b.co.uk".to_string()));
// IP / localhost → None (no registrable domain).
assert_eq!(key_for("http://127.0.0.1/foo"), None);
assert_eq!(key_for("http://localhost:3000/bar"), None);
}
#[test]
fn record_then_query_returns_hint() {
let sink = InMemorySink::new();
let store = SiteMemoryStore::new(Box::new(sink));
let entry = SiteMemoryEntry {
etld1: "google.com".into(),
url_pattern: "https://mail.google.com/inbox".into(),
intent: "click".into(),
role: "button".into(),
accessible_name: "Compose".into(),
selector: Some("div[gh=cm]".into()),
from_secret: false,
};
store.record(entry.clone());
let results = store.query("google.com");
assert_eq!(results.len(), 1);
assert_eq!(results[0].role, "button");
assert_eq!(results[0].accessible_name, "Compose");
assert_eq!(results[0].selector, Some("div[gh=cm]".to_string()));
// Different eTLD+1 returns empty.
let results2 = store.query("github.com");
assert!(results2.is_empty());
}
#[test]
fn record_skips_secret_sourced_descriptor() {
let sink = InMemorySink::new();
let store = SiteMemoryStore::new(Box::new(sink));
// Case 1: from_secret = true → dropped.
let secret_entry = SiteMemoryEntry {
etld1: "bank.com".into(),
url_pattern: "https://bank.com/login".into(),
intent: "type".into(),
role: "textbox".into(),
accessible_name: "Password".into(),
selector: Some("#pw".into()),
from_secret: true,
};
store.record(secret_entry);
assert!(store.query("bank.com").is_empty(), "from_secret=true must be dropped");
// Case 2: accessible_name is a redaction placeholder → dropped.
let redacted_entry = SiteMemoryEntry {
etld1: "bank.com".into(),
url_pattern: "https://bank.com/login".into(),
intent: "click".into(),
role: "textbox".into(),
accessible_name: "[KNOWN_SECRET_REDACTED]".into(),
selector: Some("#secret-field".into()),
from_secret: false,
};
store.record(redacted_entry);
assert!(store.query("bank.com").is_empty(), "redaction placeholder must be dropped");
// Case 3: Another redaction marker variant.
let redacted_entry2 = SiteMemoryEntry {
etld1: "bank.com".into(),
url_pattern: "https://bank.com/login".into(),
intent: "type".into(),
role: "textbox".into(),
accessible_name: "OTP [REDACTED]".into(),
selector: None,
from_secret: false,
};
store.record(redacted_entry2);
assert!(store.query("bank.com").is_empty(), "[REDACTED] in name must be dropped");
// Case 4: Normal (non-secret) entry IS persisted.
let normal_entry = SiteMemoryEntry {
etld1: "bank.com".into(),
url_pattern: "https://bank.com/dashboard".into(),
intent: "click".into(),
role: "button".into(),
accessible_name: "Transfer".into(),
selector: Some("#transfer-btn".into()),
from_secret: false,
};
store.record(normal_entry);
let results = store.query("bank.com");
assert_eq!(results.len(), 1, "non-secret entry should persist");
assert_eq!(results[0].accessible_name, "Transfer");
}
#[test]
fn stale_descriptor_invalidated_on_role_mismatch() {
let sink = InMemorySink::new();
let store = SiteMemoryStore::new(Box::new(sink));
// Record two entries with selectors.
let entry_a = SiteMemoryEntry {
etld1: "example.com".into(),
url_pattern: "https://example.com/page".into(),
intent: "click".into(),
role: "button".into(),
accessible_name: "Submit".into(),
selector: Some("#submit-btn".into()),
from_secret: false,
};
let entry_b = SiteMemoryEntry {
etld1: "example.com".into(),
url_pattern: "https://example.com/page".into(),
intent: "click".into(),
role: "link".into(),
accessible_name: "Help".into(),
selector: Some("a.help".into()),
from_secret: false,
};
store.record(entry_a);
store.record(entry_b);
assert_eq!(store.query("example.com").len(), 2);
// Current observe: #submit-btn is now a "link" with name "Back" (role mismatch → stale).
// a.help still matches.
let mut current_by_selector = HashMap::new();
current_by_selector.insert("#submit-btn".to_string(), ("link".to_string(), "Back".to_string()));
current_by_selector.insert("a.help".to_string(), ("link".to_string(), "Help".to_string()));
store.reconcile("example.com", &current_by_selector);
let remaining = store.query("example.com");
assert_eq!(remaining.len(), 1, "stale entry should be dropped");
assert_eq!(remaining[0].accessible_name, "Help");
assert_eq!(remaining[0].selector, Some("a.help".to_string()));
}
nomifun-tauri/crates/agent/nomi-browser/tests/site_memory_e2e.rs
+85
View File
@@ -0,0 +1,85 @@
//! **P7A: Site-memory real-Chrome smoke test** (`#[ignore]`, requires NOMIFUN_CHROME_BINARY).
//!
//! Navigates to `https://example.com`, hovers an element (non-navigating action),
//! verifies site memory records the element, then observes again to confirm hints
//! are attached.
//!
//! Run:
//! export NOMIFUN_CHROME_BINARY="/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
//! cargo nextest run -p nomi-browser --run-ignored all -E 'test(site_memory_real_chrome)'
use std::sync::Arc;
use nomi_browser::site_memory::{InMemorySink, SiteMemoryStore};
use nomi_browser::BrowserTool;
use nomi_tools::Tool;
use serde_json::json;
fn isolated_data_dir() -> std::path::PathBuf {
std::env::temp_dir().join("nomifun-p7a-site-memory-smoke")
}
/// **Real-Chrome smoke**: navigate example.com, hover a heading (non-navigating),
/// verify site memory records the element; then observe again and verify hints appear.
#[tokio::test]
#[ignore = "requires NOMIFUN_CHROME_BINARY + network access to example.com"]
async fn site_memory_real_chrome_remember_across_navigations() {
let sink = InMemorySink::new();
let store = Arc::new(SiteMemoryStore::new(Box::new(sink)));
let tool = BrowserTool::with_data_dir(isolated_data_dir(), false)
.with_site_memory(store.clone());
// ── 1. Navigate to example.com ────────────────────────────────────────────
let nav = tool
.execute(json!({"action": "navigate", "url": "https://example.com"}))
.await;
assert!(!nav.is_error, "navigate should succeed: {}", nav.content);
// ── 2. Observe: get the page structure ────────────────────────────────────
let obs1 = tool.execute(json!({"action": "observe"})).await;
assert!(!obs1.is_error, "observe should succeed: {}", obs1.content);
let obs_text = &obs1.content;
// Find a heading ref ("Example Domain") — hover it (non-navigating action).
let heading_ref = obs_text
.lines()
.find(|line| line.contains("heading") && line.contains("Example Domain") && line.contains("[ref="))
.and_then(|line| {
let start = line.find("[ref=")? + 5;
let end = line[start..].find(']')? + start;
Some(line[start..end].to_string())
})
.expect("should find a ref for the 'Example Domain' heading");
// ── 3. Hover the heading → triggers site-memory recording ─────────────────
let hover = tool
.execute(json!({"action": "hover", "ref": heading_ref}))
.await;
assert!(!hover.is_error, "hover should succeed: {}", hover.content);
// ── 4. Verify site memory recorded the hover ──────────────────────────────
let hints = store.query("example.com");
assert!(
!hints.is_empty(),
"site memory should have recorded at least one entry for example.com"
);
assert!(
hints.iter().any(|h| h.accessible_name.contains("Example Domain")),
"site memory should remember the 'Example Domain' heading; got: {hints:?}"
);
// ── 5. Observe again — hints should appear in the output ──────────────────
let obs2 = tool.execute(json!({"action": "observe"})).await;
assert!(!obs2.is_error, "2nd observe should succeed: {}", obs2.content);
// The 2nd observe should include site-memory hints.
let obs2_text = &obs2.content;
assert!(
obs2_text.contains("site-memory-hints"),
"2nd observe should include site-memory hints; got:\n{obs2_text}"
);
assert!(
obs2_text.contains("Example Domain"),
"hints should mention the remembered 'Example Domain' heading"
);
}
nomifun-tauri/crates/agent/nomi-browser/tests/visual_fallback.rs
+270
View File
@@ -0,0 +1,270 @@
//! Tests for the visual fallback module (P7B).
//!
//! These are pure-logic tests that do NOT require a Chrome binary.
use nomi_browser::visual_fallback::{
should_try_visual, som_overlay, to_css_point, ElementRect, PixelBox, VisualFallback,
VisualLocateResult, VisualLocator,
};
use nomi_browser_engine::BrowserError;
/// **THE KEYSTONE TEST**: vision models return device/image pixels. The engine's input layer
/// is DPR-free (CSS pixels). The facade MUST divide by DPR before dispatching.
///
/// `to_css_point(200, 400, dpr=2.0)` => `(100.0, 200.0)` (divides by DPR).
/// `to_css_point(200, 400, dpr=1.0)` => `(200.0, 400.0)` (identity when DPR is 1).
#[test]
fn pixel_to_css_divides_by_dpr() {
// DPR 2.0: Retina display — device pixels are 2x CSS pixels.
let (cx, cy) = to_css_point(200.0, 400.0, 2.0);
assert_eq!(cx, 100.0, "x must be divided by DPR");
assert_eq!(cy, 200.0, "y must be divided by DPR");
// DPR 1.0: identity — device pixels == CSS pixels.
let (cx, cy) = to_css_point(200.0, 400.0, 1.0);
assert_eq!(cx, 200.0, "dpr=1.0 must be identity for x");
assert_eq!(cy, 400.0, "dpr=1.0 must be identity for y");
// DPR 1.5: fractional scale factor.
let (cx, cy) = to_css_point(300.0, 450.0, 1.5);
assert_eq!(cx, 200.0, "x/1.5 = 200");
assert_eq!(cy, 300.0, "y/1.5 = 300");
}
/// Visual fallback must ONLY be attempted when DOM/aria anchoring fails with
/// NodeStale or NotConnected. It must NOT run when `resolve_ref` succeeds, and
/// must NOT run on unrelated errors (timeout, session lost, blocked, etc.).
#[test]
fn fallback_only_invoked_on_anchor_failure() {
// Anchor succeeded — never try visual.
assert!(!should_try_visual(&Ok(())), "must NOT fallback on successful anchor");
// NodeStale — ref from old generation, should try visual.
assert!(
should_try_visual(&Err(BrowserError::NodeStale { generation: 5 })),
"must fallback on NodeStale"
);
// NotConnected — element detached from DOM, should try visual.
assert!(
should_try_visual(&Err(BrowserError::NotConnected)),
"must fallback on NotConnected"
);
// SessionLost — NOT a visual-fallback candidate.
assert!(
!should_try_visual(&Err(BrowserError::SessionLost { recoverable: false })),
"must NOT fallback on SessionLost"
);
// Timeout — NOT a visual-fallback candidate.
assert!(
!should_try_visual(&Err(BrowserError::Timeout {
phase: nomi_browser_engine::NavPhase::Action
})),
"must NOT fallback on Timeout"
);
// Blocked — NOT a visual-fallback candidate.
assert!(
!should_try_visual(&Err(BrowserError::Blocked {
reason: "denied".into()
})),
"must NOT fallback on Blocked"
);
// Other — NOT a visual-fallback candidate (generic errors are not anchor-specific).
assert!(
!should_try_visual(&Err(BrowserError::Other("something went wrong".into()))),
"must NOT fallback on Other"
);
}
/// A fake vision locator that returns a fixed pixel bounding box (simulating what a
/// real vision model would return after analyzing a screenshot).
struct FakeLocator {
/// The pixel-space bounding box the fake "finds".
pixel_box: PixelBox,
}
#[async_trait::async_trait]
impl VisualLocator for FakeLocator {
async fn locate(
&self,
_screenshot: &[u8],
_instruction: &str,
) -> Result<VisualLocateResult, String> {
Ok(VisualLocateResult {
pixel_box: self.pixel_box,
confidence: 0.95,
})
}
}
/// VisualFallback::locate_and_target calls the locator with the redacted screenshot,
/// receives pixel coords, and maps them to CSS pixels via DPR division.
#[tokio::test]
async fn visual_fallback_locates_and_maps() {
// Fake locator returns a box centered at (200, 400) in device pixels.
let locator = FakeLocator {
pixel_box: PixelBox {
x: 180.0,
y: 380.0,
width: 40.0,
height: 40.0,
},
};
let fallback = VisualFallback::new(std::sync::Arc::new(locator));
// DPR = 2.0 → center pixel (200, 400) → CSS (100, 200).
let fake_screenshot = b"fake-png-data";
let result = fallback
.locate_and_target(fake_screenshot, "Click the Submit button", 2.0)
.await
.expect("locate_and_target should succeed with a fake locator");
assert_eq!(result.x, 100.0, "CSS x = pixel_center_x / dpr = 200/2");
assert_eq!(result.y, 200.0, "CSS y = pixel_center_y / dpr = 400/2");
// DPR = 1.0 → identity.
let result = fallback
.locate_and_target(fake_screenshot, "Click the Submit button", 1.0)
.await
.expect("locate_and_target should succeed");
assert_eq!(result.x, 200.0, "CSS x = pixel_center_x / 1.0 = 200");
assert_eq!(result.y, 400.0, "CSS y = pixel_center_y / 1.0 = 400");
}
/// SoM overlay assigns deterministic 1..N labels to element rects, sorted by position
/// (top-to-bottom, left-to-right). The numbering is stable across repeated calls.
#[test]
fn som_overlay_numbers_boxes_stably() {
let rects = vec![
// Bottom-right element (should be numbered LAST due to sort order).
ElementRect { x: 300.0, y: 200.0, width: 50.0, height: 30.0 },
// Top-left element (should be numbered FIRST).
ElementRect { x: 10.0, y: 10.0, width: 100.0, height: 40.0 },
// Middle element (between top and bottom).
ElementRect { x: 150.0, y: 100.0, width: 80.0, height: 30.0 },
// Same y as first, but further right (should be numbered second).
ElementRect { x: 200.0, y: 10.0, width: 60.0, height: 40.0 },
];
let fake_png = b"fake-png-bytes";
let result = som_overlay(fake_png, &rects);
// Should have 4 labels.
assert_eq!(result.label_map.len(), 4);
// Label 1: top-left (y=10, x=10) — the topmost, leftmost.
assert_eq!(result.label_map[0].number, 1);
assert_eq!(result.label_map[0].rect.x, 10.0);
assert_eq!(result.label_map[0].rect.y, 10.0);
// Label 2: top-right (y=10, x=200) — same row as label 1, but further right.
assert_eq!(result.label_map[1].number, 2);
assert_eq!(result.label_map[1].rect.x, 200.0);
assert_eq!(result.label_map[1].rect.y, 10.0);
// Label 3: middle (y=100, x=150).
assert_eq!(result.label_map[2].number, 3);
assert_eq!(result.label_map[2].rect.x, 150.0);
assert_eq!(result.label_map[2].rect.y, 100.0);
// Label 4: bottom-right (y=200, x=300).
assert_eq!(result.label_map[3].number, 4);
assert_eq!(result.label_map[3].rect.x, 300.0);
assert_eq!(result.label_map[3].rect.y, 200.0);
// Stability: calling with the same rects produces the same numbering.
let result2 = som_overlay(fake_png, &rects);
assert_eq!(result.label_map, result2.label_map, "numbering must be deterministic");
// Empty rects → empty label map.
let empty_result = som_overlay(fake_png, &[]);
assert!(empty_result.label_map.is_empty());
// With invalid PNG bytes, annotated_png falls back to input unchanged.
assert_eq!(result.annotated_png, fake_png.as_slice());
}
/// SoM overlay with a real PNG: annotated output must (a) decode as valid PNG,
/// (b) differ from input (proving drawing happened), (c) label_map is unchanged.
#[test]
fn som_overlay_draws_on_real_png() {
use image::{ImageFormat, RgbaImage, Rgba};
use std::io::Cursor;
// Create a small 200×200 solid-gray PNG.
let img = RgbaImage::from_pixel(200, 200, Rgba([128, 128, 128, 255]));
let mut input_buf = Cursor::new(Vec::new());
img.write_to(&mut input_buf, ImageFormat::Png).unwrap();
let input_png = input_buf.into_inner();
let rects = vec![
ElementRect { x: 20.0, y: 50.0, width: 80.0, height: 40.0 },
ElementRect { x: 10.0, y: 10.0, width: 60.0, height: 30.0 },
ElementRect { x: 100.0, y: 120.0, width: 50.0, height: 25.0 },
];
let result = som_overlay(&input_png, &rects);
// (a) annotated_png is a valid PNG and decodes successfully.
let decoded = image::load_from_memory_with_format(&result.annotated_png, ImageFormat::Png);
assert!(decoded.is_ok(), "annotated_png must be a valid PNG");
// (b) annotated_png DIFFERS from the input (drawing happened).
assert_ne!(
result.annotated_png, input_png,
"annotated_png must differ from input (overlay was drawn)"
);
// (c) label_map numbering is correct and stable.
assert_eq!(result.label_map.len(), 3);
// Sorted by y then x: (10,10)=1, (20,50)=2, (100,120)=3
assert_eq!(result.label_map[0].number, 1);
assert_eq!(result.label_map[0].rect.x, 10.0);
assert_eq!(result.label_map[0].rect.y, 10.0);
assert_eq!(result.label_map[1].number, 2);
assert_eq!(result.label_map[1].rect.x, 20.0);
assert_eq!(result.label_map[1].rect.y, 50.0);
assert_eq!(result.label_map[2].number, 3);
assert_eq!(result.label_map[2].rect.x, 100.0);
assert_eq!(result.label_map[2].rect.y, 120.0);
// Verify output dimensions match input.
let out_img = decoded.unwrap().to_rgba8();
assert_eq!(out_img.dimensions(), (200, 200));
}
/// Edge case: rects that are partially or fully off-screen must not panic.
#[test]
fn som_overlay_clips_offscreen_rects() {
use image::{ImageFormat, RgbaImage, Rgba};
use std::io::Cursor;
let img = RgbaImage::from_pixel(100, 100, Rgba([0, 0, 0, 255]));
let mut buf = Cursor::new(Vec::new());
img.write_to(&mut buf, ImageFormat::Png).unwrap();
let input_png = buf.into_inner();
let rects = vec![
// Partially off-screen (extends beyond image bounds).
ElementRect { x: 80.0, y: 80.0, width: 50.0, height: 50.0 },
// Fully off-screen.
ElementRect { x: 200.0, y: 200.0, width: 30.0, height: 30.0 },
// Negative coords.
ElementRect { x: -10.0, y: -10.0, width: 50.0, height: 50.0 },
// Zero-size rect (degenerate).
ElementRect { x: 50.0, y: 50.0, width: 0.0, height: 0.0 },
];
// Must not panic.
let result = som_overlay(&input_png, &rects);
// All 4 rects get labels even if drawing is clipped.
assert_eq!(result.label_map.len(), 4);
// Output is a valid PNG.
assert!(image::load_from_memory_with_format(&result.annotated_png, ImageFormat::Png).is_ok());
}