From 0df293186a6e8a5a5908b9cbffe2c1ba99c40aa7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E7=8E=89=E6=BE=9C?= Date: Thu, 13 Aug 2026 15:20:09 +0800 Subject: [PATCH] feat(cli): harden tool search ranking, guardrails and observability - default engine: fielded BM25 ensemble (action_v1 remains a shadow candidate) - add --explain score_breakdown: per-field contributions, structured multiplier, query classification and avoid_when penalty - add DWS_TOOL_SEARCH_* env overrides (algorithm, k1/b, field weights, explain) for release-independent tuning and rollback - tokenizer: CJK unigram+bigram so single-character intents match; align isToolSearchCJK with unicode.Han used by the evaluation slices - structured routing: extend vocabularies with English synonyms, word-boundary matching for ASCII values, restrict the rerank gate to technical identifiers only - avoid_when demotion layer wraps every lexical algorithm (Forbidden@5 proxy 37.55% -> 0); intent-overlap matching rejected after control showed it demoted gold answers (Recall@5 0.8664 -> 0.7845) - remove the OA task_id +1.00 fixture-specific multiplier - build the default engine once per process for long-lived hosts - evaluation baseline: intent-proxy R@5 990/1123 (88.16%); update integration sentinels accordingly --- Makefile | 2 +- docs/rfc-tool-search-progressive-discovery.md | 20 +- docs/tool-search-ranking-evaluation.md | 36 +-- internal/app/tool_search_integration_test.go | 16 +- internal/cli/schema_search_command.go | 16 +- internal/cli/schema_search_command_test.go | 2 +- internal/cli/tool_search.go | 191 ++++++++++-- internal/cli/tool_search_evaluation.go | 2 +- .../cli/tool_search_independent_evaluation.go | 23 +- internal/cli/tool_search_lexical.go | 283 ++++++++++++++---- internal/cli/tool_search_test.go | 150 +++++++++- .../cmd_tool_search_comparison/main.go | 4 +- scripts/policy/check-generated-drift.sh | 1 + .../check-shortcut-public-catalog-drift.py | 68 +++++ scripts/policy/skill-command-check/main.go | 15 + .../policy/skill-command-check/main_test.go | 12 +- .../testdata/tool_search_eval_manifest.json | 6 +- skills/mono/SKILL.md | 20 +- skills/mono/references/products/event.md | 2 +- skills/mono/references/products/sheet.md | 2 +- .../references/schema-usage.md | 26 +- 21 files changed, 715 insertions(+), 182 deletions(-) create mode 100644 scripts/policy/check-shortcut-public-catalog-drift.py diff --git a/Makefile b/Makefile index 924fcb6c..f591a99c 100644 --- a/Makefile +++ b/Makefile @@ -87,7 +87,7 @@ fmt: $(GO_SOURCE_LIST) > "$$go_files"; \ xargs -0 sh -c 'if [ "$$#" -gt 0 ]; then exec gofmt -w -- "$$@"; fi' sh < "$$go_files" -policy: test-auth-legacy-compat tool-search-evaluation-harness +policy: test-auth-legacy-compat tool-search-evaluation-harness skill-mono-multi-content @mkdir -p "$(POLICY_GOTMPDIR)" @$(POLICY_ENV) ./scripts/policy/check-open-source-assets.sh @$(POLICY_ENV) ./scripts/policy/check-skill-context-budget.sh diff --git a/docs/rfc-tool-search-progressive-discovery.md b/docs/rfc-tool-search-progressive-discovery.md index 91a9d708..2bd1f24e 100644 --- a/docs/rfc-tool-search-progressive-discovery.md +++ b/docs/rfc-tool-search-progressive-discovery.md @@ -112,16 +112,16 @@ DWS 已经具备： | 能力 | 当前证据 | 可支持的结论 | |---|---:|---| -| Go 默认动作重排 BM25 | R@1 65.63%；R@5 86.38%；MRR@5 0.7417 | 当前本地默认候选，尚非独立发布结论 | -| Go 原始字段 BM25 control | R@1 65.09%；R@5 86.64%；MRR@5 0.7370 | 默认动作重排的 R@5 低 0.27 pp，但 R@1/MRR@5、workflow 与负例暴露更好 | +| Go 字段 BM25 默认 | R@1 65.00%；R@5 86.64%；MRR@5 0.7366 | 独立 qrels 缺失期间采用的保守默认 | +| Go action ranker shadow | R@1 65.54%；R@5 86.38%；MRR@5 0.7414 | R@5 低 0.27 pp；R@1/MRR@5、workflow 与负例暴露在同源 proxy 上更好，等待独立门禁 | | 精确身份 | Exact Guard 后 canonical / CLI Top-1 100% | Exact Guard 是穷举契约门禁 | -| 多工具完整性 | 整句 Complete@5 40% / required recall 66.67%；reviewed 拆解后 90% / 95% | 动作级检索有效；拆解结果是人工上界，不是端到端成功率 | -| 中文召回 | 默认算法纯中文 R@5 81.59%，中英混合 89.04% | 中文 bigram + 标识符保留可用，但纯中文仍是重点优化项 | -| 负例暴露 | Forbidden@1 5.83%，Forbidden@5 36.83% | Top-1 已较低，Top-5 仍高，必须继续建设 alternative gold / contradiction gate | +| 多工具完整性 | 默认整句 Complete@5 40% / required recall 61.67%；reviewed 拆解后 50% / 75% | 拆解结果是人工上界，不是端到端成功率；action shadow 为 90% / 95% | +| 中文召回 | 默认算法纯中文 R@5 82.34%，中英混合 89.04% | 中文 bigram + 标识符保留可用，但纯中文仍是重点优化项 | +| 负例暴露 | 默认 Forbidden@1 6.12%，Forbidden@5 37.55% | Top-5 仍高，必须继续建设 alternative gold / contradiction gate | 相对最简单的 IDF keyword overlap，TF-IDF 在当前 proxy 集上的 R@5 从 76.91% 提升到 84.55%，提高 7.64 个百分点。这个数字可以说明“需要一个正式词法 ranker”，但因为 query 与索引元数据来自同一批作者，不能外推为线上业务收益。 -当前分支的纯 Go shipped-runtime 对比器直接消费运行时装配的 typed Catalog，不读取仓库中的生成 JSON。当前默认 `fielded_bm25_action_v1` 相对原始 `fielded_bm25_ensemble`：R@1 **+0.53 pp**、R@5 **-0.27 pp**、MRR@5 **+0.47 pp**；raw workflow required recall **+5 pp**，reviewed 拆解的 Complete@5 / required recall **+40 / +20 pp**；Forbidden@5 **下降 0.72 pp**。这支持继续 shadow 当前动作重排，但仍受同源 query 和人工拆解限制，不能替代独立 test 或真实 Agent A/B。 +当前分支的纯 Go shipped-runtime 对比器直接消费运行时装配的 typed Catalog，不读取仓库中的生成 JSON。action shadow `fielded_bm25_action_v1` 相对当前默认 `fielded_bm25_ensemble`：R@1 **+0.53 pp**、R@5 **-0.27 pp**、MRR@5 **+0.47 pp**；raw workflow required recall **+5 pp**，reviewed 拆解的 Complete@5 / required recall **+40 / +20 pp**；Forbidden@5 **下降 0.72 pp**。由于这些结果来自同源 proxy，且主召回指标没有改善，action 只能继续 shadow；切换默认必须通过 sealed 独立 test 的非劣效与最小增益门禁。 合并主线前曾用固定 `gpt-5.6-sol` 做过一轮 answer-free、无业务执行的规划 smoke A/B：同一批 10 条 workflow，各 arm 一个 batch/一次 trial。该 run 绑定旧 572 工具 Catalog，已因当前 1,098 工具 surface 变化判为过期，只保留方法记录，不能作为现行效果证明。旧结果中两臂 required-tool complete/recall 都是 **100% / 100%**；精确最小计划率分别为 **90% / 70%**，plan precision 为 **95.45% / 87.50%**，额外步骤为 **1 / 3**。单 trial 无区间，不能据此通过默认开启门禁。 @@ -131,13 +131,13 @@ DWS 已经具备： | 载荷 | 大小 | |---|---:| -| compact 全量 Schema | 17,851,126 bytes | -| 平均 Search + gold Inspect | 4,486.87 bytes | +| compact 全量 Schema | 17,876,084 bytes | +| 平均 Search + gold Inspect | 4,489.58 bytes | | 理想化 overview + 正确 product + Inspect | 122,193 bytes | 渐进发现把“预加载全部 Schema”变成“约 3 KB 候选 + 选中 leaf 的数 KB Schema”。实际模型 token 下降比例取决于 tokenizer 和对话编排，因此 RFC 只以字节数作为已验证证据，不给出未经测量的 token 节省承诺。 -用 Go 对比器逐条渲染当前 compact JSON envelope 后，Search + gold leaf Inspect 平均 **4,486.87 bytes**；相对 17,851,126 bytes 的全量 Schema 减少 **99.9749%**，相对已经假设 oracle 知道正确 product 的理想化导航也减少 **96.3297%**。评测器直接 Inspect gold，即使 Search miss 也不计额外尝试；因此这是 JSON byte 容量上界，不是 tokenizer token、真实 Agent 成本或任务成功率承诺。 +用 Go 对比器逐条渲染当前 compact JSON envelope 后，Search + gold leaf Inspect 平均 **4,489.58 bytes**；相对 17,876,084 bytes 的全量 Schema 减少 **99.9749%**，相对已经假设 oracle 知道正确 product 的理想化导航也减少 **96.3311%**。评测器直接 Inspect gold，即使 Search miss 也不计额外尝试；因此这是 JSON byte 容量上界，不是 tokenizer token、真实 Agent 成本或任务成功率承诺。 ### 3.3 CLI 与工程收益 @@ -465,7 +465,7 @@ CLI transport 固定为 `dws schema search --request-json -`：stdin 只接受 "idempotency": "idempotent", "rank": 1, "matched_fields": ["summary", "parameters"], - "rank_sources": ["fielded_bm25_action_v1"], + "rank_sources": ["fielded_bm25_ensemble"], "requires_inspect": true } ``` diff --git a/docs/tool-search-ranking-evaluation.md b/docs/tool-search-ranking-evaluation.md index dfee962d..6ba1b523 100644 --- a/docs/tool-search-ranking-evaluation.md +++ b/docs/tool-search-ranking-evaluation.md @@ -13,7 +13,7 @@ > > 分支：`test/tool-search-ranking-eval` > -> 当前 Catalog：1,098 个工具，`source_hash=sha256:106a3d2389f672f7fd0ec302c45e2391e9edaac2d34523e15d42c395abb6c499`，`surface_hash=sha256:7f9839f1ff428b52f1d032d2fa6493815c3755650b7d9fa653839e7d589482e5` +> 当前 Catalog：1,098 个工具，`source_hash=sha256:02c633c075f4af915ea097d383bdcbc25fe549c172e2d3336152e30cf2906e80`，`surface_hash=sha256:7f9839f1ff428b52f1d032d2fa6493815c3755650b7d9fa653839e7d589482e5` > > 评测代码：[`scripts/dev/eval_tool_search_ranking.py`](../scripts/dev/eval_tool_search_ranking.py) @@ -23,19 +23,19 @@ 当前结果由 `cmd_tool_search_comparison` 直接消费 Go 运行时声明装配结果，不读取仓库中的 Catalog JSON。评测/CI 如需检查分片，只能先运行 `cmd_schema_catalog`，在 `.worktrees/policy-tmp/tool-search-schema-catalog` 动态生成；该目录被忽略且不是生产输入。 -| 指标 | `fielded_bm25_action_v1` | 原始 `fielded_bm25_ensemble` control | +| 指标 | `fielded_bm25_ensemble` 当前默认 | `fielded_bm25_action_v1` shadow | |---|---:|---:| | Intent cases | 1,123（另有 10 条超预算并显式排除） | 1,123 | -| R@1 / R@5 / MRR@5 | **65.63% / 86.38% / 0.7417** | 65.09% / **86.64%** / 0.7370 | -| 纯中文（402）R@1 / R@5 | **50.75%** / 81.59% | 49.75% / **82.34%** | -| 中文混 ASCII（721）R@1 / R@5 | **73.93%** / 89.04% | 73.65% / 89.04% | -| 整句 workflow Complete@5 / Recall@5 | 40% / **66.67%** | 40% / 61.67% | -| reviewed 拆解 Complete@5 / Recall@5 | **90% / 95%** | 50% / 75% | -| Forbidden@1 / @5（越低越好） | **5.83% / 36.83%** | 6.12% / 37.55% | +| R@1 / R@5 / MRR@5 | 65.00% / **86.64%** / 0.7366 | **65.54%** / 86.38% / **0.7414** | +| 纯中文（402）R@1 / R@5 | 49.75% / **82.34%** | **50.75%** / 81.59% | +| 中文混 ASCII（721）R@1 / R@5 | 73.65% / 89.04% | **73.93%** / 89.04% | +| 整句 workflow Complete@5 / Recall@5 | 40% / 61.67% | 40% / **66.67%** | +| reviewed 拆解 Complete@5 / Recall@5 | 50% / 75% | **90% / 95%** | +| Forbidden@1 / @5（越低越好） | 6.12% / 37.55% | **5.83% / 36.83%** | -身份与完整性门禁：1,098 个 canonical、1,098 个 primary CLI、19 个 reviewed alias、1,098 个 NFKC identity 及 1,098 个 `exact_filtered` 全部 100%；5,801 个响应的 Catalog 绑定失败、unknown candidate、ineligible candidate、response budget violation 均为 0。上下文方面，平均 Search + gold Inspect 为 4,486.87 bytes，相对 17,851,126 bytes 的 compact 全量 Schema 减少 **99.9749%**，相对假设 oracle 已选对产品的理想导航仍减少 **96.3297%**。两条评测路径都直接 Inspect gold leaf，即使 Search miss 也不计额外尝试，因此是容量上界，不是实际 Agent 成本。 +身份与完整性门禁：1,098 个 canonical、1,098 个 primary CLI、19 个 reviewed alias、1,098 个 NFKC identity 及 1,098 个 `exact_filtered` 全部 100%；5,801 个响应的 Catalog 绑定失败、unknown candidate、ineligible candidate、response budget violation 均为 0。上下文方面，平均 Search + gold Inspect 为 4,489.58 bytes，相对 17,876,084 bytes 的 compact 全量 Schema 减少 **99.9749%**，相对假设 oracle 已选对产品的理想导航仍减少 **96.3311%**。两条评测路径都直接 Inspect gold leaf，即使 Search miss 也不计额外尝试，因此是容量上界，不是实际 Agent 成本。 -这些数据说明当前实现达到了“显著缩小上下文、身份不退化、中文自然语言 Top-5 约 86%”的工程目标，但不等价于线上任务成功率。Forbidden@5 仍有 36.83%，独立 qrels、英文 ≥100、workflow ≥80、真实同模型 Agent A/B 仍为空；因此不能据此默认开启。 +这些数据说明当前实现达到了“显著缩小上下文、身份不退化、中文自然语言 Top-5 约 87%”的工程目标，但不等价于线上任务成功率。当前默认的 Forbidden@5 仍有 37.55%，独立 qrels、英文 ≥100、workflow ≥80、真实同模型 Agent A/B 仍为空；因此不能据此把 action 重排提升为默认。 ### 1.2 合并前算法研究记录（2026-08-12，历史基线） @@ -370,16 +370,16 @@ make generate-tool-search-comparison | 项目 | 结果 | |---|---:| -| 1,123 条预算内同源 intent R@1 / R@5 / MRR@5 | 65.63% / 86.38% / 0.7417 | -| Go 原始字段 BM25 control R@1 / R@5 / MRR@5 | 65.09% / 86.64% / 0.7370 | -| Go TF-IDF shadow R@1 / R@5 / MRR@5 | 45.59% / 75.51% / 0.5701 | -| 10 条 workflow 整句 Complete@5 / required recall | 40% / 66.67% | -| reviewed subquery 后 Complete@5 / required recall | 90% / 95% | -| Search + gold Inspect 平均 compact JSON | 4,486.87 bytes（oracle-assisted capacity upper bound） | -| 相对 oracle 导航 `overview → product → inspect` 的 byte reduction | 96.3297% | +| 1,123 条预算内同源 intent R@1 / R@5 / MRR@5 | 65.00% / 86.64% / 0.7366 | +| Go action ranker shadow R@1 / R@5 / MRR@5 | 65.54% / 86.38% / 0.7414 | +| Go TF-IDF shadow R@1 / R@5 / MRR@5 | 45.59% / 75.51% / 0.5698 | +| 10 条 workflow 整句 Complete@5 / required recall | 40% / 61.67% | +| reviewed subquery 后 Complete@5 / required recall | 50% / 75% | +| Search + gold Inspect 平均 compact JSON | 4,489.58 bytes（oracle-assisted capacity upper bound） | +| 相对 oracle 导航 `overview → product → inspect` 的 byte reduction | 96.3311% | | 相对全量紧凑 Schema 的 byte reduction | 99.9749% | -这解决了“Python 离线结果和 Go 生产实现混在一起”的问题：Python 表格用于记录候选算法研究，Go 表格锁定 shipped runtime。当前默认增加有界 action/product/entity 重排，相对原始字段 BM25 的 R@5 低 0.27 pp，但 R@1、MRR、workflow 与 Forbidden exposure 更优，继续作为独立 test 前的默认候选。Go TF-IDF 使用 fielded/raw-TF，而 Python 头条 TF-IDF 使用 unfielded/log-TF；当前 Go shadow 明显更差，不能因为历史 Python 的 84.55% 就切换生产默认。两边由于 Catalog、投影和 TF 定义不同，不声称逐 case parity。 +这解决了“Python 离线结果和 Go 生产实现混在一起”的问题：Python 表格用于记录候选算法研究，Go 表格锁定 shipped runtime。由于 proxy 同源且 action ranker 的整体与纯中文 R@5 分别低 0.27 / 0.75 pp，当前默认回到更简单的字段 BM25；action/product/entity 重排仅作为 shadow，由 sealed 独立 qrels 上的配对 product-cluster CI 决定是否具备切换资格。Go TF-IDF 使用 fielded/raw-TF，而 Python 头条 TF-IDF 使用 unfielded/log-TF；当前 Go shadow 明显更差，不能因为历史 Python 的 84.55% 就切换生产默认。两边由于 Catalog、投影和 TF 定义不同，不声称逐 case parity。 ### 8.1 当前 release binary 冷进程与包体（Apple M3 Pro） diff --git a/internal/app/tool_search_integration_test.go b/internal/app/tool_search_integration_test.go index 91e0d9e1..cddf5170 100644 --- a/internal/app/tool_search_integration_test.go +++ b/internal/app/tool_search_integration_test.go @@ -18,9 +18,14 @@ import ( func TestToolSearchDeliveryActionSiblingRouting(t *testing.T) { registerSchemaRuntimeDelivery() - engine, err := cli.NewDeliveryToolSearchEngine() + // This test pins the action_v1 structured sibling routing itself, decoupled + // from the production default (now the fielded BM25 ensemble, which has no + // structured multiplier and cannot right-size these sibling queries). + config := cli.DefaultToolSearchConfig() + config.LexicalAlgorithm = cli.ToolSearchLexicalBM25Action + engine, err := cli.NewDeliveryToolSearchEngineWithConfig(config) if err != nil { - t.Fatalf("NewDeliveryToolSearchEngine() error = %v", err) + t.Fatalf("NewDeliveryToolSearchEngineWithConfig() error = %v", err) } tests := []struct { query string @@ -72,7 +77,12 @@ func TestToolSearchDeliveryDiagnosticTrustAndChineseSlices(t *testing.T) { if report.IntentProxy.Cases != 1123 || report.IntentExcludedOverBudget != 10 { t.Fatalf("intent population = %+v excluded=%d", report.IntentProxy, report.IntentExcludedOverBudget) } - if math.Abs(report.IntentProxy.RecallAt5-0.8637577916295637) > 1e-12 { + // The default lexical algorithm is the fielded BM25 ensemble; this sentinel + // is its current hit count (990 of 1123 intent-proxy cases). History: + // 970/1123 = action_v1 default era; 973/1123 = ensemble with bigram-only + // tokenization; 990/1123 after unigram+bigram tokenization, the English + // vocabulary extension and the relaxed technical-identifier gate. + if math.Abs(report.IntentProxy.RecallAt5-990.0/1123.0) > 1e-12 { t.Fatalf("Recall@5 = %.12f", report.IntentProxy.RecallAt5) } if report.IntentLanguageSlices["chinese_only"].Cases != 402 || report.IntentLanguageSlices["mixed_chinese_ascii"].Cases != 721 { diff --git a/internal/cli/schema_search_command.go b/internal/cli/schema_search_command.go index 0b5cae5f..fe44196f 100644 --- a/internal/cli/schema_search_command.go +++ b/internal/cli/schema_search_command.go @@ -36,7 +36,18 @@ type ToolSearchV1Request struct { ExcludeCanonical []string `json:"exclude_canonical,omitempty"` } -var schemaSearchNewEngine = NewDeliveryToolSearchEngine +var schemaSearchNewEngine = NewDeliveryToolSearchEngineWithConfig + +// schemaSearchEngineConfig resolves the search config the same way +// NewDeliveryToolSearchEngine does (defaults + DWS_TOOL_SEARCH_* env +// overrides), then lets an explicit --explain flag win over the env value. +func schemaSearchEngineConfig(cmd *cobra.Command) ToolSearchConfig { + config := ResolveToolSearchConfigFromEnv(DefaultToolSearchConfig()) + if explain, _ := cmd.Flags().GetBool("explain"); explain { + config.Explain = true + } + return config +} func newSchemaSearchCommand() *cobra.Command { cmd := &cobra.Command{ @@ -66,7 +77,7 @@ DWS 对 Agent 是一个元工具：已知子命令直接执行，未知子命令 if err != nil { return err } - engine, err := schemaSearchNewEngine() + engine, err := schemaSearchNewEngine(schemaSearchEngineConfig(cmd)) if err != nil { return err } @@ -103,6 +114,7 @@ DWS 对 Agent 是一个元工具：已知子命令直接执行，未知子命令 cmd.Flags().StringSlice("effect", nil, "限制 effect，可重复或逗号分隔") cmd.Flags().StringSlice("exclude", nil, "排除 canonical path，可重复或逗号分隔") cmd.Flags().String("request-json", "", `从 stdin 读取 JSON；值必须为 -；必含 version="tool-search.v1"，动作分解字段为 subqueries`) + cmd.Flags().Bool("explain", false, "在每个 candidate 上输出 score_breakdown 排序诊断（会增大响应体积，建议配合 --limit 3）") return cmd } diff --git a/internal/cli/schema_search_command_test.go b/internal/cli/schema_search_command_test.go index bd536789..cdeae355 100644 --- a/internal/cli/schema_search_command_test.go +++ b/internal/cli/schema_search_command_test.go @@ -81,7 +81,7 @@ func TestSchemaSearchHelpDocumentsStructuredRequest(t *testing.T) { func installSchemaSearchTestEngine(t *testing.T) { t.Helper() - testseam.Swap(t, &schemaSearchNewEngine, func() (*ToolSearchEngine, error) { + testseam.Swap(t, &schemaSearchNewEngine, func(ToolSearchConfig) (*ToolSearchEngine, error) { return newToolSearchTestEngine(t), nil }) } diff --git a/internal/cli/tool_search.go b/internal/cli/tool_search.go index e7d45dd3..7a4880b8 100644 --- a/internal/cli/tool_search.go +++ b/internal/cli/tool_search.go @@ -18,9 +18,12 @@ import ( "encoding/json" "fmt" "math" + "os" "regexp" "sort" + "strconv" "strings" + "sync" "unicode" "unicode/utf8" @@ -40,6 +43,8 @@ const ( maxToolSearchSummaryRunes = 256 maxToolSearchResponseBytes = 8 * 1024 maxToolSearchRequestBytes = 64 * 1024 + defaultToolSearchBM25K1 = 0.9 + defaultToolSearchBM25B = 0.4 ) var toolSearchIdentifierPattern = regexp.MustCompile(`[A-Za-z0-9]+(?:[._-][A-Za-z0-9]+)+`) @@ -65,6 +70,9 @@ type ToolSearchConfig struct { DefaultCandidates int LexicalAlgorithm string IncludeUseWhen bool + Explain bool + BM25K1 float64 + BM25B float64 CatalogSourceHash string CatalogSurfaceHash string FieldWeights ToolSearchFieldWeights @@ -75,8 +83,11 @@ func DefaultToolSearchConfig() ToolSearchConfig { return ToolSearchConfig{ DefaultLimit: defaultToolSearchLimit, DefaultCandidates: defaultToolSearchCandidateLimit, - LexicalAlgorithm: ToolSearchLexicalBM25Action, + LexicalAlgorithm: ToolSearchLexicalBM25, IncludeUseWhen: false, + Explain: false, + BM25K1: defaultToolSearchBM25K1, + BM25B: defaultToolSearchBM25B, FieldWeights: ToolSearchFieldWeights{ Identity: 8, Summary: 5, @@ -98,24 +109,47 @@ type ToolSearchRequest struct { ExcludeCanonicalPaths []string } +// ToolSearchExplainQueryClass echoes the structured query classification used +// by the action_v1 retriever. It is only populated under --explain so the +// default wire response stays byte-identical. +type ToolSearchExplainQueryClass struct { + Actions []string `json:"actions,omitempty"` + Products []string `json:"products,omitempty"` + Entities []string `json:"entities,omitempty"` + Effect string `json:"effect,omitempty"` +} + +// ToolSearchScoreBreakdown explains why a candidate ranked where it did. It is +// attached to a ToolReference only when the engine is built with Explain=true. +// FieldScores are the per-field weighted contributions; Multiplier/QueryClass +// are filled only by the action_v1 retriever. +type ToolSearchScoreBreakdown struct { + Score float64 `json:"score"` + FieldScores map[string]float64 `json:"field_scores,omitempty"` + Multiplier *float64 `json:"multiplier,omitempty"` + QueryClass *ToolSearchExplainQueryClass `json:"query_class,omitempty"` + AvoidWhenPenalty string `json:"avoid_when_penalty,omitempty"` +} + // ToolReference is intentionally smaller than a ToolSpec. An Agent must // inspect the selected canonical path before execution instead of guessing // parameters from search output. type ToolReference struct { - CanonicalPath string `json:"canonical_path"` - PrimaryCLIPath string `json:"primary_cli_path"` - ProductID string `json:"product_id"` - Title string `json:"title,omitempty"` - AgentSummary string `json:"agent_summary,omitempty"` - Effect string `json:"effect,omitempty"` - Risk string `json:"risk,omitempty"` - Confirmation string `json:"confirmation,omitempty"` - Idempotency string `json:"idempotency,omitempty"` - Rank int `json:"rank"` - MatchedFields []string `json:"matched_fields,omitempty"` - RankSources []string `json:"rank_sources"` - TruncatedFields []string `json:"truncated_fields,omitempty"` - RequiresInspect bool `json:"requires_inspect"` + CanonicalPath string `json:"canonical_path"` + PrimaryCLIPath string `json:"primary_cli_path"` + ProductID string `json:"product_id"` + Title string `json:"title,omitempty"` + AgentSummary string `json:"agent_summary,omitempty"` + Effect string `json:"effect,omitempty"` + Risk string `json:"risk,omitempty"` + Confirmation string `json:"confirmation,omitempty"` + Idempotency string `json:"idempotency,omitempty"` + Rank int `json:"rank"` + MatchedFields []string `json:"matched_fields,omitempty"` + RankSources []string `json:"rank_sources"` + TruncatedFields []string `json:"truncated_fields,omitempty"` + RequiresInspect bool `json:"requires_inspect"` + ScoreBreakdown *ToolSearchScoreBreakdown `json:"score_breakdown,omitempty"` score float64 } @@ -197,15 +231,35 @@ type ToolSearchEngine struct { lexical LexicalRetriever } +var ( + deliveryToolSearchEngineOnce sync.Once + deliveryToolSearchEngine *ToolSearchEngine + deliveryToolSearchEngineErr error +) + // NewDeliveryToolSearchEngine indexes the immutable in-memory Catalog // assembled from live Go declarations by RegisterSchemaSourceRoot → // ResolveSchemaBuild. There is no committed or runtime-read Catalog JSON. +// The default engine is immutable for the process lifetime: the delivery +// Catalog is frozen and DWS_TOOL_SEARCH_* overrides are read once, so the +// engine is built once per process (a CLI invocation constructs it exactly +// once either way; long-lived hosts such as a daemon or MCP server reuse it). func NewDeliveryToolSearchEngine() (*ToolSearchEngine, error) { + deliveryToolSearchEngineOnce.Do(func() { + deliveryToolSearchEngine, deliveryToolSearchEngineErr = + NewDeliveryToolSearchEngineWithConfig(ResolveToolSearchConfigFromEnv(DefaultToolSearchConfig())) + }) + return deliveryToolSearchEngine, deliveryToolSearchEngineErr +} + +// NewDeliveryToolSearchEngineWithConfig builds the delivery engine from an +// explicit, already-resolved config. Callers that override fields (such as +// --explain) pass a fully populated config here. +func NewDeliveryToolSearchEngineWithConfig(config ToolSearchConfig) (*ToolSearchEngine, error) { if err := deliverySchemaCatalogError(); err != nil { return nil, fmt.Errorf("assemble typed Schema registry for tool search: %w", err) } loaded := deliverySchemaCatalog() - config := DefaultToolSearchConfig() config.CatalogSourceHash = loaded.Snapshot.SourceHash config.CatalogSurfaceHash = loaded.Snapshot.SurfaceHash return NewToolSearchEngine(loaded.Registry, config) @@ -275,6 +329,9 @@ func (e *ToolSearchEngine) Search(ctx context.Context, request ToolSearchRequest response.Strategy = "exact_guard" reference := toolReference(exact, 1, []string{"identity"}, []string{"exact"}) reference.Rank = 1 + if e.config.Explain { + reference.ScoreBreakdown = &ToolSearchScoreBreakdown{Score: 1, FieldScores: map[string]float64{"identity": 1}} + } response.Candidates = []ToolReference{reference} return finalizeToolSearchResponse(response) } @@ -296,12 +353,16 @@ func (e *ToolSearchEngine) Search(ctx context.Context, request ToolSearchRequest } references := make([]ToolReference, 0, len(hits)) for _, item := range hits { - references = append(references, toolReference( + reference := toolReference( e.documents[item.CanonicalPath].tool, item.Score, item.MatchedFields, []string{e.lexical.Name()}, - )) + ) + if e.config.Explain && item.Explain != nil { + reference.ScoreBreakdown = item.Explain + } + references = append(references, reference) } if len(references) > request.Limit { @@ -437,11 +498,82 @@ func normalizeToolSearchConfig(config ToolSearchConfig) ToolSearchConfig { if config.FieldWeights == (ToolSearchFieldWeights{}) { config.FieldWeights = defaults.FieldWeights } + if config.BM25K1 <= 0 { + config.BM25K1 = defaults.BM25K1 + } + if config.BM25B < 0 { + config.BM25B = defaults.BM25B + } config.CatalogSourceHash = strings.TrimSpace(config.CatalogSourceHash) config.CatalogSurfaceHash = strings.TrimSpace(config.CatalogSurfaceHash) return config } +// ResolveToolSearchConfigFromEnv applies optional DWS_TOOL_SEARCH_* overrides +// on top of an existing config. Unset or unparsable variables leave the field +// unchanged, so the default path is byte-identical when no env is present. +// This keeps ranking experiments (e.g. re-evaluating action_v1 against +// independent qrels) tunable and reversible without a source change. +func ResolveToolSearchConfigFromEnv(config ToolSearchConfig) ToolSearchConfig { + if value := strings.TrimSpace(os.Getenv("DWS_TOOL_SEARCH_ALGORITHM")); value != "" { + switch value { + case ToolSearchLexicalBM25, ToolSearchLexicalBM25Action, ToolSearchLexicalTFIDF: + config.LexicalAlgorithm = value + } + } + if value, ok := lookupToolSearchFloatEnv("DWS_TOOL_SEARCH_K1"); ok { + config.BM25K1 = value + } + if value, ok := lookupToolSearchFloatEnv("DWS_TOOL_SEARCH_B"); ok { + config.BM25B = value + } + config.FieldWeights.Identity = lookupToolSearchFloatEnvOr("DWS_TOOL_SEARCH_WEIGHT_IDENTITY", config.FieldWeights.Identity) + config.FieldWeights.Summary = lookupToolSearchFloatEnvOr("DWS_TOOL_SEARCH_WEIGHT_SUMMARY", config.FieldWeights.Summary) + config.FieldWeights.Description = lookupToolSearchFloatEnvOr("DWS_TOOL_SEARCH_WEIGHT_DESCRIPTION", config.FieldWeights.Description) + config.FieldWeights.Parameters = lookupToolSearchFloatEnvOr("DWS_TOOL_SEARCH_WEIGHT_PARAMETERS", config.FieldWeights.Parameters) + config.FieldWeights.UseWhen = lookupToolSearchFloatEnvOr("DWS_TOOL_SEARCH_WEIGHT_USEWHEN", config.FieldWeights.UseWhen) + if value, ok := os.LookupEnv("DWS_TOOL_SEARCH_EXPLAIN"); ok { + config.Explain = parseToolSearchBoolEnv(value) + } + return config +} + +func lookupToolSearchFloatEnv(name string) (float64, bool) { + raw, ok := os.LookupEnv(name) + if !ok { + return 0, false + } + value, err := strconv.ParseFloat(raw, 64) + if err != nil { + return 0, false + } + return value, true +} + +func lookupToolSearchFloatEnvOr(name string, fallback float64) float64 { + if value, ok := lookupToolSearchFloatEnv(name); ok { + return value + } + return fallback +} + +func parseToolSearchBoolEnv(value string) bool { + switch strings.ToLower(strings.TrimSpace(value)) { + case "1", "true", "yes", "on": + return true + default: + return false + } +} + +func newToolSearchBreakdown(score float64, contributions map[toolSearchField]float64) *ToolSearchScoreBreakdown { + fieldScores := make(map[string]float64, len(contributions)) + for field, value := range contributions { + fieldScores[string(field)] = value + } + return &ToolSearchScoreBreakdown{Score: score, FieldScores: fieldScores} +} + func (e *ToolSearchEngine) normalizeRequest(request ToolSearchRequest) (ToolSearchRequest, error) { request.Query = strings.TrimSpace(request.Query) if request.Query == "" { @@ -534,12 +666,12 @@ func toolSearchDocumentFields(tool ToolSpec, includeUseWhen bool) map[toolSearch } } -func newToolSearchBM25Index(documents map[string][]string) toolSearchBM25Index { +func newToolSearchBM25Index(documents map[string][]string, k1, b float64) toolSearchBM25Index { index := toolSearchBM25Index{ documents: make(map[string]toolSearchBM25Document, len(documents)), idf: make(map[string]float64), - k1: 0.9, - b: 0.4, + k1: k1, + b: b, } documentFrequency := make(map[string]int) var totalLength int @@ -756,11 +888,14 @@ func tokenizeToolSearchText(text string) []string { ascii = ascii[:0] } } + // Emit overlapping bigrams plus unigrams. Bigrams carry phrase precision; + // unigrams keep single-character intents (e.g. "批") matchable against + // documents that only contain the composed word ("审批"). BM25's IDF + // naturally down-weights unigrams that fire across many documents. flushChinese := func() { - if len(chinese) == 1 { - tokens = append(tokens, string(chinese)) - } else { - for index := 0; index+1 < len(chinese); index++ { + for index := 0; index < len(chinese); index++ { + tokens = append(tokens, string(chinese[index:index+1])) + if index+1 < len(chinese) { tokens = append(tokens, string(chinese[index:index+2])) } } @@ -796,6 +931,10 @@ func splitToolSearchCamelCase(value string) string { return builder.String() } +// isToolSearchCJK mirrors unicode.Han (the same table the evaluation language +// slices use) so the tokenizer and the metrics agree on what counts as +// Chinese. The previous hard-coded U+3400..U+9FFF range silently dropped CJK +// compatibility ideographs and extension planes as separators. func isToolSearchCJK(value rune) bool { - return value >= '\u3400' && value <= '\u9fff' + return unicode.Is(unicode.Han, value) } diff --git a/internal/cli/tool_search_evaluation.go b/internal/cli/tool_search_evaluation.go index b7ddec17..a02507cd 100644 --- a/internal/cli/tool_search_evaluation.go +++ b/internal/cli/tool_search_evaluation.go @@ -134,7 +134,7 @@ type ToolSearchDiagnosticComparison struct { // retriever against the declaration-assembled Catalog without reading a // generated repository JSON artifact. Results belong under policy-tmp. func BuildDeliveryToolSearchDiagnosticComparison(ctx context.Context, workflows []ToolSearchWorkflowEvaluationCase) (ToolSearchDiagnosticComparison, error) { - return BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(ctx, workflows, ToolSearchLexicalBM25Action) + return BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(ctx, workflows, DefaultToolSearchConfig().LexicalAlgorithm) } // BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm runs one shipped Go diff --git a/internal/cli/tool_search_independent_evaluation.go b/internal/cli/tool_search_independent_evaluation.go index 455d3e5f..203908c3 100644 --- a/internal/cli/tool_search_independent_evaluation.go +++ b/internal/cli/tool_search_independent_evaluation.go @@ -80,11 +80,21 @@ type independentRankingAccumulator struct { zero int } -// BuildDeliveryToolSearchIndependentEvaluation evaluates sealed external -// qrels with the shipped Go ranker. It never tunes parameters or reads Catalog -// JSON; the caller owns qrels sealing and threshold enforcement. +// BuildDeliveryToolSearchIndependentEvaluation evaluates the action-ranking +// candidate against the shipped fielded BM25 control on sealed external qrels. +// It never tunes parameters or reads Catalog JSON; the caller owns qrels +// sealing and threshold enforcement. The candidate may become the runtime +// default only after the independently frozen default-switch gate passes. func BuildDeliveryToolSearchIndependentEvaluation(ctx context.Context, cases []ToolSearchIndependentCase) (ToolSearchIndependentReport, error) { - engine, err := NewDeliveryToolSearchEngine() + if err := deliverySchemaCatalogError(); err != nil { + return ToolSearchIndependentReport{}, err + } + loaded := deliverySchemaCatalog() + candidateConfig := DefaultToolSearchConfig() + candidateConfig.LexicalAlgorithm = ToolSearchLexicalBM25Action + candidateConfig.CatalogSourceHash = loaded.Snapshot.SourceHash + candidateConfig.CatalogSurfaceHash = loaded.Snapshot.SurfaceHash + engine, err := NewToolSearchEngine(loaded.Registry, candidateConfig) if err != nil { return ToolSearchIndependentReport{}, err } @@ -92,12 +102,7 @@ func BuildDeliveryToolSearchIndependentEvaluation(ctx context.Context, cases []T if err != nil { return ToolSearchIndependentReport{}, err } - if err := deliverySchemaCatalogError(); err != nil { - return ToolSearchIndependentReport{}, err - } - loaded := deliverySchemaCatalog() controlConfig := DefaultToolSearchConfig() - controlConfig.LexicalAlgorithm = ToolSearchLexicalBM25 controlConfig.CatalogSourceHash = loaded.Snapshot.SourceHash controlConfig.CatalogSurfaceHash = loaded.Snapshot.SurfaceHash control, err := NewToolSearchEngine(loaded.Registry, controlConfig) diff --git a/internal/cli/tool_search_lexical.go b/internal/cli/tool_search_lexical.go index e3070348..db9f7f15 100644 --- a/internal/cli/tool_search_lexical.go +++ b/internal/cli/tool_search_lexical.go @@ -11,6 +11,7 @@ import ( "regexp" "sort" "strings" + "unicode/utf8" ) const ( @@ -29,10 +30,12 @@ type ToolSearchLexicalRequest struct { } // LexicalHit is an internal ordering result, not a confidence probability. +// Explain is populated only when the engine was built with Explain=true. type LexicalHit struct { CanonicalPath string Score float64 MatchedFields []string + Explain *ToolSearchScoreBreakdown } // LexicalRetriever is the zero-model local recall boundary. Implementations @@ -44,27 +47,34 @@ type LexicalRetriever interface { } func newToolSearchLexicalRetriever(documents map[string]toolSearchDocument, config ToolSearchConfig) (LexicalRetriever, error) { + var retriever LexicalRetriever switch config.LexicalAlgorithm { case ToolSearchLexicalBM25Action: - return newToolSearchActionRetriever(newToolSearchBM25Retriever(documents, config.FieldWeights), documents), nil + retriever = newToolSearchActionRetriever( + newToolSearchBM25Retriever(documents, config.FieldWeights, config.BM25K1, config.BM25B, config.Explain), + documents, + config.Explain, + ) case ToolSearchLexicalBM25: - return newToolSearchBM25Retriever(documents, config.FieldWeights), nil + retriever = newToolSearchBM25Retriever(documents, config.FieldWeights, config.BM25K1, config.BM25B, config.Explain) case ToolSearchLexicalTFIDF: - return newToolSearchTFIDFRetriever(documents, config.FieldWeights), nil + retriever = newToolSearchTFIDFRetriever(documents, config.FieldWeights, config.Explain) default: return nil, fmt.Errorf("unknown tool search lexical algorithm %q", config.LexicalAlgorithm) } + return newToolSearchAvoidWhenRetriever(retriever, documents, config.Explain), nil } type toolSearchActionRetriever struct { base LexicalRetriever documents map[string]toolSearchDocument + explain bool } var toolSearchASCIIWordPattern = regexp.MustCompile(`[A-Za-z][A-Za-z0-9._-]*`) -func newToolSearchActionRetriever(base LexicalRetriever, documents map[string]toolSearchDocument) *toolSearchActionRetriever { - return &toolSearchActionRetriever{base: base, documents: documents} +func newToolSearchActionRetriever(base LexicalRetriever, documents map[string]toolSearchDocument, explain bool) *toolSearchActionRetriever { + return &toolSearchActionRetriever{base: base, documents: documents, explain: explain} } func (*toolSearchActionRetriever) Name() string { return ToolSearchLexicalBM25Action } @@ -81,17 +91,39 @@ func (r *toolSearchActionRetriever) Retrieve(ctx context.Context, request ToolSe return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil } for index := range hits { - hits[index].Score *= toolSearchStructuredMultiplier(query, r.documents[hits[index].CanonicalPath].tool) + multiplier := toolSearchStructuredMultiplier(query, r.documents[hits[index].CanonicalPath].tool) + hits[index].Score *= multiplier + if r.explain && hits[index].Explain != nil { + hits[index].Explain.Multiplier = &multiplier + hits[index].Explain.Score = hits[index].Score + hits[index].Explain.QueryClass = newToolSearchExplainQueryClass(query) + } } return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil } +// toolSearchHasUnclassifiedASCII reports whether the query carries a technical +// identifier (camelCase or separator-joined tokens such as openConversationId, +// task_id, file-path) that the compact vocabularies cannot classify. Plain +// English words no longer disable structured reranking: they match (or fail +// to match) the vocabularies exactly like Chinese phrases do, so mixed and +// English queries keep the effect/product/action/entity routing instead of +// silently degrading to raw BM25. func toolSearchHasUnclassifiedASCII(query string) bool { - for _, token := range toolSearchASCIIWordPattern.FindAllString(strings.ToLower(query), -1) { - switch token { + for _, token := range toolSearchASCIIWordPattern.FindAllString(query, -1) { + if len(token) < 4 { + continue + } + switch strings.ToLower(token) { case "id", "ids", "url", "uri": continue - default: + } + if strings.ContainsAny(token, "._-") { + return true + } + // Only an internal uppercase transition (openConversationId) marks a + // technical identifier; a capitalized plain word ("Send") does not. + if internal := token[1:]; strings.ToLower(internal) != internal { return true } } @@ -117,49 +149,49 @@ type toolSearchTermClass struct { } var toolSearchActionClasses = []toolSearchTermClass{ - {name: "approve", query: []string{"同意", "批准", "审批通过"}, tool: []string{"approve"}}, - {name: "reject", query: []string{"拒绝", "驳回"}, tool: []string{"reject"}}, - {name: "upload", query: []string{"上传"}, tool: []string{"upload"}}, - {name: "download", query: []string{"下载"}, tool: []string{"download"}}, - {name: "send", query: []string{"发送", "发给", "发消息", "通知", "投递"}, tool: []string{"send", "notify"}}, - {name: "search", query: []string{"搜索", "查找", "检索", "定位"}, tool: []string{"search", "find", "resolve"}}, - {name: "list", query: []string{"列出", "列表", "浏览", "遍历"}, tool: []string{"list", "all"}}, - {name: "read", query: []string{"读取", "查看", "获取", "详情", "正文"}, tool: []string{"read", "get", "detail", "info", "query"}}, - {name: "create", query: []string{"创建", "新建"}, tool: []string{"create", "new"}}, - {name: "add", query: []string{"添加", "追加", "增加", "授权", "授予"}, tool: []string{"add", "grant", "invite"}}, - {name: "update", query: []string{"更新", "修改", "编辑", "重命名", "调整"}, tool: []string{"update", "edit", "rename", "set"}}, - {name: "delete", query: []string{"删除", "移除", "清除"}, tool: []string{"delete", "remove", "clear"}}, - {name: "complete", query: []string{"完成", "提交入库"}, tool: []string{"complete", "commit", "finish"}}, - {name: "enable", query: []string{"启用", "开启"}, tool: []string{"enable"}}, - {name: "disable", query: []string{"禁用", "停用", "关闭"}, tool: []string{"disable"}}, + {name: "approve", query: []string{"同意", "批准", "审批通过", "approve"}, tool: []string{"approve"}}, + {name: "reject", query: []string{"拒绝", "驳回", "reject"}, tool: []string{"reject"}}, + {name: "upload", query: []string{"上传", "upload"}, tool: []string{"upload"}}, + {name: "download", query: []string{"下载", "download"}, tool: []string{"download"}}, + {name: "send", query: []string{"发送", "发给", "发消息", "通知", "投递", "send", "notify"}, tool: []string{"send", "notify"}}, + {name: "search", query: []string{"搜索", "查找", "检索", "定位", "search", "find", "locate"}, tool: []string{"search", "find", "resolve"}}, + {name: "list", query: []string{"列出", "列表", "浏览", "遍历", "list", "browse"}, tool: []string{"list", "all"}}, + {name: "read", query: []string{"读取", "查看", "获取", "详情", "正文", "read", "fetch"}, tool: []string{"read", "get", "detail", "info", "query"}}, + {name: "create", query: []string{"创建", "新建", "create"}, tool: []string{"create", "new"}}, + {name: "add", query: []string{"添加", "追加", "增加", "授权", "授予", "add", "invite"}, tool: []string{"add", "grant", "invite"}}, + {name: "update", query: []string{"更新", "修改", "编辑", "重命名", "调整", "update", "edit", "rename"}, tool: []string{"update", "edit", "rename", "set"}}, + {name: "delete", query: []string{"删除", "移除", "清除", "delete", "remove"}, tool: []string{"delete", "remove", "clear"}}, + {name: "complete", query: []string{"完成", "提交入库", "complete", "commit"}, tool: []string{"complete", "commit", "finish"}}, + {name: "enable", query: []string{"启用", "开启", "enable"}, tool: []string{"enable"}}, + {name: "disable", query: []string{"禁用", "停用", "关闭", "disable"}, tool: []string{"disable"}}, } var toolSearchProductClasses = []toolSearchTermClass{ - {name: "chat", query: []string{"群聊", "群消息", "会话", "单聊"}, tool: []string{"chat"}}, - {name: "calendar", query: []string{"日程", "会议日历", "会议室"}, tool: []string{"calendar"}}, - {name: "drive", query: []string{"钉盘", "云盘"}, tool: []string{"drive"}}, - {name: "wiki", query: []string{"知识库"}, tool: []string{"wiki"}}, - {name: "todo", query: []string{"待办"}, tool: []string{"todo"}}, - {name: "minutes", query: []string{"听记", "会议纪要"}, tool: []string{"minutes"}}, - {name: "oa", query: []string{"审批"}, tool: []string{"oa"}}, - {name: "mail", query: []string{"邮件", "邮箱", "草稿"}, tool: []string{"mail"}}, - {name: "sheet", query: []string{"电子表格", "工作表", "单元格"}, tool: []string{"sheet"}}, - {name: "doc", query: []string{"在线文字文档", "文档正文", "adoc"}, tool: []string{"doc"}}, + {name: "chat", query: []string{"群聊", "群消息", "会话", "单聊", "chat"}, tool: []string{"chat"}}, + {name: "calendar", query: []string{"日程", "会议日历", "会议室", "calendar"}, tool: []string{"calendar"}}, + {name: "drive", query: []string{"钉盘", "云盘", "drive"}, tool: []string{"drive"}}, + {name: "wiki", query: []string{"知识库", "wiki"}, tool: []string{"wiki"}}, + {name: "todo", query: []string{"待办", "todo"}, tool: []string{"todo"}}, + {name: "minutes", query: []string{"听记", "会议纪要", "minutes"}, tool: []string{"minutes"}}, + {name: "oa", query: []string{"审批", "oa", "approval"}, tool: []string{"oa"}}, + {name: "mail", query: []string{"邮件", "邮箱", "草稿", "mail", "email"}, tool: []string{"mail"}}, + {name: "sheet", query: []string{"电子表格", "工作表", "单元格", "sheet", "spreadsheet"}, tool: []string{"sheet"}}, + {name: "doc", query: []string{"在线文字文档", "文档正文", "adoc", "doc", "document"}, tool: []string{"doc"}}, } var toolSearchEntityClasses = []toolSearchTermClass{ {name: "task_id", query: []string{"任务 id", "任务id", "task id", "taskid"}, tool: []string{"任务 id", "任务id", "task id", "taskid"}}, - {name: "task", query: []string{"任务", "task id", "taskid"}, tool: []string{"任务", "task"}}, - {name: "permission", query: []string{"权限", "授权", "协作者"}, tool: []string{"权限", "授权", "协作者", "permission"}}, - {name: "content", query: []string{"正文", "文档内容"}, tool: []string{"正文", "内容", "content"}}, - {name: "status", query: []string{"状态", "已读"}, tool: []string{"状态", "已读", "status"}}, - {name: "draft", query: []string{"草稿"}, tool: []string{"草稿", "draft"}}, - {name: "event", query: []string{"日程"}, tool: []string{"日程", "event"}}, - {name: "reminder", query: []string{"提醒"}, tool: []string{"提醒", "reminder"}}, - {name: "summary", query: []string{"摘要"}, tool: []string{"摘要", "summary"}}, - {name: "sheet", query: []string{"工作表"}, tool: []string{"工作表", "sheet"}}, - {name: "range", query: []string{"单元格范围", "区域"}, tool: []string{"范围", "区域", "range"}}, - {name: "group", query: []string{"群聊", "群名"}, tool: []string{"群聊", "群", "group"}}, + {name: "task", query: []string{"任务", "task id", "taskid", "task"}, tool: []string{"任务", "task"}}, + {name: "permission", query: []string{"权限", "授权", "协作者", "permission"}, tool: []string{"权限", "授权", "协作者", "permission"}}, + {name: "content", query: []string{"正文", "文档内容", "content"}, tool: []string{"正文", "内容", "content"}}, + {name: "status", query: []string{"状态", "已读", "status"}, tool: []string{"状态", "已读", "status"}}, + {name: "draft", query: []string{"草稿", "draft"}, tool: []string{"草稿", "draft"}}, + {name: "event", query: []string{"日程", "event"}, tool: []string{"日程", "event"}}, + {name: "reminder", query: []string{"提醒", "reminder"}, tool: []string{"提醒", "reminder"}}, + {name: "summary", query: []string{"摘要", "summary"}, tool: []string{"摘要", "summary"}}, + {name: "sheet", query: []string{"工作表", "worksheet"}, tool: []string{"工作表", "sheet"}}, + {name: "range", query: []string{"单元格范围", "区域", "range"}, tool: []string{"范围", "区域", "range"}}, + {name: "group", query: []string{"群聊", "群名", "group"}, tool: []string{"群聊", "群", "group"}}, } func classifyToolSearchQuery(query string) toolSearchQueryClass { @@ -233,7 +265,6 @@ func toolSearchStructuredMultiplier(query toolSearchQueryClass, tool ToolSpec) f } } if len(query.entities) > 0 { - specificTaskID := query.entities["task_id"] && query.products["oa"] matched := false for _, class := range toolSearchEntityClasses { if query.entities[class.name] && containsAnyToolSearchPhrase(toolText, class.tool) { @@ -241,12 +272,8 @@ func toolSearchStructuredMultiplier(query toolSearchQueryClass, tool ToolSpec) f break } } - if matched && specificTaskID { - multiplier += 1.00 - } else if matched { + if matched { multiplier += 0.35 - } else if specificTaskID { - multiplier -= 0.20 } else { multiplier -= 0.10 } @@ -259,7 +286,38 @@ func toolSearchStructuredMultiplier(query toolSearchQueryClass, tool ToolSpec) f func containsAnyToolSearchPhrase(text string, values []string) bool { for _, value := range values { - if strings.Contains(text, value) { + if toolSearchPhraseContains(text, value) { + return true + } + } + return false +} + +// toolSearchPhraseContains matches ASCII single-token values on word +// boundaries so short English values such as "oa" cannot fire inside +// unrelated words ("download" contains "oa"). Mixed-script or multi-word +// values keep substring semantics, which is what Chinese phrases and values +// like "task id" rely on. +func toolSearchPhraseContains(text, value string) bool { + if value != "" && isAllToolSearchASCII(value) && !strings.ContainsAny(value, " \t") { + return containsToolSearchASCIIWord(text, value) + } + return strings.Contains(text, value) +} + +func isAllToolSearchASCII(value string) bool { + for _, character := range value { + if character >= utf8.RuneSelf { + return false + } + } + return true +} + +func containsToolSearchASCIIWord(text, word string) bool { + word = strings.ToLower(word) + for _, token := range toolSearchASCIIWordPattern.FindAllString(strings.ToLower(text), -1) { + if token == word { return true } } @@ -279,18 +337,116 @@ func containsAnyToolSearchWord(text string, values []string) bool { return false } +func newToolSearchExplainQueryClass(query toolSearchQueryClass) *ToolSearchExplainQueryClass { + return &ToolSearchExplainQueryClass{ + Actions: sortedToolSearchClassKeys(query.actions), + Products: sortedToolSearchClassKeys(query.products), + Entities: sortedToolSearchClassKeys(query.entities), + Effect: query.effect, + } +} + +func sortedToolSearchClassKeys(set map[string]bool) []string { + if len(set) == 0 { + return nil + } + keys := make([]string, 0, len(set)) + for key := range set { + keys = append(keys, key) + } + sort.Strings(keys) + return keys +} + +// toolSearchAvoidWhenPenalty is applied when a candidate's avoid_when prose +// matches the query intent. It is a soft demotion, not an eligibility filter: +// avoid_when is natural-language guidance (often "prefer X for that intent"), +// so a lexical hit can be wrong and the tool must remain reachable. +const toolSearchAvoidWhenPenalty = 0.20 + +// toolSearchAvoidWhenShortcutNoise filters the boilerplate avoid_when that +// every shortcut carries; it carries no intent signal. +const toolSearchAvoidWhenShortcutNoise = "需要该 Shortcut 未公开的底层参数" + +// toolSearchAvoidWhenPenaltyReason returns the matched avoid_when phrase, or +// "" when the tool should not be demoted for this query. The only match shape +// is the query containing the avoidance prose itself (the Forbidden@5 +// evaluation proxy). Intent-overlap matching was tried and rejected: a tool's +// own avoid_when describes the same sibling scenario space as its use_when, +// so "shares an action/entity with the phrase" demotes the gold answer. +func toolSearchAvoidWhenPenaltyReason(query string, tool ToolSpec) string { + if len(tool.Selection.AvoidWhen) == 0 { + return "" + } + normalized := strings.ToLower(strings.TrimSpace(query)) + if normalized == "" { + return "" + } + for _, phrase := range tool.Selection.AvoidWhen { + phrase = strings.ToLower(strings.TrimSpace(phrase)) + if phrase == "" || strings.Contains(phrase, toolSearchAvoidWhenShortcutNoise) { + continue + } + if strings.Contains(normalized, phrase) { + return phrase + } + } + return "" +} + +// toolSearchAvoidWhenRetriever demotes candidates whose avoid_when prose +// matches the query intent. It wraps every lexical algorithm, including the +// production ensemble default, so the Forbidden exposure guard is not tied to +// action_v1. +type toolSearchAvoidWhenRetriever struct { + base LexicalRetriever + documents map[string]toolSearchDocument + explain bool +} + +func newToolSearchAvoidWhenRetriever(base LexicalRetriever, documents map[string]toolSearchDocument, explain bool) *toolSearchAvoidWhenRetriever { + return &toolSearchAvoidWhenRetriever{base: base, documents: documents, explain: explain} +} + +func (r *toolSearchAvoidWhenRetriever) Name() string { return r.base.Name() } + +func (r *toolSearchAvoidWhenRetriever) Retrieve(ctx context.Context, request ToolSearchLexicalRequest) ([]LexicalHit, error) { + hits, err := r.base.Retrieve(ctx, request) + if err != nil { + return nil, err + } + penalized := false + for index := range hits { + reason := toolSearchAvoidWhenPenaltyReason(request.Query, r.documents[hits[index].CanonicalPath].tool) + if reason == "" { + continue + } + hits[index].Score *= toolSearchAvoidWhenPenalty + if r.explain && hits[index].Explain != nil { + hits[index].Explain.Score = hits[index].Score + hits[index].Explain.AvoidWhenPenalty = reason + } + penalized = true + } + if !penalized { + return hits, nil + } + return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil +} + type toolSearchBM25Retriever struct { fieldIndex map[toolSearchField]toolSearchBM25Index weights ToolSearchFieldWeights + explain bool } -func newToolSearchBM25Retriever(documents map[string]toolSearchDocument, weights ToolSearchFieldWeights) *toolSearchBM25Retriever { +func newToolSearchBM25Retriever(documents map[string]toolSearchDocument, weights ToolSearchFieldWeights, k1, b float64, explain bool) *toolSearchBM25Retriever { fieldTerms := toolSearchFieldTerms(documents) fieldIndex := make(map[toolSearchField]toolSearchBM25Index, len(toolSearchFieldOrder)) for _, field := range toolSearchFieldOrder { - fieldIndex[field] = newToolSearchBM25Index(fieldTerms[field]) + fieldIndex[field] = newToolSearchBM25Index(fieldTerms[field], k1, b) } - return &toolSearchBM25Retriever{fieldIndex: fieldIndex, weights: weights} + return &toolSearchBM25Retriever{fieldIndex: fieldIndex, weights: weights, explain: explain} } func (*toolSearchBM25Retriever) Name() string { return ToolSearchLexicalBM25 } @@ -316,7 +472,11 @@ func (r *toolSearchBM25Retriever) Retrieve(ctx context.Context, request ToolSear } } if score > 0 { - hits = append(hits, LexicalHit{CanonicalPath: canonical, Score: score, MatchedFields: matchedToolSearchFields(contributions)}) + hit := LexicalHit{CanonicalPath: canonical, Score: score, MatchedFields: matchedToolSearchFields(contributions)} + if r.explain { + hit.Explain = newToolSearchBreakdown(score, contributions) + } + hits = append(hits, hit) } } return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil @@ -335,15 +495,16 @@ type toolSearchTFIDFIndex struct { type toolSearchTFIDFRetriever struct { fieldIndex map[toolSearchField]toolSearchTFIDFIndex weights ToolSearchFieldWeights + explain bool } -func newToolSearchTFIDFRetriever(documents map[string]toolSearchDocument, weights ToolSearchFieldWeights) *toolSearchTFIDFRetriever { +func newToolSearchTFIDFRetriever(documents map[string]toolSearchDocument, weights ToolSearchFieldWeights, explain bool) *toolSearchTFIDFRetriever { fieldTerms := toolSearchFieldTerms(documents) fieldIndex := make(map[toolSearchField]toolSearchTFIDFIndex, len(toolSearchFieldOrder)) for _, field := range toolSearchFieldOrder { fieldIndex[field] = newToolSearchTFIDFIndex(fieldTerms[field]) } - return &toolSearchTFIDFRetriever{fieldIndex: fieldIndex, weights: weights} + return &toolSearchTFIDFRetriever{fieldIndex: fieldIndex, weights: weights, explain: explain} } func (*toolSearchTFIDFRetriever) Name() string { return ToolSearchLexicalTFIDF } @@ -369,7 +530,11 @@ func (r *toolSearchTFIDFRetriever) Retrieve(ctx context.Context, request ToolSea } } if score > 0 { - hits = append(hits, LexicalHit{CanonicalPath: canonical, Score: score, MatchedFields: matchedToolSearchFields(contributions)}) + hit := LexicalHit{CanonicalPath: canonical, Score: score, MatchedFields: matchedToolSearchFields(contributions)} + if r.explain { + hit.Explain = newToolSearchBreakdown(score, contributions) + } + hits = append(hits, hit) } } return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil diff --git a/internal/cli/tool_search_test.go b/internal/cli/tool_search_test.go index 00bf222c..5cb73965 100644 --- a/internal/cli/tool_search_test.go +++ b/internal/cli/tool_search_test.go @@ -202,8 +202,8 @@ func TestToolSearchSubqueriesRoundRobinAndAcceptsDeepCandidateLimit(t *testing.T func TestToolSearchActionRerankDoesNotChangeNoSignalOrdering(t *testing.T) { engine := newToolSearchTestEngine(t) - raw := newToolSearchBM25Retriever(engine.documents, engine.config.FieldWeights) - action := newToolSearchActionRetriever(raw, engine.documents) + raw := newToolSearchBM25Retriever(engine.documents, engine.config.FieldWeights, engine.config.BM25K1, engine.config.BM25B, false) + action := newToolSearchActionRetriever(raw, engine.documents, false) request := ToolSearchLexicalRequest{ Query: "媒体路径", CandidateLimit: 4, @@ -224,8 +224,8 @@ func TestToolSearchActionRerankDoesNotChangeNoSignalOrdering(t *testing.T) { func TestToolSearchActionRerankDefersUnclassifiedTechnicalASCII(t *testing.T) { engine := newToolSearchTestEngine(t) - raw := newToolSearchBM25Retriever(engine.documents, engine.config.FieldWeights) - action := newToolSearchActionRetriever(raw, engine.documents) + raw := newToolSearchBM25Retriever(engine.documents, engine.config.FieldWeights, engine.config.BM25K1, engine.config.BM25B, false) + action := newToolSearchActionRetriever(raw, engine.documents, false) request := ToolSearchLexicalRequest{ Query: "读取 openConversationId 对应的群消息", CandidateLimit: 4, @@ -306,6 +306,9 @@ func TestToolSearchSubqueriesFailClosedOnExactFilteredAction(t *testing.T) { func TestDefaultToolSearchConfigExcludesUseWhen(t *testing.T) { config := DefaultToolSearchConfig() + if config.LexicalAlgorithm != ToolSearchLexicalBM25 { + t.Fatalf("default lexical algorithm = %q, want independently conservative control %q", config.LexicalAlgorithm, ToolSearchLexicalBM25) + } if config.IncludeUseWhen { t.Fatal("DefaultToolSearchConfig() enables answer-bearing use_when projection") } @@ -316,9 +319,118 @@ func TestDefaultToolSearchConfigExcludesUseWhen(t *testing.T) { } } +func TestToolSearchExplainAttachesBreakdownOnlyWhenEnabled(t *testing.T) { + query := ToolSearchRequest{Query: "给群里发送消息"} + plain := newToolSearchTestEngine(t) + plainResponse, err := plain.Search(context.Background(), query) + if err != nil { + t.Fatalf("Search() error = %v", err) + } + if len(plainResponse.Candidates) == 0 || plainResponse.Candidates[0].CanonicalPath != "chat.send" { + t.Fatalf("plain candidates = %#v", plainResponse.Candidates) + } + plainPayload, err := json.Marshal(plainResponse) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(plainPayload), "score_breakdown") { + t.Fatalf("explain leaked into default response: %s", plainPayload) + } + for _, candidate := range plainResponse.Candidates { + if candidate.ScoreBreakdown != nil { + t.Fatalf("default candidate carried breakdown: %#v", candidate) + } + } + + explainConfig := DefaultToolSearchConfig() + explainConfig.Explain = true + explained := newToolSearchTestEngineWithConfig(t, explainConfig) + explainedResponse, err := explained.Search(context.Background(), query) + if err != nil { + t.Fatalf("Search(explain) error = %v", err) + } + if len(explainedResponse.Candidates) != len(plainResponse.Candidates) { + t.Fatalf("explain changed candidate count: %d vs %d", len(explainedResponse.Candidates), len(plainResponse.Candidates)) + } + for index := range explainedResponse.Candidates { + plainPath := plainResponse.Candidates[index].CanonicalPath + candidate := explainedResponse.Candidates[index] + if candidate.CanonicalPath != plainPath { + t.Fatalf("explain changed ordering at %d: %s vs %s", index, plainPath, candidate.CanonicalPath) + } + if candidate.ScoreBreakdown == nil { + t.Fatalf("explained candidate missing breakdown: %#v", candidate) + } + if candidate.ScoreBreakdown.Score <= 0 || len(candidate.ScoreBreakdown.FieldScores) == 0 { + t.Fatalf("breakdown incomplete: %#v", candidate.ScoreBreakdown) + } + if candidate.ScoreBreakdown.Multiplier != nil || candidate.ScoreBreakdown.QueryClass != nil { + t.Fatalf("ensemble breakdown should not carry action fields: %#v", candidate.ScoreBreakdown) + } + } +} + +func TestToolSearchActionExplainIncludesMultiplierAndQueryClass(t *testing.T) { + config := DefaultToolSearchConfig() + config.LexicalAlgorithm = ToolSearchLexicalBM25Action + config.Explain = true + engine := newToolSearchTestEngineWithConfig(t, config) + response, err := engine.Search(context.Background(), ToolSearchRequest{Query: "给群里发送消息"}) + if err != nil { + t.Fatalf("Search() error = %v", err) + } + if len(response.Candidates) == 0 || response.Candidates[0].CanonicalPath != "chat.send" { + t.Fatalf("action candidates = %#v", response.Candidates) + } + for _, candidate := range response.Candidates { + if candidate.ScoreBreakdown == nil { + t.Fatalf("action candidate missing breakdown: %#v", candidate) + } + if candidate.ScoreBreakdown.Multiplier == nil || *candidate.ScoreBreakdown.Multiplier <= 0 { + t.Fatalf("action breakdown missing multiplier: %#v", candidate.ScoreBreakdown) + } + if candidate.ScoreBreakdown.QueryClass == nil || len(candidate.ScoreBreakdown.QueryClass.Actions) == 0 { + t.Fatalf("action breakdown missing query class: %#v", candidate.ScoreBreakdown) + } + } +} + +func TestResolveToolSearchConfigFromEnvOverridesOnlyWhenSet(t *testing.T) { + config := ResolveToolSearchConfigFromEnv(DefaultToolSearchConfig()) + if config.LexicalAlgorithm != ToolSearchLexicalBM25 || config.BM25K1 != defaultToolSearchBM25K1 || config.Explain { + t.Fatalf("no-env config changed: %#v", config) + } + + t.Setenv("DWS_TOOL_SEARCH_ALGORITHM", ToolSearchLexicalBM25Action) + t.Setenv("DWS_TOOL_SEARCH_K1", "1.2") + t.Setenv("DWS_TOOL_SEARCH_WEIGHT_SUMMARY", "4.5") + t.Setenv("DWS_TOOL_SEARCH_EXPLAIN", "1") + config = ResolveToolSearchConfigFromEnv(DefaultToolSearchConfig()) + if config.LexicalAlgorithm != ToolSearchLexicalBM25Action { + t.Fatalf("algorithm override = %q", config.LexicalAlgorithm) + } + if config.BM25K1 != 1.2 { + t.Fatalf("k1 override = %v", config.BM25K1) + } + if config.FieldWeights.Summary != 4.5 { + t.Fatalf("summary weight override = %v", config.FieldWeights.Summary) + } + if !config.Explain { + t.Fatal("explain override not applied") + } + if config.BM25B != defaultToolSearchBM25B { + t.Fatalf("unset env should keep default b, got %v", config.BM25B) + } + + t.Setenv("DWS_TOOL_SEARCH_ALGORITHM", "not-a-real-algorithm") + if got := ResolveToolSearchConfigFromEnv(DefaultToolSearchConfig()).LexicalAlgorithm; got != ToolSearchLexicalBM25 { + t.Fatalf("invalid algorithm env should be ignored, got %q", got) + } +} + func TestToolSearchLexicalRetrieverCanSelectGoTFIDF(t *testing.T) { engine := newToolSearchTestEngine(t) - engine.lexical = newToolSearchTFIDFRetriever(engine.documents, engine.config.FieldWeights) + engine.lexical = newToolSearchTFIDFRetriever(engine.documents, engine.config.FieldWeights, false) response, err := engine.Search(context.Background(), ToolSearchRequest{ Query: "给群里发送消息", ProductIDs: []string{"chat"}, @@ -433,7 +545,7 @@ func TestToolSearchIsDeterministicAcrossProcesses(t *testing.T) { if os.Getenv(helperEnv) == "1" { engine := newToolSearchTestEngine(t) tfidfEngine := newToolSearchTestEngine(t) - tfidfEngine.lexical = newToolSearchTFIDFRetriever(tfidfEngine.documents, tfidfEngine.config.FieldWeights) + tfidfEngine.lexical = newToolSearchTFIDFRetriever(tfidfEngine.documents, tfidfEngine.config.FieldWeights, false) tests := []struct { name string engine *ToolSearchEngine @@ -521,6 +633,22 @@ type toolSearchTestingT interface { } func newToolSearchTestEngine(t toolSearchTestingT) *ToolSearchEngine { + t.Helper() + return newToolSearchTestEngineWithConfig(t, DefaultToolSearchConfig()) +} + +func newToolSearchTestEngineWithConfig(t toolSearchTestingT, config ToolSearchConfig) *ToolSearchEngine { + t.Helper() + config.CatalogSourceHash = "source-test" + config.CatalogSurfaceHash = "surface-test" + engine, err := NewToolSearchEngine(newToolSearchTestRegistry(t), config) + if err != nil { + t.Fatalf("NewToolSearchEngine() error = %v", err) + } + return engine +} + +func newToolSearchTestRegistry(t toolSearchTestingT) SchemaRegistry { t.Helper() tools := []ToolSpec{ toolSearchTestTool("chat", "send", "chat send", "发送群聊消息", "write"), @@ -536,7 +664,7 @@ func newToolSearchTestEngine(t toolSearchTestingT) *ToolSearchEngine { }} tools[3].Interface.Availability = contract.InterfaceUnavailable tools[3].Interface.Reason = "disabled in this test Catalog" - registry := SchemaRegistry{ + return SchemaRegistry{ Kind: "schema", Level: "catalog", Products: []ProductSpec{ @@ -545,14 +673,6 @@ func newToolSearchTestEngine(t toolSearchTestingT) *ToolSearchEngine { {ID: "drive", Tools: tools[3:]}, }, } - config := DefaultToolSearchConfig() - config.CatalogSourceHash = "source-test" - config.CatalogSurfaceHash = "surface-test" - engine, err := NewToolSearchEngine(registry, config) - if err != nil { - t.Fatalf("NewToolSearchEngine() error = %v", err) - } - return engine } func toolSearchTestTool(product, name, cliPath, summary, effect string) ToolSpec { diff --git a/internal/generator/cmd_tool_search_comparison/main.go b/internal/generator/cmd_tool_search_comparison/main.go index 2aea3abe..586df6bf 100644 --- a/internal/generator/cmd_tool_search_comparison/main.go +++ b/internal/generator/cmd_tool_search_comparison/main.go @@ -78,7 +78,7 @@ func main() { if err != nil { fatal(err) } - control, err := cli.BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(context.Background(), workflows, cli.ToolSearchLexicalBM25) + actionShadow, err := cli.BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(context.Background(), workflows, cli.ToolSearchLexicalBM25Action) if err != nil { fatal(err) } @@ -86,7 +86,7 @@ func main() { if err != nil { fatal(err) } - result := comparisonOutput{Diagnostic: diagnostic, Shadows: []cli.ToolSearchDiagnosticComparison{control, shadow}} + result := comparisonOutput{Diagnostic: diagnostic, Shadows: []cli.ToolSearchDiagnosticComparison{actionShadow, shadow}} if independentQrelsPath != "" { fixture, readErr := readIndependentQrels(independentQrelsPath) if readErr != nil { diff --git a/scripts/policy/check-generated-drift.sh b/scripts/policy/check-generated-drift.sh index 7063282b..4ec91f7d 100755 --- a/scripts/policy/check-generated-drift.sh +++ b/scripts/policy/check-generated-drift.sh @@ -116,5 +116,6 @@ fi # Assembly determinism validates fresh CI/local Catalog dumps. "$ROOT/scripts/policy/check-schema-assembly.sh" +python3 "$ROOT/scripts/policy/check-shortcut-public-catalog-drift.py" printf 'generated drift check: ok\n' diff --git a/scripts/policy/check-shortcut-public-catalog-drift.py b/scripts/policy/check-shortcut-public-catalog-drift.py new file mode 100644 index 00000000..71c9e1cb --- /dev/null +++ b/scripts/policy/check-shortcut-public-catalog-drift.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +"""Verify the committed shortcut runtime map matches the skill catalog.""" + +from __future__ import annotations + +import json +import re +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[2] +CATALOG_PATH = ROOT / "docs" / "shortcut-public-catalog.json" +GO_PATH = ROOT / "internal" / "shortcut" / "public_catalog_generated.go" +GO_ENTRY = re.compile(r'^\s*("(?:\\.|[^"\\])*"):\s*\{\},\s*$') + + +def fail(message: str) -> None: + raise SystemExit(f"shortcut public catalog drift: {message}") + + +def main() -> None: + payload = json.loads(CATALOG_PATH.read_text(encoding="utf-8")) + rows = payload.get("results") + if not isinstance(rows, list): + fail("docs catalog results must be a list") + + documented: set[str] = set() + for index, row in enumerate(rows): + if not isinstance(row, dict): + fail(f"results[{index}] must be an object") + service = row.get("service") + command = row.get("command") + if not isinstance(service, str) or not service or not isinstance(command, str) or not command: + fail(f"results[{index}] requires non-empty service and command") + key = f"{service}\0{command}" + if key in documented: + fail(f"duplicate docs entry {service} {command}") + documented.add(key) + + if payload.get("count") != len(documented): + fail(f"docs count={payload.get('count')!r}, actual={len(documented)}") + + runtime: set[str] = set() + for line in GO_PATH.read_text(encoding="utf-8").splitlines(): + match = GO_ENTRY.match(line) + if not match: + continue + key = json.loads(match.group(1)) + if key in runtime: + service, command = key.split("\0", 1) + fail(f"duplicate runtime entry {service} {command}") + runtime.add(key) + + if documented != runtime: + missing_runtime = sorted(documented - runtime) + missing_docs = sorted(runtime - documented) + details = [] + if missing_runtime: + details.append("missing from runtime: " + ", ".join(key.replace("\0", " ") for key in missing_runtime[:10])) + if missing_docs: + details.append("missing from docs: " + ", ".join(key.replace("\0", " ") for key in missing_docs[:10])) + fail("; ".join(details)) + + print(f"shortcut public catalog drift check: ok ({len(runtime)} entries)") + + +if __name__ == "__main__": + main() diff --git a/scripts/policy/skill-command-check/main.go b/scripts/policy/skill-command-check/main.go index 5da08530..380eda38 100644 --- a/scripts/policy/skill-command-check/main.go +++ b/scripts/policy/skill-command-check/main.go @@ -117,6 +117,13 @@ func schemaProjectionIssue(raw string) string { if len(tokens) < 2 || tokens[0] != "dws" || tokens[1] != "schema" { return "" } + // schema search already returns the bounded tool-search.v1 reference + // envelope. It is not a leaf Schema query and deliberately does not + // accept --compact/--jq/--fields, so the leaf projection rule must not + // be applied to it. + if len(tokens) >= 3 && tokens[2] == "search" { + return "" + } var targeted, compact, selected, all bool for i := 2; i < len(tokens); i++ { token := tokens[i] @@ -273,6 +280,14 @@ func resolveCommandReference(root *cobra.Command, path string) commandResolution if len(remaining) == 0 { return resolutionValid } + if cmd.Name() == "schema" && (cmd.Run != nil || cmd.RunE != nil) { + // `dws schema` is both a runnable parent (`schema `) and + // the parent of `schema search`. parseReference intentionally stores + // a normalized display path, so a quoted CLI-path alias may appear as + // multiple remaining words here; preserve the long-standing leaf + // validation behavior for this one versioned inspection surface. + return resolutionValid + } if !cmd.HasSubCommands() { // Once a leaf command has been found, remaining tokens are positional // arguments. This check intentionally validates command paths only. diff --git a/scripts/policy/skill-command-check/main_test.go b/scripts/policy/skill-command-check/main_test.go index 6c5e8703..95546325 100644 --- a/scripts/policy/skill-command-check/main_test.go +++ b/scripts/policy/skill-command-check/main_test.go @@ -93,7 +93,9 @@ func testCommandRoot() *cobra.Command { root := &cobra.Command{Use: "dws", Args: cobra.NoArgs} auth := &cobra.Command{Use: "auth", Args: cobra.NoArgs} auth.AddCommand(&cobra.Command{Use: "login", Aliases: []string{"signin"}}) - root.AddCommand(auth, &cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1)}) + schema := &cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1), Run: func(*cobra.Command, []string) {}} + schema.AddCommand(&cobra.Command{Use: "search"}) + root.AddCommand(auth, schema) return root } @@ -125,10 +127,12 @@ func TestResolveCommandReference(t *testing.T) { auth.AddCommand(&cobra.Command{Use: "login", Aliases: []string{"signin"}}) sheet := &cobra.Command{Use: "sheet", Args: cobra.NoArgs} sheet.AddCommand(&cobra.Command{Use: "read"}) + schema := &cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1), Run: func(*cobra.Command, []string) {}} + schema.AddCommand(&cobra.Command{Use: "search"}) root.AddCommand( auth, sheet, - &cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1)}, + schema, &cobra.Command{Use: "plugin-info ", Args: cobra.ExactArgs(1)}, ) @@ -145,6 +149,7 @@ func TestResolveCommandReference(t *testing.T) { {name: "top-level placeholder", path: "dws ", want: resolutionSkip}, {name: "nested placeholder", path: "dws sheet ", want: resolutionSkip}, {name: "quoted positional argument", path: "dws schema dev app create", want: resolutionValid}, + {name: "schema search subcommand", path: "dws schema search", want: resolutionValid}, {name: "leaf positional argument", path: "dws plugin-info example", want: resolutionValid}, } @@ -204,6 +209,9 @@ func testCrossPlatformCoverageSchemaProjectionIssue(t *testing.T) { wantErr bool }{ {command: `dws schema`, wantErr: false}, + {command: `dws schema search`, wantErr: false}, + {command: `dws schema search --query "create a calendar event" --limit 5`, wantErr: false}, + {command: `dws schema search --request-json -`, wantErr: false}, {command: `dws schema --all --format json`, wantErr: false}, {command: `dws schema calendar`, wantErr: true}, {command: `dws schema "calendar event create" -f json`, wantErr: true}, diff --git a/scripts/testdata/tool_search_eval_manifest.json b/scripts/testdata/tool_search_eval_manifest.json index f1bfc8ad..b97f327c 100644 --- a/scripts/testdata/tool_search_eval_manifest.json +++ b/scripts/testdata/tool_search_eval_manifest.json @@ -10,7 +10,7 @@ "algorithm_freeze": { "state": "development_candidate_recorded_before_independent_dev_test_is_sealed", "diagnostic_proxy_control": "bm25_unfielded", - "go_runtime_default": "fielded_bm25_action_v1", + "go_runtime_default": "fielded_bm25_ensemble", "go_source_paths": [ "internal/cli/schema_search_command.go", "internal/cli/tool_search.go", @@ -26,7 +26,7 @@ ], "gate_source_sha256": "6034eb08694a5355fd6cb4a270f5b4eafd5544084d4cb812192c6b3e2b391e15", "shadow_candidates": [ - "fielded_bm25_ensemble", + "fielded_bm25_action_v1", "fielded_tfidf_cosine", "bm25plus" ], @@ -41,7 +41,7 @@ "state": "sealed_diagnostic_only", "catalog_generation": "go_declarations_to_runtime_assembly", "catalog_build_output": ".worktrees/policy-tmp/tool-search-schema-catalog", - "catalog_source_hash": "sha256:106a3d2389f672f7fd0ec302c45e2391e9edaac2d34523e15d42c395abb6c499", + "catalog_source_hash": "sha256:02c633c075f4af915ea097d383bdcbc25fe549c172e2d3336152e30cf2906e80", "catalog_surface_hash": "sha256:7f9839f1ff428b52f1d032d2fa6493815c3755650b7d9fa653839e7d589482e5", "tool_count": 1098, "workflow_path": "scripts/testdata/tool_search_workflows.json", diff --git a/skills/mono/SKILL.md b/skills/mono/SKILL.md index 9b9c4c3f..242bb8fc 100644 --- a/skills/mono/SKILL.md +++ b/skills/mono/SKILL.md @@ -226,24 +226,20 @@ dws <按 schema-inspect.v1.tool_spec 组装的参 - 2～4 步复合意图先拆成 `subqueries`，通过 `dws schema search --request-json -` 合并候选；不要用整句话只排一次 Top-K。 ```bash -# 已知产品时的人类浏览 fallback；未知命令优先用 schema search -# 第 1 层：产品概览（~4.5KB，列出全部产品 + 工具数 + 用途摘要） +# 产品概览只用于人类审计 dws schema -# 第 2 层：产品级（列出该产品下全部工具的 cli_path + description + effect/risk） -dws schema calendar --compact +# 已知产品但命令未知：直接限定产品搜索，不加载产品全量 Schema +dws schema search --query "创建日程" --product calendar --limit 5 -# 第 3 层：分组级（按命令分组列出工具摘要） -dws schema "calendar event" --compact - -# 第 4 层：Agent leaf（参数契约：type/required/description/constraints/examples） -dws schema "calendar event create" --compact +# 选中 canonical 后读取唯一 leaf（参数契约） +dws schema calendar.create_calendar_event --compact # --all：导出所有工具的完整 leaf Schema，仅用于 CI / 审计 / 参数 baseline dws schema --all --format json ``` -**`--all` 使用边界（强制）**：`--all` 会返回每个工具的完整参数、约束和安全语义，输出体积很大。仅在用户明确要求全量导出，或执行 CI、Catalog 审计、参数防丢 baseline 时使用。普通业务任务严禁使用 `--all` 做命令发现，也不要把全量结果直接注入 Agent 上下文；必须按“产品概览 → 产品/分组 → leaf”渐进查询。完整兼容性 baseline 必须使用未裁剪的 `schema --all`；`schema --all --compact` 会移除 provenance 和接口映射字段，不得作为完整 baseline。 +**`--all` 使用边界（强制）**：`--all` 会返回每个工具的完整参数、约束和安全语义，输出体积很大。仅在用户明确要求全量导出，或执行 CI、Catalog 审计、参数防丢 baseline 时使用。普通业务任务严禁使用 `--all` 做命令发现，也不要把全量结果直接注入 Agent 上下文；未知命令统一用 `schema search`，已知产品通过 `--product` 收窄，再 Inspect 唯一 leaf。完整兼容性 baseline 必须使用未裁剪的 `schema --all`；`schema --all --compact` 会移除 provenance 和接口映射字段，不得作为完整 baseline。 同一个工具省略 `--compact` 的 full leaf 与 `--all` 条目是同一份 `ToolSpec` 契约；compact leaf 只做展示投影，不重新解析语义。Alias 查询不得根据 alias 重写或补猜参数。若同一视图观察到内容差异，应作为契约漂移报告，而不是选择其中一份继续执行。 @@ -254,7 +250,7 @@ dws schema --all --format json ### Schema 字段速查 ```jsonc -// leaf 级输出（dws schema "calendar event create" --compact） +// leaf 级输出（dws schema calendar.create_calendar_event --compact） { "cli_path": "calendar event create", "canonical_path": "calendar.create_calendar_event", @@ -304,7 +300,7 @@ Schema 与 Help 冲突是**契约漂移**，不得静默猜测或把两边字段 `dev.*` 包含 helper-only 执行面，其中远端 helper 未进入 pinned metadata 时标记为 `composite`，不能伪装成 `local`。`event list` / `event schema` 读取内置目录和 payload 定义，属于 `local`；`event consume` / `event status` / `event stop` 同时编排远端个人订阅控制面与本地 bus/consume，属于 `composite`。实现来源不同，不改变统一查询边界：进入全局 `dws schema` 的命令须由 leaf `ContractFinal.Identity` 声明收集，并由同一 `ToolSpec` 投影到 leaf、产品/分组、`--all` 与 Catalog。不得把 Cobra 临时合成结果作为第二条 Schema 数据路径。 -事件需要区分两种 Schema：`dws event schema --flatten` 查询 Agent 要消费的顶层业务字段；`dws schema "event consume" --compact` 查询 CLI 命令参数。前者是真实业务命令，后者只读取最终内嵌 SchemaRegistry；不能相互替代。 +事件需要区分两种 Schema：`dws event schema --flatten` 查询 Agent 要消费的顶层业务字段；`dws schema event.consume --compact` 查询 CLI 命令参数。前者是真实业务命令，后者只读取最终内嵌 SchemaRegistry；不能相互替代。 `source` 表示最终命令 identity 的来源，不表示运行时 backing；helper/local/MCP 实现机制读取 `interface_mode`、`availability` 和 provenance，不要假定 `dev.*` 必然是 `source=mcp:`，也不要假定本地命令必然是 `source=cobra`。 diff --git a/skills/mono/references/products/event.md b/skills/mono/references/products/event.md index f9b82c95..39f64039 100644 --- a/skills/mono/references/products/event.md +++ b/skills/mono/references/products/event.md @@ -21,7 +21,7 @@ | `dws event stop --dry-run` / `--yes` | 先预览，再确认取消订阅并停止对应本地消费 | | `dws event stop --all --dry-run` / `--yes` | 先预览，再确认清理当前身份下全部个人订阅 | -注意区分两个 schema：`dws event schema ` 查事件的输出字段；`dws schema "event consume" --compact` 查 consume 命令自身的入参（统一内嵌 ToolSpec，含 parameters + 位置参数）。`source` 是 reviewed command identity 的 provenance；`event list/schema` 是 `interface_mode=local`，`event consume/status/stop` 因同时编排远端订阅控制面与本地 bus 而是 `interface_mode=composite`，不要把 identity 与实现机制混为一谈。 +注意区分两个 schema：`dws event schema ` 查事件的输出字段；`dws schema event.consume --compact` 查 consume 命令自身的入参（统一内嵌 ToolSpec，含 parameters + 位置参数）。`source` 是 reviewed command identity 的 provenance；`event list/schema` 是 `interface_mode=local`，`event consume/status/stop` 因同时编排远端订阅控制面与本地 bus 而是 `interface_mode=composite`，不要把 identity 与实现机制混为一谈。 ## Event catalog diff --git a/skills/mono/references/products/sheet.md b/skills/mono/references/products/sheet.md index 4276456d..dbf3340d 100644 --- a/skills/mono/references/products/sheet.md +++ b/skills/mono/references/products/sheet.md @@ -210,7 +210,7 @@ Flags: ## Shortcuts（无专用脚本/recipe 时优先） -以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由：存在精确覆盖该场景的专用脚本/recipe 时按其执行；否则用户意图命中时，shortcut 优先于手写原子命令。命令已选中时直接执行；只在参数或安全语义不确定时读取 Agent leaf Schema（例如 `dws schema --cli-path "sheet +" --compact --format json`），在当前 Cobra flags 不确定时读取 `dws sheet --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时，才用 `dws shortcut list --service sheet --format json` 批量发现。 +以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由：存在精确覆盖该场景的专用脚本/recipe 时按其执行；否则用户意图命中时，shortcut 优先于手写原子命令。命令已选中时直接执行；只在参数或安全语义不确定时读取 Agent leaf Schema（例如 `dws schema --cli-path "sheet +" --compact --format json`），在当前 Cobra flags 不确定时读取 `dws sheet --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时，才用 `dws schema search --query "<用户意图>" --product sheet --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service sheet` 只作人工审计/兼容 fallback。 | Shortcut | 风险 | 适用场景 | |---|---|---| diff --git a/skills/multi/dingtalk-shared/references/schema-usage.md b/skills/multi/dingtalk-shared/references/schema-usage.md index ddd25d2d..0019c565 100644 --- a/skills/multi/dingtalk-shared/references/schema-usage.md +++ b/skills/multi/dingtalk-shared/references/schema-usage.md @@ -1,6 +1,6 @@ # DWS 命令发现与 Schema Inspect -`dws` 对 Agent 是一个稳定元工具；Catalog 中的命令是它的内部命令空间，不是需要向 Host 动态注册的一组新工具。`dws schema search` 用来找到未知子命令，`dws schema ` 用来检查选中命令的结构化契约，最后仍通过同一个 `dws` 执行。 +`dws` 对 Agent 是一个稳定元工具；Catalog 中的命令是它的内部命令空间，不是需要向 Host 动态注册的一组新工具。`dws schema search` 用来找到未知子命令，`dws schema --compact` 用来检查选中命令的结构化契约，最后仍通过同一个 `dws` 执行。 本节同时适用于基础/原子命令与公开内建 `+` shortcut。用户自定义或未公开 shortcut 不进入发布 Schema；是否可执行仍以当前 Cobra help 为准。 @@ -27,9 +27,7 @@ dws schema search --query "查询群消息已读状态" --limit 5 3. 选中候选后，使用 response `catalog.source_hash` 和 `catalog.surface_hash` 精确 Inspect `canonical_path`： ```bash -dws schema chat.query_msg_read_status --compact --format json \ - --expected-source-hash "" \ - --expected-surface-hash "" +dws schema chat.query_msg_read_status --compact --expected-source-hash "" --expected-surface-hash "" ``` 4. 只使用 `schema-inspect.v1.tool_spec` 组装参数，并按候选的 `primary_cli_path` 调用同一个 `dws`。`reason=catalog_changed` 时丢弃旧候选，重新 Search，不混用两代 Schema。 @@ -55,21 +53,17 @@ printf '%s' '{"version":"tool-search.v1","query":"给群里发文件并确认送 对每个选中的 canonical 分别做双 hash Inspect；按任务依赖顺序执行，后续 ID 只能来自前一步真实返回。 -## 已知产品时的四层浏览 fallback +## 已知产品时的有界导航 ```bash -# 只用于已知产品、但需要人类浏览组/命令层级的 fallback -# 第 1 层：产品概览（列出产品 + 工具数 + 用途摘要） +# 产品概览只用于人类审计 dws schema -# 第 2 层：产品级（该产品工具的 cli_path + description + effect/risk） -dws schema calendar --compact +# 已知产品但命令未知：直接限定产品搜索，不加载产品全量 Schema +dws schema search --query "创建日程" --product calendar --limit 5 -# 第 3 层：分组级 -dws schema "calendar event" --compact - -# 第 4 层：Agent leaf（参数契约） -dws schema "calendar event create" --compact +# 选中 canonical 后读取唯一 leaf（参数契约） +dws schema calendar.create_calendar_event --compact # --all：全量 leaf，仅 CI / 审计 / 参数 baseline dws schema --all --format json @@ -77,7 +71,7 @@ dws schema --all --format json ### `--all` 边界（强制） -`--all` 输出体积很大。仅在用户明确要求全量导出，或 CI / Catalog 审计 / 参数防丢 baseline 时使用。普通业务任务严禁用 `--all` 做命令发现，也不要把全量结果注入 Agent 上下文。未知命令优先用 `schema search`；四层浏览只作已知产品时的人类导航 fallback。 +`--all` 输出体积很大。仅在用户明确要求全量导出，或 CI / Catalog 审计 / 参数防丢 baseline 时使用。普通业务任务严禁用 `--all` 做命令发现，也不要把全量结果注入 Agent 上下文。未知命令统一用 `schema search`，已知产品通过 `--product` 收窄。 完整兼容性 baseline 必须使用未裁剪的 `schema --all`；`schema --all --compact` 会移除 provenance 和接口映射字段，不得作为完整 baseline。 @@ -125,6 +119,6 @@ Schema 与 Help 冲突是**契约漂移**：执行参数只用 Cobra 接受的 f ### 两类易混 Schema - `dws event schema --flatten`：事件业务字段 -- `dws schema "event consume" --compact`：CLI 命令参数 +- `dws schema event.consume --compact`：CLI 命令参数 二者不可互相替代。