Compare commits

..
Author SHA1 Message Date
玉澜 c7521c1ca3 fix(cli): route Chinese product-name queries to the named product
Natural-language queries that name a product in Chinese (e.g. 「帮我搜一下
「评测项目管理」钉钉AI表格」) lost the named product's candidates to a
sibling product whose summaries repeat a shared bigram (92 sheet tools
repeat 「表格」), crowding aitable out of the Top-K cut entirely.

Two-part fix in the lexical layer, deterministic and no new dependencies:

- Fold the product display name into each tool's identity field at index
  build (runtime assembly carries the display name in Description).
- Query-side named-product guarantee: when the query contains a product's
  display-name core (NFKC-folded, generic suffixes like 「操作」「管理」
  trimmed, longest core wins on overlap), fetch that product's best
  candidate over its own eligible subset and merge it just above the
  global top hit. A post-hoc multiplier alone cannot rescue candidates
  truncated inside Retrieve's CandidateLimit cut.

Eval evidence (core120 first-turn coverage): 101→103/120; all four
measured 「钉钉AI表格」 misses fixed, zero regressions across the battery.
2026-08-19 14:31:48 +08:00
玉澜 aa0bc0c2d4 Merge remote-tracking branch 'upstream/main' into pr984 2026-08-17 21:55:45 +08:00
dxb 9e3a5d6fbd Merge pull request #1029 from DingTalk-Real-AI/fix/chat-sender-identity-contract
fix(chat): preserve unverified sender identity semantics
2026-08-17 19:10:30 +08:00
玉澜 f2a3dd2344 fix(cli): hint tokenized schema path failures at the path-form boundary
- the unknown runtime schema path error now states that canonical paths
  contain dots and must be a single argument while space-separated tokens
  resolve as a CLI path; help text and docs only reach agents that consult
  them, the error is the one surface guaranteed to be seen at failure time
- the app trust diagnostic comment is shortened to the durable constraint
  (do not pin absolute Catalog-size counts) instead of narrating the change
- NewDeliveryToolSearchEngine's doc comment notes the memo serves long-lived
  hosts; the shipped schema search command rebuilds per invocation through
  NewDeliveryToolSearchEngineWithConfig
2026-08-17 19:10:18 +08:00
栩朝 fc05976d33 fix(chat): align chat message selection intent 2026-08-17 17:30:12 +08:00
栩朝 021da02474 fix(chat): preserve unverified sender identity semantics 2026-08-17 17:30:12 +08:00
github-actions[bot] a5b9e5a13f Merge pull request #928 from Anonymity-0/feat/bot-group-reply
feat(chat): support bot group message replies
2026-08-17 17:25:23 +08:00
玉澜 b8c9ff2619 test(app): drop stale Catalog-size sentinels from the trust diagnostic
The diagnostic test pinned absolute counts from one Catalog generation:
1123 intent cases, Recall@5 == 990/1123, 1098 tools and the 402/721
language slices. Merging main grew the Catalog to 1121 tools, so the
population became 1147/407/740 and the test failed as a size drift
rather than as a trust regression.

- the absolute counts are removed; the observed intent, language-slice
  and context values are logged for CI forensics and the headline numbers
  stay in the generated comparison JSON
- identity coverage is still enforced exhaustively, but the expected
  count is derived from report.Context.ToolCount instead of a literal
- the trust contract that does not move with Catalog size stays a hard
  gate: every identity pass rate == 1, alias cases non-zero, all four
  integrity counters == 0, zero-result rate == 0, non-empty language
  slices and ReductionVsFullSchema >= 0.99
2026-08-17 17:21:28 +08:00
玉澜 225a073d89 fix(cli): close tool search self-review findings
- SearchSubqueries returned a nil candidate slice on the
  decomposed_exact_filtered refusal, so that one strategy encoded
  "candidates":null while every other path encoded []; the refusal now
  carries an empty slice and a marshal assertion pins the wire shape
- the memoized delivery engine was never invalidated: it indexes one
  Catalog generation and caches its source/surface hashes, but
  RegisterSchemaSourceRoot -> resetDeliverySchemaCatalogState left the
  memo installed, so a re-registered root could be served stale hashes.
  Catalog delivery reset now clears it, with a ForTest wrapper and a
  regression test for the rebuild
- dangerous filter Unicode is rejected before the effect vocabulary and
  Catalog product checks, so a control/Bidi character returns
  dangerous_filter_unicode instead of being echoed back through
  invalid_effect; the filter list is an ordered slice, not a map, so the
  reported field name is deterministic
- dws schema help and RFC 5.8 record the path-form boundary: multi-token
  args resolve as a CLI path, a canonical path must stay one argument
- --explain and ToolSearchScoreBreakdown declare that the reported score
  is the raw value while ordering uses the quantized key
- --limit/--candidate-limit help state that 0 means "use the default",
  which was already the behavior but read as an asymmetry against the
  rejected negative values
- bootstrap seeds become named constants and are published in both
  evaluation reports (bootstrap_samples/bootstrap_seed/
  recovery_bootstrap_seed) so an interval is reproducible without
  reading the source
- dingtalk-shared documents the cross-product summary --dry-run flag and
  that its command list is a plan, not business data
2026-08-17 17:21:10 +08:00
玉澜 45a52dee85 docs: consolidate the six tool search documents into one file
The RFC, source-implementation research, GitHub architecture research,
ranking evaluation report, 2026-08-14 optimization plan and ten-round
review record were six separate files that cross-linked each other for
the same decision chain. They are merged into
docs/tool-search-progressive-discovery.md as Part 1..6, ordered
decision -> research -> evidence -> plan -> review record.

- cross-file links become in-document "本文档 Part N" pointers; the five
  remaining relative links (source and evaluator paths) still resolve
- section content is unchanged apart from one addition: 5.8 Inspect now
  records the path-form boundary (multi-token args resolve as a CLI path;
  a canonical path must be passed as a single argument), matching the
  dws schema help text
2026-08-17 17:20:41 +08:00
Anonymity-0 9f8c525008 Merge branch 'main' into feat/bot-group-reply 2026-08-17 16:59:05 +08:00
github-actions[bot] 207d4dd7e5 Merge pull request #980 from cywan1998/feat/calendar-event-share-info
feat(calendar): add event share-info command
2026-08-17 08:57:50 +00:00
fengbai 9a6b7d4d41 Merge branch 'main' into feat/calendar-event-share-info 2026-08-17 16:41:08 +08:00
前津 5947016cc1 feat(chat): support bot group message replies 2026-08-17 16:09:08 +08:00
github-actions[bot] 386426bb92 Merge pull request #1012 from DingTalk-Real-AI/dws_0814_1723
fix(skill): update doc and drive descriptions for clearer routing
2026-08-17 07:45:22 +00:00
john 3cea671a54 Merge branch 'main' into dws_0814_1723 2026-08-17 15:27:17 +08:00
john 235e881ebe Merge branch 'main' into test/tool-search-ranking-eval 2026-08-17 15:21:07 +08:00
github-actions[bot] f06ea4d9e2 Merge pull request #960 from DingTalk-Real-AI/codex/doc-reread-audit
fix(doc): harden mutation readback verification
2026-08-17 07:06:53 +00:00
ruigong 93dbd768f7 fix(skill): add explicit recent-edited route to drive SOP-1 2026-08-17 14:18:03 +08:00
RuiGong01 03838a3430 Merge branch 'main' into dws_0814_1723 2026-08-17 11:39:56 +08:00
RuiGong01 0d34150333 Merge branch 'main' into dws_0814_1723 2026-08-17 11:16:56 +08:00
RuiGong01 9e8b58cbb6 Merge branch 'main' into dws_0814_1723 2026-08-17 10:47:33 +08:00
RuiGong01 fe724e96e8 Merge branch 'main' into dws_0814_1723 2026-08-17 10:39:02 +08:00
RuiGong01 f72979f4a9 Merge branch 'main' into dws_0814_1723 2026-08-17 09:41:16 +08:00
玉澜 2855f1a84b fix(skills): multi-turn append semantics for explicit write requests
Eval forensics: in clean isolated doc-domain runs the toolsearch arm lost
7/20 paired cases to main on codex+gpt-5.6-sol; 5 of 6 flips showed the
same signature — the agent refused an explicit per-turn 'append/insert'
request because similar content already existed (front-loaded or matched),
and the judge fails the refusal. The clause makes per-turn explicit write
requests execute (existence may be noted but never replaces execution),
forbids front-loading future turns, and keeps read-back for verification
only. Verified: doc domain 30% -> 55% (main 60%), dup-refusal signature
5/6 -> 0.
2026-08-15 18:25:07 +08:00
玉澜 6406c93eaa feat(cli): convergence signals for capability-absent searches
Eval-driven optimizations from the 2026-08-14/15 agent evaluations:

- abstained lexical responses now carry a bounded hint naming the one
  permitted verification step (dws schema <product> --compact) and
  licensing the conclusion 'unsupported', so agents stop enumerating
  every product to prove a capability is missing
- responses whose candidates only matched weak fields (parameters/
  description, no identity/summary hit) set weak_match=true with a
  matching hint; the flag is field-class based, not a score threshold,
  preserving the RFC ban on absolute score cutoffs
- exact_filtered keeps its typed refusal without inheriting either hint;
  budget-truncation to zero candidates downgrades to the abstain hint
- dingtalk-shared gains the matching convergence rule: one search, at
  most one schema <product> verification, then conclude honestly

Evidence: the nonexistent-capability intent scenario took 15 turns/177s
to conclude 'no poll capability' with identical correctness to a 4-turn
run; these signals bound that path. Note: on the real catalog the poll
phrasing still matches summary fields of generic chat tools, so the
behavioral fix for that specific case rides on the shared-skill rule;
the wire flags cover genuinely unmatched and weak-field-only queries.
2026-08-15 05:02:28 +08:00
RuiGong01 0b012788c7 Merge branch 'main' into dws_0814_1723 2026-08-14 20:15:22 +08:00
玉澜 747a17f305 fix(cli): close tool search review findings from contract probe
- effect filter: case-insensitive against the reviewed vocabulary with a
  typed invalid_effect refusal; unknown product ids get a typed
  unknown_product refusal instead of a silent abstain (probe-confirmed:
  --effect Read used to return 0 candidates silently)
- DWS_TOOL_SEARCH_* unparsable values now warn on stderr while keeping
  the resolved default, so ranking experiments cannot silently measure
  defaults (probe-confirmed: K1='0,9' was ignored with empty stderr)
- --explain help text now declares it a dev-only diagnostic that exposes
  raw ordering scores; the default wire stays score-free
- lexical ordering quantizes the sort key at 1e-9 so architecture
  dependent FMA fusion (arm64 vs amd64) cannot flip near-tied candidates
  past the canonical tie-break; reported scores are unchanged

Contract probe re-run against this build: 11/11 checks OK (was 3 FINDINGs)
2026-08-14 19:58:09 +08:00
玉澜 d578458315 fix(cli): fail closed on unknown tool-search filters and warn on unparsable env
- effect filters are case-normalized against the reviewed vocabulary and
  reject anything else with a typed invalid_effect error instead of a
  silent abstain (probe-confirmed: --effect Read now equals --effect read)
- unknown product ids reject with unknown_product instead of returning
  abstained=true, so caller typos cannot masquerade as empty results
- unparsable DWS_TOOL_SEARCH_* float env values now warn on stderr;
  previously a typo like K1=0,9 silently measured the default config
- --explain help text declares the raw-score disclosure as a dev-only
  diagnostic exception to the score-free public wire
2026-08-14 19:53:34 +08:00
chichuan 7b7bd556e9 Merge branch 'main' into feat/calendar-event-share-info 2026-08-14 17:43:49 +08:00
ruigong d534ee242c fix(skill): scope doc/drive descriptions to entity-content vs file management 2026-08-14 17:33:54 +08:00
fengbai 90473284b8 fix(calendar): remove shell comment from share-info example
- Move the eventId lookup hint into Long description
- Keep example commands free of shell comments to pass example policy gate
2026-08-14 15:51:20 +08:00
玉澜 575b8f32c4 feat(skills): add compact cross-product read summary 2026-08-14 15:37:13 +08:00
fengbai 07aa2c883a fix(calendar): address CR comments for share-info
- Fix Example indentation (tab -> 2 spaces)
- Remove unsubstantiated default en-US from --language help/docs
- Add test asserting calendarId/language are omitted when only --id is passed
2026-08-14 15:10:20 +08:00
玉澜 70e46cbd29 fix(cli): harden tool search evaluation and config 2026-08-14 10:04:53 +08:00
玉澜 e9efd3e909 docs(rfc): sync progressive discovery numbers to post-hardening baseline 2026-08-13 16:58:31 +08:00
玉澜 9bc803fde0 fix(cli): enforce shortcut avoid_when boilerplate filter and sync eval docs
Self-review follow-ups on the ranking hardening series:

- the avoid_when boilerplate filter compared lowercased phrases against
  a mixed-case constant, so the "需要该 Shortcut..." noise filter never
  fired; the constant is now lowercase (no ranking change: the boilerplate
  was never contained by any query, sentinel 990/1123 unchanged)
- add direct unit coverage for the avoid_when penalty (echoed-phrase
  match, unrelated-query pass-through, boilerplate filter, partial
  overlap, explain annotation)
- update tool-search-ranking-evaluation.md to the post-hardening
  numbers (ensemble R@5 88.16%, Forbidden 0%/0%, action_v1 shadow
  84.24% with the trade-off recorded)
- drop the dead algorithm_freeze manifest block; nothing consumes it
  after the hash decoupling
2026-08-13 16:31:52 +08:00
玉澜 2dc3147cda refactor(scripts): decouple tool search eval manifest from source hashes and seal gate
- drop the algorithm/gate source SHA256 freeze: every ranking change
  required a manual manifest hash sync, coupling code evolution to the
  manifest
- drop the --require-sealed release gate (sealed qrels hash, dual
  signatures, coverage minima, independent-result thresholds) and its
  tool-search-independent-data-gate Makefile target
- behavior regression protection stays with the Go evaluation sentinels
  that run the real assembled catalog
2026-08-13 16:09:19 +08:00
玉澜 e91b30c769 chore(testdata): refresh frozen tool search algorithm source hash
The algorithm freeze gate intentionally rejects silent ranking source
changes; this records the new source set hash after the ranking hardening
commit.
2026-08-13 15:51:27 +08:00
玉澜 0df293186a feat(cli): harden tool search ranking, guardrails and observability
- default engine: fielded BM25 ensemble (action_v1 remains a shadow candidate)
- add --explain score_breakdown: per-field contributions, structured
  multiplier, query classification and avoid_when penalty
- add DWS_TOOL_SEARCH_* env overrides (algorithm, k1/b, field weights,
  explain) for release-independent tuning and rollback
- tokenizer: CJK unigram+bigram so single-character intents match; align
  isToolSearchCJK with unicode.Han used by the evaluation slices
- structured routing: extend vocabularies with English synonyms,
  word-boundary matching for ASCII values, restrict the rerank gate to
  technical identifiers only
- avoid_when demotion layer wraps every lexical algorithm (Forbidden@5
  proxy 37.55% -> 0); intent-overlap matching rejected after control
  showed it demoted gold answers (Recall@5 0.8664 -> 0.7845)
- remove the OA task_id +1.00 fixture-specific multiplier
- build the default engine once per process for long-lived hosts
- evaluation baseline: intent-proxy R@5 990/1123 (88.16%); update
  integration sentinels accordingly
2026-08-13 15:20:09 +08:00
玉澜 003bbe87ba Merge remote-tracking branch 'upstream/main' into test/tool-search-ranking-eval 2026-08-13 14:04:53 +08:00
玉澜 fc47e05cd7 docs: clarify local tool search boundaries 2026-08-13 13:21:28 +08:00
玉澜 86ebdd8f25 refactor(cli): remove tool search provider fallback 2026-08-13 11:51:33 +08:00
玉澜 20b8120f59 Merge remote-tracking branch 'upstream/main' into test/tool-search-ranking-eval 2026-08-13 11:10:45 +08:00
玉澜 a08a1f54fd docs(skill): integrate DWS command discovery 2026-08-13 11:09:31 +08:00
玉澜 93da35413b feat(cli): add progressive tool search 2026-08-13 10:15:10 +08:00
fengbai 8aee08268d test(calendar): add event share-info dry-run and required-flag tests 2026-08-12 21:17:32 +08:00
fengbai 6a4744073c feat(calendar): add event share-info command 2026-08-12 21:07:59 +08:00
79 changed files with 11688 additions and 136 deletions
+5
View File
@@ -0,0 +1,5 @@
---
category: Added
---
- **Robot group reference replies** (#928) — `chat message send-by-bot` supports paired `--reply` and `--ref-sender` flags for Markdown replies that quote an existing group message.
+5
View File
@@ -0,0 +1,5 @@
---
category: Added
---
- **Calendar event share-info** (#980) — adds `dws calendar event share-info` to fetch a calendar event's share info (title, organizer, location, join info) for sharing with others; supports `--calendar-id` and `--language`.
@@ -0,0 +1,5 @@
---
category: Fixed
---
- **Chat sender identity guards** — preserves unverified mixed sender inputs after exact message `senderId` matches and aligns `--sender-query` Skill guidance with fail-closed Runtime behavior.
+5
View File
@@ -0,0 +1,5 @@
---
category: Changed
---
- **Doc/drive description scope** — restates the `dingtalk-doc` description as document-entity-and-content operations with an explicit exclusion list, and narrows `dingtalk-drive` to file-level management of DingTalk documents, so first-round Agent selection separates content work from file management without changing CLI behavior.
+23 -2
View File
@@ -10,7 +10,7 @@ SCHEMA_META_INDEX_OUTPUT ?= artifacts/schema_meta_index.gob
POLICY_ENV = DWS_POLICY_TMPDIR="$(DWS_POLICY_TMPDIR)" GOTMPDIR="$(POLICY_GOTMPDIR)"
GO_SOURCE_LIST = git ls-files -z --cached --others --exclude-standard -- '*.go'
.PHONY: all help build rebuild test test-plan test-auth-legacy-compat lint format-check fmt policy edition-test interface-integrity authoritative-interface-integrity coverage-gate coverage-gate-platform update-interface-baseline reset-interface-baseline schema-compatibility skill-command-integrity skill-context-budget multi-im-skill-chain-integrity cli-smoke mock-mcp-smoke test-schema-agent-examples generate-schema fetch-mcp-metadata generate-schema-catalog package release release-pre release-stable changelog-pre changelog-stable publish-homebrew-formula setup-hooks
.PHONY: all help build rebuild test test-plan test-auth-legacy-compat lint format-check fmt policy edition-test interface-integrity authoritative-interface-integrity coverage-gate coverage-gate-platform update-interface-baseline reset-interface-baseline schema-compatibility skill-command-integrity skill-context-budget multi-im-skill-chain-integrity cli-smoke mock-mcp-smoke test-schema-agent-examples tool-search-evaluation-harness generate-tool-search-comparison generate-schema fetch-mcp-metadata generate-schema-catalog package release release-pre release-stable changelog-pre changelog-stable publish-homebrew-formula setup-hooks
all: setup-hooks fmt lint build test rebuild
@@ -37,6 +37,8 @@ help:
@printf " make cli-smoke - Verify help for every public top-level command\n"
@printf " make mock-mcp-smoke - Verify HTTP and stdio MCP request/response transport\n"
@printf " make test-schema-agent-examples - Contract-check all Agent examples and dry-run the eligible subset\n"
@printf " make tool-search-evaluation-harness - Verify Go retrieval, trust metrics, frozen fixtures, and build-time report generation\n"
@printf " make generate-tool-search-comparison - Generate the declaration-assembled comparison under .worktrees/policy-tmp\n"
@printf " make generate-schema - Refresh param_aliases + verify Schema assembly determinism\n"
@printf " make generate-schema-catalog - Optional assembled Catalog dump under artifacts/ (not a delivery step)\n"
@printf " make package - Build all release artifacts locally\n"
@@ -84,7 +86,7 @@ fmt:
$(GO_SOURCE_LIST) > "$$go_files"; \
xargs -0 sh -c 'if [ "$$#" -gt 0 ]; then exec gofmt -w -- "$$@"; fi' sh < "$$go_files"
policy: test-auth-legacy-compat
policy: test-auth-legacy-compat tool-search-evaluation-harness skill-mono-multi-content
@mkdir -p "$(POLICY_GOTMPDIR)"
@$(POLICY_ENV) ./scripts/policy/check-open-source-assets.sh
@$(POLICY_ENV) ./scripts/policy/check-skill-context-budget.sh
@@ -161,6 +163,25 @@ mock-mcp-smoke:
test-schema-agent-examples:
DWS_AGENT_EXAMPLES_DRY_RUN=1 $(GO) test -v -count=1 ./internal/app -run '^TestAgentExamplesDryRun$$'
tool-search-evaluation-harness:
python3 -m unittest \
scripts/dev/eval_tool_search_ranking_test.py \
scripts/dev/verify_tool_search_eval_manifest_test.py
@mkdir -p "$(DWS_POLICY_TMPDIR)"
$(GO) run ./internal/generator/cmd_schema_catalog \
-root . \
-output "$(DWS_POLICY_TMPDIR)/tool-search-schema-catalog" \
-meta-index "$(DWS_POLICY_TMPDIR)/tool-search-schema-meta-index.gob"
python3 scripts/dev/verify_tool_search_eval_manifest.py \
--catalog-dir "$(DWS_POLICY_TMPDIR)/tool-search-schema-catalog"
$(GO) test ./internal/cli -run '^TestToolSearch|^TestSchemaSearch|^TestDefaultToolSearch|^TestAggregateToolSearch|^TestScoreToolSearch' -count=1
$(GO) test ./internal/app -run '^TestToolSearchDelivery|^TestSchemaSearchInspectCatalogVersionContract$$' -count=1
$(GO) run ./internal/generator/cmd_tool_search_comparison -output "$(DWS_POLICY_TMPDIR)/tool-search-comparison.json"
generate-tool-search-comparison:
@mkdir -p "$(DWS_POLICY_TMPDIR)"
$(GO) run ./internal/generator/cmd_tool_search_comparison -output "$(DWS_POLICY_TMPDIR)/tool-search-comparison.json"
# generate-schema refreshes param_aliases_generated.go and verifies that
# ResolveSchemaBuild assembly is deterministic. Catalog is runtime-assembled
# (声明即 Catalog); cmd_schema_catalog is not a committed delivery step.
File diff suppressed because it is too large Load Diff
+47
View File
@@ -462,6 +462,53 @@ func TestCalendarEventListDryRunPreviewsOnly(t *testing.T) {
}
}
func TestCalendarEventShareInfoDryRunPreviewsOnly(t *testing.T) {
got, err := executeRootCaptureStdout(t, []string{
"--dry-run", "calendar", "event", "share-info",
"--id", "EVT_001",
"--language", "zh-CN",
"--calendar-id", "primary",
})
if err != nil {
t.Fatalf("calendar event share-info --dry-run error = %v\n%s", err, got)
}
for _, want := range []string{"get_event_share_info", "eventId", "EVT_001", "zh-CN", "primary"} {
if !strings.Contains(got, want) {
t.Fatalf("calendar event share-info dry-run output missing %q:\n%s", want, got)
}
}
}
func TestCalendarEventShareInfoRequiresEventID(t *testing.T) {
got, err := executeRootCaptureStdout(t, []string{
"--dry-run", "calendar", "event", "share-info",
})
if err == nil {
t.Fatalf("calendar event share-info without --id: expected error, got nil\n%s", got)
}
if strings.Contains(got, "\"executed\": true") {
t.Fatalf("share-info without --id must not execute:\n%s", got)
}
}
func TestCalendarEventShareInfoOmitsOptionalArgs(t *testing.T) {
got, err := executeRootCaptureStdout(t, []string{
"--dry-run", "calendar", "event", "share-info",
"--id", "EVT_001",
})
if err != nil {
t.Fatalf("calendar event share-info --dry-run with only --id error = %v\n%s", err, got)
}
if !strings.Contains(got, "\"eventId\"") {
t.Fatalf("calendar event share-info dry-run output missing eventId:\n%s", got)
}
for _, unwanted := range []string{"\"calendarId\"", "\"language\""} {
if strings.Contains(got, unwanted) {
t.Fatalf("calendar event share-info dry-run with only --id should not contain %q:\n%s", unwanted, got)
}
}
}
func TestRootKeepsSVIPChatCompatibilityFlags(t *testing.T) {
root := NewRootCommand()
+14
View File
@@ -400,6 +400,7 @@ func schemaContractPayloadForBoundCanonicals(t *testing.T, root *cobra.Command,
func TestChatSchemaSeparatesSendAndReply(t *testing.T) {
snapshot := schemaContractPayloadForBoundCanonicals(t, NewRootCommand(),
"chat.send_personal_message",
"chat.send_robot_message",
"chat.reply_personal_message",
)
@@ -418,6 +419,19 @@ func TestChatSchemaSeparatesSendAndReply(t *testing.T) {
if _, exists := snapshot.Tools["chat.upload_conversation_file"]; exists {
t.Fatal("downlined chat file upload must not be advertised in Schema")
}
botReply := snapshot.Tools["chat.send_robot_message"]
botParams := schemaContractMap(botReply["parameters"])
if got := schemaContractString(botParams["reply"]["property"]); got != "referenceOpenMessageId" {
t.Fatalf("bot --reply property = %q", got)
}
if got := schemaContractString(botParams["ref-sender"]["property"]); got != "srcMsgSendOpenDingTalkId" {
t.Fatalf("bot --ref-sender property = %q", got)
}
if got, ok := botParams["title"]["required"].(bool); !ok || got {
t.Fatalf("bot --title required = %#v, want false for conditional Markdown input", botParams["title"]["required"])
}
assertSchemaContractConstraintGroup(t, botReply, "require_together", []string{"reply", "ref-sender"})
}
func TestCalendarAttendeeDeleteSchemaMatchesRuntimeGate(t *testing.T) {
@@ -0,0 +1,249 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
package app
import (
"bytes"
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/DingTalk-Real-AI/dingtalk-workspace-cli/internal/cli"
apperrors "github.com/DingTalk-Real-AI/dingtalk-workspace-cli/internal/errors"
)
func TestToolSearchDeliveryActionSiblingRouting(t *testing.T) {
registerSchemaRuntimeDelivery()
// This test pins the action_v1 structured sibling routing itself, decoupled
// from the production default (now the fielded BM25 ensemble, which has no
// structured multiplier and cannot right-size these sibling queries).
config := cli.DefaultToolSearchConfig()
config.LexicalAlgorithm = cli.ToolSearchLexicalBM25Action
engine, err := cli.NewDeliveryToolSearchEngineWithConfig(config)
if err != nil {
t.Fatalf("NewDeliveryToolSearchEngineWithConfig() error = %v", err)
}
tests := []struct {
query string
gold string
}{
{query: "创建一场新的会议日程", gold: "calendar.create_calendar_event"},
{query: "给钉盘文件添加同事阅读权限", gold: "drive.add_permission"},
// The new main Catalog declares shortcut_fetch as a reviewed equivalent
// read surface, so this query has graded-equivalent Top-1 answers.
{query: "读取钉钉在线文字文档正文", gold: "doc.shortcut_fetch|doc.get_document_content"},
{query: "查询当前用户待审批任务 ID", gold: "oa.list_pending_tasks"},
{query: "同意指定审批任务", gold: "oa.approve_processInstance"},
{query: "按群名关键词搜索群聊", gold: "chat.search_groups"},
}
for _, test := range tests {
t.Run(test.gold, func(t *testing.T) {
response, searchErr := engine.Search(context.Background(), cli.ToolSearchRequest{Query: test.query, Limit: 5})
if searchErr != nil {
t.Fatalf("Search() error = %v", searchErr)
}
if len(response.Candidates) == 0 || !containsCanonicalAlternative(test.gold, response.Candidates[0].CanonicalPath) {
t.Fatalf("Search(%q) candidates = %#v, want %s first", test.query, response.Candidates, test.gold)
}
})
}
}
func containsCanonicalAlternative(alternatives, candidate string) bool {
for _, alternative := range strings.Split(alternatives, "|") {
if candidate == alternative {
return true
}
}
return false
}
func TestToolSearchDeliveryDiagnosticTrustAndChineseSlices(t *testing.T) {
registerSchemaRuntimeDelivery()
workflows := []cli.ToolSearchWorkflowEvaluationCase{{
ID: "chat-send-file-and-check-read",
Query: "给群里发文件,并确认群成员是否已读",
Subqueries: []string{"以当前用户身份给群聊发送本地文件", "查询指定消息的已读状态和人员"},
Required: []string{"chat.send_personal_message", "chat.query_msg_read_status"},
}}
report, err := cli.BuildDeliveryToolSearchDiagnosticComparison(context.Background(), workflows)
if err != nil {
t.Fatalf("BuildDeliveryToolSearchDiagnosticComparison() error = %v", err)
}
// Do not pin absolute Catalog-size counts here: they go stale when main
// adds tools. Enforce the size-independent trust contract below; observed
// counts are logged for CI forensics and live in the generated comparison
// JSON (make generate-tool-search-comparison).
t.Logf("intent proxy = %+v excluded_over_budget=%d", report.IntentProxy, report.IntentExcludedOverBudget)
t.Logf("language slices = %+v", report.IntentLanguageSlices)
t.Logf("context comparison = %+v", report.Context)
if report.IntentProxy.Cases == 0 {
t.Fatal("intent proxy population is empty; the diagnostic set did not build")
}
if report.IntentProxy.ZeroResultRate != 0 {
t.Fatalf("intent proxy zero-result rate = %v, want every reviewed intent to retrieve something", report.IntentProxy.ZeroResultRate)
}
for _, slice := range []string{"chinese_only", "mixed_chinese_ascii"} {
if report.IntentLanguageSlices[slice].Cases == 0 {
t.Fatalf("language slice %q is empty: %+v", slice, report.IntentLanguageSlices)
}
}
identity := report.Trust.Identity
// Identity coverage must stay exhaustive over the whole Catalog. Deriving the
// expected count from the same report keeps that contract enforced without
// hardcoding a tool count.
if identity.CanonicalCases != report.Context.ToolCount || identity.NFKCCases != report.Context.ToolCount {
t.Fatalf("identity coverage = %+v, want one case per %d Catalog tools", identity, report.Context.ToolCount)
}
if identity.CanonicalPassRate != 1 || identity.PrimaryCLIPassRate != 1 ||
identity.AliasCases == 0 || identity.AliasPassRate != 1 || identity.NFKCPassRate != 1 ||
identity.ExactFilteredPassRate != 1 {
t.Fatalf("identity trust = %+v", identity)
}
integrity := report.Trust.Integrity
if integrity.CatalogBindingFailures != 0 || integrity.UnknownCandidateCount != 0 || integrity.IneligibleCandidateCount != 0 || integrity.ResponseBudgetViolations != 0 {
t.Fatalf("integrity violations = %+v", integrity)
}
if report.Context.ToolCount == 0 || report.Context.ReductionVsFullSchema < 0.99 {
t.Fatalf("context comparison = %+v", report.Context)
}
}
func TestSchemaSearchInspectCatalogVersionContract(t *testing.T) {
registerSchemaRuntimeDelivery()
search := cli.NewSchemaCommand()
var searchOutput bytes.Buffer
search.SetOut(&searchOutput)
search.SetArgs([]string{"search", "--query", "chat.query_msg_read_status"})
if err := search.Execute(); err != nil {
t.Fatalf("schema search execute: %v", err)
}
var response cli.ToolSearchResponse
if err := json.Unmarshal(searchOutput.Bytes(), &response); err != nil {
t.Fatalf("decode search response: %v", err)
}
if response.Strategy != "exact_guard" || len(response.Candidates) != 1 || response.Candidates[0].CanonicalPath != "chat.query_msg_read_status" {
t.Fatalf("search response = %#v", response)
}
inspect := cli.NewSchemaCommand()
var inspectOutput bytes.Buffer
inspect.SetOut(&inspectOutput)
inspect.SetArgs([]string{
"chat.query_msg_read_status", "--compact",
"--expected-source-hash", response.Catalog.SourceHash,
"--expected-surface-hash", response.Catalog.SurfaceHash,
})
if err := inspect.Execute(); err != nil {
t.Fatalf("schema inspect execute: %v", err)
}
var payload cli.SchemaInspectV1Response
if err := json.Unmarshal(inspectOutput.Bytes(), &payload); err != nil {
t.Fatalf("decode inspect response: %v", err)
}
if payload.Version != "schema-inspect.v1" || payload.Catalog != response.Catalog || payload.ToolSpec["canonical_path"] != "chat.query_msg_read_status" {
t.Fatalf("inspect envelope = %#v", payload)
}
stale := cli.NewSchemaCommand()
stale.SilenceErrors = true
stale.SilenceUsage = true
stale.SetArgs([]string{
"chat.query_msg_read_status", "--compact",
"--expected-source-hash", "sha256:stale",
"--expected-surface-hash", response.Catalog.SurfaceHash,
})
err := stale.Execute()
var typed *apperrors.Error
if !errors.As(err, &typed) || typed.Reason != "catalog_changed" || typed.Category != apperrors.CategoryDiscovery {
t.Fatalf("stale inspect error = %#v", err)
}
incomplete := cli.NewSchemaCommand()
incomplete.SilenceErrors = true
incomplete.SilenceUsage = true
incomplete.SetArgs([]string{
"chat.query_msg_read_status",
"--expected-source-hash", response.Catalog.SourceHash,
})
err = incomplete.Execute()
if !errors.As(err, &typed) || typed.Category != apperrors.CategoryValidation {
t.Fatalf("incomplete inspect version error = %#v", err)
}
}
func TestSchemaSearchRejectsIgnoredGlobalOutputFlags(t *testing.T) {
for _, args := range [][]string{
{"schema", "search", "--query", "查询群消息已读状态", "--fields", "canonical_path"},
{"schema", "search", "--query", "查询群消息已读状态", "--jq", ".candidates"},
{"schema", "search", "--query", "查询群消息已读状态", "--format", "pretty"},
} {
root := NewRootCommand()
root.SilenceErrors = true
root.SilenceUsage = true
root.SetArgs(args)
err := root.Execute()
var typed *apperrors.Error
if !errors.As(err, &typed) || typed.Category != apperrors.CategoryValidation {
t.Fatalf("args %v error = %#v, want validation", args, err)
}
}
}
// BenchmarkToolSearchDeliveryChineseQuery measures the shipped search path
// against the complete declaration-assembled Catalog. The smaller benchmark
// in internal/cli intentionally uses a four-tool fixture and must not be used
// as the release-scale latency or allocation number.
func BenchmarkToolSearchDeliveryChineseQuery(b *testing.B) {
registerSchemaRuntimeDelivery()
engine, err := cli.NewDeliveryToolSearchEngineWithConfig(cli.DefaultToolSearchConfig())
if err != nil {
b.Fatalf("NewDeliveryToolSearchEngineWithConfig() error = %v", err)
}
request := cli.ToolSearchRequest{Query: "给群里发文件并确认消息已读", Limit: 5, CandidateLimit: 20}
b.ReportAllocs()
b.ResetTimer()
for index := 0; index < b.N; index++ {
if _, searchErr := engine.Search(context.Background(), request); searchErr != nil {
b.Fatalf("Search() error = %v", searchErr)
}
}
}
func BenchmarkToolSearchDeliveryDecomposedWorkflow(b *testing.B) {
registerSchemaRuntimeDelivery()
engine, err := cli.NewDeliveryToolSearchEngineWithConfig(cli.DefaultToolSearchConfig())
if err != nil {
b.Fatalf("NewDeliveryToolSearchEngineWithConfig() error = %v", err)
}
request := cli.ToolSearchRequest{Query: "给群里发文件,并确认群成员是否已读", Limit: 5, CandidateLimit: 20}
subqueries := []string{"以当前用户身份给群聊发送本地文件", "查询指定消息的已读状态和人员"}
b.ReportAllocs()
b.ResetTimer()
for index := 0; index < b.N; index++ {
if _, searchErr := engine.SearchSubqueries(context.Background(), subqueries, request); searchErr != nil {
b.Fatalf("SearchSubqueries() error = %v", searchErr)
}
}
}
func BenchmarkToolSearchDeliveryIndexBuild(b *testing.B) {
registerSchemaRuntimeDelivery()
config := cli.DefaultToolSearchConfig()
// Prime and cache declaration assembly so this benchmark isolates building
// the Tool Search projection/index. Cold process startup is measured at the
// release-binary boundary and reported separately.
if _, err := cli.NewDeliveryToolSearchEngineWithConfig(config); err != nil {
b.Fatalf("prime delivery Catalog: %v", err)
}
b.ReportAllocs()
b.ResetTimer()
for index := 0; index < b.N; index++ {
if _, err := cli.NewDeliveryToolSearchEngineWithConfig(config); err != nil {
b.Fatalf("NewDeliveryToolSearchEngineWithConfig() error = %v", err)
}
}
}
+53 -2
View File
@@ -23,6 +23,15 @@ import (
"github.com/spf13/cobra"
)
// SchemaInspectV1Response is the versioned Search -> Inspect wire contract.
// Legacy schema queries without expected hashes keep their historical flat
// payload for human and script compatibility.
type SchemaInspectV1Response struct {
Version string `json:"version"`
Catalog CatalogVersionRef `json:"catalog"`
ToolSpec map[string]any `json:"tool_spec"`
}
// schemaCommandCatalogError / payloads use deliverySchemaCatalog
// (RegisterSchemaSourceRoot → ResolveSchemaBuild). There is no committed
// Schema Catalog embed fallback.
@@ -71,12 +80,20 @@ func NewSchemaCommand() *cobra.Command {
Short: "渐进查看命令 Schema (产品 / 分组 / 工具参数)",
Long: `查看当前可运行命令的 Schema 元数据。
不带参数时列出产品和工具数量;传产品或分组路径逐层展开;传具体工具路径输出扁平参数 Schema(对齐 GWS:parameters 内联 required,键为 CLI flag)。普通 Agent 查询应使用 --compact:它按稳定字段白名单输出选参、约束、安全语义和已评审的返回契约。省略 --compact 的 full leaf 保留参数映射、接口绑定和 provenance,仅用于定向审计;--all 输出全部工具的完整 leaf Schema,用于审计/CI。helper、MCP 与本地 Cobra 命令均须通过 ContractFinal.Identity 声明进入收集的身份集,并从同一声明装配的 ToolSpec 投影;查询不执行服务发现或临时合成第二份 Schema。`,
Args: cobra.MaximumNArgs(1),
不带参数时列出产品和工具数量;传产品或分组路径逐层展开;传具体工具路径输出扁平参数 Schema(对齐 GWS:parameters 内联 required,键为 CLI flag)。普通 Agent 查询应使用 --compact:它按稳定字段白名单输出选参、约束、安全语义和已评审的返回契约。省略 --compact 的 full leaf 保留参数映射、接口绑定和 provenance,仅用于定向审计;--all 输出全部工具的完整 leaf Schema,用于审计/CI。helper、MCP 与本地 Cobra 命令均须通过 ContractFinal.Identity 声明进入收集的身份集,并从同一声明装配的 ToolSpec 投影;查询不执行服务发现或临时合成第二份 Schema。
路径形态:多个位置参数会按空格拼接并作为 CLI path 解析,等价于加引号的形式和 --cli-path,例如 dws schema chat message read-status 与 dws schema "chat message read-status" 相同。canonical path 含点号,必须作为单个参数整体传入(dws schema chat.query_msg_read_status);把 canonical 拆成多个 token 会被当作 CLI path 而解析失败。`,
Args: cobra.MaximumNArgs(16),
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, args []string) error {
all, _ := cmd.Flags().GetBool("all")
compact, _ := cmd.Flags().GetBool("compact")
expectedSourceHash, _ := cmd.Flags().GetString("expected-source-hash")
expectedSurfaceHash, _ := cmd.Flags().GetString("expected-surface-hash")
if (expectedSourceHash == "") != (expectedSurfaceHash == "") {
return apperrors.NewValidation("--expected-source-hash and --expected-surface-hash must be provided together", apperrors.WithReason("incomplete_catalog_version"))
}
versionedInspect := expectedSourceHash != ""
cliPath, _ := cmd.Flags().GetString("cli-path")
cliPath = strings.TrimSpace(cliPath)
if cliPath != "" && len(args) > 0 {
@@ -90,10 +107,34 @@ func NewSchemaCommand() *cobra.Command {
}
if cliPath != "" {
args = []string{cliPath}
} else if len(args) > 1 {
// Preserve the quoted path form while accepting the equivalent
// natural tokenized form: dws schema chat message.
args = []string{strings.Join(args, " ")}
}
if versionedInspect {
if all || len(args) == 0 || !compact {
return apperrors.NewValidation("versioned Schema inspect requires one tool path and --compact", apperrors.WithReason("invalid_inspect_request"))
}
for _, name := range []string{"fields", "jq"} {
if flag := cmd.Flag(name); flag != nil && flag.Changed {
return apperrors.NewValidation("schema-inspect.v1 does not support --"+name, apperrors.WithReason("unsupported_output_filter"))
}
}
if flag := cmd.Flag("format"); flag != nil && flag.Changed && strings.TrimSpace(flag.Value.String()) != "json" {
return apperrors.NewValidation("schema-inspect.v1 only supports --format json", apperrors.WithReason("unsupported_output_format"))
}
}
if err := schemaCommandCatalogError(); err != nil {
return fmt.Errorf("load typed Schema registry: %w", err)
}
loaded := deliverySchemaCatalog()
if expectedSourceHash != "" && expectedSourceHash != loaded.Snapshot.SourceHash {
return apperrors.NewDiscovery("Catalog changed before Schema inspect", apperrors.WithReason("catalog_changed"))
}
if expectedSurfaceHash != "" && expectedSurfaceHash != loaded.Snapshot.SurfaceHash {
return apperrors.NewDiscovery("Catalog surface changed before Schema inspect", apperrors.WithReason("catalog_changed"))
}
var payload map[string]any
var err error
if all {
@@ -109,12 +150,22 @@ func NewSchemaCommand() *cobra.Command {
if compact {
payload = stripSchemaPayloadCompact(payload)
}
if versionedInspect {
return output.WriteFiltered(cmd.OutOrStdout(), output.FormatJSON, SchemaInspectV1Response{
Version: "schema-inspect.v1",
Catalog: CatalogVersionRef{SourceHash: loaded.Snapshot.SourceHash, SurfaceHash: loaded.Snapshot.SurfaceHash},
ToolSpec: payload,
}, "", "")
}
return output.WriteFiltered(cmd.OutOrStdout(), output.ResolveFormat(cmd, output.FormatJSON), payload, output.ResolveFields(cmd), output.ResolveJQ(cmd))
},
}
cmd.Flags().Bool("all", false, "输出全部工具的完整 leaf Schema(包括参数和约束,用于审计/CI)")
cmd.Flags().Bool("compact", false, "按稳定字段白名单输出 Agent 选参、约束、安全语义和返回契约")
cmd.Flags().String("cli-path", "", "按 CLI 命令路径查询")
cmd.Flags().String("expected-source-hash", "", "要求 Inspect 使用指定 Catalog source hash")
cmd.Flags().String("expected-surface-hash", "", "要求 Inspect 使用指定 Catalog surface hash")
cmd.AddCommand(newSchemaSearchCommand())
return cmd
}
+6
View File
@@ -35,6 +35,12 @@ func resetSchemaSourceRootAtomicForTest() {
schemaSourceRoot = atomic.Value{}
}
// resetDeliveryToolSearchEngineStateForTest drops the memoized default Tool
// Search engine so a test can rebuild it against a freshly installed Catalog.
func resetDeliveryToolSearchEngineStateForTest() {
resetDeliveryToolSearchEngineState()
}
// clearDeclaredDryRunCapabilitiesForTest resets the declared index (tests only).
func clearDeclaredDryRunCapabilitiesForTest() {
declaredDryRunCapabilities.Range(func(key, _ any) bool {
+4 -1
View File
@@ -325,7 +325,10 @@ func schemaPayloadFromLoadedCatalog(loaded loadedSchemaCatalog, args []string) (
}
}
}
return nil, apperrors.NewValidation("unknown runtime schema path " + strconvQuote(raw))
// The message carries the path-form boundary because it is the one surface
// an Agent is guaranteed to see at failure time.
return nil, apperrors.NewValidation("unknown runtime schema path " + strconvQuote(raw) +
": canonical paths contain dots and must be a single argument (e.g. dws schema chat.query_msg_read_status), while space-separated tokens resolve as a CLI path")
}
func schemaCatalogSnapshotHash(snapshot SchemaCatalogSnapshot) string {
+1 -1
View File
@@ -51,7 +51,7 @@ var reviewedRuntimeSchemaExclusionGroups = []runtimeSchemaExclusionGroup{
"profile list",
"profile switch",
"profile use",
"schema",
"schema search",
"skill get",
"skill install",
"skill search",
+203
View File
@@ -0,0 +1,203 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cli
import (
"bytes"
"encoding/json"
"fmt"
"io"
"strings"
apperrors "github.com/DingTalk-Real-AI/dingtalk-workspace-cli/internal/errors"
"github.com/spf13/cobra"
)
// ToolSearchV1Request is the stable stdin transport used by Agent hosts.
type ToolSearchV1Request struct {
Version string `json:"version"`
Query string `json:"query"`
Subqueries []string `json:"subqueries,omitempty"`
Limit int `json:"limit,omitempty"`
CandidateLimit int `json:"candidate_limit,omitempty"`
ProductIDs []string `json:"product_ids,omitempty"`
Effects []string `json:"effects,omitempty"`
ExcludeCanonical []string `json:"exclude_canonical,omitempty"`
}
var schemaSearchNewEngine = NewDeliveryToolSearchEngineWithConfig
// schemaSearchEngineConfig resolves the search config the same way
// NewDeliveryToolSearchEngine does (defaults + DWS_TOOL_SEARCH_* env
// overrides), then lets an explicit --explain flag win over the env value.
func schemaSearchEngineConfig(cmd *cobra.Command) ToolSearchConfig {
config := ResolveToolSearchConfigFromEnv(DefaultToolSearchConfig())
if explain, _ := cmd.Flags().GetBool("explain"); explain {
config.Explain = true
}
return config
}
func newSchemaSearchCommand() *cobra.Command {
cmd := &cobra.Command{
Use: "search",
Short: "按自然语言搜索可用命令 Schema",
Long: `按自然语言或精确身份检索 ToolReference。
输出固定为完整、紧凑的 tool-search.v1 JSON envelope。为保证版本、Catalog
绑定和响应字节预算可验证,本命令不支持 --fields、--jq 或非 JSON --format。
DWS 对 Agent 是一个元工具:已知子命令直接执行,未知子命令才使用 search。
选中候选后,必须使用 response catalog 的 source/surface hash Inspect canonical_path,
再按 primary_cli_path 执行;Search 结果本身不授权、不执行。`,
Example: ` dws schema search --query "查询群消息已读状态"
dws schema chat.query_msg_read_status --compact --format json \
--expected-source-hash "<search.catalog.source_hash>" \
--expected-surface-hash "<search.catalog.surface_hash>"
printf '%s' '{"version":"tool-search.v1","query":"发群文件并确认已读","subqueries":["给群里发送文件消息","查询群消息已读状态"]}' |
dws schema search --request-json -`,
Args: cobra.NoArgs,
DisableAutoGenTag: true,
RunE: func(cmd *cobra.Command, _ []string) error {
if err := validateSchemaSearchOutputFlags(cmd); err != nil {
return err
}
request, err := schemaSearchRequest(cmd)
if err != nil {
return err
}
engine, err := schemaSearchNewEngine(schemaSearchEngineConfig(cmd))
if err != nil {
return err
}
internal := ToolSearchRequest{
Query: request.Query,
Limit: request.Limit,
CandidateLimit: request.CandidateLimit,
ProductIDs: request.ProductIDs,
Effects: request.Effects,
ExcludeCanonicalPaths: request.ExcludeCanonical,
}
var response ToolSearchResponse
if len(request.Subqueries) > 0 {
response, err = engine.SearchSubqueries(cmd.Context(), request.Subqueries, internal)
} else {
response, err = engine.Search(cmd.Context(), internal)
}
if err != nil {
return err
}
// The response budget is defined over this compact wire encoding. Do
// not run it through pretty/table formatting or post-filtering because
// Agent hosts require one complete versioned envelope.
if err := json.NewEncoder(cmd.OutOrStdout()).Encode(response); err != nil {
return fmt.Errorf("encode tool-search.v1 response: %w", err)
}
return nil
},
}
cmd.Flags().String("query", "", "自然语言意图或 canonical/CLI path")
cmd.Flags().Int("limit", defaultToolSearchLimit, fmt.Sprintf("返回引用数量(1-%d;0 表示沿用默认值 %d)", maxToolSearchLimit, defaultToolSearchLimit))
cmd.Flags().Int("candidate-limit", defaultToolSearchCandidateLimit, fmt.Sprintf("每路召回候选数量(1-%d;0 表示沿用默认值 %d)", maxToolSearchCandidateLimit, defaultToolSearchCandidateLimit))
cmd.Flags().StringSlice("product", nil, "限制产品 ID,可重复或逗号分隔")
cmd.Flags().StringSlice("effect", nil, "限制 effect,可重复或逗号分隔")
cmd.Flags().StringSlice("exclude", nil, "排除 canonical path,可重复或逗号分隔")
cmd.Flags().String("request-json", "", `从 stdin 读取 JSON;值必须为 -;必含 version="tool-search.v1",动作分解字段为 subqueries`)
cmd.Flags().Bool("explain", false, "开发诊断:输出 score_breakdown 含原始排序分(公开默认响应不含分数;仅用于离线调参,不建议线上 Agent 使用;会增大响应体积,建议配合 --limit 3)。score_breakdown.score 是未量化的原始分,排序键使用量化到 1e-9 的值,两者末位可能不同,不要用它复算候选顺序")
return cmd
}
func validateSchemaSearchOutputFlags(cmd *cobra.Command) error {
for _, name := range []string{"fields", "jq"} {
if flag := cmd.Flag(name); flag != nil && flag.Changed {
return apperrors.NewValidation(
"schema search returns one complete tool-search.v1 envelope and does not support --"+name,
apperrors.WithReason("unsupported_output_filter"),
)
}
}
if flag := cmd.Flag("format"); flag != nil && flag.Changed && strings.TrimSpace(flag.Value.String()) != "json" {
return apperrors.NewValidation(
"schema search only supports --format json",
apperrors.WithReason("unsupported_output_format"),
)
}
return nil
}
func schemaSearchRequest(cmd *cobra.Command) (ToolSearchV1Request, error) {
requestJSON, _ := cmd.Flags().GetString("request-json")
requestJSON = strings.TrimSpace(requestJSON)
if requestJSON != "" {
if requestJSON != "-" {
return ToolSearchV1Request{}, apperrors.NewValidation("--request-json only accepts - (stdin)", apperrors.WithReason("invalid_transport"))
}
for _, name := range []string{"query", "limit", "candidate-limit", "product", "effect", "exclude"} {
if cmd.Flags().Changed(name) {
return ToolSearchV1Request{}, apperrors.NewValidation("--request-json cannot be combined with --"+name, apperrors.WithReason("ambiguous_input"))
}
}
request, err := decodeToolSearchV1Request(cmd.InOrStdin())
if err != nil {
return ToolSearchV1Request{}, err
}
return validateToolSearchV1Request(request)
}
query, _ := cmd.Flags().GetString("query")
limit, _ := cmd.Flags().GetInt("limit")
candidateLimit, _ := cmd.Flags().GetInt("candidate-limit")
products, _ := cmd.Flags().GetStringSlice("product")
effects, _ := cmd.Flags().GetStringSlice("effect")
excluded, _ := cmd.Flags().GetStringSlice("exclude")
return validateToolSearchV1Request(ToolSearchV1Request{
Version: "tool-search.v1",
Query: query,
Limit: limit,
CandidateLimit: candidateLimit,
ProductIDs: products,
Effects: effects,
ExcludeCanonical: excluded,
})
}
func decodeToolSearchV1Request(reader io.Reader) (ToolSearchV1Request, error) {
payload, err := io.ReadAll(io.LimitReader(reader, maxToolSearchRequestBytes+1))
if err != nil {
return ToolSearchV1Request{}, apperrors.NewValidation("read tool search request: "+err.Error(), apperrors.WithReason("read_request_failed"))
}
if len(payload) > maxToolSearchRequestBytes {
return ToolSearchV1Request{}, apperrors.NewValidation(fmt.Sprintf("tool search request exceeds %d bytes", maxToolSearchRequestBytes), apperrors.WithReason("request_too_large"))
}
decoder := json.NewDecoder(bytes.NewReader(payload))
decoder.DisallowUnknownFields()
var request ToolSearchV1Request
if err := decoder.Decode(&request); err != nil {
return ToolSearchV1Request{}, apperrors.NewValidation("decode tool-search.v1 request: "+err.Error(), apperrors.WithReason("invalid_request_json"))
}
var trailing any
if err := decoder.Decode(&trailing); err != io.EOF {
if err == nil {
return ToolSearchV1Request{}, apperrors.NewValidation("tool search request must contain exactly one JSON object", apperrors.WithReason("invalid_request_json"))
}
return ToolSearchV1Request{}, apperrors.NewValidation("decode trailing tool search input: "+err.Error(), apperrors.WithReason("invalid_request_json"))
}
return request, nil
}
func validateToolSearchV1Request(request ToolSearchV1Request) (ToolSearchV1Request, error) {
if request.Version != "tool-search.v1" {
return ToolSearchV1Request{}, apperrors.NewValidation(`tool search request must include "version":"tool-search.v1"; action decomposition uses the "subqueries" array`, apperrors.WithReason("unsupported_version"))
}
return request, nil
}
@@ -0,0 +1,95 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
package cli
import (
"bytes"
"encoding/json"
"errors"
"strings"
"testing"
apperrors "github.com/DingTalk-Real-AI/dingtalk-workspace-cli/internal/errors"
"github.com/DingTalk-Real-AI/dingtalk-workspace-cli/internal/testseam"
"github.com/spf13/cobra"
)
func TestSchemaSearchCommandExactAndVersioned(t *testing.T) {
installSchemaSearchTestEngine(t)
root := &cobra.Command{Use: "dws"}
root.AddCommand(NewSchemaCommand())
var stdout bytes.Buffer
root.SetOut(&stdout)
root.SetArgs([]string{"schema", "search", "--query", "chat.read_status"})
if err := root.Execute(); err != nil {
t.Fatalf("schema search execute: %v", err)
}
var response ToolSearchResponse
if err := json.Unmarshal(stdout.Bytes(), &response); err != nil {
t.Fatalf("decode response: %v\n%s", err, stdout.String())
}
if response.Version != "tool-search.v1" || response.Strategy != "exact_guard" {
t.Fatalf("response = %#v", response)
}
if len(response.Candidates) != 1 || response.Candidates[0].CanonicalPath != "chat.read_status" {
t.Fatalf("candidates = %#v", response.Candidates)
}
wantCatalog := CatalogVersionRef{SourceHash: "source-test", SurfaceHash: "surface-test"}
if response.Catalog != wantCatalog {
t.Fatalf("Catalog = %#v, want %#v", response.Catalog, wantCatalog)
}
for _, removed := range []string{"external_ranking", "degraded", "degraded_reason_code", "warning_codes", "_fallback"} {
if strings.Contains(stdout.String(), removed) {
t.Fatalf("removed fallback field %q leaked into response: %s", removed, stdout.String())
}
}
}
func TestSchemaSearchRequestRejectsUnboundedOrAmbiguousInput(t *testing.T) {
if _, err := decodeToolSearchV1Request(strings.NewReader(`{"version":"tool-search.v1","query":"x","unknown":true}`)); validationReason(err) != "invalid_request_json" {
t.Fatal("unknown field was accepted")
}
if _, err := decodeToolSearchV1Request(strings.NewReader(`{"version":"tool-search.v1","query":"x","external_ranking":{}}`)); validationReason(err) != "invalid_request_json" {
t.Fatal("removed external_ranking field was accepted")
}
oversized := strings.NewReader(strings.Repeat("x", maxToolSearchRequestBytes+1))
if _, err := decodeToolSearchV1Request(oversized); validationReason(err) != "request_too_large" {
t.Fatal("oversized request was accepted")
}
_, err := validateToolSearchV1Request(ToolSearchV1Request{Query: "x", Subqueries: []string{"x"}})
if validationReason(err) != "unsupported_version" || !strings.Contains(err.Error(), `"version":"tool-search.v1"`) || !strings.Contains(err.Error(), "subqueries") {
t.Fatalf("missing-version guidance = %v", err)
}
}
func TestSchemaSearchHelpDocumentsStructuredRequest(t *testing.T) {
command := newSchemaSearchCommand()
var output bytes.Buffer
command.SetOut(&output)
command.SetArgs([]string{"--help"})
if err := command.Execute(); err != nil {
t.Fatalf("schema search help: %v", err)
}
for _, required := range []string{`"version":"tool-search.v1"`, `"subqueries"`, "--request-json -", "不支持 --fields、--jq", "非 JSON --format"} {
if !strings.Contains(output.String(), required) {
t.Fatalf("schema search help omits %q:\n%s", required, output.String())
}
}
}
func installSchemaSearchTestEngine(t *testing.T) {
t.Helper()
testseam.Swap(t, &schemaSearchNewEngine, func(ToolSearchConfig) (*ToolSearchEngine, error) {
return newToolSearchTestEngine(t), nil
})
}
func validationReason(err error) string {
var typed *apperrors.Error
if errors.As(err, &typed) && typed.Category == apperrors.CategoryValidation {
return typed.Reason
}
return ""
}
+4 -1
View File
@@ -63,12 +63,15 @@ func storeSchemaSourceRootFn(fn func() *cobra.Command) {
}
// resetDeliverySchemaCatalogState clears the lazy Catalog delivery Once/caches
// so the next deliverySchemaCatalog() reassembles.
// so the next deliverySchemaCatalog() reassembles. The memoized Tool Search
// engine is a Catalog derivative (it indexes one generation and caches its
// source/surface hashes), so it is invalidated here as well.
func resetDeliverySchemaCatalogState() {
runtimeDeliverySchemaCatalogOnce = sync.Once{}
runtimeDeliverySchemaCatalog = loadedSchemaCatalog{}
runtimeDeliverySchemaCatalogErr = nil
runtimeDeliverySchemaCatalogLazyCount.Store(0)
resetDeliveryToolSearchEngineState()
}
// resetSchemaDeliveryState clears ResolveMeta lookup state and Catalog delivery
+19
View File
@@ -7,11 +7,30 @@ package cli
import (
"bytes"
"encoding/json"
"strings"
"testing"
"github.com/spf13/cobra"
)
func TestSchemaHelpDocumentsCanonicalAndCLIPathForms(t *testing.T) {
// Positional args are joined with spaces and resolved as a CLI path, so the
// tokenized form is not a canonical-path spelling. Help must keep saying so:
// an Agent that splits a canonical path on the dot otherwise gets an
// "unknown runtime schema path" error with no explanation of the boundary.
long := NewSchemaCommand().Long
for _, required := range []string{
"CLI path",
"canonical path",
`dws schema "chat message read-status"`,
"dws schema chat.query_msg_read_status",
} {
if !strings.Contains(long, required) {
t.Fatalf("schema long help omits the path-form boundary %q:\n%s", required, long)
}
}
}
func TestSchemaUsesDeliveryCatalogWithoutRuntimeLoad(t *testing.T) {
root := &cobra.Command{Use: "dws"}
root.AddCommand(NewSchemaCommand())
File diff suppressed because it is too large Load Diff
+490
View File
@@ -0,0 +1,490 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cli
import (
"fmt"
"math/rand"
"sort"
"strings"
)
const (
ToolSearchAgentArmDirectSchema = "direct_schema"
ToolSearchAgentArmSearchInspect = "search_inspect"
)
// Bootstrap resampling uses fixed seeds so a report can be re-derived from the
// same paired runs. The seeds are published in the report so an auditor never
// has to read them out of the source to reproduce an interval. The recovery
// slice uses a distinct seed because it resamples a different pair subset.
const (
toolSearchAgentABBootstrapSamples = 10000
toolSearchAgentABBootstrapSeed = 927
toolSearchAgentABRecoveryBootstrapSeed = 928
)
// ToolSearchAgentABInput is populated by an Agent runner. Every case/trial
// must have both arms so the comparison remains paired.
type ToolSearchAgentABInput struct {
Version string `json:"version"`
Catalog CatalogVersionRef `json:"catalog"`
Runs []ToolSearchAgentABRun `json:"runs"`
}
// ToolSearchAgentABRun records end-to-end outcomes rather than retrieval
// proxies. ContextTokens must come from the selected model's tokenizer.
type ToolSearchAgentABRun struct {
CaseID string `json:"case_id"`
Trial int `json:"trial"`
Arm string `json:"arm"`
TaskCompleted bool `json:"task_completed"`
CorrectToolPlan bool `json:"correct_tool_plan"`
UnsafeAction bool `json:"unsafe_action"`
RecoveryAttempted bool `json:"recovery_attempted"`
RecoverySucceeded bool `json:"recovery_succeeded"`
ContextTokens float64 `json:"context_tokens"`
ToolCalls float64 `json:"tool_calls"`
LatencyMS float64 `json:"latency_ms"`
}
type ToolSearchAgentABArmMetrics struct {
Runs int `json:"runs"`
TaskSuccessRate float64 `json:"task_success_rate"`
CorrectPlanRate float64 `json:"correct_plan_rate"`
UnsafeActionRate float64 `json:"unsafe_action_rate"`
RecoverySuccessRate float64 `json:"recovery_success_rate"`
RecoveryAttempts int `json:"recovery_attempts"`
AverageContextTokens float64 `json:"average_context_tokens"`
AverageToolCalls float64 `json:"average_tool_calls"`
AverageLatencyMS float64 `json:"average_latency_ms"`
}
// ToolSearchAgentABDelta is search_inspect minus direct_schema. A positive
// value is desirable for success/plan/recovery and undesirable for unsafe
// actions, context, calls, and latency.
type ToolSearchAgentABDelta struct {
DirectSchema float64 `json:"direct_schema"`
SearchInspect float64 `json:"search_inspect"`
Delta float64 `json:"delta"`
CI95Low float64 `json:"ci95_low"`
CI95High float64 `json:"ci95_high"`
}
type ToolSearchAgentABReport struct {
Version string `json:"version"`
Catalog CatalogVersionRef `json:"catalog"`
Cases int `json:"cases"`
PairedTrials int `json:"paired_trials"`
DirectSchema ToolSearchAgentABArmMetrics `json:"direct_schema"`
SearchInspect ToolSearchAgentABArmMetrics `json:"search_inspect"`
Deltas map[string]ToolSearchAgentABDelta `json:"deltas"`
// BootstrapSamples/BootstrapSeed/RecoveryBootstrapSeed are published so an
// auditor can reproduce every interval without reading the source.
BootstrapSamples int `json:"bootstrap_samples"`
BootstrapSeed int64 `json:"bootstrap_seed"`
RecoveryBootstrapSeed int64 `json:"recovery_bootstrap_seed"`
Method string `json:"method"`
}
// ToolSearchAgentPlanOutput is the strict final response of a non-executing
// Agent planning run. Gold tools remain in the local workflow fixture and are
// never sent to the evaluated model.
type ToolSearchAgentPlanOutput struct {
Results []ToolSearchAgentPlanResult `json:"results"`
}
type ToolSearchAgentPlanResult struct {
ID string `json:"id"`
CanonicalPaths []string `json:"canonical_paths"`
}
type ToolSearchAgentPlanMetrics struct {
Cases int `json:"cases"`
CompleteRate float64 `json:"complete_rate"`
ExactMinimalRate float64 `json:"exact_minimal_rate"`
RequiredToolRecall float64 `json:"required_tool_recall"`
PlanPrecision float64 `json:"plan_precision"`
RequiredTools int `json:"required_tools"`
PredictedTools int `json:"predicted_tools"`
UnnecessaryTools int `json:"unnecessary_tools"`
}
type ToolSearchAgentPlanningABReport struct {
Version string `json:"version"`
Model string `json:"model"`
DirectSchema ToolSearchAgentPlanMetrics `json:"direct_schema"`
SearchInspect ToolSearchAgentPlanMetrics `json:"search_inspect"`
Delta map[string]float64 `json:"delta_search_minus_direct"`
Limitations []string `json:"limitations"`
}
// ScoreToolSearchAgentPlanningAB scores an answer-free paired planning smoke
// run against reviewed workflow requirements. It measures plan coverage and
// minimality only; it does not claim that any business task was executed.
func ScoreToolSearchAgentPlanningAB(model string, workflows []ToolSearchWorkflowEvaluationCase, direct, search ToolSearchAgentPlanOutput) (ToolSearchAgentPlanningABReport, error) {
if err := deliverySchemaCatalogError(); err != nil {
return ToolSearchAgentPlanningABReport{}, fmt.Errorf("assemble typed Schema registry for Agent planning score: %w", err)
}
return scoreToolSearchAgentPlanningAB(deliverySchemaCatalog().Index, model, workflows, direct, search)
}
func scoreToolSearchAgentPlanningAB(index SchemaIndex, model string, workflows []ToolSearchWorkflowEvaluationCase, direct, search ToolSearchAgentPlanOutput) (ToolSearchAgentPlanningABReport, error) {
if strings.TrimSpace(model) == "" {
return ToolSearchAgentPlanningABReport{}, fmt.Errorf("Agent planning A/B requires a model identifier")
}
directMetrics, err := scoreToolSearchAgentPlans(index, workflows, direct)
if err != nil {
return ToolSearchAgentPlanningABReport{}, fmt.Errorf("score direct_schema plans: %w", err)
}
searchMetrics, err := scoreToolSearchAgentPlans(index, workflows, search)
if err != nil {
return ToolSearchAgentPlanningABReport{}, fmt.Errorf("score search_inspect plans: %w", err)
}
return ToolSearchAgentPlanningABReport{
Version: "tool-search-agent-planning-ab.v1",
Model: strings.TrimSpace(model),
DirectSchema: directMetrics,
SearchInspect: searchMetrics,
Delta: map[string]float64{
"complete_rate": searchMetrics.CompleteRate - directMetrics.CompleteRate,
"exact_minimal_rate": searchMetrics.ExactMinimalRate - directMetrics.ExactMinimalRate,
"required_tool_recall": searchMetrics.RequiredToolRecall - directMetrics.RequiredToolRecall,
"plan_precision": searchMetrics.PlanPrecision - directMetrics.PlanPrecision,
"unnecessary_tools": float64(searchMetrics.UnnecessaryTools - directMetrics.UnnecessaryTools),
},
Limitations: []string{
"one batch and one model trial per arm; no confidence interval",
"reviewed ten-workflow diagnostic fixture is not an independent release test",
"planning only: no parameters, business execution, recovery, latency, or tokenizer usage is scored",
"extra prerequisites can be operationally valid even when absent from the minimal reviewed gold",
},
}, nil
}
func scoreToolSearchAgentPlans(index SchemaIndex, workflows []ToolSearchWorkflowEvaluationCase, output ToolSearchAgentPlanOutput) (ToolSearchAgentPlanMetrics, error) {
byID := make(map[string]ToolSearchAgentPlanResult, len(output.Results))
for _, result := range output.Results {
if strings.TrimSpace(result.ID) == "" || len(result.CanonicalPaths) == 0 {
return ToolSearchAgentPlanMetrics{}, fmt.Errorf("plan result requires id and at least one canonical path")
}
if _, exists := byID[result.ID]; exists {
return ToolSearchAgentPlanMetrics{}, fmt.Errorf("duplicate plan result %q", result.ID)
}
seen := make(map[string]bool, len(result.CanonicalPaths))
for _, canonical := range result.CanonicalPaths {
if _, ok := index.Resolve(canonical); !ok {
return ToolSearchAgentPlanMetrics{}, fmt.Errorf("plan result %q contains unknown canonical %q", result.ID, canonical)
}
if seen[canonical] {
return ToolSearchAgentPlanMetrics{}, fmt.Errorf("plan result %q repeats canonical %q", result.ID, canonical)
}
seen[canonical] = true
}
byID[result.ID] = result
}
metrics := ToolSearchAgentPlanMetrics{Cases: len(workflows)}
var completeCases, exactCases, hits int
for _, workflow := range workflows {
result, ok := byID[workflow.ID]
if !ok {
return ToolSearchAgentPlanMetrics{}, fmt.Errorf("missing plan result %q", workflow.ID)
}
delete(byID, workflow.ID)
gold := stringSet(workflow.Required)
caseHits := 0
for _, canonical := range result.CanonicalPaths {
if gold[canonical] {
caseHits++
}
}
hits += caseHits
metrics.RequiredTools += len(workflow.Required)
metrics.PredictedTools += len(result.CanonicalPaths)
if caseHits == len(workflow.Required) {
completeCases++
}
if stringSlicesEqual(result.CanonicalPaths, workflow.Required) {
exactCases++
}
}
if len(byID) > 0 {
unexpected := make([]string, 0, len(byID))
for id := range byID {
unexpected = append(unexpected, id)
}
sort.Strings(unexpected)
return ToolSearchAgentPlanMetrics{}, fmt.Errorf("unexpected plan results: %s", strings.Join(unexpected, ", "))
}
if metrics.Cases > 0 {
metrics.CompleteRate = float64(completeCases) / float64(metrics.Cases)
metrics.ExactMinimalRate = float64(exactCases) / float64(metrics.Cases)
}
if metrics.RequiredTools > 0 {
metrics.RequiredToolRecall = float64(hits) / float64(metrics.RequiredTools)
}
if metrics.PredictedTools > 0 {
metrics.PlanPrecision = float64(hits) / float64(metrics.PredictedTools)
}
metrics.UnnecessaryTools = metrics.PredictedTools - hits
return metrics, nil
}
func stringSlicesEqual(left, right []string) bool {
if len(left) != len(right) {
return false
}
for index := range left {
if left[index] != right[index] {
return false
}
}
return true
}
type toolSearchAgentPair struct {
caseID string
direct ToolSearchAgentABRun
search ToolSearchAgentABRun
}
// AggregateToolSearchAgentAB validates a paired experiment and computes
// case-cluster bootstrap confidence intervals with a fixed seed. Trials of the
// same task are averaged inside a case before resampling so repeated model
// seeds are not treated as independent tasks.
func AggregateToolSearchAgentAB(input ToolSearchAgentABInput) (ToolSearchAgentABReport, error) {
if input.Version != "tool-search-agent-ab.v1" {
return ToolSearchAgentABReport{}, fmt.Errorf("unsupported Agent A/B input version %q", input.Version)
}
if strings.TrimSpace(input.Catalog.SourceHash) == "" || strings.TrimSpace(input.Catalog.SurfaceHash) == "" {
return ToolSearchAgentABReport{}, fmt.Errorf("Agent A/B input requires both Catalog hashes")
}
type key struct {
caseID string
trial int
}
paired := make(map[key]*toolSearchAgentPair)
for _, run := range input.Runs {
if strings.TrimSpace(run.CaseID) == "" || run.Trial < 0 {
return ToolSearchAgentABReport{}, fmt.Errorf("Agent A/B run requires case_id and non-negative trial")
}
if run.ContextTokens < 0 || run.ToolCalls < 0 || run.LatencyMS < 0 {
return ToolSearchAgentABReport{}, fmt.Errorf("Agent A/B run %s/%d has a negative cost", run.CaseID, run.Trial)
}
if run.RecoverySucceeded && !run.RecoveryAttempted {
return ToolSearchAgentABReport{}, fmt.Errorf("Agent A/B run %s/%d marks recovery success without an attempt", run.CaseID, run.Trial)
}
itemKey := key{caseID: run.CaseID, trial: run.Trial}
pair := paired[itemKey]
if pair == nil {
pair = &toolSearchAgentPair{caseID: run.CaseID}
paired[itemKey] = pair
}
switch run.Arm {
case ToolSearchAgentArmDirectSchema:
if pair.direct.Arm != "" {
return ToolSearchAgentABReport{}, fmt.Errorf("duplicate direct_schema run for %s/%d", run.CaseID, run.Trial)
}
pair.direct = run
case ToolSearchAgentArmSearchInspect:
if pair.search.Arm != "" {
return ToolSearchAgentABReport{}, fmt.Errorf("duplicate search_inspect run for %s/%d", run.CaseID, run.Trial)
}
pair.search = run
default:
return ToolSearchAgentABReport{}, fmt.Errorf("unknown Agent A/B arm %q", run.Arm)
}
}
if len(paired) == 0 {
return ToolSearchAgentABReport{}, fmt.Errorf("Agent A/B input contains no runs")
}
pairs := make([]toolSearchAgentPair, 0, len(paired))
caseSet := make(map[string]bool)
for itemKey, pair := range paired {
if pair.direct.Arm == "" || pair.search.Arm == "" {
return ToolSearchAgentABReport{}, fmt.Errorf("Agent A/B case %s/%d is not paired", itemKey.caseID, itemKey.trial)
}
pairs = append(pairs, *pair)
caseSet[itemKey.caseID] = true
}
sort.Slice(pairs, func(i, j int) bool {
if pairs[i].caseID == pairs[j].caseID {
return pairs[i].direct.Trial < pairs[j].direct.Trial
}
return pairs[i].caseID < pairs[j].caseID
})
directMetrics := aggregateToolSearchAgentArm(pairs, false)
searchMetrics := aggregateToolSearchAgentArm(pairs, true)
metrics := []struct {
name string
value func(ToolSearchAgentABRun) float64
direct float64
search float64
}{
{"task_success_rate", func(run ToolSearchAgentABRun) float64 { return boolFloat(run.TaskCompleted) }, directMetrics.TaskSuccessRate, searchMetrics.TaskSuccessRate},
{"correct_plan_rate", func(run ToolSearchAgentABRun) float64 { return boolFloat(run.CorrectToolPlan) }, directMetrics.CorrectPlanRate, searchMetrics.CorrectPlanRate},
{"unsafe_action_rate", func(run ToolSearchAgentABRun) float64 { return boolFloat(run.UnsafeAction) }, directMetrics.UnsafeActionRate, searchMetrics.UnsafeActionRate},
{"context_tokens", func(run ToolSearchAgentABRun) float64 { return run.ContextTokens }, directMetrics.AverageContextTokens, searchMetrics.AverageContextTokens},
{"tool_calls", func(run ToolSearchAgentABRun) float64 { return run.ToolCalls }, directMetrics.AverageToolCalls, searchMetrics.AverageToolCalls},
{"latency_ms", func(run ToolSearchAgentABRun) float64 { return run.LatencyMS }, directMetrics.AverageLatencyMS, searchMetrics.AverageLatencyMS},
}
deltas := make(map[string]ToolSearchAgentABDelta, len(metrics)+1)
for _, metric := range metrics {
low, high := toolSearchCaseClusterBootstrap(pairs, metric.value, toolSearchAgentABBootstrapSamples, toolSearchAgentABBootstrapSeed)
deltas[metric.name] = ToolSearchAgentABDelta{
DirectSchema: metric.direct,
SearchInspect: metric.search,
Delta: metric.search - metric.direct,
CI95Low: low,
CI95High: high,
}
}
if directMetrics.RecoveryAttempts > 0 && searchMetrics.RecoveryAttempts > 0 {
// Recovery is conditionally defined over attempted recoveries, so its
// interval uses the same paired tasks but maps non-attempts to zero only
// after both arms attempted recovery for that trial.
recoveryPairs := make([]toolSearchAgentPair, 0, len(pairs))
for _, pair := range pairs {
if pair.direct.RecoveryAttempted && pair.search.RecoveryAttempted {
recoveryPairs = append(recoveryPairs, pair)
}
}
if len(recoveryPairs) > 0 {
value := func(run ToolSearchAgentABRun) float64 { return boolFloat(run.RecoverySucceeded) }
low, high := toolSearchCaseClusterBootstrap(recoveryPairs, value, toolSearchAgentABBootstrapSamples, toolSearchAgentABRecoveryBootstrapSeed)
directRecovery := meanAgentRunValue(recoveryPairs, false, value)
searchRecovery := meanAgentRunValue(recoveryPairs, true, value)
deltas["recovery_success_rate_paired_attempts"] = ToolSearchAgentABDelta{
DirectSchema: directRecovery,
SearchInspect: searchRecovery,
Delta: searchRecovery - directRecovery,
CI95Low: low,
CI95High: high,
}
}
}
return ToolSearchAgentABReport{
Version: "tool-search-agent-ab-report.v1",
Catalog: input.Catalog,
Cases: len(caseSet),
PairedTrials: len(pairs),
DirectSchema: directMetrics,
SearchInspect: searchMetrics,
Deltas: deltas,
BootstrapSamples: toolSearchAgentABBootstrapSamples,
BootstrapSeed: toolSearchAgentABBootstrapSeed,
RecoveryBootstrapSeed: toolSearchAgentABRecoveryBootstrapSeed,
Method: fmt.Sprintf(
"paired by case_id/trial; 95%% percentile bootstrap clustered by case_id; %d resamples; fixed seed %d (recovery slice seed %d)",
toolSearchAgentABBootstrapSamples,
toolSearchAgentABBootstrapSeed,
toolSearchAgentABRecoveryBootstrapSeed,
),
}, nil
}
func aggregateToolSearchAgentArm(pairs []toolSearchAgentPair, search bool) ToolSearchAgentABArmMetrics {
metrics := ToolSearchAgentABArmMetrics{Runs: len(pairs)}
for _, pair := range pairs {
run := pair.direct
if search {
run = pair.search
}
metrics.TaskSuccessRate += boolFloat(run.TaskCompleted)
metrics.CorrectPlanRate += boolFloat(run.CorrectToolPlan)
metrics.UnsafeActionRate += boolFloat(run.UnsafeAction)
metrics.AverageContextTokens += run.ContextTokens
metrics.AverageToolCalls += run.ToolCalls
metrics.AverageLatencyMS += run.LatencyMS
if run.RecoveryAttempted {
metrics.RecoveryAttempts++
metrics.RecoverySuccessRate += boolFloat(run.RecoverySucceeded)
}
}
if metrics.Runs > 0 {
denominator := float64(metrics.Runs)
metrics.TaskSuccessRate /= denominator
metrics.CorrectPlanRate /= denominator
metrics.UnsafeActionRate /= denominator
metrics.AverageContextTokens /= denominator
metrics.AverageToolCalls /= denominator
metrics.AverageLatencyMS /= denominator
}
if metrics.RecoveryAttempts > 0 {
metrics.RecoverySuccessRate /= float64(metrics.RecoveryAttempts)
}
return metrics
}
func toolSearchCaseClusterBootstrap(pairs []toolSearchAgentPair, value func(ToolSearchAgentABRun) float64, samples int, seed int64) (float64, float64) {
byCase := make(map[string][]float64)
for _, pair := range pairs {
byCase[pair.caseID] = append(byCase[pair.caseID], value(pair.search)-value(pair.direct))
}
caseIDs := make([]string, 0, len(byCase))
caseMeans := make(map[string]float64, len(byCase))
for caseID, values := range byCase {
caseIDs = append(caseIDs, caseID)
for _, value := range values {
caseMeans[caseID] += value / float64(len(values))
}
}
sort.Strings(caseIDs)
if len(caseIDs) == 1 {
value := caseMeans[caseIDs[0]]
return value, value
}
random := rand.New(rand.NewSource(seed))
distribution := make([]float64, samples)
for sample := range samples {
var total float64
for range caseIDs {
total += caseMeans[caseIDs[random.Intn(len(caseIDs))]]
}
distribution[sample] = total / float64(len(caseIDs))
}
sort.Float64s(distribution)
return percentile(distribution, 0.025), percentile(distribution, 0.975)
}
func percentile(sorted []float64, quantile float64) float64 {
if len(sorted) == 0 {
return 0
}
index := int(quantile * float64(len(sorted)-1))
return sorted[index]
}
func meanAgentRunValue(pairs []toolSearchAgentPair, search bool, value func(ToolSearchAgentABRun) float64) float64 {
if len(pairs) == 0 {
return 0
}
var total float64
for _, pair := range pairs {
run := pair.direct
if search {
run = pair.search
}
total += value(run)
}
return total / float64(len(pairs))
}
func boolFloat(value bool) float64 {
if value {
return 1
}
return 0
}
+587
View File
@@ -0,0 +1,587 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cli
import (
"context"
"encoding/json"
"fmt"
"math"
"sort"
"unicode"
"unicode/utf8"
)
// ToolSearchWorkflowEvaluationCase is a reviewed diagnostic workflow. It is
// intentionally separate from release qrels: the small in-repository fixture
// measures mechanics and cannot approve a ranking configuration.
type ToolSearchWorkflowEvaluationCase struct {
ID string `json:"id"`
Query string `json:"query"`
Subqueries []string `json:"subqueries"`
Required []string `json:"required"`
}
// ToolSearchRetrievalMetrics reports deterministic ranking metrics. Values are
// fractions in [0,1], not percentages.
type ToolSearchRetrievalMetrics struct {
Cases int `json:"cases"`
RecallAt1 float64 `json:"recall_at_1"`
RecallAt5 float64 `json:"recall_at_5"`
MeanReciprocalRankAt5 float64 `json:"mean_reciprocal_rank_at_5"`
ZeroResultRate float64 `json:"zero_result_rate"`
}
// ToolSearchWorkflowMetrics measures whether every required tool is present in
// the shared Top-5 budget and the average required-tool recall.
type ToolSearchWorkflowMetrics struct {
Cases int `json:"cases"`
CompleteAt5 float64 `json:"complete_at_5"`
RequiredRecallAt5 float64 `json:"required_recall_at_5"`
}
// ToolSearchIdentityTrustMetrics exercises the complete delivered identity
// set. Exact filtered checks deliberately exclude each canonical path and
// verify that search abstains instead of falling through to a fuzzy sibling.
type ToolSearchIdentityTrustMetrics struct {
CanonicalCases int `json:"canonical_cases"`
CanonicalPassRate float64 `json:"canonical_pass_rate"`
PrimaryCLICases int `json:"primary_cli_cases"`
PrimaryCLIPassRate float64 `json:"primary_cli_pass_rate"`
AliasCases int `json:"alias_cases"`
AliasPassRate float64 `json:"alias_pass_rate"`
NFKCCases int `json:"nfkc_cases"`
NFKCPassRate float64 `json:"nfkc_pass_rate"`
ExactFilteredCases int `json:"exact_filtered_cases"`
ExactFilteredPassRate float64 `json:"exact_filtered_pass_rate"`
}
// ToolSearchNegativeTrustMetrics is a same-source diagnostic only. A
// forbidden exposure means the tool that owns an avoid_when sentence appears
// in the ranking; it does not prove that a model would call that tool.
type ToolSearchNegativeTrustMetrics struct {
Cases int `json:"cases"`
ExcludedOverBudget int `json:"excluded_over_budget"`
ForbiddenAt1 float64 `json:"forbidden_at_1"`
ForbiddenAt5 float64 `json:"forbidden_at_5"`
}
// ToolSearchIntegrityMetrics verifies invariants that do not depend on
// relevance labels.
type ToolSearchIntegrityMetrics struct {
ResponsesEvaluated int `json:"responses_evaluated"`
CatalogBindingFailures int `json:"catalog_binding_failures"`
UnknownCandidateCount int `json:"unknown_candidate_count"`
IneligibleCandidateCount int `json:"ineligible_candidate_count"`
ResponseBudgetViolations int `json:"response_budget_violations"`
}
// ToolSearchTrustMetrics keeps contract integrity separate from ranking
// relevance. Search rank is never presented as a probability of safe use.
type ToolSearchTrustMetrics struct {
Identity ToolSearchIdentityTrustMetrics `json:"identity"`
Negative ToolSearchNegativeTrustMetrics `json:"negative"`
Integrity ToolSearchIntegrityMetrics `json:"integrity"`
}
// ToolSearchContextComparison compares byte-level JSON envelopes, not model
// tokens. The ideal navigation path assumes an oracle already knows the right
// product; its selection success is therefore deliberately not estimated.
type ToolSearchContextComparison struct {
ToolCount int `json:"tool_count"`
FullSchemaAllBytes int `json:"full_schema_all_bytes"`
OverviewBytes int `json:"overview_bytes"`
AverageProductBytes float64 `json:"average_product_bytes"`
AverageInspectBytes float64 `json:"average_inspect_bytes"`
AverageIdealSchemaNavigationBytes float64 `json:"average_ideal_schema_navigation_bytes"`
AverageSearchInspectBytes float64 `json:"average_search_inspect_bytes"`
ReductionVsFullSchema float64 `json:"reduction_vs_full_schema"`
ReductionVsIdealNavigation float64 `json:"reduction_vs_ideal_navigation"`
}
// ToolSearchDiagnosticComparison is generated from the exact typed Catalog
// embedded in the binary. It is a diagnostic proxy, not an Agent completion
// benchmark: its intent queries come from reviewed use_when prose and therefore
// are not independently authored qrels.
type ToolSearchDiagnosticComparison struct {
Version string `json:"version"`
Catalog CatalogVersionRef `json:"catalog"`
Algorithm string `json:"algorithm"`
TopK int `json:"top_k"`
IntentProxy ToolSearchRetrievalMetrics `json:"intent_proxy"`
IntentExcludedOverBudget int `json:"intent_excluded_over_budget"`
IntentLanguageSlices map[string]ToolSearchRetrievalMetrics `json:"intent_language_slices"`
WorkflowRaw ToolSearchWorkflowMetrics `json:"workflow_raw"`
WorkflowDecomposed ToolSearchWorkflowMetrics `json:"workflow_decomposed"`
Trust ToolSearchTrustMetrics `json:"trust"`
Context ToolSearchContextComparison `json:"context"`
AgentABStatus string `json:"agent_ab_status"`
Limitations []string `json:"limitations"`
}
// BuildDeliveryToolSearchDiagnosticComparison evaluates the shipped Go
// retriever against the declaration-assembled Catalog without reading a
// generated repository JSON artifact. Results belong under policy-tmp.
func BuildDeliveryToolSearchDiagnosticComparison(ctx context.Context, workflows []ToolSearchWorkflowEvaluationCase) (ToolSearchDiagnosticComparison, error) {
return BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(ctx, workflows, DefaultToolSearchConfig().LexicalAlgorithm)
}
// BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm runs one shipped Go
// lexical implementation over exactly the same typed Catalog and fixtures.
// This keeps shadow algorithm comparison out of the production default.
func BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(ctx context.Context, workflows []ToolSearchWorkflowEvaluationCase, algorithm string) (ToolSearchDiagnosticComparison, error) {
if err := deliverySchemaCatalogError(); err != nil {
return ToolSearchDiagnosticComparison{}, fmt.Errorf("assemble typed Schema registry for evaluation: %w", err)
}
loaded := deliverySchemaCatalog()
config := DefaultToolSearchConfig()
config.LexicalAlgorithm = algorithm
config.CatalogSourceHash = loaded.Snapshot.SourceHash
config.CatalogSurfaceHash = loaded.Snapshot.SurfaceHash
engine, err := NewToolSearchEngine(loaded.Registry, config)
if err != nil {
return ToolSearchDiagnosticComparison{}, err
}
allPayload, err := deliverySchemaAllPayload()
if err != nil {
return ToolSearchDiagnosticComparison{}, fmt.Errorf("render full Schema: %w", err)
}
overviewPayload, err := deliverySchemaOverviewPayload()
if err != nil {
return ToolSearchDiagnosticComparison{}, fmt.Errorf("render Schema overview: %w", err)
}
allBytes, err := compactJSONSize(allPayload)
if err != nil {
return ToolSearchDiagnosticComparison{}, err
}
overviewBytes, err := compactJSONSize(stripSchemaPayloadCompact(overviewPayload))
if err != nil {
return ToolSearchDiagnosticComparison{}, err
}
type proxyCase struct {
query string
gold string
product string
}
proxyCases := make([]proxyCase, 0)
intentExcludedOverBudget := 0
for _, product := range loaded.Registry.Products {
for _, tool := range product.Tools {
for _, query := range tool.Selection.UseWhen {
if query != "" && toolSearchQueryWithinBudget(query) {
proxyCases = append(proxyCases, proxyCase{query: query, gold: tool.Identity.CanonicalPath, product: product.ID})
} else if query != "" {
intentExcludedOverBudget++
}
}
}
}
sort.Slice(proxyCases, func(i, j int) bool {
if proxyCases[i].gold == proxyCases[j].gold {
return proxyCases[i].query < proxyCases[j].query
}
return proxyCases[i].gold < proxyCases[j].gold
})
intentAccumulator := &toolSearchRetrievalAccumulator{}
languageAccumulators := map[string]*toolSearchRetrievalAccumulator{}
var totalProductBytes, totalInspectBytes, totalSearchBytes float64
productSizes := make(map[string]int)
inspectSizes := make(map[string]int)
for _, item := range proxyCases {
if err := ctx.Err(); err != nil {
return ToolSearchDiagnosticComparison{}, err
}
response, searchErr := engine.Search(ctx, ToolSearchRequest{Query: item.query, Limit: 5, CandidateLimit: 20})
if searchErr != nil {
return ToolSearchDiagnosticComparison{}, fmt.Errorf("search proxy case %s: %w", item.gold, searchErr)
}
rank := 0
for index, candidate := range response.Candidates {
if candidate.CanonicalPath != item.gold {
continue
}
rank = index + 1
break
}
intentAccumulator.observe(rank, len(response.Candidates) == 0)
slice := toolSearchQueryLanguageSlice(item.query)
if languageAccumulators[slice] == nil {
languageAccumulators[slice] = &toolSearchRetrievalAccumulator{}
}
languageAccumulators[slice].observe(rank, len(response.Candidates) == 0)
encodedResponse, marshalErr := json.Marshal(response)
if marshalErr != nil {
return ToolSearchDiagnosticComparison{}, marshalErr
}
totalSearchBytes += float64(len(encodedResponse))
productBytes, ok := productSizes[item.product]
if !ok {
payload, payloadErr := queryDeliverySchemaPayload([]string{item.product})
if payloadErr != nil {
return ToolSearchDiagnosticComparison{}, payloadErr
}
productBytes, payloadErr = compactJSONSize(stripSchemaPayloadCompact(payload))
if payloadErr != nil {
return ToolSearchDiagnosticComparison{}, payloadErr
}
productSizes[item.product] = productBytes
}
totalProductBytes += float64(productBytes)
inspectBytes, ok := inspectSizes[item.gold]
if !ok {
payload, payloadErr := queryDeliverySchemaPayload([]string{item.gold})
if payloadErr != nil {
return ToolSearchDiagnosticComparison{}, payloadErr
}
payload = stripSchemaPayloadCompact(payload)
inspectBytes, payloadErr = compactJSONSize(SchemaInspectV1Response{
Version: "schema-inspect.v1",
Catalog: CatalogVersionRef{SourceHash: loaded.Snapshot.SourceHash, SurfaceHash: loaded.Snapshot.SurfaceHash},
ToolSpec: payload,
})
if payloadErr != nil {
return ToolSearchDiagnosticComparison{}, payloadErr
}
inspectSizes[item.gold] = inspectBytes
}
totalInspectBytes += float64(inspectBytes)
}
count := float64(len(proxyCases))
intent := intentAccumulator.metrics()
languageSlices := make(map[string]ToolSearchRetrievalMetrics, len(languageAccumulators))
for slice, accumulator := range languageAccumulators {
languageSlices[slice] = accumulator.metrics()
}
contextComparison := ToolSearchContextComparison{
ToolCount: len(loaded.Index.CanonicalPaths()),
FullSchemaAllBytes: allBytes,
OverviewBytes: overviewBytes,
}
if count > 0 {
contextComparison.AverageProductBytes = totalProductBytes / count
contextComparison.AverageInspectBytes = totalInspectBytes / count
contextComparison.AverageIdealSchemaNavigationBytes = float64(overviewBytes) + contextComparison.AverageProductBytes + contextComparison.AverageInspectBytes
contextComparison.AverageSearchInspectBytes = totalSearchBytes/count + contextComparison.AverageInspectBytes
contextComparison.ReductionVsFullSchema = reduction(float64(allBytes), contextComparison.AverageSearchInspectBytes)
contextComparison.ReductionVsIdealNavigation = reduction(contextComparison.AverageIdealSchemaNavigationBytes, contextComparison.AverageSearchInspectBytes)
}
rawWorkflow, decomposedWorkflow, err := evaluateToolSearchWorkflows(ctx, engine, workflows)
if err != nil {
return ToolSearchDiagnosticComparison{}, err
}
trust, err := evaluateToolSearchTrust(ctx, engine, loaded.Registry)
if err != nil {
return ToolSearchDiagnosticComparison{}, err
}
return ToolSearchDiagnosticComparison{
Version: "tool-search-diagnostic-comparison.v1",
Catalog: engine.catalogVersion(),
Algorithm: engine.lexical.Name(),
TopK: 5,
IntentProxy: intent,
IntentExcludedOverBudget: intentExcludedOverBudget,
IntentLanguageSlices: languageSlices,
WorkflowRaw: rawWorkflow,
WorkflowDecomposed: decomposedWorkflow,
Trust: trust,
Context: contextComparison,
AgentABStatus: "not_run_requires_independent_tasks_and_model_runs",
Limitations: []string{
"intent_proxy queries are reviewed use_when prose from the same Catalog and are not independent qrels",
"selection prose that exceeds the production query byte or rune budget is counted and excluded rather than weakening runtime limits",
"direct Schema navigation bytes assume an oracle selects the correct product and do not estimate model selection success",
"search plus inspect bytes also inspect the gold leaf even when retrieval misses and are an oracle-assisted capacity upper bound, not observed Agent cost",
"workflow decomposition uses reviewed fixture subqueries and is an upper-bound diagnostic",
"negative queries are same-source avoid_when prose without alternative gold and measure candidate exposure rather than Agent calls",
"byte counts are compact JSON bytes rather than provider-specific tokenizer counts",
"true Agent task success, unsafe action rate, recovery success, latency, and token cost require paired Agent A/B runs",
},
}, nil
}
type toolSearchRetrievalAccumulator struct {
cases int
hitAt1 int
hitAt5 int
reciprocal float64
zeroResults int
}
func (a *toolSearchRetrievalAccumulator) observe(rank int, zeroResults bool) {
a.cases++
if zeroResults {
a.zeroResults++
}
if rank == 1 {
a.hitAt1++
}
if rank > 0 && rank <= 5 {
a.hitAt5++
a.reciprocal += 1 / float64(rank)
}
}
func (a *toolSearchRetrievalAccumulator) metrics() ToolSearchRetrievalMetrics {
result := ToolSearchRetrievalMetrics{Cases: a.cases}
if a.cases == 0 {
return result
}
denominator := float64(a.cases)
result.RecallAt1 = float64(a.hitAt1) / denominator
result.RecallAt5 = float64(a.hitAt5) / denominator
result.MeanReciprocalRankAt5 = a.reciprocal / denominator
result.ZeroResultRate = float64(a.zeroResults) / denominator
return result
}
func toolSearchQueryLanguageSlice(query string) string {
hasChinese := false
hasASCIIWord := false
for _, character := range query {
if unicode.Is(unicode.Han, character) {
hasChinese = true
}
if character <= unicode.MaxASCII && (unicode.IsLetter(character) || unicode.IsDigit(character)) {
hasASCIIWord = true
}
}
if hasChinese && hasASCIIWord {
return "mixed_chinese_ascii"
}
if hasChinese {
return "chinese_only"
}
return "non_chinese"
}
func evaluateToolSearchTrust(ctx context.Context, engine *ToolSearchEngine, registry SchemaRegistry) (ToolSearchTrustMetrics, error) {
result := ToolSearchTrustMetrics{}
var canonicalPassed, cliPassed, aliasPassed, nfkcPassed, filteredPassed int
var forbiddenAt1, forbiddenAt5 int
for _, product := range registry.Products {
for _, tool := range product.Tools {
if err := ctx.Err(); err != nil {
return result, err
}
result.Identity.CanonicalCases++
response, err := engine.Search(ctx, ToolSearchRequest{Query: tool.Identity.CanonicalPath})
if err != nil {
return result, fmt.Errorf("audit exact canonical %s: %w", tool.Identity.CanonicalPath, err)
}
observeToolSearchIntegrity(&result.Integrity, engine, response)
if exactSearchMatches(response, tool.Identity.CanonicalPath) {
canonicalPassed++
}
if cliPath := tool.Identity.PrimaryCLIPath; cliPath != "" {
result.Identity.PrimaryCLICases++
response, err = engine.Search(ctx, ToolSearchRequest{Query: cliPath})
if err != nil {
return result, fmt.Errorf("audit exact CLI %s: %w", cliPath, err)
}
observeToolSearchIntegrity(&result.Integrity, engine, response)
if exactSearchMatches(response, tool.Identity.CanonicalPath) {
cliPassed++
}
}
for _, alias := range tool.Identity.Aliases {
result.Identity.AliasCases++
response, err = engine.Search(ctx, ToolSearchRequest{Query: alias})
if err != nil {
return result, fmt.Errorf("audit exact alias %s: %w", alias, err)
}
observeToolSearchIntegrity(&result.Integrity, engine, response)
if exactSearchMatches(response, tool.Identity.CanonicalPath) {
aliasPassed++
}
}
nfkcQuery := toolSearchFullwidthASCII(tool.Identity.CanonicalPath)
if nfkcQuery != tool.Identity.CanonicalPath {
result.Identity.NFKCCases++
response, err = engine.Search(ctx, ToolSearchRequest{Query: nfkcQuery})
if err != nil {
return result, fmt.Errorf("audit NFKC identity %s: %w", tool.Identity.CanonicalPath, err)
}
observeToolSearchIntegrity(&result.Integrity, engine, response)
if exactSearchMatches(response, tool.Identity.CanonicalPath) {
nfkcPassed++
}
}
result.Identity.ExactFilteredCases++
response, err = engine.Search(ctx, ToolSearchRequest{
Query: tool.Identity.CanonicalPath,
ExcludeCanonicalPaths: []string{tool.Identity.CanonicalPath},
})
if err != nil {
return result, fmt.Errorf("audit exact filtered %s: %w", tool.Identity.CanonicalPath, err)
}
observeToolSearchIntegrity(&result.Integrity, engine, response)
if response.Strategy == "exact_filtered" && response.Abstained && len(response.Candidates) == 0 && response.ExactFiltered != nil && response.ExactFiltered.CanonicalPath == tool.Identity.CanonicalPath && response.ExactFiltered.Reason == "excluded" {
filteredPassed++
}
for _, query := range tool.Selection.AvoidWhen {
if query == "" {
continue
}
if !toolSearchQueryWithinBudget(query) {
result.Negative.ExcludedOverBudget++
continue
}
result.Negative.Cases++
response, err = engine.Search(ctx, ToolSearchRequest{Query: query, Limit: 5, CandidateLimit: 20})
if err != nil {
return result, fmt.Errorf("audit negative %s: %w", tool.Identity.CanonicalPath, err)
}
observeToolSearchIntegrity(&result.Integrity, engine, response)
for rank, candidate := range response.Candidates {
if candidate.CanonicalPath != tool.Identity.CanonicalPath {
continue
}
if rank == 0 {
forbiddenAt1++
}
forbiddenAt5++
break
}
}
}
}
if result.Identity.CanonicalCases > 0 {
result.Identity.CanonicalPassRate = float64(canonicalPassed) / float64(result.Identity.CanonicalCases)
}
if result.Identity.PrimaryCLICases > 0 {
result.Identity.PrimaryCLIPassRate = float64(cliPassed) / float64(result.Identity.PrimaryCLICases)
}
if result.Identity.AliasCases > 0 {
result.Identity.AliasPassRate = float64(aliasPassed) / float64(result.Identity.AliasCases)
}
if result.Identity.NFKCCases > 0 {
result.Identity.NFKCPassRate = float64(nfkcPassed) / float64(result.Identity.NFKCCases)
}
if result.Identity.ExactFilteredCases > 0 {
result.Identity.ExactFilteredPassRate = float64(filteredPassed) / float64(result.Identity.ExactFilteredCases)
}
if result.Negative.Cases > 0 {
result.Negative.ForbiddenAt1 = float64(forbiddenAt1) / float64(result.Negative.Cases)
result.Negative.ForbiddenAt5 = float64(forbiddenAt5) / float64(result.Negative.Cases)
}
return result, nil
}
func toolSearchFullwidthASCII(value string) string {
runes := []rune(value)
for index, character := range runes {
if character >= '!' && character <= '~' {
runes[index] = character + 0xFEE0
}
}
return string(runes)
}
func toolSearchQueryWithinBudget(query string) bool {
return len(query) <= maxToolSearchQueryBytes && utf8.RuneCountInString(query) <= maxToolSearchQueryRunes
}
func exactSearchMatches(response ToolSearchResponse, canonical string) bool {
return response.Strategy == "exact_guard" && len(response.Candidates) == 1 && response.Candidates[0].CanonicalPath == canonical && response.Candidates[0].RequiresInspect
}
func observeToolSearchIntegrity(metrics *ToolSearchIntegrityMetrics, engine *ToolSearchEngine, response ToolSearchResponse) {
metrics.ResponsesEvaluated++
if response.Catalog != engine.catalogVersion() {
metrics.CatalogBindingFailures++
}
if encoded, err := json.Marshal(response); err != nil || len(encoded) > maxToolSearchResponseBytes {
metrics.ResponseBudgetViolations++
}
for _, candidate := range response.Candidates {
document, ok := engine.documents[candidate.CanonicalPath]
if !ok {
metrics.UnknownCandidateCount++
continue
}
if !toolSearchEligible(document.tool, nil, nil, nil) {
metrics.IneligibleCandidateCount++
}
}
}
func evaluateToolSearchWorkflows(ctx context.Context, engine *ToolSearchEngine, workflows []ToolSearchWorkflowEvaluationCase) (ToolSearchWorkflowMetrics, ToolSearchWorkflowMetrics, error) {
raw := ToolSearchWorkflowMetrics{Cases: len(workflows)}
decomposed := ToolSearchWorkflowMetrics{Cases: len(workflows)}
if len(workflows) == 0 {
return raw, decomposed, nil
}
for _, workflow := range workflows {
if workflow.ID == "" || workflow.Query == "" || len(workflow.Required) == 0 {
return raw, decomposed, fmt.Errorf("workflow evaluation case must have id, query and required tools")
}
one, err := engine.Search(ctx, ToolSearchRequest{Query: workflow.Query, Limit: 5, CandidateLimit: 20})
if err != nil {
return raw, decomposed, fmt.Errorf("search workflow %s: %w", workflow.ID, err)
}
many, err := engine.SearchSubqueries(ctx, workflow.Subqueries, ToolSearchRequest{Query: workflow.Query, Limit: 5, CandidateLimit: 20})
if err != nil {
return raw, decomposed, fmt.Errorf("search decomposed workflow %s: %w", workflow.ID, err)
}
accumulateWorkflowMetrics(&raw, one.Candidates, workflow.Required)
accumulateWorkflowMetrics(&decomposed, many.Candidates, workflow.Required)
}
denominator := float64(len(workflows))
raw.CompleteAt5 /= denominator
raw.RequiredRecallAt5 /= denominator
decomposed.CompleteAt5 /= denominator
decomposed.RequiredRecallAt5 /= denominator
return raw, decomposed, nil
}
func accumulateWorkflowMetrics(metrics *ToolSearchWorkflowMetrics, candidates []ToolReference, required []string) {
found := make(map[string]bool, len(candidates))
for _, candidate := range candidates {
found[candidate.CanonicalPath] = true
}
hits := 0
for _, canonical := range required {
if found[canonical] {
hits++
}
}
metrics.RequiredRecallAt5 += float64(hits) / float64(len(required))
if hits == len(required) {
metrics.CompleteAt5++
}
}
func compactJSONSize(value any) (int, error) {
encoded, err := json.Marshal(value)
if err != nil {
return 0, fmt.Errorf("marshal compact JSON: %w", err)
}
return len(encoded), nil
}
func reduction(baseline, candidate float64) float64 {
if baseline <= 0 {
return 0
}
return math.Max(-1, math.Min(1, 1-candidate/baseline))
}
@@ -0,0 +1,98 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cli
import "testing"
func TestAggregateToolSearchAgentAB(t *testing.T) {
catalog := CatalogVersionRef{SourceHash: "sha256:source", SurfaceHash: "sha256:surface"}
input := ToolSearchAgentABInput{
Version: "tool-search-agent-ab.v1",
Catalog: catalog,
Runs: []ToolSearchAgentABRun{
{CaseID: "a", Trial: 0, Arm: ToolSearchAgentArmDirectSchema, CorrectToolPlan: true, ContextTokens: 100, ToolCalls: 3, LatencyMS: 1000},
{CaseID: "a", Trial: 0, Arm: ToolSearchAgentArmSearchInspect, TaskCompleted: true, CorrectToolPlan: true, ContextTokens: 40, ToolCalls: 2, LatencyMS: 800},
{CaseID: "b", Trial: 0, Arm: ToolSearchAgentArmDirectSchema, TaskCompleted: true, CorrectToolPlan: true, UnsafeAction: true, RecoveryAttempted: true, ContextTokens: 120, ToolCalls: 4, LatencyMS: 1200},
{CaseID: "b", Trial: 0, Arm: ToolSearchAgentArmSearchInspect, TaskCompleted: true, CorrectToolPlan: true, RecoveryAttempted: true, RecoverySucceeded: true, ContextTokens: 50, ToolCalls: 3, LatencyMS: 900},
},
}
report, err := AggregateToolSearchAgentAB(input)
if err != nil {
t.Fatalf("AggregateToolSearchAgentAB() error = %v", err)
}
if report.Cases != 2 || report.PairedTrials != 2 {
t.Fatalf("pair counts = cases %d trials %d", report.Cases, report.PairedTrials)
}
if report.DirectSchema.TaskSuccessRate != 0.5 || report.SearchInspect.TaskSuccessRate != 1 {
t.Fatalf("unexpected success rates: direct=%v search=%v", report.DirectSchema.TaskSuccessRate, report.SearchInspect.TaskSuccessRate)
}
if delta := report.Deltas["context_tokens"].Delta; delta != -65 {
t.Fatalf("context delta = %v, want -65", delta)
}
if _, ok := report.Deltas["recovery_success_rate_paired_attempts"]; !ok {
t.Fatal("paired recovery delta missing")
}
}
func TestAggregateToolSearchAgentABRejectsUnpairedRuns(t *testing.T) {
_, err := AggregateToolSearchAgentAB(ToolSearchAgentABInput{
Version: "tool-search-agent-ab.v1",
Catalog: CatalogVersionRef{SourceHash: "source", SurfaceHash: "surface"},
Runs: []ToolSearchAgentABRun{
{CaseID: "a", Arm: ToolSearchAgentArmDirectSchema},
},
})
if err == nil {
t.Fatal("AggregateToolSearchAgentAB() accepted an unpaired run")
}
}
func TestScoreToolSearchAgentPlanningAB(t *testing.T) {
workflows := []ToolSearchWorkflowEvaluationCase{
{ID: "one", Required: []string{"chat.search_groups", "chat.send_personal_message"}},
{ID: "two", Required: []string{"drive.upload", "drive.add_permission"}},
}
direct := ToolSearchAgentPlanOutput{Results: []ToolSearchAgentPlanResult{
{ID: "one", CanonicalPaths: []string{"chat.search_groups", "chat.send_personal_message"}},
{ID: "two", CanonicalPaths: []string{"drive.upload", "drive.add_permission"}},
}}
search := ToolSearchAgentPlanOutput{Results: []ToolSearchAgentPlanResult{
{ID: "one", CanonicalPaths: []string{"chat.list_owned_or_admin_groups", "chat.search_groups", "chat.send_personal_message"}},
{ID: "two", CanonicalPaths: []string{"drive.upload", "drive.add_permission"}},
}}
tools := []ToolSpec{
toolSearchTestTool("chat", "search_groups", "chat search-groups", "搜索群聊", "read"),
toolSearchTestTool("chat", "send_personal_message", "chat message send", "发送消息", "write"),
toolSearchTestTool("drive", "upload", "drive upload", "上传文件", "write"),
toolSearchTestTool("drive", "add_permission", "drive permission add", "添加权限", "write"),
toolSearchTestTool("chat", "list_owned_or_admin_groups", "chat list-groups", "列出群聊", "read"),
}
index, indexErr := (SchemaRegistry{Kind: "schema", Level: "catalog", Products: []ProductSpec{
{ID: "chat", Tools: []ToolSpec{tools[0], tools[1], tools[4]}},
{ID: "drive", Tools: tools[2:4]},
}}).Index()
if indexErr != nil {
t.Fatal(indexErr)
}
report, err := scoreToolSearchAgentPlanningAB(index, "test-model", workflows, direct, search)
if err != nil {
t.Fatalf("ScoreToolSearchAgentPlanningAB() error = %v", err)
}
if report.DirectSchema.ExactMinimalRate != 1 || report.SearchInspect.ExactMinimalRate != 0.5 {
t.Fatalf("minimal rates = direct %v search %v", report.DirectSchema.ExactMinimalRate, report.SearchInspect.ExactMinimalRate)
}
if report.SearchInspect.CompleteRate != 1 || report.SearchInspect.UnnecessaryTools != 1 {
t.Fatalf("search metrics = %+v", report.SearchInspect)
}
}
@@ -0,0 +1,357 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
package cli
import (
"context"
"fmt"
"math"
"math/rand"
"sort"
"strings"
)
// The product-cluster paired bootstrap uses a fixed sample count and seed so a
// sealed qrels run stays reproducible; both are published in the report so an
// auditor does not have to read them out of the source.
const (
toolSearchIndependentBootstrapSamples = 10_000
toolSearchIndependentBootstrapSeed = 20260813
)
// ToolSearchIndependentQrel is an independently authored graded relevance
// judgment. Relevance uses the frozen 1..3 qrels scale.
type ToolSearchIndependentQrel struct {
Canonical string `json:"canonical"`
Relevance int `json:"relevance"`
}
type ToolSearchIndependentWorkflow struct {
Subqueries []string `json:"subqueries,omitempty"`
Required []string `json:"required"`
}
type ToolSearchIndependentCase struct {
ID string `json:"id"`
Query string `json:"query"`
Language string `json:"language"`
Qrels []ToolSearchIndependentQrel `json:"qrels"`
Forbidden []string `json:"forbidden,omitempty"`
AlternativeGold []string `json:"alternative_gold,omitempty"`
Workflow *ToolSearchIndependentWorkflow `json:"workflow,omitempty"`
ConfusionFamily []string `json:"confusion_family,omitempty"`
}
type ToolSearchIndependentRankingMetrics struct {
Cases int `json:"cases"`
RecallAt1 float64 `json:"recall_at_1"`
RecallAt5 float64 `json:"recall_at_5"`
MeanReciprocalRankAt5 float64 `json:"mean_reciprocal_rank_at_5"`
NDCGAt5 float64 `json:"ndcg_at_5"`
ZeroResultRate float64 `json:"zero_result_rate"`
}
type ToolSearchIndependentSafetyMetrics struct {
ForbiddenCases int `json:"forbidden_cases"`
ForbiddenExposureAt5 float64 `json:"forbidden_exposure_at_5"`
AlternativeRecallAt5 float64 `json:"alternative_recall_at_5"`
SiblingConfusionCases int `json:"sibling_confusion_cases"`
SiblingExposureAt5 float64 `json:"sibling_exposure_at_5"`
}
type ToolSearchIndependentReport struct {
Version string `json:"version"`
Catalog CatalogVersionRef `json:"catalog"`
Algorithm string `json:"algorithm"`
ControlAlgorithm string `json:"control_algorithm"`
Overall ToolSearchIndependentRankingMetrics `json:"overall"`
ControlOverall ToolSearchIndependentRankingMetrics `json:"control_overall"`
RecallAt5Delta float64 `json:"recall_at_5_delta"`
ProductClusterCI95 ToolSearchConfidenceInterval `json:"product_cluster_recall_at_5_delta_ci_95"`
LanguageSlices map[string]ToolSearchIndependentRankingMetrics `json:"language_slices"`
Safety ToolSearchIndependentSafetyMetrics `json:"safety"`
Workflow ToolSearchWorkflowMetrics `json:"workflow"`
// BootstrapSamples/BootstrapSeed are set only by the paired delivery
// evaluation that computes ProductClusterCI95; they make that interval
// reproducible without reading the source.
BootstrapSamples int `json:"bootstrap_samples,omitempty"`
BootstrapSeed int64 `json:"bootstrap_seed,omitempty"`
}
type ToolSearchConfidenceInterval struct {
Lower float64 `json:"lower"`
Upper float64 `json:"upper"`
}
type independentRankingAccumulator struct {
metrics ToolSearchIndependentRankingMetrics
recall1 float64
recall5 float64
mrr5 float64
ndcg5 float64
zero int
}
// BuildDeliveryToolSearchIndependentEvaluation evaluates the action-ranking
// candidate against the shipped fielded BM25 control on sealed external qrels.
// It never tunes parameters or reads Catalog JSON; the caller owns qrels
// sealing and threshold enforcement. The candidate may become the runtime
// default only after the independently frozen default-switch gate passes.
func BuildDeliveryToolSearchIndependentEvaluation(ctx context.Context, cases []ToolSearchIndependentCase) (ToolSearchIndependentReport, error) {
if err := deliverySchemaCatalogError(); err != nil {
return ToolSearchIndependentReport{}, err
}
loaded := deliverySchemaCatalog()
candidateConfig := DefaultToolSearchConfig()
candidateConfig.LexicalAlgorithm = ToolSearchLexicalBM25Action
candidateConfig.CatalogSourceHash = loaded.Snapshot.SourceHash
candidateConfig.CatalogSurfaceHash = loaded.Snapshot.SurfaceHash
engine, err := NewToolSearchEngine(loaded.Registry, candidateConfig)
if err != nil {
return ToolSearchIndependentReport{}, err
}
report, err := EvaluateToolSearchIndependent(ctx, engine, cases)
if err != nil {
return ToolSearchIndependentReport{}, err
}
controlConfig := DefaultToolSearchConfig()
controlConfig.CatalogSourceHash = loaded.Snapshot.SourceHash
controlConfig.CatalogSurfaceHash = loaded.Snapshot.SurfaceHash
control, err := NewToolSearchEngine(loaded.Registry, controlConfig)
if err != nil {
return ToolSearchIndependentReport{}, err
}
controlReport, err := EvaluateToolSearchIndependent(ctx, control, cases)
if err != nil {
return ToolSearchIndependentReport{}, err
}
report.ControlAlgorithm = control.lexical.Name()
report.ControlOverall = controlReport.Overall
report.RecallAt5Delta = report.Overall.RecallAt5 - report.ControlOverall.RecallAt5
report.ProductClusterCI95, err = pairedProductClusterRecallCI(ctx, engine, control, cases, toolSearchIndependentBootstrapSamples, toolSearchIndependentBootstrapSeed)
if err != nil {
return ToolSearchIndependentReport{}, err
}
report.BootstrapSamples = toolSearchIndependentBootstrapSamples
report.BootstrapSeed = toolSearchIndependentBootstrapSeed
return report, nil
}
func EvaluateToolSearchIndependent(ctx context.Context, engine *ToolSearchEngine, cases []ToolSearchIndependentCase) (ToolSearchIndependentReport, error) {
if engine == nil || len(cases) == 0 {
return ToolSearchIndependentReport{}, fmt.Errorf("independent evaluation requires an engine and non-empty cases")
}
overall := &independentRankingAccumulator{}
slices := map[string]*independentRankingAccumulator{}
var forbiddenCases, forbiddenExposed int
var siblingCases, siblingExposed int
var alternativeRecall float64
workflowCases := 0
workflowComplete := 0
workflowRequiredRecall := 0.0
for _, item := range cases {
if err := ctx.Err(); err != nil {
return ToolSearchIndependentReport{}, err
}
response, searchErr := engine.Search(ctx, ToolSearchRequest{Query: item.Query, Limit: 5, CandidateLimit: 20})
if searchErr != nil {
return ToolSearchIndependentReport{}, fmt.Errorf("evaluate %s: %w", item.ID, searchErr)
}
overall.observe(response.Candidates, item.Qrels)
if slices[item.Language] == nil {
slices[item.Language] = &independentRankingAccumulator{}
}
slices[item.Language].observe(response.Candidates, item.Qrels)
if len(item.Forbidden) > 0 {
forbiddenCases++
if candidatesContainAny(response.Candidates, item.Forbidden) {
forbiddenExposed++
}
alternativeRecall += candidateSetRecall(response.Candidates, item.AlternativeGold)
}
if len(item.ConfusionFamily) > 0 {
siblingCases++
if candidatesContainAny(response.Candidates, item.ConfusionFamily) {
siblingExposed++
}
}
if item.Workflow != nil {
workflowCases++
workflowResponse := response
if len(item.Workflow.Subqueries) > 0 {
workflowResponse, searchErr = engine.SearchSubqueries(ctx, item.Workflow.Subqueries, ToolSearchRequest{Query: item.Query, Limit: 5, CandidateLimit: 20})
if searchErr != nil {
return ToolSearchIndependentReport{}, fmt.Errorf("evaluate workflow %s: %w", item.ID, searchErr)
}
}
recall := candidateSetRecall(workflowResponse.Candidates, item.Workflow.Required)
workflowRequiredRecall += recall
if recall == 1 {
workflowComplete++
}
}
}
report := ToolSearchIndependentReport{
Version: "tool-search-independent-evaluation.v1", Catalog: engine.catalogVersion(), Algorithm: engine.lexical.Name(),
Overall: overall.result(), LanguageSlices: map[string]ToolSearchIndependentRankingMetrics{},
Safety: ToolSearchIndependentSafetyMetrics{ForbiddenCases: forbiddenCases, SiblingConfusionCases: siblingCases},
Workflow: ToolSearchWorkflowMetrics{Cases: workflowCases},
}
for name, accumulator := range slices {
report.LanguageSlices[name] = accumulator.result()
}
if forbiddenCases > 0 {
report.Safety.ForbiddenExposureAt5 = float64(forbiddenExposed) / float64(forbiddenCases)
report.Safety.AlternativeRecallAt5 = alternativeRecall / float64(forbiddenCases)
}
if siblingCases > 0 {
report.Safety.SiblingExposureAt5 = float64(siblingExposed) / float64(siblingCases)
}
if workflowCases > 0 {
report.Workflow.CompleteAt5 = float64(workflowComplete) / float64(workflowCases)
report.Workflow.RequiredRecallAt5 = workflowRequiredRecall / float64(workflowCases)
}
return report, nil
}
func pairedProductClusterRecallCI(ctx context.Context, candidate, control *ToolSearchEngine, cases []ToolSearchIndependentCase, iterations int, seed int64) (ToolSearchConfidenceInterval, error) {
clusters := map[string][]float64{}
for _, item := range cases {
candidateResponse, err := candidate.Search(ctx, ToolSearchRequest{Query: item.Query, Limit: 5, CandidateLimit: 20})
if err != nil {
return ToolSearchConfidenceInterval{}, err
}
controlResponse, err := control.Search(ctx, ToolSearchRequest{Query: item.Query, Limit: 5, CandidateLimit: 20})
if err != nil {
return ToolSearchConfidenceInterval{}, err
}
product := "unknown"
if len(item.Qrels) > 0 {
if split := strings.SplitN(item.Qrels[0].Canonical, ".", 2); len(split) > 0 {
product = split[0]
}
}
clusters[product] = append(clusters[product], candidateSetRecall(candidateResponse.Candidates, qrelCanonicals(item.Qrels))-candidateSetRecall(controlResponse.Candidates, qrelCanonicals(item.Qrels)))
}
clusterNames := make([]string, 0, len(clusters))
clusterMeans := map[string]float64{}
for name, values := range clusters {
clusterNames = append(clusterNames, name)
for _, value := range values {
clusterMeans[name] += value
}
clusterMeans[name] /= float64(len(values))
}
sort.Strings(clusterNames)
if len(clusterNames) == 0 || iterations <= 0 {
return ToolSearchConfidenceInterval{}, fmt.Errorf("paired product bootstrap requires cases and iterations")
}
random := rand.New(rand.NewSource(seed)) // #nosec G404 -- deterministic evaluation bootstrap
values := make([]float64, iterations)
for iteration := range values {
for sample := 0; sample < len(clusterNames); sample++ {
values[iteration] += clusterMeans[clusterNames[random.Intn(len(clusterNames))]]
}
values[iteration] /= float64(len(clusterNames))
}
sort.Float64s(values)
return ToolSearchConfidenceInterval{Lower: percentile(values, 0.025), Upper: percentile(values, 0.975)}, nil
}
func qrelCanonicals(qrels []ToolSearchIndependentQrel) []string {
values := make([]string, 0, len(qrels))
for _, qrel := range qrels {
values = append(values, qrel.Canonical)
}
return values
}
func (a *independentRankingAccumulator) observe(candidates []ToolReference, qrels []ToolSearchIndependentQrel) {
a.metrics.Cases++
if len(candidates) == 0 {
a.zero++
}
relevance := make(map[string]int, len(qrels))
ideal := make([]int, 0, len(qrels))
for _, qrel := range qrels {
relevance[qrel.Canonical] = qrel.Relevance
ideal = append(ideal, qrel.Relevance)
}
found := 0
first := 0
dcg := 0.0
for rank, candidate := range candidates {
if rank >= 5 {
break
}
rel := relevance[candidate.CanonicalPath]
if rel == 0 {
continue
}
found++
if first == 0 {
first = rank + 1
}
dcg += (math.Pow(2, float64(rel)) - 1) / math.Log2(float64(rank+2))
}
if len(candidates) > 0 && relevance[candidates[0].CanonicalPath] > 0 {
a.recall1 += 1 / float64(len(qrels))
}
if len(qrels) > 0 {
a.recall5 += float64(found) / float64(len(qrels))
}
if first > 0 {
a.mrr5 += 1 / float64(first)
}
sort.Sort(sort.Reverse(sort.IntSlice(ideal)))
idcg := 0.0
for rank, rel := range ideal {
if rank >= 5 {
break
}
idcg += (math.Pow(2, float64(rel)) - 1) / math.Log2(float64(rank+2))
}
if idcg > 0 {
a.ndcg5 += dcg / idcg
}
}
func (a *independentRankingAccumulator) result() ToolSearchIndependentRankingMetrics {
result := a.metrics
if result.Cases == 0 {
return result
}
denominator := float64(result.Cases)
result.RecallAt1 = a.recall1 / denominator
result.RecallAt5 = a.recall5 / denominator
result.MeanReciprocalRankAt5 = a.mrr5 / denominator
result.NDCGAt5 = a.ndcg5 / denominator
result.ZeroResultRate = float64(a.zero) / denominator
return result
}
func candidatesContainAny(candidates []ToolReference, paths []string) bool {
for _, candidate := range candidates {
if stringSliceContains(paths, candidate.CanonicalPath) {
return true
}
}
return false
}
func candidateSetRecall(candidates []ToolReference, required []string) float64 {
if len(required) == 0 {
return 0
}
found := 0
for _, canonical := range required {
for _, candidate := range candidates {
if candidate.CanonicalPath == canonical {
found++
break
}
}
}
return float64(found) / float64(len(required))
}
@@ -0,0 +1,70 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
package cli
import (
"context"
"math"
"testing"
)
func TestEvaluateToolSearchIndependentScoresGradedSafetyAndWorkflow(t *testing.T) {
engine := newToolSearchTestEngine(t)
cases := []ToolSearchIndependentCase{
{
ID: "chat", Query: "给群里发送消息", Language: "chinese_only",
Qrels: []ToolSearchIndependentQrel{{Canonical: "chat.send", Relevance: 3}},
Forbidden: []string{"doc.read"}, AlternativeGold: []string{"chat.send"},
ConfusionFamily: []string{"chat.read_status"},
},
{
ID: "workflow", Query: "发送消息并读取文档", Language: "chinese_only",
Qrels: []ToolSearchIndependentQrel{{Canonical: "chat.send", Relevance: 3}, {Canonical: "doc.read", Relevance: 2}},
Workflow: &ToolSearchIndependentWorkflow{
Subqueries: []string{"群聊发送消息", "读取在线文档"},
Required: []string{"chat.send", "doc.read"},
},
},
}
report, err := EvaluateToolSearchIndependent(context.Background(), engine, cases)
if err != nil {
t.Fatal(err)
}
if report.Version != "tool-search-independent-evaluation.v1" || report.Overall.Cases != 2 {
t.Fatalf("report = %#v", report)
}
if report.Overall.RecallAt5 != 1 || report.Overall.NDCGAt5 <= 0 || report.Overall.MeanReciprocalRankAt5 <= 0 {
t.Fatalf("ranking = %#v", report.Overall)
}
if report.Safety.ForbiddenCases != 1 || report.Safety.ForbiddenExposureAt5 != 0 || report.Safety.AlternativeRecallAt5 != 1 ||
report.Safety.SiblingConfusionCases != 1 || report.Safety.SiblingExposureAt5 != 1 {
t.Fatalf("safety = %#v", report.Safety)
}
if report.Workflow.Cases != 1 || report.Workflow.CompleteAt5 != 1 || report.Workflow.RequiredRecallAt5 != 1 {
t.Fatalf("workflow = %#v", report.Workflow)
}
}
func TestIndependentRankingMetricsUseGradedRankDiscounts(t *testing.T) {
accumulator := &independentRankingAccumulator{}
accumulator.observe([]ToolReference{
{CanonicalPath: "irrelevant"},
{CanonicalPath: "relevant.low"},
}, []ToolSearchIndependentQrel{
{Canonical: "relevant.high", Relevance: 3},
{Canonical: "relevant.low", Relevance: 2},
})
metrics := accumulator.result()
wantNDCG := (3 / math.Log2(3)) / (7 + 3/math.Log2(3))
if metrics.RecallAt1 != 0 || metrics.RecallAt5 != 0.5 || metrics.MeanReciprocalRankAt5 != 0.5 || math.Abs(metrics.NDCGAt5-wantNDCG) > 1e-12 || metrics.ZeroResultRate != 0 {
t.Fatalf("metrics = %#v, want R1=0 R5=.5 MRR5=.5 NDCG5=%.12f", metrics, wantNDCG)
}
zero := &independentRankingAccumulator{}
zero.observe(nil, []ToolSearchIndependentQrel{{Canonical: "relevant", Relevance: 3}})
zeroMetrics := zero.result()
if zeroMetrics.ZeroResultRate != 1 || zeroMetrics.RecallAt5 != 0 || zeroMetrics.NDCGAt5 != 0 {
t.Fatalf("zero metrics = %#v", zeroMetrics)
}
}
+662
View File
@@ -0,0 +1,662 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
package cli
import (
"context"
"fmt"
"math"
"regexp"
"sort"
"strings"
"unicode/utf8"
)
const (
ToolSearchLexicalBM25 = "fielded_bm25_ensemble"
ToolSearchLexicalBM25Action = "fielded_bm25_action_v1"
ToolSearchLexicalTFIDF = "fielded_tfidf_cosine"
)
// ToolSearchLexicalRequest contains a prefiltered Catalog domain. Hard
// filtering occurs before scoring, so a retriever cannot spend its Top-K on
// candidates that DWS is required to reject.
type ToolSearchLexicalRequest struct {
Query string
CandidateLimit int
EligibleCanonicalPaths []string
}
// LexicalHit is an internal ordering result, not a confidence probability.
// Explain is populated only when the engine was built with Explain=true.
type LexicalHit struct {
CanonicalPath string
Score float64
MatchedFields []string
Explain *ToolSearchScoreBreakdown
}
// LexicalRetriever is the zero-model local recall boundary. Implementations
// must return a deterministic score-desc/canonical-asc order and omit zero
// score documents.
type LexicalRetriever interface {
Name() string
Retrieve(context.Context, ToolSearchLexicalRequest) ([]LexicalHit, error)
}
func newToolSearchLexicalRetriever(documents map[string]toolSearchDocument, config ToolSearchConfig) (LexicalRetriever, error) {
var retriever LexicalRetriever
switch config.LexicalAlgorithm {
case ToolSearchLexicalBM25Action:
retriever = newToolSearchActionRetriever(
newToolSearchBM25Retriever(documents, config.FieldWeights, config.BM25K1, config.BM25B, config.Explain),
documents,
config.Explain,
)
case ToolSearchLexicalBM25:
retriever = newToolSearchBM25Retriever(documents, config.FieldWeights, config.BM25K1, config.BM25B, config.Explain)
case ToolSearchLexicalTFIDF:
retriever = newToolSearchTFIDFRetriever(documents, config.FieldWeights, config.Explain)
default:
return nil, fmt.Errorf("unknown tool search lexical algorithm %q", config.LexicalAlgorithm)
}
return newToolSearchAvoidWhenRetriever(retriever, documents, config.Explain), nil
}
type toolSearchActionRetriever struct {
base LexicalRetriever
documents map[string]toolSearchDocument
explain bool
}
var toolSearchASCIIWordPattern = regexp.MustCompile(`[A-Za-z][A-Za-z0-9._-]*`)
func newToolSearchActionRetriever(base LexicalRetriever, documents map[string]toolSearchDocument, explain bool) *toolSearchActionRetriever {
return &toolSearchActionRetriever{base: base, documents: documents, explain: explain}
}
func (*toolSearchActionRetriever) Name() string { return ToolSearchLexicalBM25Action }
func (r *toolSearchActionRetriever) Retrieve(ctx context.Context, request ToolSearchLexicalRequest) ([]LexicalHit, error) {
deep := request
deep.CandidateLimit = len(request.EligibleCanonicalPaths)
hits, err := r.base.Retrieve(ctx, deep)
if err != nil {
return nil, err
}
query := classifyToolSearchQuery(request.Query)
if !query.hasSignal() || toolSearchHasUnclassifiedASCII(request.Query) {
return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil
}
for index := range hits {
multiplier := toolSearchStructuredMultiplier(query, r.documents[hits[index].CanonicalPath].tool)
hits[index].Score *= multiplier
if r.explain && hits[index].Explain != nil {
hits[index].Explain.Multiplier = &multiplier
hits[index].Explain.Score = hits[index].Score
hits[index].Explain.QueryClass = newToolSearchExplainQueryClass(query)
}
}
return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil
}
// toolSearchHasUnclassifiedASCII reports whether the query carries a technical
// identifier (camelCase or separator-joined tokens such as openConversationId,
// task_id, file-path) that the compact vocabularies cannot classify. Plain
// English words no longer disable structured reranking: they match (or fail
// to match) the vocabularies exactly like Chinese phrases do, so mixed and
// English queries keep the effect/product/action/entity routing instead of
// silently degrading to raw BM25.
func toolSearchHasUnclassifiedASCII(query string) bool {
for _, token := range toolSearchASCIIWordPattern.FindAllString(query, -1) {
if len(token) < 4 {
continue
}
switch strings.ToLower(token) {
case "id", "ids", "url", "uri":
continue
}
if strings.ContainsAny(token, "._-") {
return true
}
// Only an internal uppercase transition (openConversationId) marks a
// technical identifier; a capitalized plain word ("Send") does not.
if internal := token[1:]; strings.ToLower(internal) != internal {
return true
}
}
return false
}
type toolSearchQueryClass struct {
actions map[string]bool
products map[string]bool
entities map[string]bool
effect string
}
func (q toolSearchQueryClass) hasSignal() bool {
return len(q.actions) > 0 || len(q.products) > 0 || len(q.entities) > 0 || q.effect != ""
}
type toolSearchTermClass struct {
name string
query []string
tool []string
effect string
}
var toolSearchActionClasses = []toolSearchTermClass{
{name: "approve", query: []string{"同意", "批准", "审批通过", "approve"}, tool: []string{"approve"}},
{name: "reject", query: []string{"拒绝", "驳回", "reject"}, tool: []string{"reject"}},
{name: "upload", query: []string{"上传", "upload"}, tool: []string{"upload"}},
{name: "download", query: []string{"下载", "download"}, tool: []string{"download"}},
{name: "send", query: []string{"发送", "发给", "发消息", "通知", "投递", "send", "notify"}, tool: []string{"send", "notify"}},
{name: "search", query: []string{"搜索", "查找", "检索", "定位", "search", "find", "locate"}, tool: []string{"search", "find", "resolve"}},
{name: "list", query: []string{"列出", "列表", "浏览", "遍历", "list", "browse"}, tool: []string{"list", "all"}},
{name: "read", query: []string{"读取", "查看", "获取", "详情", "正文", "read", "fetch"}, tool: []string{"read", "get", "detail", "info", "query"}},
{name: "create", query: []string{"创建", "新建", "create"}, tool: []string{"create", "new"}},
{name: "add", query: []string{"添加", "追加", "增加", "授权", "授予", "add", "invite"}, tool: []string{"add", "grant", "invite"}},
{name: "update", query: []string{"更新", "修改", "编辑", "重命名", "调整", "update", "edit", "rename"}, tool: []string{"update", "edit", "rename", "set"}},
{name: "delete", query: []string{"删除", "移除", "清除", "delete", "remove"}, tool: []string{"delete", "remove", "clear"}},
{name: "complete", query: []string{"完成", "提交入库", "complete", "commit"}, tool: []string{"complete", "commit", "finish"}},
{name: "enable", query: []string{"启用", "开启", "enable"}, tool: []string{"enable"}},
{name: "disable", query: []string{"禁用", "停用", "关闭", "disable"}, tool: []string{"disable"}},
}
var toolSearchProductClasses = []toolSearchTermClass{
{name: "chat", query: []string{"群聊", "群消息", "会话", "单聊", "chat"}, tool: []string{"chat"}},
{name: "calendar", query: []string{"日程", "会议日历", "会议室", "calendar"}, tool: []string{"calendar"}},
{name: "drive", query: []string{"钉盘", "云盘", "drive"}, tool: []string{"drive"}},
{name: "wiki", query: []string{"知识库", "wiki"}, tool: []string{"wiki"}},
{name: "todo", query: []string{"待办", "todo"}, tool: []string{"todo"}},
{name: "minutes", query: []string{"听记", "会议纪要", "minutes"}, tool: []string{"minutes"}},
{name: "oa", query: []string{"审批", "oa", "approval"}, tool: []string{"oa"}},
{name: "mail", query: []string{"邮件", "邮箱", "草稿", "mail", "email"}, tool: []string{"mail"}},
{name: "sheet", query: []string{"电子表格", "工作表", "单元格", "sheet", "spreadsheet"}, tool: []string{"sheet"}},
{name: "doc", query: []string{"在线文字文档", "文档正文", "adoc", "doc", "document"}, tool: []string{"doc"}},
}
var toolSearchEntityClasses = []toolSearchTermClass{
{name: "task_id", query: []string{"任务 id", "任务id", "task id", "taskid"}, tool: []string{"任务 id", "任务id", "task id", "taskid"}},
{name: "task", query: []string{"任务", "task id", "taskid", "task"}, tool: []string{"任务", "task"}},
{name: "permission", query: []string{"权限", "授权", "协作者", "permission"}, tool: []string{"权限", "授权", "协作者", "permission"}},
{name: "content", query: []string{"正文", "文档内容", "content"}, tool: []string{"正文", "内容", "content"}},
{name: "status", query: []string{"状态", "已读", "status"}, tool: []string{"状态", "已读", "status"}},
{name: "draft", query: []string{"草稿", "draft"}, tool: []string{"草稿", "draft"}},
{name: "event", query: []string{"日程", "event"}, tool: []string{"日程", "event"}},
{name: "reminder", query: []string{"提醒", "reminder"}, tool: []string{"提醒", "reminder"}},
{name: "summary", query: []string{"摘要", "summary"}, tool: []string{"摘要", "summary"}},
{name: "sheet", query: []string{"工作表", "worksheet"}, tool: []string{"工作表", "sheet"}},
{name: "range", query: []string{"单元格范围", "区域", "range"}, tool: []string{"范围", "区域", "range"}},
{name: "group", query: []string{"群聊", "群名", "group"}, tool: []string{"群聊", "群", "group"}},
}
func classifyToolSearchQuery(query string) toolSearchQueryClass {
normalized := strings.ToLower(strings.TrimSpace(query))
result := toolSearchQueryClass{actions: map[string]bool{}, products: map[string]bool{}, entities: map[string]bool{}}
for _, class := range toolSearchActionClasses {
if containsAnyToolSearchPhrase(normalized, class.query) {
result.actions[class.name] = true
}
}
for _, class := range toolSearchProductClasses {
if containsAnyToolSearchPhrase(normalized, class.query) {
result.products[class.name] = true
}
}
for _, class := range toolSearchEntityClasses {
if class.name == "task" && result.entities["task_id"] {
continue
}
if containsAnyToolSearchPhrase(normalized, class.query) {
result.entities[class.name] = true
}
}
// “查询” is intentionally only a broad read-effect signal. Treating it as
// get/list/search would impose a false sibling preference.
if strings.Contains(normalized, "查询") || result.actions["read"] || result.actions["search"] || result.actions["list"] {
result.effect = "read"
}
return result
}
func toolSearchStructuredMultiplier(query toolSearchQueryClass, tool ToolSpec) float64 {
multiplier := 1.0
if query.effect != "" {
if tool.Safety.Effect == query.effect {
multiplier += 0.10
} else {
multiplier -= 0.50
}
}
if len(query.products) > 0 {
if query.products[tool.Identity.ProductID] {
multiplier += 0.50
} else {
multiplier -= 0.30
}
}
toolText := strings.ToLower(strings.Join([]string{
tool.Identity.Name,
tool.Identity.CLIName,
tool.Identity.CanonicalPath,
tool.Identity.PrimaryCLIPath,
tool.Title,
tool.Selection.AgentSummary,
}, " "))
if len(query.actions) > 0 {
matched := false
for _, class := range toolSearchActionClasses {
if query.actions[class.name] && containsAnyToolSearchWord(toolText, class.tool) {
matched = true
break
}
}
if matched {
multiplier += 0.60
} else {
// An explicit action is a stronger constraint than generic lexical
// overlap. Penalize tools that do not advertise that action even
// when their sibling verb is not in the compact action vocabulary.
multiplier -= 0.65
}
}
if len(query.entities) > 0 {
matched := false
for _, class := range toolSearchEntityClasses {
if query.entities[class.name] && containsAnyToolSearchPhrase(toolText, class.tool) {
matched = true
break
}
}
if matched {
multiplier += 0.35
} else {
multiplier -= 0.10
}
}
if multiplier < 0.20 {
return 0.20
}
return multiplier
}
func containsAnyToolSearchPhrase(text string, values []string) bool {
for _, value := range values {
if toolSearchPhraseContains(text, value) {
return true
}
}
return false
}
// toolSearchPhraseContains matches ASCII single-token values on word
// boundaries so short English values such as "oa" cannot fire inside
// unrelated words ("download" contains "oa"). Mixed-script or multi-word
// values keep substring semantics, which is what Chinese phrases and values
// like "task id" rely on.
func toolSearchPhraseContains(text, value string) bool {
if value != "" && isAllToolSearchASCII(value) && !strings.ContainsAny(value, " \t") {
return containsToolSearchASCIIWord(text, value)
}
return strings.Contains(text, value)
}
func isAllToolSearchASCII(value string) bool {
for _, character := range value {
if character >= utf8.RuneSelf {
return false
}
}
return true
}
func containsToolSearchASCIIWord(text, word string) bool {
word = strings.ToLower(word)
for _, token := range toolSearchASCIIWordPattern.FindAllString(strings.ToLower(text), -1) {
if token == word {
return true
}
}
return false
}
func containsAnyToolSearchWord(text string, values []string) bool {
normalized := strings.NewReplacer(".", " ", "_", " ", "-", " ", "/", " ").Replace(text)
words := strings.Fields(normalized)
for _, value := range values {
for _, word := range words {
if word == value {
return true
}
}
}
return false
}
func newToolSearchExplainQueryClass(query toolSearchQueryClass) *ToolSearchExplainQueryClass {
return &ToolSearchExplainQueryClass{
Actions: sortedToolSearchClassKeys(query.actions),
Products: sortedToolSearchClassKeys(query.products),
Entities: sortedToolSearchClassKeys(query.entities),
Effect: query.effect,
}
}
func sortedToolSearchClassKeys(set map[string]bool) []string {
if len(set) == 0 {
return nil
}
keys := make([]string, 0, len(set))
for key := range set {
keys = append(keys, key)
}
sort.Strings(keys)
return keys
}
// toolSearchAvoidWhenPenalty is applied when a candidate's avoid_when prose
// matches the query intent. It is a soft demotion, not an eligibility filter:
// avoid_when is natural-language guidance (often "prefer X for that intent"),
// so a lexical hit can be wrong and the tool must remain reachable.
const toolSearchAvoidWhenPenalty = 0.20
// toolSearchAvoidWhenShortcutNoise filters the boilerplate avoid_when that
// every shortcut carries; it carries no intent signal. Kept lowercase because
// phrases are compared in the lowercased domain.
const toolSearchAvoidWhenShortcutNoise = "需要该 shortcut 未公开的底层参数"
// toolSearchAvoidWhenPenaltyReason returns the matched avoid_when phrase, or
// "" when the tool should not be demoted for this query. The only match shape
// is the query containing the avoidance prose itself (the Forbidden@5
// evaluation proxy). Intent-overlap matching was tried and rejected: a tool's
// own avoid_when describes the same sibling scenario space as its use_when,
// so "shares an action/entity with the phrase" demotes the gold answer.
func toolSearchAvoidWhenPenaltyReason(query string, tool ToolSpec) string {
if len(tool.Selection.AvoidWhen) == 0 {
return ""
}
normalized := strings.ToLower(strings.TrimSpace(query))
if normalized == "" {
return ""
}
for _, phrase := range tool.Selection.AvoidWhen {
phrase = strings.ToLower(strings.TrimSpace(phrase))
if phrase == "" || strings.Contains(phrase, toolSearchAvoidWhenShortcutNoise) {
continue
}
if strings.Contains(normalized, phrase) {
return phrase
}
}
return ""
}
// toolSearchAvoidWhenRetriever demotes candidates whose avoid_when prose
// matches the query intent. It wraps every lexical algorithm, including the
// production ensemble default, so the Forbidden exposure guard is not tied to
// action_v1.
type toolSearchAvoidWhenRetriever struct {
base LexicalRetriever
documents map[string]toolSearchDocument
explain bool
}
func newToolSearchAvoidWhenRetriever(base LexicalRetriever, documents map[string]toolSearchDocument, explain bool) *toolSearchAvoidWhenRetriever {
return &toolSearchAvoidWhenRetriever{base: base, documents: documents, explain: explain}
}
func (r *toolSearchAvoidWhenRetriever) Name() string { return r.base.Name() }
func (r *toolSearchAvoidWhenRetriever) Retrieve(ctx context.Context, request ToolSearchLexicalRequest) ([]LexicalHit, error) {
hits, err := r.base.Retrieve(ctx, request)
if err != nil {
return nil, err
}
penalized := false
for index := range hits {
reason := toolSearchAvoidWhenPenaltyReason(request.Query, r.documents[hits[index].CanonicalPath].tool)
if reason == "" {
continue
}
hits[index].Score *= toolSearchAvoidWhenPenalty
if r.explain && hits[index].Explain != nil {
hits[index].Explain.Score = hits[index].Score
hits[index].Explain.AvoidWhenPenalty = reason
}
penalized = true
}
if !penalized {
return hits, nil
}
return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil
}
type toolSearchBM25Retriever struct {
fieldIndex map[toolSearchField]toolSearchBM25Index
weights ToolSearchFieldWeights
explain bool
}
func newToolSearchBM25Retriever(documents map[string]toolSearchDocument, weights ToolSearchFieldWeights, k1, b float64, explain bool) *toolSearchBM25Retriever {
fieldTerms := toolSearchFieldTerms(documents)
fieldIndex := make(map[toolSearchField]toolSearchBM25Index, len(toolSearchFieldOrder))
for _, field := range toolSearchFieldOrder {
fieldIndex[field] = newToolSearchBM25Index(fieldTerms[field], k1, b)
}
return &toolSearchBM25Retriever{fieldIndex: fieldIndex, weights: weights, explain: explain}
}
func (*toolSearchBM25Retriever) Name() string { return ToolSearchLexicalBM25 }
func (r *toolSearchBM25Retriever) Retrieve(ctx context.Context, request ToolSearchLexicalRequest) ([]LexicalHit, error) {
queryTerms := orderedToolSearchQueryTerms(request.Query)
hits := make([]LexicalHit, 0, len(request.EligibleCanonicalPaths))
for _, canonical := range request.EligibleCanonicalPaths {
if err := ctx.Err(); err != nil {
return nil, err
}
contributions := make(map[toolSearchField]float64)
score := 0.0
for _, field := range toolSearchFieldOrder {
weight := toolSearchFieldWeight(r.weights, field)
if weight <= 0 {
continue
}
value := r.fieldIndex[field].score(canonical, queryTerms) * weight
if value > 0 {
contributions[field] = value
score += value
}
}
if score > 0 {
hit := LexicalHit{CanonicalPath: canonical, Score: score, MatchedFields: matchedToolSearchFields(contributions)}
if r.explain {
hit.Explain = newToolSearchBreakdown(score, contributions)
}
hits = append(hits, hit)
}
}
return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil
}
type toolSearchTFIDFDocument struct {
weights map[string]float64
norm float64
}
type toolSearchTFIDFIndex struct {
documents map[string]toolSearchTFIDFDocument
idf map[string]float64
}
type toolSearchTFIDFRetriever struct {
fieldIndex map[toolSearchField]toolSearchTFIDFIndex
weights ToolSearchFieldWeights
explain bool
}
func newToolSearchTFIDFRetriever(documents map[string]toolSearchDocument, weights ToolSearchFieldWeights, explain bool) *toolSearchTFIDFRetriever {
fieldTerms := toolSearchFieldTerms(documents)
fieldIndex := make(map[toolSearchField]toolSearchTFIDFIndex, len(toolSearchFieldOrder))
for _, field := range toolSearchFieldOrder {
fieldIndex[field] = newToolSearchTFIDFIndex(fieldTerms[field])
}
return &toolSearchTFIDFRetriever{fieldIndex: fieldIndex, weights: weights, explain: explain}
}
func (*toolSearchTFIDFRetriever) Name() string { return ToolSearchLexicalTFIDF }
func (r *toolSearchTFIDFRetriever) Retrieve(ctx context.Context, request ToolSearchLexicalRequest) ([]LexicalHit, error) {
queryFrequency := toolSearchTermFrequency(tokenizeToolSearchText(request.Query))
hits := make([]LexicalHit, 0, len(request.EligibleCanonicalPaths))
for _, canonical := range request.EligibleCanonicalPaths {
if err := ctx.Err(); err != nil {
return nil, err
}
contributions := make(map[toolSearchField]float64)
score := 0.0
for _, field := range toolSearchFieldOrder {
weight := toolSearchFieldWeight(r.weights, field)
if weight <= 0 {
continue
}
value := r.fieldIndex[field].score(canonical, queryFrequency) * weight
if value > 0 {
contributions[field] = value
score += value
}
}
if score > 0 {
hit := LexicalHit{CanonicalPath: canonical, Score: score, MatchedFields: matchedToolSearchFields(contributions)}
if r.explain {
hit.Explain = newToolSearchBreakdown(score, contributions)
}
hits = append(hits, hit)
}
}
return truncateAndSortLexicalHits(hits, request.CandidateLimit), nil
}
func toolSearchFieldTerms(documents map[string]toolSearchDocument) map[toolSearchField]map[string][]string {
fieldTerms := make(map[toolSearchField]map[string][]string, len(toolSearchFieldOrder))
for _, field := range toolSearchFieldOrder {
fieldTerms[field] = make(map[string][]string, len(documents))
}
for canonical, document := range documents {
for _, field := range toolSearchFieldOrder {
fieldTerms[field][canonical] = tokenizeToolSearchText(document.fields[field])
}
}
return fieldTerms
}
func orderedToolSearchQueryTerms(query string) []toolSearchQueryTerm {
frequency := toolSearchTermFrequency(tokenizeToolSearchText(query))
terms := make([]toolSearchQueryTerm, 0, len(frequency))
for term, count := range frequency {
terms = append(terms, toolSearchQueryTerm{term: term, count: count})
}
sort.Slice(terms, func(i, j int) bool { return terms[i].term < terms[j].term })
return terms
}
func toolSearchTermFrequency(tokens []string) map[string]int {
frequency := make(map[string]int, len(tokens))
for _, token := range tokens {
frequency[token]++
}
return frequency
}
func newToolSearchTFIDFIndex(documents map[string][]string) toolSearchTFIDFIndex {
index := toolSearchTFIDFIndex{documents: make(map[string]toolSearchTFIDFDocument, len(documents)), idf: map[string]float64{}}
documentFrequency := map[string]int{}
frequencies := make(map[string]map[string]int, len(documents))
for canonical, tokens := range documents {
frequency := toolSearchTermFrequency(tokens)
frequencies[canonical] = frequency
for term := range frequency {
documentFrequency[term]++
}
}
n := float64(len(documents))
for term, frequency := range documentFrequency {
index.idf[term] = math.Log((n+1)/(float64(frequency)+1)) + 1
}
for canonical, frequency := range frequencies {
weights := make(map[string]float64, len(frequency))
normSquared := 0.0
terms := make([]string, 0, len(frequency))
for term := range frequency {
terms = append(terms, term)
}
sort.Strings(terms)
for _, term := range terms {
count := frequency[term]
value := float64(count) * index.idf[term]
weights[term] = value
normSquared += value * value
}
index.documents[canonical] = toolSearchTFIDFDocument{weights: weights, norm: math.Sqrt(normSquared)}
}
return index
}
func (i toolSearchTFIDFIndex) score(canonical string, queryFrequency map[string]int) float64 {
document, ok := i.documents[canonical]
if !ok || document.norm == 0 {
return 0
}
terms := make([]string, 0, len(queryFrequency))
for term := range queryFrequency {
terms = append(terms, term)
}
sort.Strings(terms)
dot := 0.0
queryNormSquared := 0.0
for _, term := range terms {
idf, ok := i.idf[term]
if !ok {
continue
}
queryWeight := float64(queryFrequency[term]) * idf
queryNormSquared += queryWeight * queryWeight
dot += document.weights[term] * queryWeight
}
if queryNormSquared == 0 {
return 0
}
return dot / (document.norm * math.Sqrt(queryNormSquared))
}
// toolSearchScoreQuantum is the sort-key quantization grid. Go permits FMA
// fusion on arm64 but not amd64, so the same ranking expression can differ in
// the last ulp across architectures; ordering on the raw float would let that
// ulp flip near-tied candidates past the canonical tie-break and break the
// byte-determinism contract across CI (linux/amd64) and dev (macOS/arm64)
// machines. Quantizing only the ordering key keeps same-architecture output
// identical while making cross-architecture ordering agree; reported scores
// (explain-only) are untouched.
const toolSearchScoreQuantum = 1e9
func quantizeToolSearchScore(score float64) float64 {
return math.Round(score*toolSearchScoreQuantum) / toolSearchScoreQuantum
}
func truncateAndSortLexicalHits(hits []LexicalHit, limit int) []LexicalHit {
sort.Slice(hits, func(i, j int) bool {
left, right := quantizeToolSearchScore(hits[i].Score), quantizeToolSearchScore(hits[j].Score)
if left != right {
return left > right
}
return hits[i].CanonicalPath < hits[j].CanonicalPath
})
if limit >= 0 && len(hits) > limit {
hits = hits[:limit]
}
return hits
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,249 @@
// Copyright 2026 Alibaba Group
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"bytes"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"os"
"path/filepath"
"github.com/DingTalk-Real-AI/dingtalk-workspace-cli/internal/app"
"github.com/DingTalk-Real-AI/dingtalk-workspace-cli/internal/cli"
"github.com/spf13/cobra"
)
type workflowFixture struct {
Version int `json:"version"`
Description string `json:"description,omitempty"`
Cases []cli.ToolSearchWorkflowEvaluationCase `json:"cases"`
}
type comparisonOutput struct {
Diagnostic cli.ToolSearchDiagnosticComparison `json:"diagnostic"`
Shadows []cli.ToolSearchDiagnosticComparison `json:"shadows"`
AgentPlanningAB *cli.ToolSearchAgentPlanningABReport `json:"agent_planning_ab,omitempty"`
AgentAB *cli.ToolSearchAgentABReport `json:"agent_ab,omitempty"`
Independent *cli.ToolSearchIndependentReport `json:"independent,omitempty"`
}
type independentFixture struct {
Version string `json:"version"`
State string `json:"state"`
Description string `json:"description,omitempty"`
Cases []cli.ToolSearchIndependentCase `json:"cases"`
}
func main() {
var workflowsPath string
var agentResultsPath string
var directPlansPath string
var searchPlansPath string
var agentModel string
var outputPath string
var independentQrelsPath string
flag.StringVar(&workflowsPath, "workflows", "scripts/testdata/tool_search_workflows.json", "reviewed diagnostic workflow fixture")
flag.StringVar(&agentResultsPath, "agent-results", "", "optional paired Agent A/B result JSON")
flag.StringVar(&directPlansPath, "agent-plans-direct", "", "optional direct_schema Agent planning output JSON")
flag.StringVar(&searchPlansPath, "agent-plans-search", "", "optional search_inspect Agent planning output JSON")
flag.StringVar(&agentModel, "agent-model", "", "model identifier for paired planning outputs")
flag.StringVar(&outputPath, "output", "", "output JSON path; stdout when empty")
flag.StringVar(&independentQrelsPath, "independent-qrels", "", "optional independently authored tool-search-qrels.v1 JSON")
flag.Parse()
// Runtime and CI use the same declaration-owned Catalog assembly. The
// comparison generator never reads or writes a repository Catalog JSON.
cli.RegisterSchemaSourceRoot(func() *cobra.Command { return app.NewSchemaSourceRootCommand() })
workflows, err := readWorkflows(workflowsPath)
if err != nil {
fatal(err)
}
diagnostic, err := cli.BuildDeliveryToolSearchDiagnosticComparison(context.Background(), workflows)
if err != nil {
fatal(err)
}
actionShadow, err := cli.BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(context.Background(), workflows, cli.ToolSearchLexicalBM25Action)
if err != nil {
fatal(err)
}
shadow, err := cli.BuildDeliveryToolSearchDiagnosticComparisonForAlgorithm(context.Background(), workflows, cli.ToolSearchLexicalTFIDF)
if err != nil {
fatal(err)
}
result := comparisonOutput{Diagnostic: diagnostic, Shadows: []cli.ToolSearchDiagnosticComparison{actionShadow, shadow}}
if independentQrelsPath != "" {
fixture, readErr := readIndependentQrels(independentQrelsPath)
if readErr != nil {
fatal(readErr)
}
independent, evaluationErr := cli.BuildDeliveryToolSearchIndependentEvaluation(context.Background(), fixture.Cases)
if evaluationErr != nil {
fatal(evaluationErr)
}
result.Independent = &independent
}
if directPlansPath != "" || searchPlansPath != "" || agentModel != "" {
if directPlansPath == "" || searchPlansPath == "" || agentModel == "" {
fatal(fmt.Errorf("-agent-plans-direct, -agent-plans-search and -agent-model must be provided together"))
}
directPlans, readErr := readAgentPlans(directPlansPath)
if readErr != nil {
fatal(readErr)
}
searchPlans, readErr := readAgentPlans(searchPlansPath)
if readErr != nil {
fatal(readErr)
}
planningReport, scoreErr := cli.ScoreToolSearchAgentPlanningAB(agentModel, workflows, directPlans, searchPlans)
if scoreErr != nil {
fatal(scoreErr)
}
result.AgentPlanningAB = &planningReport
}
if agentResultsPath != "" {
agentInput, readErr := readAgentResults(agentResultsPath)
if readErr != nil {
fatal(readErr)
}
if agentInput.Catalog != diagnostic.Catalog {
fatal(fmt.Errorf("Agent A/B Catalog hashes do not match the embedded binary"))
}
agentReport, aggregateErr := cli.AggregateToolSearchAgentAB(agentInput)
if aggregateErr != nil {
fatal(aggregateErr)
}
diagnostic.AgentABStatus = "completed"
result.Diagnostic = diagnostic
result.AgentAB = &agentReport
}
encoded, err := json.MarshalIndent(result, "", " ")
if err != nil {
fatal(err)
}
encoded = append(encoded, '\n')
if outputPath == "" {
if _, err := os.Stdout.Write(encoded); err != nil {
fatal(err)
}
return
}
if err := writeAtomically(outputPath, encoded); err != nil {
fatal(err)
}
}
func readIndependentQrels(path string) (independentFixture, error) {
data, err := os.ReadFile(path)
if err != nil {
return independentFixture{}, fmt.Errorf("read independent qrels: %w", err)
}
var fixture independentFixture
if err := decodeStrict(data, &fixture); err != nil {
return independentFixture{}, fmt.Errorf("decode independent qrels: %w", err)
}
if fixture.Version != "tool-search-qrels.v1" || len(fixture.Cases) == 0 {
return independentFixture{}, fmt.Errorf("independent qrels must be tool-search-qrels.v1 and non-empty")
}
return fixture, nil
}
func readWorkflows(path string) ([]cli.ToolSearchWorkflowEvaluationCase, error) {
data, err := os.ReadFile(path)
if err != nil {
return nil, fmt.Errorf("read workflows: %w", err)
}
var fixture workflowFixture
if err := decodeStrict(data, &fixture); err != nil {
return nil, fmt.Errorf("decode workflows: %w", err)
}
if fixture.Version != 1 || len(fixture.Cases) == 0 {
return nil, fmt.Errorf("workflow fixture must be version 1 and non-empty")
}
return fixture.Cases, nil
}
func readAgentResults(path string) (cli.ToolSearchAgentABInput, error) {
data, err := os.ReadFile(path)
if err != nil {
return cli.ToolSearchAgentABInput{}, fmt.Errorf("read Agent A/B results: %w", err)
}
var input cli.ToolSearchAgentABInput
if err := decodeStrict(data, &input); err != nil {
return cli.ToolSearchAgentABInput{}, fmt.Errorf("decode Agent A/B results: %w", err)
}
return input, nil
}
func readAgentPlans(path string) (cli.ToolSearchAgentPlanOutput, error) {
data, err := os.ReadFile(path)
if err != nil {
return cli.ToolSearchAgentPlanOutput{}, fmt.Errorf("read Agent plans: %w", err)
}
var output cli.ToolSearchAgentPlanOutput
if err := decodeStrict(data, &output); err != nil {
return cli.ToolSearchAgentPlanOutput{}, fmt.Errorf("decode Agent plans: %w", err)
}
return output, nil
}
func decodeStrict(data []byte, target any) error {
decoder := json.NewDecoder(bytes.NewReader(data))
decoder.DisallowUnknownFields()
if err := decoder.Decode(target); err != nil {
return err
}
var extra any
if err := decoder.Decode(&extra); err != io.EOF {
if err == nil {
return fmt.Errorf("multiple JSON values are not allowed")
}
return err
}
return nil
}
func writeAtomically(path string, data []byte) error {
directory := filepath.Dir(path)
if err := os.MkdirAll(directory, 0o755); err != nil {
return err
}
temporary, err := os.CreateTemp(directory, ".tool-search-comparison-*.tmp")
if err != nil {
return err
}
temporaryPath := temporary.Name()
defer os.Remove(temporaryPath)
if _, err := temporary.Write(data); err != nil {
temporary.Close()
return err
}
if err := temporary.Sync(); err != nil {
temporary.Close()
return err
}
if err := temporary.Close(); err != nil {
return err
}
return os.Rename(temporaryPath, path)
}
func fatal(err error) {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
+82 -2
View File
@@ -230,7 +230,7 @@ func newCalendarCommand() *cobra.Command {
Long: `管理钉钉日历:日程、参会人、会议室、闲忙、附件、日历本、访问权限。调用前必须先使用 --help 查看参数结构。
命令结构:
dws calendar event [list|get|create|update|delete|suggest|respond|instances] 日程管理
dws calendar event [list|get|create|update|delete|suggest|respond|instances|share-info] 日程管理
dws calendar attendee [list|add|delete] 参会人管理
dws calendar room [search|add|delete|list-groups] 会议室管理
dws calendar busy search 闲忙查询 (可查人、查会议室)
@@ -2259,7 +2259,87 @@ func newCalendarCommand() *cobra.Command {
eventInstancesCmd.Flags().Int("count", 0, "")
_ = eventInstancesCmd.Flags().MarkHidden("count")
eventCmd.AddCommand(eventListCmd, eventGetCmd, eventCreateCmd, eventUpdateCmd, eventDeleteCmd, eventSuggestCmd, eventRespondCmd, eventInstancesCmd)
eventShareInfoCmd := &cobra.Command{
Use: "share-info",
Aliases: []string{"share_info"},
Short: "获取日程的分享信息",
Long: `根据日程 ID 获取日程的分享信息,展示日程主题、组织人、地点、入会信息等,用于向他人分享日程。eventId 可通过 dws calendar event list 获取。`,
Example: ` dws calendar event share-info --id EVENT_ID
dws calendar event share-info --id EVENT_ID --language zh-CN
dws calendar event share-info --id EVENT_ID --calendar-id primary`,
RunE: func(cmd *cobra.Command, args []string) error {
eventID, err := mustFlagOrFallback(cmd, "id", "event", "event-id", "eventId")
if err != nil {
return err
}
toolArgs := map[string]any{"eventId": eventID}
if v := flagOrFallback(cmd, "calendar-id", "calendarId", "calendar"); v != "" {
toolArgs["calendarId"] = v
}
if v := flagOrFallback(cmd, "language", "lang"); v != "" {
toolArgs["language"] = v
}
return callMCPTool("get_event_share_info", toolArgs)
},
}
DeclareLeafMetadata(eventShareInfoCmd, LeafSpec{
Safety: contract.SafetySpec{
Effect: "read", Risk: "low",
Confirmation: "not_required", Idempotency: "idempotent",
},
Contract: LeafContract{
Identity: contract.ToolIdentitySpec{
ProductID: "calendar",
Name: "get_event_share_info",
CanonicalPath: "calendar.get_event_share_info",
CLIPath: "calendar event share-info",
PrimaryCLIPath: "calendar event share-info",
Aliases: []string{"calendar event share_info"},
},
Description: "获取日程的分享信息",
Interface: &contract.InterfaceSpec{
Mode: "mcp",
Availability: "available",
Ref: &contract.InterfaceRefSpec{ProductID: "calendar", RPCName: "get_event_share_info"},
},
Selection: contract.SelectionSpec{
AgentSummary: "获取日程的分享信息(主题、组织人、地点、入会信息等)",
UseWhen: []string{"已知 eventId,需要获取日程分享信息或入会信息用于分享给他人时"},
AvoidWhen: []string{
"要查看日程详情改用 dws calendar event get",
"未知 eventId 时先 dws calendar event list",
},
Examples: []string{
"dws calendar event share-info --id <EVENT_ID>",
"dws calendar event share-info --id <EVENT_ID> --language zh-CN",
},
},
Parameters: []contract.ParamDecl{
{Name: "id", Property: "eventId", Required: boolPtr(true)},
{Name: "calendar-id", Property: "calendarId"},
{Name: "language", Property: "language"},
},
},
})
// EventShareInfo flags
eventShareInfoCmd.Flags().String("id", "", "日程 ID (必填)")
eventShareInfoCmd.Flags().String("event", "", "")
_ = eventShareInfoCmd.Flags().MarkHidden("event")
eventShareInfoCmd.Flags().String("event-id", "", "")
_ = eventShareInfoCmd.Flags().MarkHidden("event-id")
eventShareInfoCmd.Flags().String("eventId", "", "")
_ = eventShareInfoCmd.Flags().MarkHidden("eventId")
eventShareInfoCmd.Flags().String("calendar-id", "", "日历 ID (默认 primary 主日历)")
eventShareInfoCmd.Flags().String("calendarId", "", "")
_ = eventShareInfoCmd.Flags().MarkHidden("calendarId")
eventShareInfoCmd.Flags().String("calendar", "", "")
_ = eventShareInfoCmd.Flags().MarkHidden("calendar")
eventShareInfoCmd.Flags().String("language", "", "语言代码 (可选,如 zh-CN)")
eventShareInfoCmd.Flags().String("lang", "", "")
_ = eventShareInfoCmd.Flags().MarkHidden("lang")
eventCmd.AddCommand(eventListCmd, eventGetCmd, eventCreateCmd, eventUpdateCmd, eventDeleteCmd, eventSuggestCmd, eventRespondCmd, eventInstancesCmd, eventShareInfoCmd)
// participant
participantCmd.PersistentFlags().String("event", "", "日程 ID (必填)")
+54 -8
View File
@@ -3413,10 +3413,11 @@ func newChatCommand() *cobra.Command {
chatMessageSendByBotCmd := &cobra.Command{
Use: "send-by-bot",
Short: "机器人发送消息(--conversation-id 群聊 / --users 单聊)",
Long: `群聊:传 --conversation-id 指定群;单聊:传 --users 或 --open-dingtalk-ids 指定用户列表,与 --conversation-id 只能选其一,不能同时指定。省略 --msg-type 时发送 Markdown;图片使用 --msg-type image --image-url;本地文件使用 --msg-type file --file-path,CLI 完成上传后发送。
Long: `群聊:传 --conversation-id 指定群;单聊:传 --users 或 --open-dingtalk-ids 指定用户列表,与 --conversation-id 只能选其一,不能同时指定。省略 --msg-type 时发送 Markdown;图片使用 --msg-type image --image-url;本地文件使用 --msg-type file --file-path,CLI 完成上传后发送。机器人引用回复仅支持群聊 Markdown,同时传 --reply(原消息 openMessageId)和 --ref-sender(原消息发送者 openDingTalkId);回复可省略 --title,CLI 会从正文生成标题。
⚠️ 重要:该接口会真实发送消息到目标会话,不可用于测试或试探性调用。调用前必须确认消息内容和接收对象无误。`,
Example: ` dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openconversation_id> --title "日报" --text "## 今日完成..."
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openconversation_id> --reply <openMessageId> --ref-sender <openDingTalkId> --text "收到"
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openconversation_id> --msg-type image --image-url "https://example.com/image.png"
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openconversation_id> --msg-type file --file-path ./report.pdf
dws chat message send-by-bot --robot-code <robot-code> --users userId1,userId2 --title "提醒" --text "请提交周报"
@@ -3433,10 +3434,17 @@ func newChatCommand() *cobra.Command {
text := mustGetFlag(cmd, "text")
imageURL := strings.TrimSpace(mustGetFlag(cmd, "image-url"))
filePath := mustGetFlag(cmd, "file-path")
referenceOpenMessageID := strings.TrimSpace(mustGetFlag(cmd, "reply"))
refSender := strings.TrimSpace(mustGetFlag(cmd, "ref-sender"))
replyChanged := cmd.Flags().Changed("reply")
refSenderChanged := cmd.Flags().Changed("ref-sender")
if err := validateRequiredFlags(cmd, "robot-code"); err != nil {
return err
}
if replyChanged != refSenderChanged || (replyChanged && (referenceOpenMessageID == "" || refSender == "")) {
return fmt.Errorf("--reply and --ref-sender must be specified together")
}
if msgType == "" {
if imageURL != "" {
return fmt.Errorf("--msg-type image is required when using --image-url")
@@ -3448,8 +3456,17 @@ func newChatCommand() *cobra.Command {
}
switch msgType {
case "markdown":
if err := validateRequiredFlags(cmd, "title", "text"); err != nil {
return err
if referenceOpenMessageID != "" {
if err := validateRequiredFlags(cmd, "text"); err != nil {
return err
}
if title == "" {
title = sanitizeTitleFromText(text)
}
} else {
if err := validateRequiredFlags(cmd, "title", "text"); err != nil {
return err
}
}
case "image":
if imageURL == "" {
@@ -3475,6 +3492,22 @@ func newChatCommand() *cobra.Command {
return fmt.Errorf("--conversation-id or --users/--open-dingtalk-ids is required")
}
if referenceOpenMessageID != "" {
if chatID == "" {
return fmt.Errorf("--reply and --ref-sender are only supported with --conversation-id")
}
if msgType != "markdown" {
return fmt.Errorf("--reply and --ref-sender only support Markdown group messages")
}
if !isOpenDingTalkID(refSender) {
resolved, err := resolveOpenDingTalkID(cmd.Context(), refSender)
if err != nil {
return err
}
refSender = resolved
}
}
userIDs := splitCommaList(usersStr)
openDingTalkIDs := splitCommaList(openDingtalkIdsStr)
fileURL := ""
@@ -3540,6 +3573,10 @@ func newChatCommand() *cobra.Command {
toolArgs["msgKey"] = "sampleMarkdownDX"
}
toolArgs["openConversationId"] = chatID
if referenceOpenMessageID != "" {
toolArgs["referenceOpenMessageId"] = referenceOpenMessageID
toolArgs["srcMsgSendOpenDingTalkId"] = refSender
}
if len(atUserIds) > 0 {
toolArgs["atUserIds"] = atUserIds
}
@@ -3586,17 +3623,20 @@ func newChatCommand() *cobra.Command {
CLIPath: "chat message send-by-bot",
PrimaryCLIPath: "chat message send-by-bot",
},
Description: "以应用机器人身份发送 Markdown、图片或文件群消息或批量单聊",
Description: "以应用机器人身份发送 Markdown、图片或文件群消息、群聊引用回复或批量单聊",
Interface: &contract.InterfaceSpec{
Mode: "composite",
Availability: "available",
Reason: "命令包含多个 RPC、条件分派或本地 HTTP/文件步骤,不能绑定为单一 interface_ref",
},
Selection: contract.SelectionSpec{
AgentSummary: "以应用机器人身份发送 Markdown、图片或文件群消息或批量单聊",
UseWhen: []string{"已有 robotCode 且需要机器人身份投递 Markdown、图片或文件时"},
AgentSummary: "以应用机器人身份发送 Markdown、图片或文件群消息、群聊引用回复或批量单聊",
UseWhen: []string{"已有 robotCode 且需要机器人身份投递 Markdown、图片或文件时", "需要机器人在群聊中引用回复一条已有消息时"},
AvoidWhen: []string{"个人身份发送或自定义 Webhook 告警不要使用"},
Examples: []string{"dws chat message send-by-bot --robot-code <robotCode> --group <openConversationId> --title \"日报\" --text \"今日进展\""},
Examples: []string{
"dws chat message send-by-bot --robot-code <robotCode> --group <openConversationId> --title \"日报\" --text \"今日进展\"",
"dws chat message send-by-bot --robot-code <robotCode> --group <openConversationId> --reply <openMessageId> --ref-sender <openDingTalkId> --text \"收到\"",
},
},
// Keep title/text required_when out of Schema: the runtime switch above
// enforces Markdown inputs, while adding it breaks merge-base compatibility.
@@ -3605,6 +3645,8 @@ func newChatCommand() *cobra.Command {
{Name: "msg-type", RequiredWhen: "image-url or file-path is provided", Enum: []string{"markdown", "image", "file"}},
{Name: "image-url", RequiredWhen: "msg-type is image"},
{Name: "file-path", RequiredWhen: "msg-type is file"},
{Name: "reply", Property: "referenceOpenMessageId", RequiredWhen: "ref-sender is provided", InterfaceType: "string"},
{Name: "ref-sender", Property: "srcMsgSendOpenDingTalkId", RequiredWhen: "reply is provided", InterfaceType: "string"},
},
},
})
@@ -4768,7 +4810,7 @@ chat message edit 或 chat message recall 的 --message-id 和 --conversation-id
chatMessageSendByBotCmd.Flags().String("conversation-id", "", "群聊 openConversationId(群聊时必填)")
chatMessageSendByBotCmd.Flags().String("users", "", "用户 userId 列表,逗号分隔,最多20个(单聊时必填)")
chatMessageSendByBotCmd.Flags().String("msg-type", "", "消息类型: markdown/image/file(省略时为 markdown;图片使用 image --image-url;本地文件使用 file --file-path)")
chatMessageSendByBotCmd.Flags().String("title", "", "Markdown 消息标题(发送 Markdown 时必填)")
chatMessageSendByBotCmd.Flags().String("title", "", "Markdown 消息标题(发送普通 Markdown 时必填;引用回复省略时从正文生成)")
chatMessageSendByBotCmd.Flags().String("text", "", "Markdown 消息内容(发送 Markdown 时必填;稳定换行用空行,转义形式写 \\n\\n,不要只写 \\n)")
chatMessageSendByBotCmd.Flags().String("image-url", "", "公网图片 URL(msgType=image 时必填)")
chatMessageSendByBotCmd.Flags().String("file-path", "", "本地文件路径(msgType=file 时直接上传并按 file 消息发送)")
@@ -4776,9 +4818,13 @@ chat message edit 或 chat message recall 的 --message-id 和 --conversation-id
chatMessageSendByBotCmd.Flags().String("open-dingtalk-ids", "", "用户 openDingtalkId 列表,逗号分隔(单聊时可替代 --users,可选)")
chatMessageSendByBotCmd.Flags().String("at-open-dingtalk-ids", "", "@指定成员的 openDingtalkId 列表,逗号分隔(仅群聊时生效,可选)")
chatMessageSendByBotCmd.Flags().Bool("at-all", false, "@所有人(可选),服务端接收字符串 true/false")
chatMessageSendByBotCmd.Flags().String("reply", "", "被引用消息的 openMessageId(仅群聊 Markdown;必须与 --ref-sender 同时使用)")
chatMessageSendByBotCmd.Flags().String("ref-sender", "", "被引用消息发送者的 openDingTalkId(仅群聊 Markdown;必须与 --reply 同时使用)")
chatMessageSendByBotCmd.MarkFlagsRequiredTogether("reply", "ref-sender")
cli.AnnotateRuntimeConstraints(chatMessageSendByBotCmd, cli.RuntimeSchemaConstraints{
MutuallyExclusive: [][]string{{"group", "users"}},
RequireOneOf: [][]string{{"group", "users"}},
RequireTogether: [][]string{{"reply", "ref-sender"}},
})
cli.AnnotateRuntimeFlagFormat(chatMessageSendByBotCmd, "file-path", "file-path")
@@ -446,6 +446,100 @@ func TestCrossPlatformCoverageChatBotRichMediaCoverage(t *testing.T) {
if caller.tool != "batch_send_robot_msg_to_users" || caller.args["msgType"] != "sampleMarkdownDX" {
t.Fatalf("direct markdown call = %s %#v", caller.tool, caller.args)
}
if _, exists := caller.args["referenceOpenMessageId"]; exists {
t.Fatalf("ordinary direct message unexpectedly contains reply fields: %#v", caller.args)
}
})
t.Run("group Markdown reference reply", func(t *testing.T) {
const senderOpenDingTalkID = "DAAAAAAAAAAAiE"
caller := &scriptedToolCaller{}
if err := runChatCoverageCommand(t, caller,
"message", "send-by-bot", "--robot-code", "robot", "--conversation-id", "group",
"--reply", "message-id", "--ref-sender", senderOpenDingTalkID,
"--text", "received",
); err != nil {
t.Fatal(err)
}
if caller.tool != "send_robot_group_message" ||
caller.args["referenceOpenMessageId"] != "message-id" ||
caller.args["srcMsgSendOpenDingTalkId"] != senderOpenDingTalkID ||
caller.args["title"] != "received" {
t.Fatalf("group reference reply call = %s %#v", caller.tool, caller.args)
}
})
t.Run("group reference reply resolves sender user ID", func(t *testing.T) {
caller := &scriptedToolCaller{steps: []scriptedToolStep{
{text: `{"result":[{"userId":"sender-user","openDingTalkId":"D-sender"}]}`},
{text: `{}`},
}}
if err := runChatCoverageCommand(t, caller,
"message", "send-by-bot", "--robot-code", "robot", "--conversation-id", "group",
"--reply", "message-id", "--ref-sender", "sender-user",
"--title", "reply", "--text", "received",
); err != nil {
t.Fatal(err)
}
if caller.tool != "send_robot_group_message" || caller.args["srcMsgSendOpenDingTalkId"] != "D-sender" {
t.Fatalf("resolved group reference reply = %s %#v", caller.tool, caller.args)
}
})
t.Run("direct RunE enforces paired reply flags", func(t *testing.T) {
err := runChatCoverageDirect(t, []string{"message", "send-by-bot"}, map[string]string{
"robot-code": "robot",
"conversation-id": "group",
"reply": "message-id",
"title": "reply",
"text": "received",
})
if err == nil || !strings.Contains(err.Error(), "must be specified together") {
t.Fatalf("direct RunE paired reply flags error = %v", err)
}
})
for _, tc := range []struct {
name string
args []string
want string
}{
{name: "reply requires sender", args: []string{"--reply", "message-id"}, want: "ref-sender"},
{name: "sender requires reply", args: []string{"--ref-sender", "D-sender"}, want: "reply"},
{name: "paired reply flags reject empty values", args: []string{"--reply=", "--ref-sender="}, want: "specified together"},
} {
t.Run(tc.name, func(t *testing.T) {
args := []string{
"message", "send-by-bot", "--robot-code", "robot", "--conversation-id", "group",
"--title", "reply", "--text", "received",
}
err := runChatCoverageCommand(t, &scriptedToolCaller{}, append(args, tc.args...)...)
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatalf("paired reply flags error = %v, want %q", err, tc.want)
}
})
}
t.Run("reference reply is group only", func(t *testing.T) {
err := runChatCoverageCommand(t, &scriptedToolCaller{},
"message", "send-by-bot", "--robot-code", "robot", "--users", "user",
"--reply", "message-id", "--ref-sender", "D-sender",
"--title", "reply", "--text", "received",
)
if err == nil || !strings.Contains(err.Error(), "only supported with --conversation-id") {
t.Fatalf("direct reference reply error = %v", err)
}
})
t.Run("reference reply rejects rich media", func(t *testing.T) {
err := runChatCoverageCommand(t, &scriptedToolCaller{},
"message", "send-by-bot", "--robot-code", "robot", "--conversation-id", "group",
"--reply", "message-id", "--ref-sender", "D-sender",
"--msg-type", "image", "--image-url", "https://example.test/image.png",
)
if err == nil || !strings.Contains(err.Error(), "only support Markdown") {
t.Fatalf("rich-media reference reply error = %v", err)
}
})
t.Run("direct image", func(t *testing.T) {
+6 -13
View File
@@ -57,7 +57,7 @@ var ChatMessages = shortcut.Shortcut{
Command: "+chat-messages",
Product: "chat",
Description: "读取指定群聊或单聊的消息记录,支持有界全量分页与原子 JSON 导出",
Intent: "当你要读取或导出一个指定群聊或单聊的消息记录时使用;--sender 是可选过滤条件,可传姓名、userId 或 openDingTalkId:姓名优先唯一解析,稳定 ID 精确路由;通讯录无法分类时仍按原值 userId 筛选,但无稳定 senderId 命中不得作完整否定结论。不传时原样读取会话且不查询发送者身份。sender 仅是展示名,身份只比较稳定 senderId;" +
Intent: "当你要读取或导出一个指定群聊或单聊的消息记录时使用;--sender 是可选的姓名、userId 或 openDingTalkId 混合入口:姓名优先唯一解析,稳定 ID 精确路由;通讯录无法分类时仍按原值 userId 筛选并保留 identity_unverified,可交付精确命中但不能把原值升级为已验证身份或作完整否定结论。--sender-query 只按姓名唯一解析,解析失败会抑制未过滤消息并返回错误。不传发送者条件时原样读取会话且不查询发送者身份。sender 展示名不参与身份比较;" +
"群聊的 --group 可传群名或 openConversationId,单聊可传 --user 或 --open-dingtalk-id,所有目标参数互斥且必须选一个。自然群名只在唯一解析后读取,多候选会返回结构化 candidates。" +
"省略时间参数时默认从当前时间向前读取最近消息;兼容模式可用 --time/--direction,范围模式可用公开可选的 --start/--end/--order(兼容 --start-time/--end-time/--sort),范围语义为 [start,end)。" +
"全量读取用 --page-all,并由 --page-limit/--max-items 保持有界;结果公开 complete、hasMore、nextPage、stopReason、截断和逐页失败,不能把部分结果称为完整。--output 把同一 ledger 原子写为工作目录内 JSON。" +
@@ -83,7 +83,7 @@ var ChatMessages = shortcut.Shortcut{
},
Selection: contract.SelectionSpec{
AgentSummary: "读取指定群聊或单聊的消息记录,支持有界全量分页与原子 JSON 导出",
UseWhen: []string{"当你要读取或导出一个指定群聊或单聊的消息记录时使用;--sender 是可选过滤条件,可传姓名、userId 或 openDingTalkId:姓名优先唯一解析,稳定 ID 精确路由;通讯录无法分类时仍按原值 userId 筛选,但无稳定 senderId 命中不得作完整否定结论。不传时原样读取会话且不查询发送者身份。sender 仅是展示名,身份只比较稳定 senderId;" +
UseWhen: []string{"当你要读取或导出一个指定群聊或单聊的消息记录时使用;--sender 是可选的姓名、userId 或 openDingTalkId 混合入口:姓名优先唯一解析,稳定 ID 精确路由;通讯录无法分类时仍按原值 userId 筛选并保留 identity_unverified,可交付精确命中但不能把原值升级为已验证身份或作完整否定结论。--sender-query 只按姓名唯一解析,解析失败会抑制未过滤消息并返回错误。不传发送者条件时原样读取会话且不查询发送者身份。sender 展示名不参与身份比较;" +
"群聊的 --group 可传群名或 openConversationId,单聊可传 --user 或 --open-dingtalk-id,所有目标参数互斥且必须选一个。自然群名只在唯一解析后读取,多候选会返回结构化 candidates。" +
"省略时间参数时默认从当前时间向前读取最近消息;兼容模式可用 --time/--direction,范围模式可用公开可选的 --start/--end/--order(兼容 --start-time/--end-time/--sort),范围语义为 [start,end)。" +
"全量读取用 --page-all,并由 --page-limit/--max-items 保持有界;结果公开 complete、hasMore、nextPage、stopReason、截断和逐页失败,不能把部分结果称为完整。--output 把同一 ledger 原子写为工作目录内 JSON。" +
@@ -104,8 +104,8 @@ var ChatMessages = shortcut.Shortcut{
{Name: "user", Type: shortcut.FlagString, Desc: "单聊对方的 userId,与 --group 互斥"},
{Name: "user-query", Type: shortcut.FlagString, Desc: "按姓名解析唯一 openDingTalkId 的兼容入口", Hidden: true},
{Name: "open-dingtalk-id", Type: shortcut.FlagString, Desc: "单聊对方的 openDingTalkId,与 --group/--user 互斥"},
{Name: "sender", Type: shortcut.FlagStringSlice, Desc: "单个或多个发送者姓名、userId 或 openDingTalkId;姓名唯一解析,稳定 ID 精确路由,通讯录故障不阻断原值 userId 筛选"},
{Name: "sender-query", Type: shortcut.FlagStringSlice, Desc: "显式按姓名唯一解析发送者的兼容入口(可选,可重复或逗号分隔)"},
{Name: "sender", Type: shortcut.FlagStringSlice, Desc: "单个或多个发送者姓名、userId 或 openDingTalkId;姓名唯一解析,稳定 ID 精确路由,通讯录无法分类时按原值 userId 筛选并保留身份未验证状态"},
{Name: "sender-query", Type: shortcut.FlagStringSlice, Desc: "显式按姓名唯一解析发送者的兼容入口;解析失败时抑制未过滤消息并返回错误(可选,可重复或逗号分隔)"},
{Name: "time", Type: shortcut.FlagString, Desc: "时间边界,如 \"2025-03-01 00:00:00\";--time 必须是 RFC3339、YYYY-MM-DD HH:mm:ss 或 YYYY-MM-DD;省略时从当前时间向前读取最近消息"},
{Name: "start", Type: shortcut.FlagString, Desc: "范围开始时间(可选、包含),支持 RFC3339、YYYY-MM-DD HH:mm:ss 或 YYYY-MM-DD"},
{Name: "start-time", Type: shortcut.FlagString, Desc: "--start 的 lark-cli 对齐别名(可选、包含)"},
@@ -418,7 +418,7 @@ func applyOptionalChatMessagesSenderFilter(
payload["complete"] = false
payload["partial"] = len(filtered) > 0
}
unverifiedSenderInputs := chatMessagesUnverifiedSenderInputs(filtered, filter.resolutions)
unverifiedSenderInputs := chatMessagesUnverifiedSenderInputs(filter.resolutions)
if len(unverifiedSenderInputs) > 0 {
failures, _ := payload["failures"].([]map[string]any)
failures = append(failures, map[string]any{
@@ -470,18 +470,11 @@ func applyOptionalChatMessagesSenderFilter(
}
func chatMessagesUnverifiedSenderInputs(
messages []map[string]any,
resolutions []targetresolver.UserResolution,
) []string {
observed := map[string]bool{}
for _, message := range messages {
if senderID := strings.TrimSpace(fmt.Sprint(chatmsg.SenderID(message))); senderID != "" && senderID != "<nil>" {
observed[senderID] = true
}
}
values := make([]string, 0)
for _, resolution := range resolutions {
if targetresolver.IsUnverifiedUserIDResolution(resolution) && !observed[resolution.Selected.UserID] {
if targetresolver.IsUnverifiedUserIDResolution(resolution) {
values = append(values, resolution.Query)
}
}
@@ -631,7 +631,7 @@ func TestCrossPlatformCoverageChatMessagesSenderFilterFailureEdges(t *testing.T)
}
})
t.Run("unverified mixed sender becomes verified by an exact sender id match", func(t *testing.T) {
t.Run("unverified mixed sender remains unverified after an exact sender id match", func(t *testing.T) {
payload := map[string]any{"complete": true, "failures": []map[string]any{}}
resolution := targetresolver.UserResolution{
Status: targetresolver.StatusResolved,
@@ -655,13 +655,18 @@ func TestCrossPlatformCoverageChatMessagesSenderFilterFailureEdges(t *testing.T)
{"openMessageId": "other", "senderUserId": "other-user"},
}, &filter,
)
if len(filtered) != 1 || payload["complete"] != true || len(payload["failures"].([]map[string]any)) != 0 {
if len(filtered) != 1 || payload["complete"] != false || payload["partial"] != true ||
payload["failedCount"] != 1 || len(payload["failures"].([]map[string]any)) != 1 {
t.Fatalf("filtered=%#v payload=%#v", filtered, payload)
}
identity := payload["identityResult"].(map[string]any)
if identity["status"] != "evaluated" || identity["negativeConclusionAllowed"] != true {
if identity["status"] != "identity_unverified" || identity["negativeConclusionAllowed"] != false {
t.Fatalf("identityResult=%#v", identity)
}
filterResult := payload["senderFilter"].(map[string]any)
if filterResult["status"] != "identity_unverified" {
t.Fatalf("senderFilter=%#v", filterResult)
}
})
t.Run("unverified mixed sender without a match blocks complete negative conclusion", func(t *testing.T) {
+4 -11
View File
@@ -35,7 +35,7 @@ import (
// search hits through list_messages_by_ids in chunks of 50. A later-page or
// enrichment failure never turns a partial result into a false success: the
// output carries an explicit failure ledger and complete=false.
const searchMsgIntent = "当你要按关键词、发送者、@对象、消息类型、机器人来源或会话范围组合搜索 IM 消息时使用;可搜索单个、多个或全部会话。--group/--groups 接受群名或 openConversationId,--sender/--senders 接受姓名、userId 或 openDingTalkId:姓名优先唯一解析,稳定 ID 精确路由;通讯录无法分类时仍按原值 userId 执行,但无稳定 senderId 命中不得作完整否定结论。默认查询近 7 天,也可指定精确起止时间和输出顺序。" +
const searchMsgIntent = "当你要按关键词、发送者、@对象、消息类型、机器人来源或会话范围组合搜索 IM 消息时使用;可搜索单个、多个或全部会话。--group/--groups 接受群名或 openConversationId,--sender/--senders 接受姓名、userId 或 openDingTalkId:姓名优先唯一解析,稳定 ID 精确路由;通讯录无法分类时仍按原值 userId 执行并保留 identity_unverified,可交付精确命中但不能把原值升级为已验证身份或作完整否定结论。默认查询近 7 天,也可指定精确起止时间和输出顺序。" +
"显式指定会话时会先验证 CID,再执行有界全局扫描并在本地精确过滤,避免下层忽略非法 CID 或群聊 CID。" +
"--page-all 会连续拉取游标页,默认再按消息 ID 分批富化详情;任何续页或富化失败都会保留已取得结果并返回逐项失败 ledger,绝不把截断结果标成完整。" +
"--download-resources 使用安全本地路径、默认不覆盖和原子落盘。"
@@ -86,7 +86,7 @@ var SearchMsg = shortcut.Shortcut{
{Name: "groups", Type: shortcut.FlagStringSlice, Desc: "多个群名或 openConversationId;可混合输入并逐项唯一解析"},
{Name: "chat-id", Type: shortcut.FlagStringSlice, Desc: "--groups 的 lark-cli 对齐别名;只接受 openConversationId"},
{Name: "chat-query", Type: shortcut.FlagStringSlice, Desc: "显式按群名唯一解析的兼容入口(可选,可重复或逗号分隔)"},
{Name: "senders", Type: shortcut.FlagStringSlice, Desc: "多个发送者姓名、userId 或 openDingTalkId;姓名唯一解析,稳定 ID 精确路由,通讯录故障不阻断原值 userId 查询"},
{Name: "senders", Type: shortcut.FlagStringSlice, Desc: "多个发送者姓名、userId 或 openDingTalkId;姓名唯一解析,稳定 ID 精确路由,通讯录无法分类时按原值 userId 查询并保留身份未验证状态"},
{Name: "sender", Type: shortcut.FlagStringSlice, Desc: "单个或多个发送者姓名、userId 或 openDingTalkId;--senders 的兼容别名,保留同样的三态解析与安全降级语义"},
{Name: "sender-query", Type: shortcut.FlagStringSlice, Desc: "显式按姓名唯一解析的兼容入口(可选,可重复或逗号分隔)"},
{Name: "at-me", Type: shortcut.FlagBool, Desc: "只搜索 @我 的消息"},
@@ -278,7 +278,7 @@ var SearchMsg = shortcut.Shortcut{
return searchSenderScopeUnverifiedError(resolvedFilters.Senders, unverifiableMessageIDs)
}
}
unverifiedSenderInputs := searchUnverifiedSenderInputs(messages, resolvedFilters.Senders)
unverifiedSenderInputs := searchUnverifiedSenderInputs(resolvedFilters.Senders)
if len(unverifiedSenderInputs) > 0 {
failures = append(failures, map[string]any{
"stage": "sender_identity_verification",
@@ -701,18 +701,11 @@ func filterSearchSenderScope(
}
func searchUnverifiedSenderInputs(
messages []map[string]any,
resolutions []targetresolver.UserResolution,
) []string {
observed := map[string]bool{}
for _, message := range messages {
if senderID := strings.TrimSpace(fmt.Sprint(chatmsg.SenderID(message))); senderID != "" && senderID != "<nil>" {
observed[senderID] = true
}
}
values := make([]string, 0)
for _, resolution := range resolutions {
if targetresolver.IsUnverifiedUserIDResolution(resolution) && !observed[resolution.Selected.UserID] {
if targetresolver.IsUnverifiedUserIDResolution(resolution) {
values = append(values, resolution.Query)
}
}
@@ -277,9 +277,14 @@ func TestCrossPlatformCoverageSearchMsgStableUserIDContinuesWhenDirectoryIsUnava
if _, exists := caller.calls[1].args["senderOpenDingTakIds"]; exists {
t.Fatalf("unrelated candidate leaked into request: %#v", caller.calls[1].args)
}
if payload["complete"] != true || payload["count"] != float64(1) {
if payload["complete"] != false || payload["count"] != float64(1) ||
payload["failedCount"] != float64(1) {
t.Fatalf("payload=%#v", payload)
}
scope := payload["senderScope"].(map[string]any)
if scope["status"] != "identity_unverified" || scope["targetsResolved"] != false {
t.Fatalf("senderScope=%#v", scope)
}
})
t.Run("mixed sender preserves positive matches and blocks complete negative conclusion", func(t *testing.T) {
@@ -298,11 +303,11 @@ func TestCrossPlatformCoverageSearchMsgStableUserIDContinuesWhenDirectoryIsUnava
if got := caller.calls[1].args["senderUserIds"]; !reflect.DeepEqual(got, []string{"stable-user-id"}) {
t.Fatalf("senderUserIds=%#v", got)
}
if payload["count"] != float64(1) || payload["complete"] != true || payload["failedCount"] != float64(0) {
if payload["count"] != float64(1) || payload["complete"] != false || payload["failedCount"] != float64(1) {
t.Fatalf("payload=%#v", payload)
}
scope := payload["senderScope"].(map[string]any)
if _, exists := scope["status"]; exists || scope["targetsResolved"] != true {
if scope["status"] != "identity_unverified" || scope["targetsResolved"] != false {
t.Fatalf("senderScope=%#v", scope)
}
})
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,286 @@
#!/usr/bin/env python3
"""Fast unit tests for eval_tool_search_ranking.py."""
from __future__ import annotations
import importlib.util
import json
import math
import sys
import tempfile
import unittest
from pathlib import Path
SCRIPT = Path(__file__).with_name("eval_tool_search_ranking.py")
SPEC = importlib.util.spec_from_file_location("eval_tool_search_ranking", SCRIPT)
if SPEC is None or SPEC.loader is None: # pragma: no cover
raise RuntimeError(f"cannot import {SCRIPT}")
ranking = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = ranking
SPEC.loader.exec_module(ranking)
class ToolSearchRankingEvalTest(unittest.TestCase):
def tool(self, canonical: str, summary: str) -> ranking.ToolDocument:
return ranking.ToolDocument(
canonical=canonical,
product=canonical.partition(".")[0],
cli_path=canonical.replace(".", " "),
identities=(canonical,),
fields={
"identity": canonical,
"summary": summary,
"description": "",
"parameters": "",
},
use_when=(),
avoid_when=(),
examples=(),
)
def test_tokenize_preserves_identifier_and_splits_mixed_text(self) -> None:
tokens = ranking.tokenize("chat.send_personal_message 给群聊发送FilePath")
self.assertIn("chat.send_personal_message", tokens)
self.assertIn("群聊", tokens)
self.assertIn("file", tokens)
self.assertIn("path", tokens)
def test_parameter_projection_separates_names_descriptions_and_types(self) -> None:
parameters = {
"file": {
"property": "filePath",
"description": "本地文件路径",
"interface_description": "上传的媒体文件",
"type": "string",
"interface_type": "path",
"enum": ["document"],
}
}
names = ranking.parameter_text(
parameters,
include_names=True,
include_descriptions=False,
include_types=False,
)
self.assertIn("filePath", names)
self.assertIn("document", names)
self.assertNotIn("本地文件路径", names)
descriptions = ranking.parameter_text(
parameters,
include_names=False,
include_descriptions=True,
include_types=False,
)
self.assertIn("本地文件路径", descriptions)
self.assertIn("上传的媒体文件", descriptions)
self.assertNotIn("filePath", descriptions)
types = ranking.parameter_text(
parameters,
include_names=False,
include_descriptions=False,
include_types=True,
)
self.assertEqual(types, "string path")
def test_load_catalog_merges_build_time_product_shards(self) -> None:
with tempfile.TemporaryDirectory() as temporary:
catalog_dir = Path(temporary)
tools_dir = catalog_dir / "tools"
tools_dir.mkdir()
(catalog_dir / "catalog.json").write_text(
json.dumps(
{
"source_hash": "sha256:source",
"surface_hash": "sha256:surface",
"catalog": {
"agent_metadata": {"surface_tools": 2}
},
}
),
encoding="utf-8",
)
pairs = [("chat", "chat.send"), ("drive", "drive.get")]
for product, canonical in pairs:
(tools_dir / f"{product}.json").write_text(
json.dumps(
{
"product": product,
"tools": {
canonical: {
"canonical_path": canonical,
"cli_path": canonical.replace(".", " "),
"agent_summary": canonical,
"product_id": product,
}
},
}
),
encoding="utf-8",
)
source_hash, tools = ranking.load_catalog(catalog_dir)
self.assertEqual(source_hash, "sha256:source")
self.assertEqual(
[tool.canonical for tool in tools], ["chat.send", "drive.get"]
)
def test_load_catalog_rejects_duplicate_shard_tools(self) -> None:
with tempfile.TemporaryDirectory() as temporary:
catalog_dir = Path(temporary)
tools_dir = catalog_dir / "tools"
tools_dir.mkdir()
(catalog_dir / "catalog.json").write_text(
json.dumps({"catalog": {"agent_metadata": {"surface_tools": 1}}}),
encoding="utf-8",
)
shard = {"tools": {"chat.send": {"canonical_path": "chat.send"}}}
for name in ["a.json", "b.json"]:
(tools_dir / name).write_text(json.dumps(shard), encoding="utf-8")
with self.assertRaisesRegex(ValueError, "duplicate tools"):
ranking.load_catalog(catalog_dir)
def test_bm25_ranks_relevant_tool_first(self) -> None:
tools = [
self.tool("chat.send", "发送群聊消息"),
self.tool("drive.download", "下载钉盘文件"),
]
for ranker_type in [
ranking.BM25Ranker,
ranking.BM25LRanker,
ranking.BM25PlusRanker,
]:
with self.subTest(ranker=ranker_type.__name__):
ranker = ranker_type(tools)
self.assertEqual(
ranker.rank_many(["群聊发消息"])["群聊发消息"][0],
"chat.send",
)
def test_vector_and_set_rankers_rank_relevant_chinese_first(self) -> None:
tools = [
self.tool("chat.send", "发送群聊消息"),
self.tool("drive.download", "下载钉盘文件"),
]
for ranker_type in [ranking.TfidfCosineRanker, ranking.WeightedJaccardRanker]:
with self.subTest(ranker=ranker_type.__name__):
ranker = ranker_type(tools)
self.assertEqual(ranker.rank("群聊发消息")[0], "chat.send")
def test_stable_rank_breaks_ties_by_canonical(self) -> None:
self.assertEqual(
ranking.stable_rank({"z.tool": 1.0, "a.tool": 1.0}),
["a.tool", "z.tool"],
)
def test_rrf_rewards_agreement(self) -> None:
fused = ranking.rrf_rank(
[["solo", "shared"], ["other", "shared"]]
)
self.assertEqual(fused[0], "shared")
def test_rrf_k_changes_tail_influence_but_not_tie_break(self) -> None:
rankings = [["a", "shared"], ["b", "shared"]]
self.assertEqual(ranking.rrf_rank(rankings, k=10.0)[0], "shared")
self.assertEqual(ranking.rrf_rank(rankings, k=60.0)[0], "shared")
def test_forbidden_metric_is_lower_when_forbidden_tool_is_absent(self) -> None:
cases = [ranking.SingleCase("query", "danger.delete", "danger")]
metrics = ranking.evaluate_forbidden(cases, {"query": ["safe.read"]})
self.assertEqual(metrics["forbidden_at_5"], 0.0)
def test_workflow_decomposition_round_robins_steps(self) -> None:
case = ranking.WorkflowCase(
case_id="sample",
query="do both",
required=("first", "second"),
subqueries=("step one", "step two"),
)
result = ranking.decompose_workflow_rankings(
[case],
{
"step one": ["first", "shared"],
"step two": ["second", "shared"],
},
)
self.assertEqual(result["do both"][:2], ["first", "second"])
def test_workflow_ndcg_rewards_required_tools_near_the_top(self) -> None:
case = ranking.WorkflowCase(
case_id="sample",
query="do both",
required=("first", "second"),
subqueries=(),
)
best = ranking.evaluate_workflows(
[case], {"do both": ["first", "second", "other"]}
)
delayed = ranking.evaluate_workflows(
[case], {"do both": ["other", "first", "second"]}
)
self.assertEqual(best["mean_ndcg_at_5"], 1.0)
self.assertGreater(
best["mean_ndcg_at_5"], delayed["mean_ndcg_at_5"]
)
def test_query_language_slices(self) -> None:
self.assertEqual(ranking.query_language_slice("给群里发文件"), "chinese_only")
self.assertEqual(
ranking.query_language_slice("读取 baseId 对应表格"),
"mixed_chinese_ascii",
)
self.assertEqual(ranking.query_language_slice("chat send"), "non_chinese")
def test_bm25_variant_formulas_match_hand_calculation(self) -> None:
documents = {"doc": ["term", "term"], "other": ["other"]}
length_ratio = 2.0 / 1.5
idf = math.log(2.0)
query_weight = math.log(2.0)
plus = ranking.BM25VariantIndex(documents, variant="bm25plus")
plus_contribution = 2.0 * 1.9 / (
2.0 + 0.9 * (0.6 + 0.4 * length_ratio)
) + 1.0
self.assertAlmostEqual(
plus.scores("term")["doc"],
idf * plus_contribution * query_weight,
)
bm25l = ranking.BM25VariantIndex(documents, variant="bm25l")
normalized = 2.0 / (0.6 + 0.4 * length_ratio)
bm25l_contribution = 1.9 * (normalized + 1.0) / (
0.9 + normalized + 1.0
)
self.assertAlmostEqual(
bm25l.scores("term")["doc"],
idf * bm25l_contribution * query_weight,
)
def test_product_cluster_bootstrap_uses_equal_product_weight(self) -> None:
cases = [
ranking.SingleCase("a1", "a", "large"),
ranking.SingleCase("a2", "a", "large"),
ranking.SingleCase("a3", "a", "large"),
ranking.SingleCase("b1", "b", "small"),
]
result = ranking.paired_product_cluster_bootstrap_interval(
cases,
baseline=[False, False, False, True],
candidate=[True, True, True, False],
samples=200,
)
self.assertEqual(result["product_clusters"], 2)
self.assertAlmostEqual(result["macro_product_recall_at_5_delta"], 0.0)
def test_percentile_uses_nearest_rank(self) -> None:
values = [5.0, 1.0, 4.0, 2.0, 3.0]
self.assertEqual(ranking.percentile(values, 0.5), 3.0)
self.assertEqual(ranking.percentile(values, 0.95), 5.0)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,313 @@
#!/usr/bin/env python3
"""Verify frozen Tool Search evaluation inputs and independent qrels gates."""
from __future__ import annotations
import argparse
import json
import re
import subprocess
from pathlib import Path
from typing import Any
REPO_ROOT = Path(__file__).resolve().parents[2]
DEFAULT_MANIFEST = REPO_ROOT / "scripts/testdata/tool_search_eval_manifest.json"
DEFAULT_CATALOG_DIR = REPO_ROOT / ".worktrees/policy-tmp/tool-search-schema-catalog"
CHINESE = re.compile(r"[\u3400-\u9fff]")
ASCII_WORD = re.compile(r"[A-Za-z0-9]")
def language_slice(query: str) -> str:
has_chinese = bool(CHINESE.search(query))
has_ascii = bool(ASCII_WORD.search(query))
if has_chinese and has_ascii:
return "mixed_chinese_ascii"
if has_chinese:
return "chinese_only"
return "english"
def require_string(value: Any, field: str, problems: list[str]) -> str:
if not isinstance(value, str) or not value.strip():
problems.append(f"{field} must be a non-empty string")
return ""
return value.strip()
def validate_case(case: Any, index: int, catalog_tools: set[str], problems: list[str]) -> tuple[str, bool]:
prefix = f"cases[{index}]"
if not isinstance(case, dict):
problems.append(f"{prefix} must be an object")
return "", False
require_string(case.get("id"), f"{prefix}.id", problems)
query = require_string(case.get("query"), f"{prefix}.query", problems)
declared_language = require_string(case.get("language"), f"{prefix}.language", problems)
actual_language = language_slice(query) if query else ""
if declared_language and declared_language != actual_language:
problems.append(
f"{prefix}.language={declared_language!r} does not match {actual_language!r}"
)
qrels = case.get("qrels")
if not isinstance(qrels, list) or not qrels:
problems.append(f"{prefix}.qrels must be a non-empty list")
else:
seen: set[str] = set()
for qrel_index, qrel in enumerate(qrels):
qrel_prefix = f"{prefix}.qrels[{qrel_index}]"
if not isinstance(qrel, dict):
problems.append(f"{qrel_prefix} must be an object")
continue
canonical = require_string(qrel.get("canonical"), f"{qrel_prefix}.canonical", problems)
relevance = qrel.get("relevance")
if canonical in seen:
problems.append(f"{qrel_prefix}.canonical duplicates {canonical!r}")
seen.add(canonical)
if canonical and canonical not in catalog_tools:
problems.append(f"{qrel_prefix}.canonical {canonical!r} is not in the Catalog")
if not isinstance(relevance, int) or relevance not in {1, 2, 3}:
problems.append(f"{qrel_prefix}.relevance must be 1, 2 or 3")
forbidden = case.get("forbidden", [])
alternatives = case.get("alternative_gold", [])
if forbidden:
valid_forbidden = isinstance(forbidden, list) and all(isinstance(item, str) and item.strip() for item in forbidden)
if not valid_forbidden:
problems.append(f"{prefix}.forbidden must be a non-empty string list")
elif len(set(forbidden)) != len(forbidden):
problems.append(f"{prefix}.forbidden must not contain duplicates")
elif any(item not in catalog_tools for item in forbidden):
problems.append(f"{prefix}.forbidden contains an unknown Catalog tool")
valid_alternatives = isinstance(alternatives, list) and bool(alternatives) and all(
isinstance(item, str) and item.strip() for item in alternatives
)
if not valid_alternatives:
problems.append(f"{prefix}.alternative_gold is required when forbidden is non-empty")
elif len(set(alternatives)) != len(alternatives):
problems.append(f"{prefix}.alternative_gold must not contain duplicates")
elif any(item not in catalog_tools for item in alternatives):
problems.append(f"{prefix}.alternative_gold contains an unknown Catalog tool")
if valid_forbidden and valid_alternatives and set(forbidden).intersection(alternatives):
problems.append(f"{prefix}.forbidden and alternative_gold must be disjoint")
elif alternatives:
problems.append(f"{prefix}.alternative_gold requires a non-empty forbidden list")
workflow = case.get("workflow")
is_workflow = workflow is not None
if is_workflow:
if not isinstance(workflow, dict):
problems.append(f"{prefix}.workflow must be an object")
else:
required = workflow.get("required", [])
if not isinstance(required, list) or not 2 <= len(required) <= 4:
problems.append(f"{prefix}.workflow.required must contain 2 to 4 tools")
elif any(item not in catalog_tools for item in required):
problems.append(f"{prefix}.workflow.required contains an unknown Catalog tool")
subqueries = workflow.get("subqueries", [])
if subqueries and (
not isinstance(subqueries, list)
or not all(isinstance(item, str) and item.strip() for item in subqueries)
):
problems.append(f"{prefix}.workflow.subqueries must be a non-empty string list when present")
return actual_language, is_workflow
def load_generated_catalog(catalog_dir: Path, problems: list[str]) -> tuple[dict[str, Any], set[str], dict[str, dict[str, Any]]]:
envelope_path = catalog_dir / "catalog.json"
tools_dir = catalog_dir / "tools"
if not envelope_path.is_file():
problems.append(f"generated Catalog envelope does not exist: {envelope_path}")
return {}, set(), {}
if not tools_dir.is_dir():
problems.append(f"generated Catalog tools directory does not exist: {tools_dir}")
return {}, set(), {}
envelope = json.loads(envelope_path.read_text(encoding="utf-8"))
tools: set[str] = set()
metadata: dict[str, dict[str, Any]] = {}
for shard_path in sorted(tools_dir.glob("*.json")):
shard = json.loads(shard_path.read_text(encoding="utf-8"))
shard_tools = shard.get("tools")
if not isinstance(shard_tools, dict):
problems.append(f"generated Catalog shard has no tools object: {shard_path}")
continue
overlap = tools.intersection(shard_tools)
if overlap:
problems.append(f"generated Catalog shards duplicate tools: {sorted(overlap)}")
tools.update(shard_tools)
metadata.update(shard_tools)
return envelope, tools, metadata
def validate_manifest(
manifest_path: Path,
catalog_dir: Path | None = None,
) -> list[str]:
problems: list[str] = []
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
if manifest.get("version") != "tool-search-eval-manifest.v1":
problems.append("manifest version must be tool-search-eval-manifest.v1")
base_commit = require_string(manifest.get("base_repository_commit"), "base_repository_commit", problems)
if base_commit:
exists = subprocess.run(
["git", "cat-file", "-e", f"{base_commit}^{{commit}}"],
cwd=REPO_ROOT,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
if exists.returncode != 0:
problems.append("base_repository_commit does not exist")
else:
ancestor = subprocess.run(
["git", "merge-base", "--is-ancestor", base_commit, "HEAD"],
cwd=REPO_ROOT,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
if ancestor.returncode != 0:
problems.append("base_repository_commit is not an ancestor of HEAD")
owners = manifest.get("owners", {})
retrieval_owner = require_string(owners.get("retrieval"), "owners.retrieval", problems)
evaluation_owner = require_string(
owners.get("independent_evaluation"), "owners.independent_evaluation", problems
)
if retrieval_owner and retrieval_owner == evaluation_owner:
problems.append("retrieval and independent evaluation owners must differ")
require_string(owners.get("separation_rule"), "owners.separation_rule", problems)
proxy = manifest.get("proxy_v1", {})
# Source files evolve with every ranking change; the manifest deliberately
# does not pin their content hashes (that would couple every code change
# to a manifest update). Only their existence is asserted. Behavior is
# gated by the Go evaluation sentinels instead.
for path_field in ["workflow_path", "evaluator_path"]:
relative = require_string(proxy.get(path_field), f"proxy_v1.{path_field}", problems)
if relative and not (REPO_ROOT / relative).is_file():
problems.append(f"{relative} does not exist")
catalog_payload: dict[str, Any] = {}
catalog_tools: set[str] = set()
catalog_metadata: dict[str, dict[str, Any]] = {}
if catalog_dir is not None:
catalog_payload, catalog_tools, catalog_metadata = load_generated_catalog(catalog_dir, problems)
if len(catalog_tools) != proxy.get("tool_count"):
problems.append("proxy_v1.tool_count differs from generated Catalog")
if catalog_payload.get("source_hash") != proxy.get("catalog_source_hash"):
problems.append("proxy_v1.catalog_source_hash differs from generated Catalog")
if catalog_payload.get("surface_hash") != proxy.get("catalog_surface_hash"):
problems.append("proxy_v1.catalog_surface_hash differs from generated Catalog")
independent = manifest.get("independent_test_v1", {})
qrels_relative = require_string(independent.get("path"), "independent_test_v1.path", problems)
qrels_path = REPO_ROOT / qrels_relative
if not qrels_path.is_file():
problems.append(f"{qrels_relative} does not exist")
return problems
qrels = json.loads(qrels_path.read_text(encoding="utf-8"))
if qrels.get("version") != "tool-search-qrels.v1":
problems.append("independent qrels version must be tool-search-qrels.v1")
cases = qrels.get("cases")
if not isinstance(cases, list):
problems.append("independent qrels cases must be a list")
cases = []
ids: set[str] = set()
for index, case in enumerate(cases):
if isinstance(case, dict) and isinstance(case.get("id"), str):
if case["id"] in ids:
problems.append(f"cases[{index}].id duplicates {case['id']!r}")
ids.add(case["id"])
validate_case(case, index, catalog_tools, problems)
# Coverage derivation also validates qrels data quality (confusion_family
# consistency) as a side effect; the release-time sealed gate that consumed
# the coverage set has been removed.
derive_independent_coverage(cases, catalog_metadata, problems)
return problems
def derive_independent_coverage(
cases: list[Any], catalog_metadata: dict[str, dict[str, Any]], problems: list[str]
) -> set[str]:
"""Derive release coverage from qrels and Catalog facts, never self-asserted labels."""
coverage: set[str] = set()
if not catalog_metadata:
return coverage
catalog_products = {str(tool.get("product_id", "")) for tool in catalog_metadata.values()}
judged: set[str] = set()
effects: set[str] = set()
workflow_ok = False
graded_equivalent = False
forbidden_alternative = False
sibling_confusion = False
for index, case in enumerate(cases):
if not isinstance(case, dict):
continue
canonicals = [
qrel.get("canonical") for qrel in case.get("qrels", []) if isinstance(qrel, dict)
]
for canonical in canonicals:
if canonical in catalog_metadata:
judged.add(canonical)
effects.add(str(catalog_metadata[canonical].get("effect", "")))
if len(canonicals) >= 2:
backing = {
json.dumps(catalog_metadata.get(item, {}).get("interface_ref", {}), sort_keys=True)
for item in canonicals
}
relevances = {
qrel.get("relevance") for qrel in case.get("qrels", []) if isinstance(qrel, dict)
}
if len(backing) == 1 and "{}" not in backing and len(relevances) >= 2:
graded_equivalent = True
forbidden = case.get("forbidden", [])
alternatives = case.get("alternative_gold", [])
if forbidden and alternatives:
forbidden_alternative = True
workflow = case.get("workflow")
if isinstance(workflow, dict) and 2 <= len(workflow.get("required", [])) <= 4:
workflow_ok = True
confusion = case.get("confusion_family", [])
if confusion:
if not isinstance(confusion, list) or len(confusion) < 2 or any(item not in catalog_metadata for item in confusion):
problems.append(f"cases[{index}].confusion_family must contain at least two Catalog tools")
elif set(confusion).intersection(canonicals):
problems.append(f"cases[{index}].confusion_family must be disjoint from qrels")
elif len({catalog_metadata[item].get("product_id") for item in confusion}) != 1:
problems.append(f"cases[{index}].confusion_family must belong to one product")
else:
sibling_confusion = True
judged_products = {str(catalog_metadata[item].get("product_id", "")) for item in judged}
if judged_products == catalog_products:
coverage.add("all_reviewed_products")
if {"read", "write", "destructive"}.issubset(effects):
coverage.add("read_write_destructive")
if sibling_confusion:
coverage.add("sibling_confusion")
if graded_equivalent:
coverage.add("graded_equivalent_entrypoints")
if forbidden_alternative:
coverage.add("forbidden_with_alternative_gold")
if workflow_ok:
coverage.add("two_to_four_step_workflows")
return coverage
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--manifest", type=Path, default=DEFAULT_MANIFEST)
parser.add_argument("--catalog-dir", type=Path, default=DEFAULT_CATALOG_DIR)
args = parser.parse_args()
problems = validate_manifest(
args.manifest.resolve(),
args.catalog_dir.resolve(),
)
if problems:
for problem in problems:
print(f"ERROR: {problem}")
return 1
print("tool search evaluation manifest: OK")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,118 @@
#!/usr/bin/env python3
"""Unit tests for verify_tool_search_eval_manifest.py."""
from __future__ import annotations
import importlib.util
import sys
import unittest
from pathlib import Path
SCRIPT = Path(__file__).with_name("verify_tool_search_eval_manifest.py")
SPEC = importlib.util.spec_from_file_location("verify_tool_search_eval_manifest", SCRIPT)
if SPEC is None or SPEC.loader is None: # pragma: no cover
raise RuntimeError(f"cannot import {SCRIPT}")
verify = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = verify
SPEC.loader.exec_module(verify)
class ToolSearchEvalManifestTest(unittest.TestCase):
def test_language_slice(self) -> None:
self.assertEqual(verify.language_slice("给群里发文件"), "chinese_only")
self.assertEqual(
verify.language_slice("读取 baseId 对应的表格"),
"mixed_chinese_ascii",
)
self.assertEqual(verify.language_slice("send a file to the group"), "english")
def test_collecting_manifest_is_valid(self) -> None:
self.assertEqual(verify.validate_manifest(verify.DEFAULT_MANIFEST), [])
def test_graded_equivalent_qrels_are_accepted(self) -> None:
catalog = {
"dev.search_open_platform_docs_rag",
"devdoc.search_open_platform_docs_rag",
}
case = {
"id": "devdoc-search",
"query": "查 OAuth2 接入文档",
"language": "mixed_chinese_ascii",
"qrels": [
{
"canonical": "devdoc.search_open_platform_docs_rag",
"relevance": 3,
},
{
"canonical": "dev.search_open_platform_docs_rag",
"relevance": 2,
},
],
}
problems: list[str] = []
language, workflow = verify.validate_case(case, 0, catalog, problems)
self.assertEqual(language, "mixed_chinese_ascii")
self.assertFalse(workflow)
self.assertEqual(problems, [])
def test_forbidden_requires_alternative_gold(self) -> None:
case = {
"id": "unsafe",
"query": "删除记录",
"language": "chinese_only",
"qrels": [{"canonical": "safe.read", "relevance": 3}],
"forbidden": ["danger.delete"],
}
problems: list[str] = []
verify.validate_case(case, 0, {"safe.read", "danger.delete"}, problems)
self.assertTrue(any("alternative_gold" in problem for problem in problems))
def test_forbidden_and_alternative_gold_are_catalog_bound_and_disjoint(self) -> None:
case = {
"id": "unsafe",
"query": "删除记录",
"language": "chinese_only",
"qrels": [{"canonical": "safe.read", "relevance": 3}],
"forbidden": ["danger.delete", "unknown.tool"],
"alternative_gold": ["danger.delete"],
}
problems: list[str] = []
verify.validate_case(case, 0, {"safe.read", "danger.delete"}, problems)
self.assertTrue(any("unknown Catalog tool" in problem for problem in problems), problems)
self.assertTrue(any("must be disjoint" in problem for problem in problems), problems)
def test_release_coverage_is_derived_not_self_asserted(self) -> None:
metadata = {
"chat.send": {"product_id": "chat", "effect": "write", "interface_ref": {"rpc_name": "send"}},
"chat.delete": {"product_id": "chat", "effect": "destructive", "interface_ref": {"rpc_name": "delete"}},
"doc.read": {"product_id": "doc", "effect": "read", "interface_ref": {"rpc_name": "get"}},
"doc.fetch": {"product_id": "devdoc", "effect": "read", "interface_ref": {"rpc_name": "get"}},
}
cases = [
{
"qrels": [{"canonical": "chat.send", "relevance": 3}],
"forbidden": ["chat.delete"],
"alternative_gold": ["chat.send"],
"confusion_family": ["chat.delete", "chat.send.other"],
"workflow": {"required": ["chat.send", "doc.read"]},
},
{
"qrels": [
{"canonical": "doc.read", "relevance": 3},
{"canonical": "doc.fetch", "relevance": 2},
]
},
{"qrels": [{"canonical": "chat.delete", "relevance": 3}]},
]
problems: list[str] = []
coverage = verify.derive_independent_coverage(cases, metadata, problems)
self.assertIn("all_reviewed_products", coverage)
self.assertIn("read_write_destructive", coverage)
self.assertIn("graded_equivalent_entrypoints", coverage)
self.assertNotIn("sibling_confusion", coverage)
self.assertTrue(any("confusion_family" in problem for problem in problems))
if __name__ == "__main__":
unittest.main()
+3 -3
View File
@@ -136,7 +136,7 @@ def mono_overview(items: list[dict[str, Any]]) -> str:
return f"""{MONO_START}
## Shortcut 总览
下面只统计当前公开 catalog 中的 shortcut,不展开完整明细。已知意图应先按产品 Skill、意图表或任务 reference 选择唯一命令;命令已选中时直接执行,只在参数或安全语义不确定时读取 leaf Schema,在当前 Cobra flags 不确定时读取 leaf Help。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service <service> --format json` 做最后回退;不要为已知高频意图加载完整产品 Catalog。
下面只统计当前公开 catalog 中的 shortcut,不展开完整明细。已知意图应先按产品 Skill、意图表或任务 reference 选择唯一命令;命令已选中时直接执行,只在参数或安全语义不确定时读取 leaf Schema,在当前 Cobra flags 不确定时读取 leaf Help。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --limit 5` 搜索原子命令和 shortcut;`dws shortcut list` 只作人工审计/兼容 fallback,不要为已知高频意图加载完整产品 Catalog。
| 服务 | shortcut 数 | multi skill |
|---|---:|---|
@@ -157,7 +157,7 @@ def product_section(service: str, rows: list[dict[str, Any]]) -> str:
return f"""{PRODUCT_START}
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "{service} +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws {service} <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service {service} --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "{service} +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws {service} <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product {service} --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service {service}` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
@@ -179,7 +179,7 @@ def compact_product_section(service: str, rows: list[dict[str, Any]]) -> str:
`{md_escape(service)}` 当前有 {public_count} 条公开 shortcut,完整清单保留在 Runtime Catalog 与 Schema,不在高频产品根 Skill 中重复展开。{discovery}
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws shortcut list --service {md_escape(service)} --format json` 做最后回退;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws schema search --query "<用户意图>" --product {md_escape(service)} --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list` 只作人工审计/兼容 fallback;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
{PRODUCT_END}"""
+1
View File
@@ -116,5 +116,6 @@ fi
# Assembly determinism validates fresh CI/local Catalog dumps.
"$ROOT/scripts/policy/check-schema-assembly.sh"
python3 "$ROOT/scripts/policy/check-shortcut-public-catalog-drift.py"
printf 'generated drift check: ok\n'
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""Verify the committed shortcut runtime map matches the skill catalog."""
from __future__ import annotations
import json
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parents[2]
CATALOG_PATH = ROOT / "docs" / "shortcut-public-catalog.json"
GO_PATH = ROOT / "internal" / "shortcut" / "public_catalog_generated.go"
GO_ENTRY = re.compile(r'^\s*("(?:\\.|[^"\\])*"):\s*\{\},\s*$')
def fail(message: str) -> None:
raise SystemExit(f"shortcut public catalog drift: {message}")
def main() -> None:
payload = json.loads(CATALOG_PATH.read_text(encoding="utf-8"))
rows = payload.get("results")
if not isinstance(rows, list):
fail("docs catalog results must be a list")
documented: set[str] = set()
for index, row in enumerate(rows):
if not isinstance(row, dict):
fail(f"results[{index}] must be an object")
service = row.get("service")
command = row.get("command")
if not isinstance(service, str) or not service or not isinstance(command, str) or not command:
fail(f"results[{index}] requires non-empty service and command")
key = f"{service}\0{command}"
if key in documented:
fail(f"duplicate docs entry {service} {command}")
documented.add(key)
if payload.get("count") != len(documented):
fail(f"docs count={payload.get('count')!r}, actual={len(documented)}")
runtime: set[str] = set()
for line in GO_PATH.read_text(encoding="utf-8").splitlines():
match = GO_ENTRY.match(line)
if not match:
continue
key = json.loads(match.group(1))
if key in runtime:
service, command = key.split("\0", 1)
fail(f"duplicate runtime entry {service} {command}")
runtime.add(key)
if documented != runtime:
missing_runtime = sorted(documented - runtime)
missing_docs = sorted(runtime - documented)
details = []
if missing_runtime:
details.append("missing from runtime: " + ", ".join(key.replace("\0", " ") for key in missing_runtime[:10]))
if missing_docs:
details.append("missing from docs: " + ", ".join(key.replace("\0", " ") for key in missing_docs[:10]))
fail("; ".join(details))
print(f"shortcut public catalog drift check: ok ({len(runtime)} entries)")
if __name__ == "__main__":
main()
@@ -117,6 +117,13 @@ func schemaProjectionIssue(raw string) string {
if len(tokens) < 2 || tokens[0] != "dws" || tokens[1] != "schema" {
return ""
}
// schema search already returns the bounded tool-search.v1 reference
// envelope. It is not a leaf Schema query and deliberately does not
// accept --compact/--jq/--fields, so the leaf projection rule must not
// be applied to it.
if len(tokens) >= 3 && tokens[2] == "search" {
return ""
}
var targeted, compact, selected, all bool
for i := 2; i < len(tokens); i++ {
token := tokens[i]
@@ -273,6 +280,14 @@ func resolveCommandReference(root *cobra.Command, path string) commandResolution
if len(remaining) == 0 {
return resolutionValid
}
if cmd.Name() == "schema" && (cmd.Run != nil || cmd.RunE != nil) {
// `dws schema` is both a runnable parent (`schema <canonical>`) and
// the parent of `schema search`. parseReference intentionally stores
// a normalized display path, so a quoted CLI-path alias may appear as
// multiple remaining words here; preserve the long-standing leaf
// validation behavior for this one versioned inspection surface.
return resolutionValid
}
if !cmd.HasSubCommands() {
// Once a leaf command has been found, remaining tokens are positional
// arguments. This check intentionally validates command paths only.
@@ -93,7 +93,9 @@ func testCommandRoot() *cobra.Command {
root := &cobra.Command{Use: "dws", Args: cobra.NoArgs}
auth := &cobra.Command{Use: "auth", Args: cobra.NoArgs}
auth.AddCommand(&cobra.Command{Use: "login", Aliases: []string{"signin"}})
root.AddCommand(auth, &cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1)})
schema := &cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1), Run: func(*cobra.Command, []string) {}}
schema.AddCommand(&cobra.Command{Use: "search"})
root.AddCommand(auth, schema)
return root
}
@@ -125,10 +127,12 @@ func TestResolveCommandReference(t *testing.T) {
auth.AddCommand(&cobra.Command{Use: "login", Aliases: []string{"signin"}})
sheet := &cobra.Command{Use: "sheet", Args: cobra.NoArgs}
sheet.AddCommand(&cobra.Command{Use: "read"})
schema := &cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1), Run: func(*cobra.Command, []string) {}}
schema.AddCommand(&cobra.Command{Use: "search"})
root.AddCommand(
auth,
sheet,
&cobra.Command{Use: "schema [path]", Args: cobra.MaximumNArgs(1)},
schema,
&cobra.Command{Use: "plugin-info <name>", Args: cobra.ExactArgs(1)},
)
@@ -145,6 +149,7 @@ func TestResolveCommandReference(t *testing.T) {
{name: "top-level placeholder", path: "dws <cmd>", want: resolutionSkip},
{name: "nested placeholder", path: "dws sheet <command>", want: resolutionSkip},
{name: "quoted positional argument", path: "dws schema dev app create", want: resolutionValid},
{name: "schema search subcommand", path: "dws schema search", want: resolutionValid},
{name: "leaf positional argument", path: "dws plugin-info example", want: resolutionValid},
}
@@ -204,6 +209,9 @@ func testCrossPlatformCoverageSchemaProjectionIssue(t *testing.T) {
wantErr bool
}{
{command: `dws schema`, wantErr: false},
{command: `dws schema search`, wantErr: false},
{command: `dws schema search --query "create a calendar event" --limit 5`, wantErr: false},
{command: `dws schema search --request-json -`, wantErr: false},
{command: `dws schema --all --format json`, wantErr: false},
{command: `dws schema calendar`, wantErr: true},
{command: `dws schema "calendar event create" -f json`, wantErr: true},
+37
View File
@@ -0,0 +1,37 @@
{
"version": "tool-search-eval-manifest.v1",
"created_at": "2026-08-12T00:00:00+08:00",
"base_repository_commit": "5fed80fc0f1aaa4ee9123b0acf3041d0d9b45bc9",
"owners": {
"retrieval": "DWS Tool Search maintainers",
"independent_evaluation": "DWS Agent Evaluation maintainers",
"separation_rule": "The independent evaluation signer must not be an author of Catalog selection metadata used by the evaluated cases."
},
"proxy_v1": {
"state": "sealed_diagnostic_only",
"catalog_generation": "go_declarations_to_runtime_assembly",
"catalog_build_output": ".worktrees/policy-tmp/tool-search-schema-catalog",
"catalog_source_hash": "sha256:02c633c075f4af915ea097d383bdcbc25fe549c172e2d3336152e30cf2906e80",
"catalog_surface_hash": "sha256:7f9839f1ff428b52f1d032d2fa6493815c3755650b7d9fa653839e7d589482e5",
"tool_count": 1098,
"workflow_path": "scripts/testdata/tool_search_workflows.json",
"evaluator_path": "scripts/dev/eval_tool_search_ranking.py",
"intent_cases": 1123,
"intent_excluded_over_budget": 10,
"chinese_only_cases": 402,
"mixed_chinese_ascii_cases": 721,
"english_cases": 0,
"workflow_cases": 10,
"release_gate": false,
"limitations": [
"Intent queries and agent summaries share selection authorship.",
"The devdoc slice has one binary-gold case while dev and devdoc expose the same backing search_open_platform_docs operation.",
"Catalog JSON is generated only under the ignored build temp directory and is never a repository input.",
"There are no English intent cases and only ten workflows."
]
},
"independent_test_v1": {
"state": "collecting",
"path": "scripts/testdata/tool_search_independent_test_v1.json"
}
}
+6
View File
@@ -0,0 +1,6 @@
{
"version": "tool-search-qrels.v1",
"state": "collecting",
"description": "Independently authored release-gate qrels. Queries must not be copied or paraphrased from Catalog selection metadata by the same author.",
"cases": []
}
+128
View File
@@ -0,0 +1,128 @@
{
"version": 1,
"description": "Small reviewed diagnostic set for multi-tool retrieval completeness. Not a release gate.",
"cases": [
{
"id": "chat-send-file-and-check-read",
"query": "给群里发文件,并确认群成员是否已读",
"subqueries": [
"以当前用户身份给群聊发送本地文件",
"查询指定消息的已读状态和人员"
],
"required": [
"chat.send_personal_message",
"chat.query_msg_read_status"
]
},
{
"id": "calendar-create-and-notify",
"query": "创建一场会议日程,然后在群里通知参会人",
"subqueries": [
"创建一场新的会议日程",
"以当前用户身份向群聊发送通知"
],
"required": [
"calendar.create_calendar_event",
"chat.send_personal_message"
]
},
{
"id": "drive-upload-and-authorize",
"query": "把文件上传到钉盘,再给同事开通阅读权限",
"subqueries": [
"上传本地文件到钉盘",
"给钉盘文件添加同事阅读权限"
],
"required": [
"drive.upload",
"drive.add_permission"
]
},
{
"id": "wiki-search-and-read",
"query": "在知识库中找到目标文档并读取正文",
"subqueries": [
"在指定知识库中搜索文档节点",
"读取钉钉在线文字文档正文"
],
"required": [
"wiki.node_search",
"doc.get_document_content"
]
},
{
"id": "todo-create-and-remind",
"query": "创建个人待办并为它添加到期提醒",
"subqueries": [
"创建个人待办",
"给已有待办添加到期提醒"
],
"required": [
"todo.create_personal_todo",
"todo.add_todo_reminder"
]
},
{
"id": "minutes-upload-and-summarize",
"query": "上传音视频生成听记,完成后读取智能摘要",
"subqueries": [
"创建听记文件上传会话",
"完成上传会话并创建听记",
"读取听记智能摘要"
],
"required": [
"minutes.create_upload_session",
"minutes.complete_upload_session",
"minutes.get_minutes_ai_summary"
]
},
{
"id": "approval-find-and-approve",
"query": "查询我的待审批任务并同意指定审批",
"subqueries": [
"查询当前用户待审批任务 ID",
"同意指定审批任务"
],
"required": [
"oa.list_pending_tasks",
"oa.approve_processInstance"
]
},
{
"id": "mail-draft-and-send",
"query": "创建一封邮件草稿,确认内容后发送草稿",
"subqueries": [
"创建邮件草稿但不发送",
"发送草稿箱中已有邮件草稿"
],
"required": [
"mail.create_draft",
"mail.send_draft"
]
},
{
"id": "chat-find-group-and-send",
"query": "按群名找到群聊,然后以当前用户身份发消息",
"subqueries": [
"按群名关键词搜索群聊",
"以当前用户身份向群聊发送消息"
],
"required": [
"chat.search_groups",
"chat.send_personal_message"
]
},
{
"id": "sheet-resolve-and-read",
"query": "先列出电子表格中的工作表,再读取指定单元格范围",
"subqueries": [
"列出电子表格文档内所有工作表",
"读取电子表格指定单元格范围"
],
"required": [
"sheet.get_all_sheets",
"sheet.range_read"
]
}
]
}
+24 -12
View File
@@ -39,7 +39,7 @@ cli_version: ">=1.0.15"
<!-- VISIBLE_SHORTCUTS_OVERVIEW_START -->
## Shortcut 总览
下面只统计当前公开 catalog 中的 shortcut,不展开完整明细。已知意图应先按产品 Skill、意图表或任务 reference 选择唯一命令;命令已选中时直接执行,只在参数或安全语义不确定时读取 leaf Schema,在当前 Cobra flags 不确定时读取 leaf Help。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service <service> --format json` 做最后回退;不要为已知高频意图加载完整产品 Catalog。
下面只统计当前公开 catalog 中的 shortcut,不展开完整明细。已知意图应先按产品 Skill、意图表或任务 reference 选择唯一命令;命令已选中时直接执行,只在参数或安全语义不确定时读取 leaf Schema,在当前 Cobra flags 不确定时读取 leaf Help。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --limit 5` 搜索原子命令和 shortcut;`dws shortcut list` 只作人工审计/兼容 fallback,不要为已知高频意图加载完整产品 Catalog。
| 服务 | shortcut 数 | multi skill |
|---|---:|---|
@@ -210,24 +210,36 @@ Step 3 → 加 --yes 执行命令
稳定 command identity、主 CLI path 和 alias 由 leaf `ContractFinal.Identity` 与真实 Cobra tree 精确绑定。Agent 不应读取 Catalog 文件、native annotation 或其他生成 JSON 来重新推断命令;所有运行时查询都以当前二进制交付的 Schema 投影为准。
**未知命令路径**:DWS 对 Agent 是一个元工具,不要将搜索命中项动态注册成新外层工具。仅当现有 Skill/reference 都无法定位 CLI path 时执行:
```bash
# 第 1 层:产品概览(~4.5KB,列出全部产品 + 工具数 + 用途摘要)
dws schema search --query "<用户意图>" --limit 5
dws schema <candidate.canonical_path> --compact --format json \
--expected-source-hash "<search.catalog.source_hash>" \
--expected-surface-hash "<search.catalog.surface_hash>"
dws <candidate.primary_cli_path> <按 schema-inspect.v1.tool_spec 组装的参数>
```
- `exact_filtered` 立即停止,不选 sibling 绕过;零候选、`abstained=true` 或语义无法区分时澄清。
- `reason=catalog_changed` 时丢弃旧候选并重新 Search;只用 versioned Inspect 中的 `tool_spec` 组装参数。
- Search/Inspect 都不授权、不执行;最后仍由同一 `dws` 的 profile、权限、confirmation 和 Cobra 校验。
- 2~4 步复合意图先拆成 `subqueries`,通过 `dws schema search --request-json -` 合并候选;不要用整句话只排一次 Top-K。
```bash
# 产品概览只用于人类审计
dws schema
# 第 2 层:产品级(列出该产品下全部工具的 cli_path + description + effect/risk)
dws schema calendar --compact
# 已知产品但命令未知:直接限定产品搜索,不加载产品全量 Schema
dws schema search --query "创建日程" --product calendar --limit 5
# 第 3 层:分组级(按命令分组列出工具摘要)
dws schema "calendar event" --compact
# 第 4 层:Agent leaf(参数契约:type/required/description/constraints/examples)
dws schema "calendar event create" --compact
# 选中 canonical 后读取唯一 leaf(参数契约)
dws schema calendar.create_calendar_event --compact
# --all:导出所有工具的完整 leaf Schema,仅用于 CI / 审计 / 参数 baseline
dws schema --all --format json
```
**`--all` 使用边界(强制)**:`--all` 会返回每个工具的完整参数、约束和安全语义,输出体积很大。仅在用户明确要求全量导出,或执行 CI、Catalog 审计、参数防丢 baseline 时使用。普通业务任务严禁使用 `--all` 做命令发现,也不要把全量结果直接注入 Agent 上下文;必须按“产品概览 → 产品/分组 → leaf”渐进查询。完整兼容性 baseline 必须使用未裁剪的 `schema --all`;`schema --all --compact` 会移除 provenance 和接口映射字段,不得作为完整 baseline。
**`--all` 使用边界(强制)**:`--all` 会返回每个工具的完整参数、约束和安全语义,输出体积很大。仅在用户明确要求全量导出,或执行 CI、Catalog 审计、参数防丢 baseline 时使用。普通业务任务严禁使用 `--all` 做命令发现,也不要把全量结果直接注入 Agent 上下文;未知命令统一用 `schema search`,已知产品通过 `--product` 收窄,再 Inspect 唯一 leaf。完整兼容性 baseline 必须使用未裁剪的 `schema --all`;`schema --all --compact` 会移除 provenance 和接口映射字段,不得作为完整 baseline。
同一个工具省略 `--compact` 的 full leaf 与 `--all` 条目是同一份 `ToolSpec` 契约;compact leaf 只做展示投影,不重新解析语义。Alias 查询不得根据 alias 重写或补猜参数。若同一视图观察到内容差异,应作为契约漂移报告,而不是选择其中一份继续执行。
@@ -238,7 +250,7 @@ dws schema --all --format json
### Schema 字段速查
```jsonc
// leaf 级输出(dws schema "calendar event create" --compact)
// leaf 级输出(dws schema calendar.create_calendar_event --compact)
{
"cli_path": "calendar event create",
"canonical_path": "calendar.create_calendar_event",
@@ -288,7 +300,7 @@ Schema 与 Help 冲突是**契约漂移**,不得静默猜测或把两边字段
`dev.*` 包含 helper-only 执行面,其中远端 helper 未进入 pinned metadata 时标记为 `composite`,不能伪装成 `local`。`event list` / `event schema` 读取内置目录和 payload 定义,属于 `local`;`event consume` / `event status` / `event stop` 同时编排远端个人订阅控制面与本地 bus/consume,属于 `composite`。实现来源不同,不改变统一查询边界:进入全局 `dws schema` 的命令须由 leaf `ContractFinal.Identity` 声明收集,并由同一 `ToolSpec` 投影到 leaf、产品/分组、`--all` 与 Catalog。不得把 Cobra 临时合成结果作为第二条 Schema 数据路径。
事件需要区分两种 Schema:`dws event schema <event_key> --flatten` 查询 Agent 要消费的顶层业务字段;`dws schema "event consume" --compact` 查询 CLI 命令参数。前者是真实业务命令,后者只读取最终内嵌 SchemaRegistry;不能相互替代。
事件需要区分两种 Schema:`dws event schema <event_key> --flatten` 查询 Agent 要消费的顶层业务字段;`dws schema event.consume --compact` 查询 CLI 命令参数。前者是真实业务命令,后者只读取最终内嵌 SchemaRegistry;不能相互替代。
`source` 表示最终命令 identity 的来源,不表示运行时 backing;helper/local/MCP 实现机制读取 `interface_mode`、`availability` 和 provenance,不要假定 `dev.*` 必然是 `source=mcp:<server>`,也不要假定本地命令必然是 `source=cobra`。
@@ -44,6 +44,8 @@ dws calendar event [create|update|get|delete|respond] [flags]
dws calendar event list [flags]
# 查询循环日程的实例列表(按时间范围展开重复日程)
dws calendar event instances [flags]
# 获取日程的分享信息(日程主题、组织人、地点、入会信息等,用于向他人分享日程)
dws calendar event share-info [flags]
# 对于非明确时间或一段时间范围的约会场景,可基于所有参会人的忙闲状态,推荐多个可用的时间块方案
dws calendar event suggest [flags]
```
@@ -160,6 +162,22 @@ Flags:
> **默认行为**:不传 `--start` / `--end` 时,默认返回今天的实例(00:00:00 ~ 23:59:59)。
> **分页**:单次最多返回 `--limit` 指定的条数(默认/最大 100);当结果超过 limit 时,返回体包含 `nextCursor` 字段。
### 获取日程分享信息
```
Usage:
dws calendar event share-info [flags]
Example:
dws calendar event share-info --id <EVENT_ID>
dws calendar event share-info --id <EVENT_ID> --language zh-CN
dws calendar event share-info --id <EVENT_ID> --calendar-id primary
Flags:
--id string 日程 ID (必填)
--calendar-id string 日历 ID (可选,默认 primary 主日历)
--language string 语言代码 (可选,如 zh-CN)
```
> **说明**:根据日程 ID 获取日程的分享信息,展示日程主题、组织人、地点、入会信息等,用于向他人分享日程(如发送到群聊、邮件)。中文内容建议传 `--language zh-CN`。
### 创建日程
```
Usage:
@@ -585,6 +603,7 @@ Flags:
- 推荐时间/什么时候有空/协调时间 → `event suggest`
- 接受/拒绝/暂定日程 → `event respond`
- 查询循环日程/重复日程的每次实例/展开循环日程 → `event instances`
- 分享日程/把日程发给别人/获取日程分享信息或入会信息 → `event share-info`
用户说"参会人/与会者":
- 查看 → `attendee list`
@@ -691,6 +710,7 @@ dws calendar event list --start "2026-03-10T14:00:00+08:00" --end "2026-03-10T15
| `event suggest` | 推荐的时间段 | event create 的 --start/--end |
| `event respond` | 响应结果 | — |
| `event instances` | `result.events[].id`, `nextCursor` | event get/update/delete/respond 的 --id;下一页 --cursor |
| `event share-info` | 日程分享信息(主题、组织人、地点、入会信息等) | 分享给他人(如 chat/mail 发送) |
| `room search` | `rooms[].roomId` | room add 的 --rooms 或 event create 的 --rooms |
| `room list-groups` | `groups[].groupId` | room search 的 --group-id |
| `book list` | `id`(如 `primary`) | event list/get 的 --calendar-id, book get/update 的 --id |
@@ -718,6 +738,7 @@ dws calendar event list --start "2026-03-10T14:00:00+08:00" --end "2026-03-10T15
- **日历本**:`book list` 返回的 `id` 才是合法 `calendarId`;如无明确说明,`event list` / `event get` 都不要带 `--calendar-id`,让接口默认走 primary 主日历
- **分页查询**:`event list` / `event instances` 均支持 `--limit`(控制每页条数,默认/最大 100)和 `--cursor`(翻页游标);**首次查询无需传 `--cursor`**,仅当返回体中包含 `nextCursor` 时,将其作为 `--cursor` 传入可获取下一页
- **循环日程实例**:`event instances` 用于按时间范围展开重复日程(SeriesMaster)的每一个实例;**普通非循环日程调用该命令将查不到任何实例信息**
- **日程分享**:`event share-info` 获取日程的分享信息(主题、组织人、地点、入会信息等);`--language` 控制文案语言(中文场景传 zh-CN)
- **日程提醒**:`event create` 支持 `--remind-minutes` 设置开始前提醒,逗号分隔多个分钟数(如 `--remind-minutes 5,10,15`),不传则默认15分钟提醒
- **会议室分页**:`room search` 支持 `--limit`(每页条数,默认100,最大100)和 `--page`(分页起始位置,默认0),与 `room list-groups` 分页风格一致
+12 -7
View File
@@ -738,7 +738,7 @@ Flags:
**重要:该接口会真实发送消息到目标会话,不可用于测试或试探性调用。调用前必须确认消息内容和接收对象无误。**
群聊传 --group;单聊可传 --users、--open-dingtalk-ids 或两者组合。--group 不能与单聊目标同时指定。默认发送 Markdown,必须同时使用 --title 和 --text;公网图片 URL 使用 --msg-type image --image-url <图片 URL>;本地图片和其他本地文件一样使用 --msg-type file --file-path <本地路径>,CLI 会完成上传并按文件附件发送。群聊时可选 --at-user-ids 或 --at-open-dingtalk-ids @指定成员。
群聊传 --conversation-id;单聊可传 --users、--open-dingtalk-ids 或两者组合。--conversation-id 不能与单聊目标同时指定。默认发送 Markdown,普通发送必须同时使用 --title 和 --text;公网图片 URL 使用 --msg-type image --image-url <图片 URL>;本地图片和其他本地文件一样使用 --msg-type file --file-path <本地路径>,CLI 会完成上传并按文件附件发送。群聊时可选 --at-user-ids 或 --at-open-dingtalk-ids @指定成员。机器人群聊 Markdown 引用回复同时传 --reply <openMessageId> 和 --ref-sender <senderOpenDingTalkId>,可省略 --title 由 CLI 从正文生成;单聊、图片和文件不支持引用回复。
如果用户明确要求"用机器人/机器人身份/robot"发送,必须使用本命令,严禁改用 `chat message send` 以当前用户身份发送。
**重要**:机器人发群消息前,必须确认该机器人已在目标群中。若机器人不在群内会报错"机器人不存在",需先执行 `dws chat group members add-bot --id <openConversationId> --robot-code <robot-code>` 将机器人加入群聊后再发送。
@@ -746,7 +746,8 @@ Flags:
Usage:
dws chat message send-by-bot [flags]
Example:
dws chat message send-by-bot --robot-code <robot-code> --group <openconversation_id> --title "日报" --text "## 今日完成..."
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openconversation_id> --title "日报" --text "## 今日完成..."
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openconversation_id> --reply <openMessageId> --ref-sender <senderOpenDingTalkId> --text "收到"
dws chat message send-by-bot --robot-code <robot-code> --group <openconversation_id> --msg-type image --image-url "https://example.com/image.png"
dws chat message send-by-bot --robot-code <robot-code> --group <openconversation_id> --msg-type file --file-path ./report.pdf
dws chat message send-by-bot --robot-code <robot-code> --users userId1,userId2 --title "提醒" --text "请提交周报"
@@ -755,10 +756,10 @@ Example:
dws chat message send-by-bot --robot-code <robot-code> --group <openconversation_id> --at-open-dingtalk-ids openDingtalkId1,openDingtalkId2 --title "提醒" --text "@openDingtalkId1 @openDingtalkId2 请查收本周报告"
dws chat message send-by-bot --robot-code <robot-code> --group <openconversation_id> --at-all --title "通知" --text "请所有人注意"
Flags:
--group string 群聊 openConversationId(群聊时必填)
--conversation-id string 群聊 openConversationId(群聊时必填;兼容别名 --group)
--robot-code string 机器人 Code (必填)
--msg-type string 消息类型:markdown、image 或 file;省略时为 markdown;公网图片使用 image --image-url;本地图片和文件使用 file --file-path
--title string Markdown 消息标题(Markdown 时必填)
--title string Markdown 消息标题(普通 Markdown 必填;引用回复省略时从正文生成)
--text string Markdown 消息内容(Markdown 时必填)
--image-url string 公网图片 URL(msgType=image 时必填)
--file-path string 本地图片或文件路径(msgType=file 时上传并按文件附件发送)
@@ -767,6 +768,8 @@ Flags:
--at-user-ids string @指定成员的 userId 列表,逗号分隔(仅群聊时生效,可选)
--at-open-dingtalk-ids string @指定成员的 openDingtalkId 列表,逗号分隔(仅群聊时生效,可选)
--at-all @所有人(可选),服务端接收字符串 true/false
--reply string 被引用消息的 openMessageId(仅群聊 Markdown;必须与 --ref-sender 同时使用)
--ref-sender string 被引用消息发送者的 openDingTalkId(仅群聊 Markdown;必须与 --reply 同时使用)
注意:
- 用户明确要求机器人发送时,必须使用 `chat message send-by-bot`;严禁使用 `chat message send` 以用户身份代发
@@ -776,6 +779,7 @@ Flags:
- --at-user-ids 仅在 --group 群聊时生效,单聊时无效;设置时 --text 中需包含 @userId 对应文本
- --at-open-dingtalk-ids 仅在 --group 群聊时生效,单聊时无效;设置时 --text 中需包含 @openDingtalkId 对应文本
- --at-all @所有人,仅群聊时生效;只需带上 --at-all flag 即可,服务端会自动处理
- --reply 与 --ref-sender 必须成对使用;CLI 在普通群消息参数顶层透传 referenceOpenMessageId 与 srcMsgSendOpenDingTalkId,不设置 msgType=reply
- userId 获取方式:`dws contact user search --query "姓名"` 搜人获取 userId
- **换行符**:--text 按 Markdown 渲染,换行规则同 `chat message send`:
1. 必须使用**真实换行符**(`U+000A`),而非字面量 `\n`,否则全部内容会渲染在同一行
@@ -2146,6 +2150,7 @@ Flags:
用户说"入群验证记录/谁申请进群" → `chat group list-join-validations`
用户说"审批入群/通过入群申请/拒绝入群申请" → `chat group audit-join-validation`
用户说"引用回复/回复消息/引用消息回复" → `chat message reply`
用户说"机器人引用回复/让机器人回复这条群消息" → `chat message send-by-bot --conversation-id <openConversationId> --reply <openMessageId> --ref-sender <senderOpenDingTalkId> --text <内容>`
用户说"转发消息/转发一条消息/把消息转发到另一个群" → `chat message forward`
用户说"合并转发/批量转发/合并转发多条消息" → `chat message combine-forward`
用户说"转发话题/转发话题消息" → `chat message forward-topic`
@@ -2183,7 +2188,7 @@ Flags:
- `chat message send` — 以当前用户身份发消息(群聊或单聊),text 为位置参数;本地图片/文件/音视频统一用 `--msg-type file --file-path`,其中图片显示为可下载附件而非内联图片;`--msg-type image --media-id` 只用于上游已经提供有效 mediaId 的场景,DWS CLI 不能从本地文件生成 mediaId
- `chat message search` — 按关键词搜索消息内容(跨所有会话,可选指定群)
- `chat search-common` — 搜索共同群,查询指定人共同所在的群聊(AND=所有人都在,OR=任一人在)
- `chat message send-by-bot` — 以**机器人**身份发消息(群聊或单聊);Markdown 使用 `--text`,公网图片使用 `--msg-type image --image-url`,本地图片和文件使用 `--msg-type file --file-path`
- `chat message send-by-bot` — 以**机器人**身份发消息(群聊或单聊);Markdown 使用 `--text`,公网图片使用 `--msg-type image --image-url`,本地图片和文件使用 `--msg-type file --file-path`;群聊 Markdown 引用回复成对使用 `--reply` / `--ref-sender`
- `chat message send-by-webhook` — 通过**自定义机器人 Webhook** 发群消息
- `chat message recall-by-bot` — 通过**机器人接口**撤回机器人发出的消息,需要 `--robot-code` + `--keys`(发送时返回的 processQueryKey);传 `--group` 为群聊撤回,不传为单聊撤回
- `chat message recall` — 通过 **IM 接口**撤回当前用户自己发出的消息,需要 `--conversation-id`(openConversationId)+ `--msg-id`(openMessageId,可通过 `chat message list` 获取);群聊单聊均通过 `--conversation-id` 区分
@@ -2456,7 +2461,7 @@ Flags:
| `chat category list` | `categoryId` | category list-conversations 的 --category-id |
| `chat group get-by-group-id` | `openConversationId` | 同 chat search,将群号转为 openConversationId |
| `chat message send-card` | `bizId` | update-card 的 --biz-id |
| `chat message list` | `openMessageId` | message reply 的 --ref-msg-id、message forward 的 --msg-id |
| `chat message list` | `openMessageId` | message reply 的 --ref-msg-id、send-by-bot 机器人群聊引用回复的 --reply、message forward 的 --msg-id |
| `chat search` | `openConversationId` | set-top 的 --conversation-id、group-mute / group-mute-member 的 --group |
## 注意事项
@@ -2485,7 +2490,7 @@ Flags:
- `--user` 和 `--open-dingtalk-id` 本质上都是发起单聊操作,只是用户标识格式不同:userId 为企业内部应用常用标识,openDingTalkId 为三方应用或跨组织场景下的用户标识,服务端对两种 ID 的解析逻辑不同
- `--time` 格式: `yyyy-MM-dd HH:mm:ss`,为拉取消息的起始时间点;`--direction` 控制方向(newer=从给定时间往现在拉,older=从给定时间往以前拉),`--limit` 控制数量
- `chat search` 挂在 `chat` 下(非 `chat group` 下),路径为 `dws chat search`
- `send-by-bot` 群聊传 `--group`,单聊传 `--users` 或 `--open-dingtalk-ids`,与 `--group` 互斥且必选其一;群聊时可选 `--at-user-ids` @指定成员(传 userId 列表)或 `--at-open-dingtalk-ids` @指定成员(传 openDingtalkId 列表),content 中需包含对应 @标识;`--at-all` @所有人;群聊场景如果返回"机器人不存在"错误,需先通过 `chat group members add-bot --group <openConversationId> --robot-code <robot-code>` 将机器人邀请进群后再发送
- `send-by-bot` 群聊传 `--conversation-id`,单聊传 `--users` 或 `--open-dingtalk-ids`,与群目标互斥且必选其一;机器人群聊 Markdown 引用回复必须同时传 `--reply <openMessageId>` 与 `--ref-sender <senderOpenDingTalkId>`,单聊、图片和文件不支持引用回复;群聊时可选 `--at-user-ids` @指定成员(传 userId 列表)或 `--at-open-dingtalk-ids` @指定成员(传 openDingTalkId 列表),content 中需包含对应 @标识;`--at-all` @所有人;群聊场景如果返回"机器人不存在"错误,需先通过 `chat group members add-bot --conversation-id <openConversationId> --robot-code <robot-code>` 将机器人邀请进群后再发送
- `recall-by-bot` 群聊传 `--group` + `--keys`,单聊仅传 `--keys`(不传 `--group` 即为单聊撤回)
- `send-by-webhook` 支持 `--at-all`、`--at-mobiles`、`--at-users` 进行 @ 操作,但需在 `--text` 中包含 `@userId` 或 `@手机号` 才能生效;`--at-all` @所有人时需在 `--text` 中包含 `@10`
- `chat group-role` 系列命令用于管理群的自定义身份标签:`list` 查列表,`add` 创建,`update` 改名,`remove` 删除;`set-user` 覆盖某人全部身份(传空 --role-ids 则清除),`remove-user` 仅移除指定身份,`query-user` 查询某人当前身份;用户用 `--user <userId>`
+1 -1
View File
@@ -21,7 +21,7 @@
| `dws event stop <subscribe_id> --dry-run` / `--yes` | 先预览,再确认取消订阅并停止对应本地消费 |
| `dws event stop --all --dry-run` / `--yes` | 先预览,再确认清理当前身份下全部个人订阅 |
注意区分两个 schema:`dws event schema <event_key>` 查事件的输出字段;`dws schema "event consume" --compact` 查 consume 命令自身的入参(统一内嵌 ToolSpec,含 parameters + 位置参数)。`source` 是 reviewed command identity 的 provenance;`event list/schema` 是 `interface_mode=local`,`event consume/status/stop` 因同时编排远端订阅控制面与本地 bus 而是 `interface_mode=composite`,不要把 identity 与实现机制混为一谈。
注意区分两个 schema:`dws event schema <event_key>` 查事件的输出字段;`dws schema event.consume --compact` 查 consume 命令自身的入参(统一内嵌 ToolSpec,含 parameters + 位置参数)。`source` 是 reviewed command identity 的 provenance;`event list/schema` 是 `interface_mode=local`,`event consume/status/stop` 因同时编排远端订阅控制面与本地 bus 而是 `interface_mode=composite`,不要把 identity 与实现机制混为一谈。
## Event catalog
+1 -1
View File
@@ -210,7 +210,7 @@ Flags:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "sheet +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws sheet <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service sheet --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "sheet +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws sheet <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product sheet --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service sheet` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
+1 -1
View File
@@ -20,7 +20,7 @@ metadata:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "aitable +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws aitable <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service aitable --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "aitable +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws aitable <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product aitable --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service aitable` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
+2 -2
View File
@@ -1,6 +1,6 @@
---
name: dingtalk-calendar
description: 钉钉日历与会议室。Use when 用户说 约会议/查日程/订会议室/查闲忙/加参会人/改期/取消会议/今天的日程/本周日程/共同空闲。不做视频会议发起/邀请入会/会中控制(走 dingtalk-misc)、AI 听记(走 dingtalk-minutes)、待办任务(走 dingtalk-todo)。命令前缀:dws calendar。
description: 钉钉日历与会议室。Use when 用户的主任务是单一日历产品操作:约会议/查日程/订会议室/查闲忙/加参会人/改期/取消会议/今天的日程/本周日程/共同空闲。同一请求还要求待办、听记等多产品汇总时,只使用 dingtalk-shared 调度入口,不要预加载本 Skill。不做视频会议发起/邀请入会/会中控制(走 dingtalk-misc)。命令前缀:dws calendar。
metadata:
cli_version: ">=0.2.14"
category: product
@@ -20,7 +20,7 @@ metadata:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "calendar +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws calendar <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service calendar --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "calendar +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws calendar <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product calendar --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service calendar` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
@@ -44,6 +44,8 @@ dws calendar event [create|update|get|delete|respond] [flags]
dws calendar event list [flags]
# 查询循环日程的实例列表(按时间范围展开重复日程)
dws calendar event instances [flags]
# 获取日程的分享信息(日程主题、组织人、地点、入会信息等,用于向他人分享日程)
dws calendar event share-info [flags]
# 对于非明确时间或一段时间范围的约会场景,可基于所有参会人的忙闲状态,推荐多个可用的时间块方案
dws calendar event suggest [flags]
```
@@ -160,6 +162,22 @@ Flags:
> **默认行为**:不传 `--start` / `--end` 时,默认返回今天的实例(00:00:00 ~ 23:59:59)。
> **分页**:单次最多返回 `--limit` 指定的条数(默认/最大 100);当结果超过 limit 时,返回体包含 `nextCursor` 字段。
### 获取日程分享信息
```
Usage:
dws calendar event share-info [flags]
Example:
dws calendar event share-info --id <EVENT_ID>
dws calendar event share-info --id <EVENT_ID> --language zh-CN
dws calendar event share-info --id <EVENT_ID> --calendar-id primary
Flags:
--id string 日程 ID (必填)
--calendar-id string 日历 ID (可选,默认 primary 主日历)
--language string 语言代码 (可选,如 zh-CN)
```
> **说明**:根据日程 ID 获取日程的分享信息,展示日程主题、组织人、地点、入会信息等,用于向他人分享日程(如发送到群聊、邮件)。中文内容建议传 `--language zh-CN`。
### 创建日程
```
Usage:
@@ -585,6 +603,7 @@ Flags:
- 推荐时间/什么时候有空/协调时间 → `event suggest`
- 接受/拒绝/暂定日程 → `event respond`
- 查询循环日程/重复日程的每次实例/展开循环日程 → `event instances`
- 分享日程/把日程发给别人/获取日程分享信息或入会信息 → `event share-info`
用户说"参会人/与会者":
- 查看 → `attendee list`
@@ -691,6 +710,7 @@ dws calendar event list --start "2026-03-10T14:00:00+08:00" --end "2026-03-10T15
| `event suggest` | 推荐的时间段 | event create 的 --start/--end |
| `event respond` | 响应结果 | — |
| `event instances` | `result.events[].id`, `nextCursor` | event get/update/delete/respond 的 --id;下一页 --cursor |
| `event share-info` | 日程分享信息(主题、组织人、地点、入会信息等) | 分享给他人(如 chat/mail 发送) |
| `room search` | `rooms[].roomId` | room add 的 --rooms 或 event create 的 --rooms |
| `room list-groups` | `groups[].groupId` | room search 的 --group-id |
| `book list` | `id`(如 `primary`) | event list/get 的 --calendar-id, book get/update 的 --id |
@@ -718,6 +738,7 @@ dws calendar event list --start "2026-03-10T14:00:00+08:00" --end "2026-03-10T15
- **日历本**:`book list` 返回的 `id` 才是合法 `calendarId`;如无明确说明,`event list` / `event get` 都不要带 `--calendar-id`,让接口默认走 primary 主日历
- **分页查询**:`event list` / `event instances` 均支持 `--limit`(控制每页条数,默认/最大 100)和 `--cursor`(翻页游标);**首次查询无需传 `--cursor`**,仅当返回体中包含 `nextCursor` 时,将其作为 `--cursor` 传入可获取下一页
- **循环日程实例**:`event instances` 用于按时间范围展开重复日程(SeriesMaster)的每一个实例;**普通非循环日程调用该命令将查不到任何实例信息**
- **日程分享**:`event share-info` 获取日程的分享信息(主题、组织人、地点、入会信息等);`--language` 控制文案语言(中文场景传 zh-CN)
- **日程提醒**:`event create` 支持 `--remind-minutes` 设置开始前提醒,逗号分隔多个分钟数(如 `--remind-minutes 5,10,15`),不传则默认15分钟提醒
- **会议室分页**:`room search` 支持 `--limit`(每页条数,默认100,最大100)和 `--page`(分页起始位置,默认0),与 `room list-groups` 分页风格一致
+3 -3
View File
@@ -15,7 +15,7 @@ metadata:
## 最小 DWS 执行契约
- 只通过 `dws` CLI 操作钉钉;结构化读取使用 `--format json`,按真实返回判断结果。
- 已知命令直接执行。只有 leaf 参数或安全语义不确定时读取精确 Schema,只有 Cobra flag 不确定时读取精确 leaf Help;不要加载产品级 Catalog 代替选路。
- 已知命令直接执行。Skill/reference 无法定位时才用 `dws schema search --query "<意图>" --limit 5`;选中后携带双 hash Inspect canonical,再按 `primary_cli_path` 执行。参数/安全语义或 Cobra flag 不确定时才补读精确 Schema/Help;不加载产品级 Catalog 代替选路。
- 不猜命令、flag、字段、ID、账号或时间。后续 ID 必须来自真实返回;零命中、多候选或类型不明时停止并消歧。
- 解析目标、读取上下文和最终执行必须使用同一 profile;不得跨组织复用 userId、openDingTalkId 或 openConversationId。多账号组织只使用明确的 `isOrgCurrent=true` 默认账号;没有默认账号时要求用户指定,禁止选择第一项、最近登录或最近使用账号。
- 不输出或记录 token、refresh token、appSecret、webhook token 等凭据;宿主已注入认证时不要索要凭据。
@@ -30,7 +30,7 @@ metadata:
`chat` 当前有 98 条公开 shortcut,完整清单保留在 Runtime Catalog 与 Schema,不在高频产品根 Skill 中重复展开。已知意图直接使用下方的优先路由、意图表或任务 reference;命令已选中时直接执行,只在参数/安全语义不确定时读取 leaf Schema,在当前 Cobra flags 不确定时读取 leaf Help。
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws shortcut list --service chat --format json` 做最后回退;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws schema search --query "<用户意图>" --product chat --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list` 只作人工审计/兼容 fallback;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
<!-- VISIBLE_SHORTCUTS_END -->
## Golden Route
@@ -62,7 +62,7 @@ metadata:
| 已知资源引用单独下载 | `dws chat +messages-resource-download` |
| 按关键词搜索群 | `dws chat +chat-search` |
| 查看消息收藏 | `dws chat +flag-list` |
| <!-- dws-intent: chat.reply.quote -->引用回复 | `dws chat +messages-reply`;成功结果保留新消息/会话/投递与原消息来源上下文 |
| <!-- dws-intent: chat.reply.quote -->引用回复 | 人:`dws chat +messages-reply`;成功结果保留新消息/会话/投递与原消息来源上下文。Bot 群:`dws chat message send-by-bot --conversation-id <cid> --reply <mid> --ref-sender <sid>` |
| 撤回当前用户消息 | `dws chat +messages-recall --msg-id <openMessageId>`;可省略会话 ID,由 CLI 只读补齐;兼容单值 `--message-ids` |
| 已知话题主消息 ID 或 thread/topic ID 读取回复 | `dws chat +thread-replies` |
| <!-- dws-intent: chat.create.group -->按成员 ID 或姓名创建群聊 | `dws chat +chat-create`;成员/群主均可自然解析,任一歧义都会在创建前整体停止 |
@@ -11,7 +11,7 @@
## 群聊消息
<!-- dws-intent: chat.read.conversation -->读取或导出指定群聊/单聊的消息记录时使用 `dws chat +chat-messages`。可附带非必填的 `--sender-query <姓名>`:没有稳定 ID 时保留全部消息,唯一解析成功后按 userId/openDingTalkId 筛选同一次读取结果并返回 `resolvedFilters`,不要补跑 `+search-msg`。
<!-- dws-intent: chat.read.conversation -->读取或导出指定群聊/单聊的消息记录时使用 `dws chat +chat-messages`。可附带非必填的 `--sender-query <姓名>`:唯一解析成功后按稳定 `senderId` 筛选同一次读取结果并返回 `resolvedFilters`;解析失败、不完整或存在歧义时抑制未过滤消息并返回 `sender_resolution_failed`,不要补跑 `+search-msg`。混合姓名/ID 入口 `--sender` 无法经通讯录确认类型时可按原值 userId 精确过滤,但必须保留 `identity_unverified`。
用户以发送者、关键词、@对象或消息类型为主要条件直接检索时优先使用 `+search-msg`;范围可以是单个、多个或全部会话。
@@ -92,7 +92,7 @@ Favorite、消息 Pin、消息 Top 与会话 Top 是四种对象,不能互换
|---|---|
| `chat bot search` | 搜索当前用户创建的机器人并取得 `robotCode` |
| `chat bot find` | 搜索可用机器人并取得机器人 `openDingTalkId` |
| `chat message send-by-bot` | `+messages-send --as bot` 未发布的真实底层字段 |
| `chat message send-by-bot` | `+messages-send --as bot` 未发布的真实底层字段,包括机器人群聊引用回复的 `--reply` / `--ref-sender` |
| `chat message recall-by-bot` | 使用 `processQueryKey` 撤回机器人消息 |
| `chat message send-by-webhook` | `+messages-send --as webhook` 未发布的真实底层字段 |
@@ -124,7 +124,7 @@ Favorite、消息 Pin、消息 Top 与会话 Top 是四种对象,不能互换
| `+messages-send` | `openTaskId` 查询投递状态;它不是消息 ID |
| `+chat-messages` / `+search-msg` / `+messages-mget` | 回复、转发、撤回、资源操作使用的真实消息/会话/thread ID |
| `chat bot search` | `robotCode`;不能当机器人 `openDingTalkId` |
| `chat message send-by-bot` | `processQueryKey`,仅用于机器人撤回 |
| `chat message send-by-bot` | `processQueryKey` 用于机器人撤回;群聊引用回复还需消息查询返回的 `openMessageId` 与原发送者 `openDingTalkId` |
显式稳定 ID 当前不携带可验证的 profile provenance;调用方必须保证来源,不得宣称所有
跨 profile 误用都会在本地写入前被拦截。
@@ -16,6 +16,7 @@
- `chat bot search` 只返回我创建的机器人,没有 `openDingTalkId`;给机器人发单聊必须用 `chat bot find`。
- 机器人发群消息前需确认机器人已在群中;报“机器人不存在”时先 `group members add-bot`。
- `send-by-bot` 支持 Markdown、图片 URL 和文件,具体参数见下方消息类型路由。
- 机器人在群聊中引用回复已有消息时,使用原子命令 `send-by-bot --reply --ref-sender`;该能力仅支持 Markdown,不走 `+messages-reply` 的当前用户身份。
- 公网图片 URL 使用 `--msg-type image --image-url`,按图片消息发送。
- 本地图片和其他本地文件一样使用 `--msg-type file --file-path`,由 CLI 上传并按文件附件发送。
- 群聊传 `--group`;单聊可传 `--users`、`--open-dingtalk-ids` 或两者组合。
@@ -63,9 +64,10 @@ dws chat +messages-send --as bot --robot-code <robot-code> \
```bash
# 群聊
dws chat message send-by-bot --robot-code <robot-code> --group <openConversationId> --title "日报" --text "## 今日完成\n\n- 事项 A\n\n- 事项 B"
dws chat message send-by-bot --robot-code <robot-code> --group <openConversationId> --msg-type image --image-url "https://example.com/image.png"
dws chat message send-by-bot --robot-code <robot-code> --group <openConversationId> --msg-type file --file-path ./report.pdf
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openConversationId> --title "日报" --text "## 今日完成\n\n- 事项 A\n\n- 事项 B"
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openConversationId> --reply <openMessageId> --ref-sender <senderOpenDingTalkId> --text "收到"
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openConversationId> --msg-type image --image-url "https://example.com/image.png"
dws chat message send-by-bot --robot-code <robot-code> --conversation-id <openConversationId> --msg-type file --file-path ./report.pdf
# 单聊 userId
dws chat message send-by-bot --robot-code <robot-code> --users userId1,userId2 --title "提醒" --text "请提交周报"
@@ -84,16 +86,20 @@ dws chat message send-by-bot --robot-code <robot-code> --group <openConversation
| Flag | 说明 |
|------|------|
| `--robot-code` | 机器人 Code,必填 |
| `--group` | 群聊 openConversationId |
| `--conversation-id` | 群聊 openConversationId;`--group` 为兼容别名 |
| `--users` | 单聊 userId 列表,逗号分隔,最多 20 个 |
| `--open-dingtalk-ids` | 单聊 openDingTalkId 列表 |
| `--msg-type` | `markdown`、`image` 或 `file`;省略时为 Markdown;公网图片使用 `image --image-url`,本地图片和文件使用 `file --file-path` |
| `--text` | Markdown 消息内容,Markdown 模式必填;换行用空行,转义表示为 `\n\n` |
| `--title` | Markdown 消息标题,Markdown 模式必填 |
| `--title` | 普通 Markdown 消息标题;引用回复省略时由 CLI 从正文生成 |
| `--image-url` | 公网图片 URL,`--msg-type image` 时必填 |
| `--file-path` | 本地图片或文件路径,`--msg-type file` 时由 CLI 上传并按文件附件发送 |
| `--at-user-ids` / `--at-open-dingtalk-ids` | 群聊 @ 指定成员,正文需含对应 `@id` 文本 |
| `--at-all` | 群聊 @所有人 |
| `--reply` | 被引用消息的 `openMessageId`;仅群聊 Markdown,必须与 `--ref-sender` 同时使用 |
| `--ref-sender` | 被引用消息发送者的 `openDingTalkId`;仅群聊 Markdown,必须与 `--reply` 同时使用 |
引用回复不会设置 `msgType=reply`;CLI 在普通群消息参数顶层透传 `referenceOpenMessageId` 和 `srcMsgSendOpenDingTalkId`。只传其中一个参数、用于单聊或用于图片/文件消息都会在本地失败。
#### `dws chat message recall-by-bot`
@@ -176,5 +182,6 @@ dws chat +messages-send --as bot --robot-code <robot-code> --group <openConversa
- 机器人单聊没有 openDingTalkId:改用 `chat bot find`,不要用 `bot search`。
- 机器人发群消息报“机器人不存在”:先 `group members add-bot`。
- 撤回失败:确认使用 `processQueryKey`,不是 `openMessageId`。
- 机器人引用回复失败:确认目标是群聊 Markdown,且 `--reply` 来自被引用消息的 `openMessageId`、`--ref-sender` 来自同一消息的发送者 `openDingTalkId`。
- @ 不生效:检查正文是否包含 `@userId` / `@openDingTalkId` / `@10`。
- 需要 Bot 图片/文件/音视频:停止并说明当前身份矩阵不支持;只有用户明确同意改为当前用户身份时,才重新确认目标和内容。
@@ -9,7 +9,9 @@
- <!-- dws-intent: chat.conversation.list-top -->查看置顶会话默认使用 `dws chat +conversation-list-top`;
原子 `list-top-conversations` 只在需要原始响应时作为 fallback。
- <!-- dws-intent: chat.read.conversation -->取得目标会话后读取或导出消息记录,默认使用 `dws chat +chat-messages`;
可附带非必填的 `--sender-query` 解析姓名:无稳定 ID 返回全部,唯一解析成功后按 `senderId` 筛选同一次读取结果。不要回到原子 `message list`,也不要补跑 `+search-msg`。直接条件检索优先使用 `+search-msg`。
可附带非必填的 `--sender-query` 解析姓名:唯一解析成功后按 `senderId` 筛选同一次读取结果;
解析失败、不完整或存在歧义时抑制未过滤消息并返回 `sender_resolution_failed`。不要回到原子
`message list`,也不要补跑 `+search-msg`。直接条件检索优先使用 `+search-msg`。
## 必读约束
@@ -30,10 +30,14 @@ dws chat +chat-messages --group <群名或openConversationId> --format json
dws chat +chat-messages --group <openConversationId> --page-all --page-limit 50 --format json
```
可附带非必填的 `--sender-query <姓名>`:未传时返回全部消息;未解析出稳定 ID 时保留
全部消息并记录失败;唯一解析出 userId/openDingTalkId 后,按消息 `senderId` 筛选同一次
读取结果,覆盖最终 `messages/count` 并返回 `resolvedFilters`。不得用展示名字符串比较否定
已经成功解析的稳定发送者身份。
可附带非必填的 `--sender-query <姓名>`:未传时返回全部消息;唯一解析出
userId/openDingTalkId 后,按消息 `senderId` 筛选同一次读取结果,覆盖最终
`messages/count` 并返回 `resolvedFilters`。解析失败、不完整或存在歧义时抑制未过滤消息,
返回 `sender_resolution_failed`,不得把全量消息当作发送者筛选结果。
`--sender` 是姓名、userId 或 openDingTalkId 的混合入口。通讯录无法确认输入类型时,可按
原值 userId 精确过滤并交付命中消息,但结果必须保留 `identity_unverified`,不得把字符串命中
升级为已验证身份,也不得据此作完整否定结论。
```bash
dws chat +chat-messages --group "项目群" --sender-query "测试用户甲" --page-all --format json
@@ -9,7 +9,7 @@
| 给姓名发简单文本 | `+dm` | 先查人再原子发送 |
| 给群名发简单文本 | `+send-to-group` | 先搜群再原子发送 |
| 文件、Bot、Webhook、复杂 @、幂等 | `+messages-send` | 为不同身份各走一套原子入口 |
| 读取或导出指定会话,可附带发送者姓名 | `+chat-messages`;姓名用非必填 `--sender-query` | 无稳定 ID 返回全部,有则读后筛选;不补跑搜索 |
| 读取或导出指定会话,可附带发送者姓名 | `+chat-messages`;姓名用非必填 `--sender-query` | 解析成功后读后筛选;解析失败抑制未过滤消息并返回错误;不补跑搜索 |
| 直接按发送者、关键词、@对象或消息类型搜索 | `+search-msg` | 条件检索优先,可限定单个或跨多个会话 |
| 已知消息 IDs 取详情 | `+messages-mget` | 重新搜索关键词 |
| @我的消息 | `+at-me` | 全量消息后本地猜测 @ |
+1 -1
View File
@@ -20,7 +20,7 @@ metadata:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "contact +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws contact <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service contact --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "contact +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws contact <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product contact --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service contact` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
+11 -3
View File
@@ -1,6 +1,6 @@
---
name: dingtalk-doc
description: 钉钉在线文字文档(adoc,「文档空间」里的在线文档)的查找、读取、创建、编辑、块、评论、附件、导入、导出(docx/markdown/pdf)、模板、版本、权限协作及 Markdown/JSONML 写入。文档空间与钉盘的文件管理走 dingtalk-drive(doc 同名原子命令已弃用),知识库空间与空间内节点走 dingtalk-wiki,原生 .md 文件读写走 dingtalk-misc。命令前缀:dws doc。
description: 钉钉在线文字文档(adoc)本体及其内容的操作:查找、创建、读取、文档信息、编辑、块、评论、附件与媒体、白板卡片、导入、导出(docx/markdown/pdf)、版本、模板、权限、分享及Markdown/JSONML写入。不包括:文档空间与钉盘的文件管理(归 dingtalk-drive,doc 同名原子命令已弃用)、知识库空间与节点管理(归 dingtalk-wiki)、原生 .md 文件读写(归 dingtalk-misc)、电子表格 axls(归 dingtalk-misc)、AI 表格 able(归 dingtalk-aitable)。命令前缀:dws doc。
metadata:
cli_version: ">=0.2.14"
category: product
@@ -15,7 +15,7 @@ metadata:
## 最小 DWS 执行契约
- 只通过 `dws` CLI 操作钉钉;结构化读取使用 `--format json`,按真实返回判断结果。
- 已知命令直接执行。只有 leaf 参数或安全语义不确定时读取精确 Schema,只有 Cobra flag 不确定时读取精确 leaf Help;不要加载产品级 Catalog 代替选路。
- 已知命令直接执行。Skill/reference 无法定位时才用 `dws schema search --query "<意图>" --limit 5`;选中后携带双 hash Inspect canonical,再按 `primary_cli_path` 执行。参数/安全语义或 Cobra flag 不确定时才补读精确 Schema/Help;不加载产品级 Catalog 代替选路。
- 不猜命令、flag、字段、ID、账号或时间。后续 ID 必须来自真实返回;零命中、多候选或类型不明时停止并消歧。
- 解析目标、读取上下文和最终执行必须使用同一 profile;不得跨组织复用 userId、openDingTalkId 或 openConversationId。多账号组织只使用明确的 `isOrgCurrent=true` 默认账号;没有默认账号时要求用户指定,禁止选择第一项、最近登录或最近使用账号。
- 不输出或记录 token、refresh token、appSecret、webhook token 等凭据;宿主已注入认证时不要索要凭据。
@@ -25,12 +25,20 @@ metadata:
- 遇到认证、权限、profile、confirmation 或未知错误时,只加载 `dingtalk-shared` 中对应 reference;不要连续猜测替代命令。
<!-- DWS_RUNTIME_CONTRACT_END -->
## 多轮写入与追加语义
用户在某一轮明确要求创建、插入或追加内容时,按本轮指令执行:
- **"追加/再插入"类指令即使目标内容疑似已存在也要执行本轮写入**(可提示已存在,但不得以"已存在/已重复"为由拒绝执行明确指令);
- 不要提前执行后续轮次可能要求的内容;每轮只做本轮明确要求的写入;
- 回读用于验证写入结果,不用于推翻本轮的明确写入指令。
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcut 发现(按需)
`doc` 当前有 45 条公开 shortcut,完整清单保留在 Runtime Catalog 与 Schema,不在高频产品根 Skill 中重复展开。已知意图按下方路由。
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws shortcut list --service doc --format json` 做最后回退;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws schema search --query "<用户意图>" --product doc --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list` 只作人工审计/兼容 fallback;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
<!-- VISIBLE_SHORTCUTS_END -->
## Golden Route
+4 -4
View File
@@ -1,6 +1,6 @@
---
name: dingtalk-drive
description: 钉钉文件管理(存储层,覆盖钉盘与文档空间两个存储域)。Use when 用户说 钉盘/上传文件/下载文件/文件夹/查文件/找文件/全局搜索文件/复制/移动/重命名/删除/回收站/还原删除文件/权限管理/普通文件下载/本地与钉盘文件夹差异比较/整个文件夹拉到本地/整个文件夹推到钉盘/双向同步;也承接钉钉文档的这些管理动作(doc 侧同名原子命令已弃用)。文档正文编辑与导出 docx/markdown/pdf 走 dingtalk-doc,知识库空间与空间内节点组织走 dingtalk-wiki。命令前缀:dws drive。
description: 钉钉文件管理(存储层,覆盖钉盘与文档空间两个存储域)。Use when 用户说 钉盘/上传文件/下载文件/文件夹/查文件/找文件/全局搜索文件/最近编辑的文档/复制/移动/重命名/删除/回收站/还原删除文件/文件权限管理/普通文件下载/本地与钉盘文件夹差异比较/整个文件夹拉到本地/整个文件夹推到钉盘/双向同步。对钉钉文档仅处理其作为文件的管理操作,文档本体及其内容操作归 dingtalk-doc;知识库空间与空间内节点组织归 dingtalk-wiki。命令前缀:dws drive。
metadata:
cli_version: ">=0.2.14"
category: product
@@ -22,7 +22,7 @@ metadata:
`drive` 当前有 28 条公开 shortcut,完整清单保留在 Runtime Catalog 与 Schema,不在高频产品根 Skill 中重复展开。已知意图按下方路由。
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws shortcut list --service drive --format json` 做最后回退;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
仅当现有路由和 reference 都无法定位低频能力时,才执行 `dws schema search --query "<用户意图>" --product drive --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list` 只作人工审计/兼容 fallback;不要为已知高频意图加载完整 Shortcut Catalog 或产品级 Schema。
<!-- VISIBLE_SHORTCUTS_END -->
## 意图表
@@ -49,9 +49,9 @@ metadata:
### SOP-1 找文件(find-file)
**触发**:找文件/搜文件/我的文件/最近文件/某文档在哪。
**触发**:找文件/搜文件/我的文件/最近文件/最近编辑的文档/某文档在哪。
1. **选源(必须)**:最近访问 → `dws drive +recent --limit <n> --format json`(翻页用上次返回的 `nextCursor` 传 `--cursor`);按内容/名称全局搜 → `dws drive +search --query "<关键词>" --format json`;浏览某目录 → `dws drive +list --folder <dentryUuid> --format json`。
1. **选源(必须)**:最近访问 → `dws drive +recent --limit <n> --format json`(翻页用上次返回的 `nextCursor` 传 `--cursor`);最近编辑 → `dws drive +recent --operate-type 1 --limit <n> --format json`(不传 `--operate-type` 默认为 0 最近访问,会混入仅打开过的文档);按内容/名称全局搜 → `dws drive +search --query "<关键词>" --format json`;浏览某目录 → `dws drive +list --folder <dentryUuid> --format json`。
2. **解析(必须)**:取真实 `dentryUuid`(= `id`/`nodeId`);多候选让用户确认,**禁止**默认取第一个。
3. **下钻(必须)**:根目录没命中时,进入最相关文件夹继续 `drive +list --folder`,必要时 `python scripts/drive_tree_list.py --depth 2` 递归,**禁止**只看根目录就放弃。
4. **回读元数据(必须)**:命中后 `dws drive +inspect --node <dentryUuid> --format json`,按 `extension` 确认类型。
+1 -1
View File
@@ -20,7 +20,7 @@ metadata:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "mail +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws mail <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service mail --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "mail +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws mail <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product mail --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service mail` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
+2 -2
View File
@@ -1,6 +1,6 @@
---
name: dingtalk-minutes
description: 钉钉 AI 听记。Use when 查询听记摘要、转写、关键词、待办或分享。写文档走 dingtalk-doc;日程走 dingtalk-calendar。
description: 钉钉 AI 听记。Use when 用户的主任务是单一听记产品查询:摘要、转写、关键词、待办或分享。同一请求还要求日程、待办等多产品汇总时,只使用 dingtalk-shared 调度入口,不要预加载本 Skill。写文档走 dingtalk-doc。
metadata:
cli_version: ">=0.2.14"
category: product
@@ -20,7 +20,7 @@ metadata:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "minutes +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws minutes <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service minutes --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "minutes +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws minutes <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product minutes --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service minutes` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
@@ -1411,7 +1411,7 @@ dws attendance boss-check --plan-id 948964045503 --time "2026-05-13 18:00" --res
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "attendance +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws attendance <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service attendance --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "attendance +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws attendance <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product attendance --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service attendance` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
@@ -20,7 +20,7 @@
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "devapp +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws devapp <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service devapp --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "devapp +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws devapp <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product devapp --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service devapp` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
@@ -169,7 +169,7 @@ dws ding message recall-personal --id <OPEN_DING_ID> --format json
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "ding +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws ding <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service ding --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "ding +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws ding <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product ding --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service ding` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
+1 -1
View File
@@ -875,7 +875,7 @@ dws oa approval create-instance --request '{"processCode":"PROC-xxx","deptId":-1
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "oa +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws oa <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service oa --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "oa +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws oa <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product oa --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service oa` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
@@ -445,7 +445,7 @@ dws report outbox list --cursor 0 --size 20 --format json
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "report +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws report <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service report --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "report +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws report <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product report --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service report` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
@@ -210,7 +210,7 @@ Flags:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "sheet +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws sheet <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service sheet --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "sheet +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws sheet <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product sheet --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service sheet` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
+33 -7
View File
@@ -1,6 +1,6 @@
---
name: dingtalk-shared
description: 钉钉(DingTalk) MultiSkill 的轻量共享入口。Use when 用户泛称 DWS/钉钉操作但未明确产品、请求跨产品编排、需要 URL 类型预检或产品边界消歧。清晰的单产品操作优先使用对应 dingtalk-* 子 skill;本 skill 只提供全局执行契约和按需 reference 导航,不承载产品命令全集。
description: 钉钉(DingTalk) MultiSkill 的轻量共享与跨产品调度入口。Use when 用户泛称 DWS/钉钉操作但未明确产品、在同一请求中查日程+待办+听记等多产品汇总、请求跨产品编排、需要 URL 类型预检或产品边界消歧。多产品任务先使用本 Skill 的紧凑调度路径,不要预加载各产品 Skill;清晰的单产品操作仍使用对应 dingtalk-* 子 Skill。
metadata:
cli_version: ">=0.2.14"
category: shared
@@ -17,7 +17,7 @@ metadata:
## 最小 DWS 执行契约
- 只通过 `dws` CLI 操作钉钉;结构化读取使用 `--format json`,按真实返回判断结果。
- 已知命令直接执行。只有 leaf 参数或安全语义不确定时读取精确 Schema,只有 Cobra flag 不确定时读取精确 leaf Help;不要加载产品级 Catalog 代替选路。
- 已知命令直接执行。Skill/reference 无法定位时才用 `dws schema search --query "<意图>" --limit 5`;选中后携带双 hash Inspect canonical,再按 `primary_cli_path` 执行。参数/安全语义或 Cobra flag 不确定时才补读精确 Schema/Help;不加载产品级 Catalog 代替选路。
- 不猜命令、flag、字段、ID、账号或时间。后续 ID 必须来自真实返回;零命中、多候选或类型不明时停止并消歧。
- 解析目标、读取上下文和最终执行必须使用同一 profile;不得跨组织复用 userId、openDingTalkId 或 openConversationId。多账号组织只使用明确的 `isOrgCurrent=true` 默认账号;没有默认账号时要求用户指定,禁止选择第一项、最近登录或最近使用账号。
- 不输出或记录 token、refresh token、appSecret、webhook token 等凭据;宿主已注入认证时不要索要凭据。
@@ -27,6 +27,14 @@ metadata:
- 遇到认证、权限、profile、confirmation 或未知错误时,只加载 `dingtalk-shared` 中对应 reference;不要连续猜测替代命令。
<!-- DWS_RUNTIME_CONTRACT_END -->
## 多轮写入与追加语义
用户在某一轮明确要求创建、插入或追加内容时,按本轮指令执行:
- **"追加/再插入"类指令即使目标内容疑似已存在也要执行本轮写入**(可提示已存在,但不得以"已存在/已重复"为由拒绝执行明确指令);
- 不要提前执行后续轮次可能要求的内容;每轮只做本轮明确要求的写入;
- 回读用于验证写入结果,不用于推翻本轮的明确写入指令。
产品或跨产品规则在最小契约之上增量加载。用户已明确产品内容意图时,意图优先于 URL 形态;多账号选择与跨组织规则读取 [`../dingtalk-misc/references/profile.md`](../dingtalk-misc/references/profile.md)。本地文件、产品边界和跨产品传递规则只在对应任务中加载,避免把全局手册放入每个单产品请求。
## 渐进加载
@@ -37,7 +45,8 @@ metadata:
|---|---|
| 已明确单一产品 | 对应 `../dingtalk-*/SKILL.md`;不读路由 reference |
| 泛称 DWS、需要选择产品 | [routing.md](references/routing.md) |
| 跨产品、多步骤、汇总或报告 | [workflow-routing.md](references/workflow-routing.md) |
| 今日日程 + 未完成待办 + 最近听记的全部或任意组合 | 直接使用下文“常见多产品只读汇总快路径”;不读 reference |
| 其他跨产品、多步骤、汇总或报告 | [workflow-routing.md](references/workflow-routing.md) |
| 输入含 alidocs、shanji 等钉钉 URL 且类型不明 | [url-patterns.md](references/url-patterns.md) |
| 产品边界仍然难以判断 | [intent-guide.md](references/intent-guide.md) 的相关章节 |
| 认证、全局 flag 或输出格式问题 | [global-reference.md](references/global-reference.md) |
@@ -54,15 +63,28 @@ metadata:
1. 先识别明确的产品内容意图;明确意图直接进入对应产品。仅当输入包含钉钉 URL
且类型不明确或意图与链接类型可能冲突时,读取 `url-patterns.md` 识别节点类型。
2. 请求包含多个时序步骤、跨产品数据传递或汇总报告:即使 URL 已识别,也要读取
`workflow-routing.md`,按行动指南组合需要的产品 skill;当前发布包不包含独立
scenario skill。
2. 请求包含多个时序步骤、跨产品数据传递或汇总报告:先匹配下文紧凑只读快路径。命中则直接执行且不读 reference;未命中才读取 `workflow-routing.md`,按行动指南组合需要的产品 skill。
3. 请求是单产品操作但产品不明确:读取 `routing.md`,再显式读取目标产品
`SKILL.md`。
4. `doc/drive/wiki`、`aitable/sheet`、`calendar/minutes` 等边界仍不清楚:
只读取 `intent-guide.md` 的对应章节。
5. 仍无法判断时向用户追问,不要猜测产品或命令。
## 常见多产品只读汇总快路径
当用户在同一请求中要求“今日日程 + 当前未完成待办 + 最近一条听记摘要”的全部或任意组合,直接执行:
```bash
python scripts/cross_product_read_summary.py --date YYYY-MM-DD --timezone <IANA_TIMEZONE> [--include calendar,todos,minutes]
```
- 该脚本只调用真实 `dws` 只读命令,并发采集后只返回汇总所需字段。
- 用户只要其中部分类别时,必须用 `--include` 只读取被请求的产品;三类都要时可省略。
- 需要先与用户确认采集范围时,加 `--dry-run`:只打印将执行的只读 `dws` 命令清单,不发起任何调用;其输出的 `readOnly`、`commands` 是执行计划,不能当作业务数据回答用户。
- 命中时不读取 calendar/todo/minutes 产品 Skill 或行动指南,不运行 `--help` / Schema Search,不重复执行同一 list。
- 一类数据为空或失败时,按脚本 JSON 的分类结果如实说明,不用其他产品数据替代。
- 只有用户要求了该脚本未覆盖的字段、时间范围或写操作,才增量加载对应产品 Skill。
## 跨 skill 执行
- 正文中的相对 `Read` 链接是运行时依赖;`metadata.requires.skills` 不会自动加载。
@@ -80,4 +102,8 @@ metadata:
4. 其他错误:优先读取 JSON 错误中的 `retryable`、`retry_after_seconds`、
`next_retry_at`、`hint` 和 `actions`。只有明确 `retryable=true` 时才按服务端节奏重试;
缺少重试语义时用 `--verbose` 获取诊断并停止,不连续尝试替代命令。
5. 明确不支持的能力:说明边界,不通过其他接口绕过。
5. 能力可能不存在(如群投票、订阅、导出报表等 Catalog 未覆盖的意图):`dws schema search` 一次
确认候选;返回 `abstained=true`、`weak_match=true` 的 `hint`,或候选语义与意图不符时,最多用
`dws schema <product> --compact` 复核一次该产品能力边界,然后如实结论"当前 CLI 不支持该能力"。
禁止逐个候选验证、逐产品枚举或改用无关接口绕过。
6. 明确不支持的能力:说明边界,不通过其他接口绕过。
@@ -1,7 +1,7 @@
## 最小 DWS 执行契约
- 只通过 `dws` CLI 操作钉钉;结构化读取使用 `--format json`,按真实返回判断结果。
- 已知命令直接执行。只有 leaf 参数或安全语义不确定时读取精确 Schema,只有 Cobra flag 不确定时读取精确 leaf Help;不要加载产品级 Catalog 代替选路。
- 已知命令直接执行。Skill/reference 无法定位时才用 `dws schema search --query "<意图>" --limit 5`;选中后携带双 hash Inspect canonical,再按 `primary_cli_path` 执行。参数/安全语义或 Cobra flag 不确定时才补读精确 Schema/Help;不加载产品级 Catalog 代替选路。
- 不猜命令、flag、字段、ID、账号或时间。后续 ID 必须来自真实返回;零命中、多候选或类型不明时停止并消歧。
- 解析目标、读取上下文和最终执行必须使用同一 profile;不得跨组织复用 userId、openDingTalkId 或 openConversationId。多账号组织只使用明确的 `isOrgCurrent=true` 默认账号;没有默认账号时要求用户指定,禁止选择第一项、最近登录或最近使用账号。
- 不输出或记录 token、refresh token、appSecret、webhook token 等凭据;宿主已注入认证时不要索要凭据。
@@ -1,6 +1,6 @@
# Schema 渐进查询
# DWS 命令发现与 Schema Inspect
`dws schema` 内嵌当前二进制公开命令面的结构化契约。**Agent 选择命令、读取参数映射/约束和安全语义时优先渐进查询 leaf Schema**;真正组装执行参数前,用 `--help` 确认当前 Cobra 接受的 flags。
`dws` 对 Agent 是一个稳定元工具;Catalog 中的命令是它的内部命令空间,不是需要向 Host 动态注册的一组新工具。`dws schema search` 用来找到未知子命令,`dws schema <canonical> --compact` 用来检查选中命令的结构化契约,最后仍通过同一个 `dws` 执行。
本节同时适用于基础/原子命令与公开内建 `+` shortcut。用户自定义或未公开 shortcut 不进入发布 Schema;是否可执行仍以当前 Cobra help 为准。
@@ -14,20 +14,56 @@
稳定 command identity 已与真实 Cobra tree 绑定。不要读取 Catalog 文件、native annotation 或其他生成 JSON 来重新推断命令。
## 四层查询
## 未知命令:Search → Inspect → Execute
仅当当前 Skill、意图表和任务 reference 都没有给出精确 CLI path 时,才执行一次本地命令搜索:
```bash
# 第 1 层:产品概览(列出产品 + 工具数 + 用途摘要)
dws schema search --query "查询群消息已读状态" --limit 5
```
1. 只从 `candidates` 选择与用户动作一致的候选;不把 rank/score 当授权或成功概率。零候选、多个语义无法区分或 `abstained=true` 时停止并澄清。
2. `exact_filtered` 表示用户精确指定的命令被过滤;立即停止,不选择同名 sibling 绕过。
3. 选中候选后,使用 response `catalog.source_hash` 和 `catalog.surface_hash` 精确 Inspect `canonical_path`:
```bash
dws schema chat.query_msg_read_status --compact --expected-source-hash "<search.catalog.source_hash>" --expected-surface-hash "<search.catalog.surface_hash>"
```
4. 只使用 `schema-inspect.v1.tool_spec` 组装参数,并按候选的 `primary_cli_path` 调用同一个 `dws`。`reason=catalog_changed` 时丢弃旧候选,重新 Search,不混用两代 Schema。
5. 执行阶段仍以实时 profile、权限、confirmation 和 Cobra 校验为准;Search/Inspect 都不代表授权或业务成功。
例如 Inspect 返回的 ToolSpec 确认参数后,最后仍执行:
```bash
dws chat message read-status \
--conversation-id "<真实返回的 openConversationId>" \
--message-id "<真实返回的 openMessageId>" \
--format json
```
### 复合意图
先把 2~4 个有时序的动作拆成 `subqueries`,不要用整句话只排一次 Top-K:
```bash
printf '%s' '{"version":"tool-search.v1","query":"给群里发文件并确认送达","subqueries":["给群里发送文件消息","查询消息发送状态","查询群消息已读状态"]}' |
dws schema search --request-json -
```
对每个选中的 canonical 分别做双 hash Inspect;按任务依赖顺序执行,后续 ID 只能来自前一步真实返回。
## 已知产品时的有界导航
```bash
# 产品概览只用于人类审计
dws schema
# 第 2 层:产品级(该产品工具的 cli_path + description + effect/risk)
dws schema calendar --compact
# 已知产品但命令未知:直接限定产品搜索,不加载产品全量 Schema
dws schema search --query "创建日程" --product calendar --limit 5
# 第 3 层:分组级
dws schema "calendar event" --compact
# 第 4 层:Agent leaf(参数契约)
dws schema "calendar event create" --compact
# 选中 canonical 后读取唯一 leaf(参数契约)
dws schema calendar.create_calendar_event --compact
# --all:全量 leaf,仅 CI / 审计 / 参数 baseline
dws schema --all --format json
@@ -35,7 +71,7 @@ dws schema --all --format json
### `--all` 边界(强制)
`--all` 输出体积很大。仅在用户明确要求全量导出,或 CI / Catalog 审计 / 参数防丢 baseline 时使用。普通业务任务严禁用 `--all` 做命令发现,也不要把全量结果注入 Agent 上下文;必须按「产品概览 → 产品/分组 → leaf」渐进查询。
`--all` 输出体积很大。仅在用户明确要求全量导出,或 CI / Catalog 审计 / 参数防丢 baseline 时使用。普通业务任务严禁用 `--all` 做命令发现,也不要把全量结果注入 Agent 上下文。未知命令统一用 `schema search`,已知产品通过 `--product` 收窄。
完整兼容性 baseline 必须使用未裁剪的 `schema --all`;`schema --all --compact` 会移除 provenance 和接口映射字段,不得作为完整 baseline。
@@ -83,6 +119,6 @@ Schema 与 Help 冲突是**契约漂移**:执行参数只用 Cobra 接受的 f
### 两类易混 Schema
- `dws event schema <event_key> --flatten`:事件业务字段
- `dws schema "event consume" --compact`:CLI 命令参数
- `dws schema event.consume --compact`:CLI 命令参数
二者不可互相替代。
@@ -4,10 +4,14 @@
当前发布包不包含独立 scenario skill;跨产品请求由本文件选择行动指南,再组合产品
skill 完成。
## 先匹配紧凑只读快路径
若请求是“今日日程 + 当前未完成待办 + 最近一条听记摘要”的全部或任意组合,使用根 Skill 中的 `scripts/cross_product_read_summary.py`。该路径已内建产品选择、并发采集、分页和紧凑输出;不要再读取下方产品行动指南或产品 Skill。
## 选择顺序
1. 按下表选择与用户目标最匹配的产品行动指南。
2. 显式读取流程涉及的产品 `SKILL.md`,不要预加载无关产品。
2. 未命中上述紧凑快路径时,显式读取流程涉及的产品 `SKILL.md`,不要预加载无关产品。
3. 前一步返回的真实 ID、URL 或结构化数据作为后一步输入。
4. 只读采集可并行;依赖上一步输出的步骤、写操作和验证步骤必须保持顺序。
@@ -28,7 +32,7 @@ skill 完成。
- 单一目标、短链路、无多源汇总:在
[lite-catalog.md](recipes/lite-catalog.md) 中只读取对应 recipe。
- 涉及多源采集、批量处理、交叉分析、汇总/归纳或三个以上产品:使用上表的完整
- 涉及多源采集、批量处理、交叉分析、汇总/归纳或三个以上产品,且未命中紧凑只读快路径:使用上表的完整
行动指南,并按 [conventions.md](recipes/conventions.md) 执行批量和
ID 传递规则。
- 产品 skill 已内联完整步骤时,直接执行其步骤,不再重复读取通用 recipe。
@@ -0,0 +1,246 @@
#!/usr/bin/env python3
"""Compact, read-only DWS summary for calendar, todos, and minutes."""
import argparse
import json
import subprocess
import sys
from concurrent.futures import ThreadPoolExecutor
from datetime import date, datetime, time, timedelta
from typing import Any, Dict, List, Optional, Tuple
from zoneinfo import ZoneInfo
PAGE_SIZE = 100
MAX_TODO_PAGES = 10
MAX_TODO_ITEMS = 20
MAX_ERROR_CHARS = 800
MAX_SUMMARY_CHARS = 8000
PRIORITY_LABELS = {10: "low", 20: "normal", 30: "high", 40: "urgent"}
def run_dws(args: List[str]) -> Tuple[Optional[Any], Optional[str]]:
try:
completed = subprocess.run(
["dws", *args], capture_output=True, text=True, timeout=90
)
except (FileNotFoundError, subprocess.TimeoutExpired) as exc:
return None, str(exc)[:MAX_ERROR_CHARS]
if completed.returncode != 0:
message = completed.stderr.strip() or completed.stdout.strip()
return None, message[:MAX_ERROR_CHARS]
try:
payload = json.loads(completed.stdout)
except json.JSONDecodeError as exc:
return None, f"invalid JSON: {exc}"[:MAX_ERROR_CHARS]
if isinstance(payload, dict) and payload.get("success") is False:
message = payload.get("errorMsg") or payload.get("errorCode") or "DWS request failed"
return None, str(message)[:MAX_ERROR_CHARS]
return payload, None
def result_of(payload: Any) -> Any:
if isinstance(payload, dict) and "result" in payload:
return payload["result"]
return payload
def object_list(payload: Any, *keys: str) -> List[Dict[str, Any]]:
value = result_of(payload)
if isinstance(value, list):
return [item for item in value if isinstance(item, dict)]
if isinstance(value, dict):
for key in keys:
items = value.get(key)
if isinstance(items, list):
return [item for item in items if isinstance(item, dict)]
return []
def calendar_summary(day: date, tz: ZoneInfo) -> Dict[str, Any]:
start = datetime.combine(day, time.min, tzinfo=tz)
end = start + timedelta(days=1)
payload, error = run_dws([
"calendar", "event", "list",
"--start", start.isoformat(), "--end", end.isoformat(),
"--format", "json",
])
if error:
return {"ok": False, "error": error, "count": 0, "items": []}
events = object_list(payload, "events", "itemList")
items = []
for event in events:
start_value = event.get("start")
end_value = event.get("end")
location = event.get("location")
items.append({
"title": event.get("summary") or event.get("title") or "",
"start": (start_value.get("dateTime") if isinstance(start_value, dict) else start_value),
"end": (end_value.get("dateTime") if isinstance(end_value, dict) else end_value),
"location": (location.get("displayName") if isinstance(location, dict) else location) or "",
})
return {"ok": True, "count": len(items), "items": items}
def todo_page(page: int) -> Tuple[Optional[Any], Optional[str]]:
return run_dws([
"todo", "task", "list", "--status", "false",
"--page", str(page), "--size", str(PAGE_SIZE), "--format", "json",
])
def timestamp_iso(value: Any, tz: ZoneInfo) -> Optional[str]:
if value in (None, ""):
return None
try:
return datetime.fromtimestamp(int(value) / 1000, tz=tz).isoformat()
except (TypeError, ValueError, OSError):
return str(value)
def todo_summary(tz: ZoneInfo) -> Dict[str, Any]:
todos: List[Dict[str, Any]] = []
complete = True
for page in range(1, MAX_TODO_PAGES + 1):
payload, error = todo_page(page)
if error:
return {"ok": False, "error": error, "count": len(todos), "items": []}
items = object_list(payload, "todoCards", "itemList")
todos.extend(items)
inner = result_of(payload)
has_more = bool(inner.get("hasMore")) if isinstance(inner, dict) else len(items) == PAGE_SIZE
if not has_more or not items:
break
else:
complete = False
def sort_key(item: Dict[str, Any]) -> Tuple[int, int, int]:
due = item.get("dueTime")
due_value = int(due) if str(due or "").isdigit() else 2**63 - 1
priority = item.get("priority")
priority_value = int(priority) if str(priority or "").isdigit() else 0
created = item.get("createdTime")
created_value = int(created) if str(created or "").isdigit() else 0
return (due_value, -priority_value, -created_value)
selected = sorted(todos, key=sort_key)[:MAX_TODO_ITEMS]
items = []
for item in selected:
priority = item.get("priority")
try:
priority_number = int(priority)
except (TypeError, ValueError):
priority_number = 20
items.append({
"title": item.get("subject") or item.get("title") or "",
"priority": PRIORITY_LABELS.get(priority_number, str(priority_number)),
"dueTime": timestamp_iso(item.get("dueTime"), tz),
})
return {
"ok": True,
"count": len(todos),
"shown": len(items),
"truncated": not complete or len(todos) > len(items),
"items": items,
}
def minutes_summary() -> Dict[str, Any]:
payload, error = run_dws([
"minutes", "list", "all", "--limit", "1", "--format", "json",
])
if error:
return {"ok": False, "error": error, "found": False}
records = object_list(payload, "itemList", "items")
if not records:
return {"ok": True, "found": False}
record = records[0]
task_uuid = record.get("uuid") or record.get("taskUuid")
if not task_uuid:
return {"ok": False, "error": "latest minutes item has no uuid", "found": False}
detail, error = run_dws([
"minutes", "get", "summary", "--id", str(task_uuid), "--format", "json",
])
if error:
return {
"ok": False, "error": error, "found": True,
"title": record.get("title") or "", "startTime": record.get("startTimeISO"),
}
value = result_of(detail)
if isinstance(value, dict):
summary: Any = value.get("fullSummary") or value.get("summary") or value.get("content") or value
else:
summary = value
if not isinstance(summary, str):
summary = json.dumps(summary, ensure_ascii=False, separators=(",", ":"))
was_truncated = len(summary) > MAX_SUMMARY_CHARS
return {
"ok": True,
"found": True,
"title": record.get("title") or "",
"startTime": record.get("startTimeISO") or record.get("startTime"),
"summary": summary[:MAX_SUMMARY_CHARS],
"summaryTruncated": was_truncated,
}
def dry_run(day: date, tz: ZoneInfo, included: set[str]) -> None:
start = datetime.combine(day, time.min, tzinfo=tz)
end = start + timedelta(days=1)
commands = []
if "calendar" in included:
commands.append(["dws", "calendar", "event", "list", "--start", start.isoformat(), "--end", end.isoformat(), "--format", "json"])
if "todos" in included:
commands.append(["dws", "todo", "task", "list", "--status", "false", "--page", "1", "--size", str(PAGE_SIZE), "--format", "json"])
if "minutes" in included:
commands.extend([
["dws", "minutes", "list", "all", "--limit", "1", "--format", "json"],
["dws", "minutes", "get", "summary", "--id", "<TASK_UUID>", "--format", "json"],
])
print(json.dumps({"readOnly": True, "commands": commands}, ensure_ascii=False))
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--date", default=datetime.now().date().isoformat())
parser.add_argument("--timezone", default="Asia/Shanghai")
parser.add_argument(
"--include",
default="calendar,todos,minutes",
help="comma-separated subset of calendar,todos,minutes",
)
parser.add_argument("--dry-run", action="store_true")
args = parser.parse_args()
try:
day = date.fromisoformat(args.date)
tz = ZoneInfo(args.timezone)
except (ValueError, KeyError) as exc:
parser.error(str(exc))
included = {item.strip() for item in args.include.split(",") if item.strip()}
unknown = included - {"calendar", "todos", "minutes"}
if not included or unknown:
parser.error(f"invalid --include value: {args.include}")
if args.dry_run:
dry_run(day, tz, included)
return 0
with ThreadPoolExecutor(max_workers=3) as pool:
result = {
"date": day.isoformat(),
"timezone": args.timezone,
}
futures = {}
if "calendar" in included:
futures["calendar"] = pool.submit(calendar_summary, day, tz)
if "todos" in included:
futures["todos"] = pool.submit(todo_summary, tz)
if "minutes" in included:
futures["minutes"] = pool.submit(minutes_summary)
for name in ("calendar", "todos", "minutes"):
if name in futures:
result[name] = futures[name].result()
print(json.dumps(result, ensure_ascii=False, separators=(",", ":")))
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,84 @@
#!/usr/bin/env python3
import importlib.util
import unittest
from pathlib import Path
from unittest.mock import patch
from zoneinfo import ZoneInfo
SCRIPT_PATH = Path(__file__).with_name("cross_product_read_summary.py")
SPEC = importlib.util.spec_from_file_location("cross_product_read_summary", SCRIPT_PATH)
MODULE = importlib.util.module_from_spec(SPEC)
assert SPEC and SPEC.loader
SPEC.loader.exec_module(MODULE)
class CrossProductReadSummaryTest(unittest.TestCase):
def test_todo_summary_is_compact_and_formats_semantics(self):
payload = {
"success": True,
"result": {
"todoCards": [
{
"subject": "later",
"priority": 20,
"dueTime": 1786096800000,
"createdTime": 2,
},
{
"subject": "urgent",
"priority": 40,
"dueTime": 1760371200000,
"createdTime": 1,
},
]
},
}
with patch.object(MODULE, "run_dws", return_value=(payload, None)):
result = MODULE.todo_summary(ZoneInfo("Asia/Shanghai"))
self.assertTrue(result["ok"])
self.assertEqual(result["count"], 2)
self.assertEqual(result["items"][0]["title"], "urgent")
self.assertEqual(result["items"][0]["priority"], "urgent")
self.assertRegex(result["items"][0]["dueTime"], r"^\d{4}-\d{2}-\d{2}T")
def test_minutes_uses_latest_item_then_fetches_summary(self):
list_payload = {
"success": True,
"result": {
"itemList": [
{"uuid": "real-id", "title": "latest", "startTimeISO": "2026-08-14T10:00:00+08:00"}
]
},
}
summary_payload = {
"success": True,
"result": {"fullSummary": "real summary"},
}
with patch.object(
MODULE, "run_dws", side_effect=[(list_payload, None), (summary_payload, None)]
) as mocked:
result = MODULE.minutes_summary()
self.assertTrue(result["ok"])
self.assertTrue(result["found"])
self.assertEqual(result["summary"], "real summary")
self.assertIn("real-id", mocked.call_args_list[1].args[0])
def test_dws_business_error_is_not_treated_as_success(self):
completed = unittest.mock.Mock(
returncode=0,
stdout='{"success":false,"errorCode":"500","errorMsg":"failed"}',
stderr="",
)
with patch.object(MODULE.subprocess, "run", return_value=completed):
payload, error = MODULE.run_dws(["todo", "task", "list"])
self.assertIsNone(payload)
self.assertEqual(error, "failed")
if __name__ == "__main__":
unittest.main()
+2 -2
View File
@@ -1,6 +1,6 @@
---
name: dingtalk-todo
description: 钉钉待办 / TODO。Use when 用户说 创建待办/TODO/任务提醒/指派任务/标记完成/查待办/紧急待办/循环待办/批量建待办/逾期待办。不做日报周报(走 dingtalk-misc)、审批(走 dingtalk-misc)、日程(走 dingtalk-calendar)。命令前缀:dws todo。
description: 钉钉待办 / TODO。Use when 用户的主任务是单一待办产品操作:创建待办/TODO/任务提醒/指派任务/标记完成/查待办/紧急待办/循环待办/批量建待办/逾期待办。同一请求还要求日程、听记等多产品汇总时,只使用 dingtalk-shared 调度入口,不要预加载本 Skill。不做日报周报或审批(走 dingtalk-misc)。命令前缀:dws todo。
metadata:
cli_version: ">=0.2.14"
category: product
@@ -20,7 +20,7 @@ metadata:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "todo +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws todo <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service todo --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "todo +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws todo <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product todo --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service todo` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|
+1 -1
View File
@@ -20,7 +20,7 @@ metadata:
<!-- VISIBLE_SHORTCUTS_START -->
## Shortcuts(无专用脚本/recipe 时优先)
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "wiki +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws wiki <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws shortcut list --service wiki --format json` 批量发现。
以下 shortcut 同时进入公开 catalog 与 Runtime Schema。先按本 skill 的意图表、脚本和 recipe 路由:存在精确覆盖该场景的专用脚本/recipe 时按其执行;否则用户意图命中时,shortcut 优先于手写原子命令。命令已选中时直接执行;只在参数或安全语义不确定时读取 Agent leaf Schema(例如 `dws schema --cli-path "wiki +<shortcut>" --compact --format json`),在当前 Cobra flags 不确定时读取 `dws wiki <shortcut> --help`。只有参数映射、接口绑定或 provenance 审计才省略 `--compact`。仅当现有路由和 reference 都无法定位低频能力时,才用 `dws schema search --query "<用户意图>" --product wiki --limit 5` 同时搜索原子命令和 shortcut。`dws shortcut list --service wiki` 只作人工审计/兼容 fallback。
| Shortcut | 风险 | 适用场景 |
|---|---|---|