Pular para conteúdo

AUR-580: deliberation with tools

Measured before (base e1f14ef7)

  • llm.ToolSpec/ToolCall/Message/ToolCaller (internal/llm/tools.go) and litellm.Provider.CompleteWithTools existed with no caller. The orchestrator had only Complete/CompleteMessages, each a single call with one Reserve/Commit.
  • The answer format was response_format: json_object (JSONMode); no JSON Schema was ever sent.
  • The prompt's available_tools slot (AUR-578) was never filled by cmd.
  • Every enabled quality_gates.scanners entry ran before the model, required or not.

What changed

  • internal/deliberation (new, no review semantics): Tool{Spec(); Run(ctx, args) (Result, error)}, Limits{MaxRounds, MaxCostTokens, PerToolTimeout} (each must be positive), Session.Run (the loop), ValidateArguments (the call's JSON object against the tool's schema: required properties, declared types, no undeclared property when the schema closes the object), Transcript (offered, requested, not requested, rounds, tokens, each call with redacted id/name/arguments, duration, status executed|refused|failed, summarized result, result digest) and LimitError (deliberation_limit:<limit>). A round is one model call; exceeding max_rounds, max_cost_tokens (tokens in+out summed over rounds) or per_tool_timeout_seconds returns the error and no answer: text the model wrote beside its tool calls is never an answer. An unknown tool or invalid arguments are refused before anything runs and the model is told why; a tool error is reported to the model (the tool records what it means). Tool messages are capped at 8 KiB on a rune boundary; every tool result and argument passes the redaction filter.
  • internal/llm: Orchestrator.CompleteWithTools (orchestrator_tools.go): one round per call, reserve before / commit after (release on failure), fallback only between providers found as ToolCaller through llm.As, ErrToolsUnsupported when none is; SupportsTools. SchemaOf (schema.go) derives a JSON Schema from a Go struct by reflection (no new dependency). Options.ResponseSchema/ResponseSchemaName; LiteLLM's tool request sends response_format: {type: json_schema, json_schema: {name, schema}} when a schema is set and keeps json_object otherwise (answerFormat). Complete and CompleteMessages are unchanged.
  • internal/review: Reviewer.SetDeliberation / Transcript (deliberate.go); GenerateReviewWithContext calls answer, which is the old single call without deliberation and the bounded session with it (the answer schema is llm.SchemaOf(types.ReviewResult{})). internal/review/tools (new): scanner_<engine> (one per deferred scanner; runs through an injected ScanFunc), codebase_context (internal/context resolver, only for a path of the diff), skill_section (full text of a configured skill section by its rule id), and Manifest (the prompt's tools slot: name, description, declared cost and result size). NewOfflineProvider / ToolFixtureProvider (fixture_tools.go): an offline fixture whose aurumcode_fixture declares tool_calls (tool, arguments, prompt_contains, lines_above, every_round) is a ToolCaller; every other fixture keeps the plain provider, so no existing run is offered tools.
  • internal/config: deliberation: {enabled, max_rounds, max_cost_tokens, per_tool_timeout_seconds} (deliberation.go), defaults 3 / 60000 / 120 s, negative values and unknown keys refused by the strict Parse; under a central policy a declared section decides alone (named warning), a silent policy keeps the repository's.
  • cmd/aurumcode: runScanners defers every enabled entry that is not required when deliberation is enabled (onDemand); scanEntry is the one per-entry path (the evidence phase and a model request both use it, with the same root, trust and unverified-checkout refusal). review_deliberation.go: prepareDeliberation (both sources, after the reviewer is built), scanOnRequest (a requested scan joins s.scans, so its findings count in the gate with the engine's origin and its failure is the scan's inconclusive reason), settleDeferredScans (when nothing was offered: provider without tools, profiles, skipped or failed model before the offer, the deferred scanners run as before), reportDeliberation (stderr decision line and one line per call), noteDeliberationLimit (a LimitError becomes the inconclusive model outcome). The per-file model cache is bypassed when tools were offered (errDeliberationNotCacheable); the verdict-reuse key folds ToolResultsDigest (digest of the executed calls' result digests) into cache.RequestKey. The transcript is the audit's deliberation field, only when tools were offered; every other audit keeps its bytes.
  • Docs: architecture.md (two packages, the Deliberation section), configuration.md (deliberation), tutorial docs/tutorials/deliberacao.md
  • demo/tutoriais/deliberacao (registered in docs/index.md, docs/tutorials/README.md and demo/tutoriais/README.md).

Divergences from the card text (recorded)

  • Tool names on the wire allow [a-zA-Z0-9_-] only, so the scanner tool is scanner_semgrep, not scanner:semgrep.
  • MaxCost is max_cost_tokens: a token ceiling works without a price table. A USD ceiling (--limite) is enforced per round by the reservation.
  • internal/prompt is not in the card's paths: the manifest's size estimate is part of the Cost text of prompt.ToolOffer.

Card v2: a limit goes through the gate

v1 exited 1 before the gate on a limit, so no audit, SARIF or --pr status was written. v2 (internal/gate and internal/render in paths):

  • gate.ModelDeliberationLimit is a model outcome, AlwaysNotReviewed in both sources (internal/review/session/sources.go), so the exit is 1 via gate.ExitPolicy whatever gate.inconclusive says.
  • gate.RankReason ranks it first, as deliberation_limit:<limit> (ReasonInputs.DeliberationLimit); the gate runs, the motive reaches the gate line, the audit's gate.reason, the SARIF notification and the --pr aurumcode/policy-gate status (failure under block); the aurumcode/review status is failure ("revisão por modelo inconclusiva").
  • render.AuditRecord.Deliberation (*deliberation.Transcript, field deliberation) is first class; Transcript.Limit names the exceeded limit. The cmd-side splice was removed.
  • noteDeliberationLimit (cmd) replaces the model's result with one limitation ("inconclusivo: limite de deliberação (...); nenhum parecer do modelo foi publicado") marked quality_degraded: no text of the model is kept.

  • Proof: TestAUR580RoundsExceededIsInconclusiveAndUnpublished (--base: exit 1, audit gate.reason and deliberation.limit, SARIF names the limit, no model text) and TestAUR580PullRequestRoundsExceededFailsThePolicyGate (--pr: exit != 0, policy-gate and review statuses failure, audit deliberation.limit), both in AC-002; tutorial case estoura-rodadas prints the audit's gate reason and limit.

  • Deliberation is off by default (enabled: false).

  • --pr uses the same shared steps (prepareDeliberation in buildReviewer, noteDeliberationLimit in generateReview, settleDeferredScans in joinEvidence) but has no end-to-end test of its own besides the limit test.

Proof

All Go commands ran in the shared container (.board/bin/go-shared), in series.

Item How Last line
AC-001 TestAUR580ModelAsksForTheScannerOnALargeDiff (fixture lines_above: 1: fakescan runs once, its finding fails the gate with origin fakescan in the audit, the answer cites it, the prompt carries ## Available tools with scanner_fakescan and codebase_context and their cost), TestAUR580ModelDoesNotAskOnASmallDiff (lines_above: 50: no scan, exit 0, audit not_requested has scanner_fakescan), TestAUR580ToolResultGoesBackAndAnswerEnds AUR-580/AC-001/pass
AC-002 TestAUR580MaxRoundsIsALimitErrorWithoutAnswer, ...MaxCostTokensIsALimitError, ...PerToolTimeoutIsALimitError, ...RoundsExceededIsInconclusiveAndUnpublished (exit 1, gate reason and audit deliberation.limit, SARIF, no model text), ...PullRequestRoundsExceededFailsThePolicyGate, ...CostExceededIsInconclusive AUR-580/AC-002/pass
AC-003 TestAUR580EveryToolRoundIsReservedAndCommitted (two rounds commit 3.5 USD = 1.0 + 2.5), ...RoundOverTheCeilingIsRefusedBeforeTheCall, ...NoFallbackToAProviderWithoutTools (plain provider: ErrToolsUnsupported; as fallback of a failing tool provider: 0 calls), ...ToolRoundSendsTheJSONSchema, ...SchemaOfStruct AUR-580/AC-003/pass
AC-004 TestAUR580InvalidArgumentsRefusedBeforeRun (wrong type, missing, undeclared, non-object; unknown tool), ...InvalidArgumentsRefusedAndRedactedInTheAudit, ...RequiredScannerIsNeverOptional (runs once before the model, absent from the prompt and the transcript's offered), tools, config and missing-binary tests AUR-580/AC-004/pass
AC-005 TestAUR580TranscriptRedactsArguments (redaction, 7 ms duration from an injected clock, digest), the audit tests above, demo/tutoriais/deliberacao/run.sh --check AUR-580/AC-005/tutorial/pass
MUT-001 session.go: over max_rounds, return the last reply as the answer RED: session_test.go:110: err = <nil>, want a max_rounds LimitError; aur580_test.go:135: exit=0, want non-zero with the deliberation_limit motive; AUR-580/MUT-001/rejected
MUT-002 orchestrator_tools.go: if true \|\| o.tracker == nil { (no reservation for a tool round) RED: orchestrator_tools_test.go:49: tracker committed 0.0000 USD over two rounds, want the sum of the rounds 3.5000; AUR-580/MUT-002/rejected
all bash tests/acceptance/AUR-580.sh all AUR-580/all/pass

Anchors of done acceptances

No anchor of another card moved. The removed lines of this change were checked against every tests/acceptance/*.sh: AUR-568 anchors return render.WriteAuditRecord(in.auditoriaPath, rec, filter), kept verbatim; AUR-577 MUT-001 anchors out := s.deps.scanners.Scan(s.ctx, engine, scanner.Request{, kept verbatim in scanEntry (a requested scan runs on a copy of the state carrying the tool's deadline in ctx); AUR-460 anchors the Complete request of litellm/provider.go, unchanged.

Review closing: a model that never answered decided nothing

Found by the review of 227f6e12 (cmd/aurumcode/review_deliberation.go): when the provider failed or --limite refused a round after the tools were offered, toolsOffered stayed true, the deferred scanners never ran and the transcript listed them as "not requested", a decision the model never made. settleDeferredScans now keeps the transcript only when the model decided (it answered, or its deliberation hit a limit, already inconclusive); otherwise it runs every deferred scanner the model had not already run, as before deliberation existed, and Transcript.MarkUndecided records why (undecided, with not_requested empty), on stderr and in the audit. Proof: TestAUR580DeferredScannerRunsWhenTheModelNeverAnswered (--limite refuses the first round under --seguranca: fakescan runs once, its finding is in the audit's blocking_findings with origin fakescan, the transcript is undecided with no tool declined), in AC-004.