Pular para conteúdo

AUR-505 — Invalid model output degrades the review instead of crashing

Status

Implemented in cmd/aurumcode. Unit proof: tests/unit/AUR-505.go::TestAUR505; acceptance: tests/acceptance/AUR-505.sh.

Problem

The pull-request review path (cmd/aurumcode/pr.go, runPRReview) selected a provider, called it, and — when the response did not validate against the parser's schema (validation_failed) — printed

aurumcode review: could not understand the model's response (validation_failed)

and returned 1 immediately. The deterministic half of the review (the embedded static-analysis catalog and, with --seguranca, the security pass) never ran, so nothing was published.

In CI this is noise: a weak model or a response outside the schema fails the pull request even when the deterministic analysis and security passes would have been clean (measured on pull requests #37/#40 with gpt-4o-mini and gpt-4o on a large diff).

Behavior

runPRReview now keeps a qualityDegraded flag. When reviewer.GenerateReviewWithContext returns a *prompt.ParseError:

  1. The diagnosis is still printed to stderr, plus one line stating that the run is degrading to deterministic analysis.
  2. qualityDegraded is set and result becomes an empty types.ReviewResult carrying one declared limitation (modelInvalidOutputNotice) that names the model failure. The limitation is published inside the review summary. Its only dynamic value is the ParseErrorKind enum constant, a trusted string, so it carries no model-authored bytes.
  3. The deterministic passes run exactly as on the normal path: static analysis (mergeStaticAnalysis) and, with --seguranca, the security pass.
  4. Publication, --check, --fail-on and the review memory save all behave as before, except that a degraded run does not persist model memory (there is no model answer to remember).

No finding is fabricated. result starts empty, so the only findings published are the deterministic ones.

Exit code

The invalid model response no longer forces 1. The exit code follows the deterministic gate, exactly like a normal run whose model half found nothing:

Deterministic state --check exit
a deterministic error finding failure 3
no deterministic error finding success 0

Without --check/--fail-on, a degraded run exits 0, matching the card's goal: a model that cannot be parsed must not fail a CI pull request whose deterministic passes are clean. The degradation is never silent — it is recorded in stderr and in the published limitation.

This is the AUR-458 mechanism (the deterministic pass survives a quality failure) extended from provider runtime failure to invalid output, with the invalidity represented as a declared limitation instead of a hard error.

Non-goals

  • No model change, no parser re-implementation.
  • Malformed output is not accepted as valid and no finding is invented from it.
  • The --base path (runReview) keeps AUR-458's behavior: a provider that was configured and failed still exits 1 after publishing the security pass. AUR-505 is scoped to the PR path, where the CI defect was measured.

Acceptance

tests/acceptance/AUR-505.sh selectors:

  • all — AC-001, AC-002, AC-003, MUT-001, MUT-002.
  • AC-001 — invalid response: deterministic findings published, limitation recorded, exit follows the deterministic gate (3 with a deterministic error finding, 0 on a deterministic-clean diff).
  • AC-002 — valid response merges normally.
  • AC-003 — invalid response publishes exactly the deterministic finding set and no fabricated finding; the failure is recorded.
  • MUT-001 — dropping the deterministic findings (old return 1) makes AC-001 RED; restored GREEN.
  • MUT-002 — fabricating a finding from the invalid response makes AC-003 RED; restored GREEN.

Unknown selector exits 64; a missing go exits 79.

Security

The declared limitation contains only a trusted enum value. The raw invalid response is never published and is never turned into a finding. All model fields continue to pass through internal/review's redaction.