feat(ce-review): improve signal-to-noise with confidence rubric, FP suppression, and intent verification (#434)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -22,8 +22,25 @@ Return ONLY valid JSON matching the findings schema below. No prose, no markdown
|
||||
|
||||
{schema}
|
||||
|
||||
Confidence rubric (0.0-1.0 scale):
|
||||
- 0.00-0.29: Not confident / likely false positive. Do not report.
|
||||
- 0.30-0.49: Somewhat confident. Do not report -- too speculative for actionable review.
|
||||
- 0.50-0.59: Moderately confident. Real but uncertain. Do not report unless P0 severity.
|
||||
- 0.60-0.69: Confident enough to flag. Include only when the issue is clearly actionable.
|
||||
- 0.70-0.84: Highly confident. Real and important. Report with full evidence.
|
||||
- 0.85-1.00: Certain. Verifiable from the code alone. Report.
|
||||
|
||||
Suppress threshold: 0.60. Do not emit findings below 0.60 confidence (except P0 at 0.50+).
|
||||
|
||||
False-positive categories to actively suppress:
|
||||
- Pre-existing issues unrelated to this diff (mark pre_existing: true for unchanged code the diff does not interact with; if the diff makes it newly relevant, it is secondary, not pre-existing)
|
||||
- Pedantic style nitpicks that a linter/formatter would catch
|
||||
- Code that looks wrong but is intentional (check comments, commit messages, PR description for intent)
|
||||
- Issues already handled elsewhere in the codebase (check callers, guards, middleware)
|
||||
- Suggestions that restate what the code already does in different words
|
||||
- Generic "consider adding" advice without a concrete failure mode
|
||||
|
||||
Rules:
|
||||
- Suppress any finding below your stated confidence floor (see your Confidence calibration section).
|
||||
- Every finding MUST include at least one evidence item grounded in the actual code.
|
||||
- Set pre_existing to true ONLY for issues in unchanged code that are unrelated to this diff. If the diff makes the issue newly relevant, it is NOT pre-existing.
|
||||
- You are operationally read-only. You may use non-mutating inspection commands, including read-oriented `git` / `gh` commands, to gather evidence. Do not edit files, change branches, commit, push, create PRs, or otherwise mutate the checkout or repository state.
|
||||
@@ -32,8 +49,13 @@ Rules:
|
||||
- Set `requires_verification` to true whenever the likely fix needs targeted tests, a focused re-review, or operational validation before it should be trusted.
|
||||
- suggested_fix is optional. Only include it when the fix is obvious and correct. A bad suggestion is worse than none.
|
||||
- If you find no issues, return an empty findings array. Still populate residual_risks and testing_gaps if applicable.
|
||||
- **Intent verification:** Compare the code changes against the stated intent (and PR title/body when available). If the code does something the intent does not describe, or fails to do something the intent promises, flag it as a finding. Mismatches between stated intent and actual code are high-value findings.
|
||||
</output-contract>
|
||||
|
||||
<pr-context>
|
||||
{pr_metadata}
|
||||
</pr-context>
|
||||
|
||||
<review-context>
|
||||
Intent: {intent_summary}
|
||||
|
||||
@@ -52,5 +74,6 @@ Diff:
|
||||
| `{diff_scope_rules}` | `references/diff-scope.md` content | Primary/secondary/pre-existing tier rules |
|
||||
| `{schema}` | `references/findings-schema.json` content | The JSON schema reviewers must conform to |
|
||||
| `{intent_summary}` | Stage 2 output | 2-3 line description of what the change is trying to accomplish |
|
||||
| `{pr_metadata}` | Stage 1 output | PR title, body, and URL when reviewing a PR. Empty string when reviewing a branch or standalone checkout |
|
||||
| `{file_list}` | Stage 1 output | List of changed files from the scope step |
|
||||
| `{diff}` | Stage 1 output | The actual diff content to review |
|
||||
|
||||
Reference in New Issue
Block a user