AI Verification
AI verification reads the surrounding code context plus the rule that fired and returns a structured verdict — true_positive, false_positive, or uncertain — with a confidence score, written rationale, and a suggested fix when appropriate. Run it on a single finding, in bulk, or automatically as new findings arrive.
Configuring AI verification (Settings → AI)
Section titled “Configuring AI verification (Settings → AI)”AI verification needs an LLM provider. Configure one in Settings → AI:
| Control | Effect |
|---|---|
| Provider | OpenAI, Anthropic, AWS Bedrock, or any OpenAI-compatible API (Ollama, LM Studio, vLLM, Azure OpenAI, and similar). |
| API key | The provider credential. Leave it blank when re-saving to keep the existing key. On Bedrock running on EC2 you can omit it and rely on an attached IAM role. |
| Model | The model to call — chosen from a list for OpenAI and Anthropic, or typed free-form for Bedrock and OpenAI-compatible endpoints. |
| Base URL | The endpoint for OpenAI-compatible or self-hosted models; optional for OpenAI. |
| Automatic verification | When on, new findings from each scan are verified automatically — see below. |
| Minimum severity | With auto-verify on, only verify findings at or above this severity. |
| System prompt | The base instruction sent to the model. Reset to defaults restores the built-in prompt. |
| Custom instructions | Appended to the system prompt — add context about your stack, frameworks, or security policy to sharpen verdicts. |
Running verification
Section titled “Running verification”On a single finding. Open the finding detail page and click Verify with AI. The job is queued; the verdict appears on the same page within seconds.
In bulk. Two paths from the Findings list:
- Analyze unverified — dispatches verification across every unverified finding matching the current filter (capped at 500 per click). No row selection needed.
- Re-Analyze — select rows and click this to force re-verification, even on already-verified fingerprints. Useful after editing organizational memory if you want existing verdicts to refresh.
Each finding is queued as a separate job — there’s no batching today.
Automatically. Enable Auto-verify in Settings → AI and pick a minimum severity (e.g. high+ to skip lows and mediums). A 1-minute sweeper picks up unverified findings at or above the threshold and dispatches verification jobs in batches of up to 100 per org per pass. Verdicts are advisory — they ride alongside the finding’s status without auto-closing it.
What the verdict contains
Section titled “What the verdict contains”| Field | Description |
|---|---|
verdict | true_positive / false_positive / uncertain |
confidence | low / medium / high |
reasoning | A short paragraph explaining the conclusion |
suggested_fix | (true positives only) Concrete code change |
The full block (verdict, reasoning, suggested fix) is visible on the finding detail page and ships in SARIF exports. CSV exports include the verdict and confidence columns. PR comments and Slack/Teams notifications use the verdict to gate delivery (see “Use it as a notification filter” below) but don’t render the verdict body inline today.
How verdicts are scoped
Section titled “How verdicts are scoped”Verdicts are cached by fingerprint across your entire org — once a fingerprint is verified anywhere, every other finding with the same fingerprint inherits the verdict. This includes findings on different branches and in different projects that happen to share a fingerprint. If you change the underlying code such that the fingerprint changes (different file, different line, different normalized snippet), the new finding gets a fresh verification.
What context goes into the prompt
Section titled “What context goes into the prompt”Each verification call sends Claude:
- The rule that fired (description, tags, severity).
- A snippet of code around the finding (trimmed to fit token budget).
- The finding metadata — file path, line number, package version (for SCA).
- Per-scan-type guidance — e.g. for SAST, “consider whether the sink receives attacker-controlled input”; for Secrets, “check if the secret is real or a documented placeholder”.
- For SCA findings: EPSS (exploit probability) and CISA KEV (actively exploited) signals when known.
Your organizational memory also shapes the verdict — matching memories are applied to every verification run so verdicts reflect your codebase’s conventions, not just the rule’s defaults.
Use it as a notification filter
Section titled “Use it as a notification filter”Each Slack, Teams, email, and webhook integration has an AI verdict filter with three modes that control when a per-finding alert (critical_finding / high_finding) actually fires:
| Mode | When it pings |
|---|---|
| All findings (default) | Immediately on every new critical/high finding — the historical behaviour. |
| AI-verified TPs only | Waits until verification commits, fires only when the verdict is true_positive. |
| TPs and uncertain | Waits until verification commits, fires for true_positive or uncertain, skips clear false_positive. |
The verified-only modes are the way to keep a noisy channel quiet — they trade a few seconds-to-minutes of delay for false-positive suppression.
Prerequisites. The verified-only modes need org-level AI verification turned on (provider, model/API key set, and Auto-verify in Settings → AI). The integration form blocks the verified-only modes until that’s true; the API enforces the same rule with a 400 if you bypass the UI.
Resurrected findings notify immediately. When a finding is re-detected after being auto-fixed (or first appears on a new branch where a sibling already has a verdict), it inherits the cached verdict at scan time. Verified-only integrations use that inherited verdict immediately — no second wait.
What’s exempt from the filter. scan_completed, scan_failed, and all cve_* events bypass verdict gating. Scan-level summaries fire regardless, and CVE Watch findings don’t traverse the LLM verifier at all. Recurring findings (still-open from a prior scan) never trigger per-finding events to begin with — they only show up inside the scan_completed digest.
Edge case. If verification permanently fails for a specific finding (three retried LLM errors), no per-finding alert fires for verified-only integrations on that finding. This is a narrow gap — the broader “AI is off entirely” case is prevented by the prerequisite gate.