Home
New to AI
All 47 tools →
Features
Use cases
Live demo
Impact analytics
Pricing
Blog
FAQ
About
CLAUDE.md Generator
Few-Shot Example Set Builder
MCP Config Wizard
System Prompt Generator
Prompt Optimizer
RAG Chunking & Retrieval Planner
ReAct Prompt Converter
Skill File Generator
Structured Output Schema Builder
Subagent & Slash Command Generator
Tool / Function Schema Generator
Workflow to Agent Blueprint
Free tool
Prompt Regression Diff
Prove a prompt change didn't break anything before it ships — run both versions against the same eval set and see what changed.
Get started-
Runs prompt A and prompt B against the exact same eval set and config, so a difference in pass rate is actually about the prompt — not a different test.
-
Never calls a difference a "win" on a small sample: overlapping confidence intervals are honestly reported as no detectable difference.
-
A CI-callable endpoint returns pass/fail so a bad prompt change can block a deploy, not just get noticed after the fact.
Frequently asked questions
The two prompt versions are tested against Claude models of your choice, called through Costiva's own account. A short plain-English summary of what changed is written by a small Claude model (Haiku) from the cases whose verdict flipped.
Yes. This reads a saved eval set (and optionally saved prompt versions) from your library, spends against your org's monthly run quota for both underlying runs, and enforces a real dollar cost ceiling — all of which need an account and an organization.
Each prompt's pass rate is shown with a 95% confidence interval (the same Wilson score interval the Eval Runner uses). If the two intervals overlap, the verdict is "no detectable difference" — never a forced winner on noise. Otherwise the higher pass rate is called improved or regressed.
Yes — the same check can be run from a script via a Bearer-token-authenticated endpoint, which returns pass/fail against an optional numeric pass-bar and a "never allow a regression" rule by default. Generate the token on this tool's setup page.
Approving creates a deployment marker (the same mechanism used across Costiva's Impact Analytics), so the real-world before/after cost and usage effect of switching to the new prompt shows up automatically once enough data has passed.