Home
New to AI
All 47 tools →
Features
Use cases
Live demo
Impact analytics
Pricing
Blog
FAQ
About
CLAUDE.md Generator
Few-Shot Example Set Builder
MCP Config Wizard
System Prompt Generator
Prompt Optimizer
RAG Chunking & Retrieval Planner
ReAct Prompt Converter
Skill File Generator
Structured Output Schema Builder
Subagent & Slash Command Generator
Tool / Function Schema Generator
Workflow to Agent Blueprint
Free tool
Eval Runner
Run a saved eval set against one or more models and get a scorecard: pass rate, cost, and latency — with evidence, not vibes.
Get started-
A hard cost ceiling, set before the run starts, stops the job cleanly and keeps every result so far — never a surprise bill.
-
Compare multiple models on the exact same eval set in one job, with an honest "too close to call" callout when pass rates overlap instead of a forced winner.
-
Choose your grading method: free deterministic checks, structured-field validation, LLM-as-judge against your rubric, or a human review queue — whichever fits the eval set.
Frequently asked questions
The target model(s) you choose to test are Claude models, called through Costiva's own account. When you choose LLM-as-judge grading, a Claude model (Sonnet by default) also grades each case against your eval set's rubric.
Yes. Running evals reads a saved eval set from your library, spends against your org's monthly run quota, and enforces a real dollar cost ceiling — all of which need an account and an organization to mean anything.
Generate one with the free Eval Set Generator and save it to your library, or upload your own JSON/CSV matching the same schema (featureName, cases, rubric, passBar).
The run stops cleanly before starting the next case that would exceed your ceiling. Every case already graded stays in the scorecard — nothing is discarded, and the run is clearly marked as stopped early rather than complete.
Every pass rate on this tool is shown with its sample size and a 95% confidence interval (Wilson score interval), and small samples are explicitly flagged as directional. When comparing two models, overlapping intervals are called out as "too close to call" rather than declaring a winner.