Free tool
Eval Set Generator
Turn a feature description into a starter eval set: concrete test cases, a grading rubric, and a pass bar.
Get started-
Case proportions follow a real testing playbook: happy path, edge cases, failure probes targeting YOUR stated failure modes, and adversarial/out-of-scope inputs.
-
Every case is concrete, pasteable input with pass/fail criteria a grader can check in seconds — never "responds appropriately".
-
Includes a tailored grading rubric and a suggested pass bar, plus jsonl/csv/md downloads so you can start running evals today.
Frequently asked questions
This tool is powered by Claude (Anthropic).
An eval is a set of test cases with expected behaviors, run against your AI feature to check whether it works before and after a change. Without one, you are guessing whether a prompt or model change made things better or worse.
A concrete, realistic input (not a description of one), an expected behavior specific enough that a person can mark pass/fail without guessing, and 1-3 concrete failure symptoms to check for. Vague criteria like "gives a good answer" are not testable.
No — this tool generates the eval set (cases, rubric, pass bar) as a starting point. Running those cases against your model and grading the results is a separate step you do yourself, using the downloaded jsonl/csv/md file.
No. Describe your feature, get your eval set, and download it. An email address is required to generate the file and enforce a fair-use daily limit; no account is created.