GLOSSARY
Evals (Evaluations)
Systematic tests for AI behavior: fixed task sets and graders that tell you whether a model or prompt change actually made things better.
Public benchmarks (MMLU, HumanEval, SWE-bench) let buyers compare models on chart-friendly numbers — and are gamed accordingly, since anything publicly known eventually leaks into training data. Serious teams therefore build private evals: a few hundred examples of their real tasks, graded automatically (exact match, another model as judge) or by humans, run before and after every change.
“Evals are the unit tests of AI” is the right mental model: without them, prompt and model upgrades are vibes. With them, iteration becomes measurable — the difference between teams that ship LLM features and teams that ship working ones.