GLOSSARY
Eval Harness
The test suite for an AI product: golden datasets, scorers and a gate that fails the build when quality regresses.
An eval harness is CI for model behavior. You keep a frozen set of inputs, define what “good” means — exact match, rubric, model-as-judge, human review — and run every prompt or model change against that yardstick before users see it. Braintrust and LangSmith productized the loop; many teams still keep a pytest file and a spreadsheet.
Without a harness, “the model got worse” is an anecdote. With one, it is a diff. The discipline is curating the dataset so it represents production, not the demo, and resisting the urge to tune until the eval is green while the product is not. Evals that never fail are decoration.