Ragas

Ragas is an open-source evaluation framework aimed at retrieval-augmented applications. It supplies automatic metrics, generates synthetic test sets, includes metrics specific to retrieval such as faithfulness and context relevance, and extends to monitoring quality in production.

The reason this matters is that RAG fails in places ordinary testing does not look. A retrieval pipeline can return fluent, confident, wrong answers when it retrieves the wrong passage, and there is no exception thrown and no stack trace — the failure is a sentence that reads well and is not supported by the source. Metrics built for that shape of failure are the point: faithfulness asks whether the answer is grounded in the retrieved context, context relevance asks whether the right context was retrieved at all, and the two separate a retrieval problem from a generation problem. As with any evaluator, some of the metrics are themselves model-judged, which imports the judge's blind spots into your scores.