The Evaluation Stack: Metrics That Predict Production Quality

Last Updated on August 24, 2026 by Editorial Team Author(s): Armin Norouzi, Ph.D Originally published on Towards AI. The Evaluation Stack: Metrics That Predict Production Quality The hardest engineering problem in LLM deployment is not latency or cost — it is knowing whether your model got better. Metrics that seem rigorous (BLEU, ROUGE, BERTScore) correlate weakly with human judgment. Metrics that correlate well (GPT-4 judge, task success rate) cost hundreds of dollars per evaluation run and introduce their own biases. This article maps the evaluation landscape and proposes a layered stack that delivers high signal efficiently, without breaking the bank on every pull request. After explaining why surface metrics like BLEU and ROUGE fail (they reward overlap, are weakly correlated with human preference, and can be gamed), the article details how to use LLM judges responsibly by mitigating self-preference, position, length, rubric, and benchmark-saturation issues. It then argues for functional metrics (task success, spec compliance checklists) as an underused workhorse that is cheaper, more diagnostic, and harder to game. For safety, it highlights the base-rate/precision-recall tra…