AI Evaluation

AI Benchmark

Good benchmark principles

Make sure you can detect a 1% improvement

Easy to understand the result

Hard enough (SOTA model cannot do it)

Use a standard metric and make it comparable over time (do not update often)

Extension

Can include human baseline

Includes vetting by others

Validation

Benchmarks where AI performs below human level are meaningful

AI Benchmarks

LLM Evaluation

NLP Evaluation

monotonicity

low variance

https://hai.stanford.edu/ai-index/2025-ai-index-report

AI Evaluation Notion

AI Evaluation Awareness

To measure is to know, if you cannot measure it, you cannot improve it - Lord Kelvin

OpenAI

OpenAI Evals

https://evals.openai.com/

Central Limit Theorem to fix lacked statistical rigor form Anthropic

Adding Error Bars to Evals: A Statistical Approach to Language...

Evaluations are critical for understanding the capabilities of large language models (LLMs). Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the...

https://arxiv.org/abs/2411.00640

Benchmarks are unreliable, see results from arena or trustworthy 3rd party

Jim Fan on Twitter / X

It is *incredibly* easy to game the LLM benchmarks. Training on test set is for the rookies. Here're some tricks to practice magic at home:1. Train on paraphrased examples of the test set. "LLM-decontaminator" paper from LMSys found that you can beat GPT-4 with a 13B model (!!)… pic.twitter.com/iMKHBJH4eG— Jim Fan (@DrJimFan) September 9, 2024

https://x.com/DrJimFan/status/1833160432833716715

Types

Alex Strick van Linschoten - How to think about creating a dataset for LLM finetuning evaluation

I summarise the kinds of evaluations that are needed for a structured data generation task.

https://mlops.systems/posts/2024-06-25-evaluation-finetuning-manual-dataset.html

Alex Strick van Linschoten - How to think about creating a dataset for LLM finetuning evaluation

Train-before-Test: before evaluating, fine-tune all models on the same benchmark training set, then evaluate on the test set. The goal is not to reward models that happened to see many similar problems during pretraining, but to compare which models have higher latent potential when given the same preparation. This is also analogous to

Meta Learning, in that it can be viewed as measuring a model’s fine-tuning sensitivity and generalizability.

Train-before-Test Harmonizes Language Model Rankings

Existing language model benchmarks provide contradictory model rankings, even for benchmarks that aim to capture similar skills. This dilemma of conflicting rankings hampers model selection,...

https://arxiv.org/abs/2507.05195