Seonglae ChoLLM Benchmarks
Language Model Metrics
LLM Evaluation Tools
Even a small improvement in per-step accuracy can lead to significant performance differences in long-horizon tasks due to cumulative effects. Therefore, the economic value of LLMs should be measured by how accurately they perform long tasks. Accuracy improvements result in exponential increases in achievable task length. The relationship between step accuracy p and success rate s:
While MMLU is a simple multiple-choice evaluation, even minor changes in option formatting can significantly affect performance scores. On the other hand, evaluations like BBQ Benchmark, Big-Bench, and HELM are noted for their complexity due to challenges in implementation, interpretation, and technical intricacies that make it difficult to accurately measure model performance.
Evaluating LLMs is complex so more comprehensive and purpose-specific evaluation methods is needed to assess their capabilities for various real-world applications
Every time we solve something previously out of reach, it turns out that human-level generality is even further out of reach.
