Seonglae ChoLLM Judge
Using continuous scores for LLM as a judge is not effective. LLMs perform better when making categorical judgments. It is recommended to first have LLM judges make categorical assessments, which can then be aggregated into continuous metrics if needed.
Limitation
- egocentric - prefer himself (AI Introspection)
LLM Judge Types
llm-as-a-verifierllm-as-a-verifier • Updated 2026 Aug 19 16:46
llm-as-a-verifier
llm-as-a-verifier • Updated 2026 Aug 19 16:46
llm judge weakness
political-neutrality-evalanthropics • Updated 2026 Mar 5 7:22
political-neutrality-eval
anthropics • Updated 2026 Mar 5 7:22
Inter-model evaluation agreement rates:
- Claude Sonnet 4.5 ↔ GPT-5
- 92% agreement
- Claude Opus 4.1 ↔ Sonnet 4.5
- 94% agreement
Human evaluator agreement is around 85% → Model graders are more consistent than humans.
