LLM as a Judge

Creator

Creator

Created

Created

2024 Nov 22 23:37

Editor

Editor

Edited

Edited

2025 Mar 26 23:55

Refs

Refs

LLM Judge

Cons

egocentric - prefer himself

Creating a LLM-as-a-Judge That Drives Business Results –

A step-by-step guide with my learnings from 30+ AI implementations.

Creating a LLM-as-a-Judge That Drives Business Results –

https://hamel.dev/blog/posts/llm-judge/

Creating a LLM-as-a-Judge That Drives Business Results –

Finding GPT-4’s mistakes with GPT-4

CriticGPT, a model based on GPT-4, writes critiques of ChatGPT responses to help human trainers spot mistakes during RLHF

Finding GPT-4’s mistakes with GPT-4

https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/

Finding GPT-4’s mistakes with GPT-4

Thinking-LLM-as-a-Judge (
CoT judge)

EvalPlanner is specific implementation of Thinking-LLM-as-a-Judge which plan

https://arxiv.org/pdf/2501.18099

Judge as a Judge (how to improve consistency)

Judge as A Judge: Improving the Evaluation of Retrieval-Augmented...

Retrieval-Augmented Generation (RAG) has proven its effectiveness in alleviating hallucinations for Large Language Models (LLMs). However, existing automated evaluation metrics cannot fairly...

Judge as A Judge: Improving the Evaluation of Retrieval-Augmented...

https://arxiv.org/abs/2502.18817

Judge as A Judge: Improving the Evaluation of Retrieval-Augmented...

Judge only model

Selene

Frontier AI needs frontier evaluators. Meet Selene.

Discover Atla Selene 1, the world's most accurate LLM Judge for evaluating AI responses. Outperforming frontier models from OpenAI, Anthropic, DeepSeek, and others across 11 benchmarks for evaluators, Selene provides accurate scores and actionable feedback. Make effective custom eval criteria in a few steps using our new Alignment Platform. Start for free today.

Frontier AI needs frontier evaluators. Meet Selene.

https://www.atla-ai.com/post/selene-1

Frontier AI needs frontier evaluators. Meet Selene.

Announcing Atla’s native integration with Langfuse

Atla now integrates natively with Langfuse's LLM observability platform. Use Selene 1 as an “LLM-as-a-Judge” to run evals in Langfuse. Monitor app performance in production and run experiments over datasets. Discover how this powerful integration enhances LLM evaluations.

Announcing Atla’s native integration with Langfuse

https://www.atla-ai.com/post/langfuse-native-integration

Announcing Atla’s native integration with Langfuse

Recommendations

////////