What is LLM as a judge?

An LLM-as-a-judge is a model that scores another model's output against a rubric you write. It is used because LLM applications often produce conversational text with no single correct answer: https://docs.langchain.com/langsmith/llm-as-judge

When should you add an LLM judge?

Later than you think. Build the feedback loop first: real chats, a place to leave human feedback on them, and evals built from that feedback. Add the judge after that and check it against the feedback.

Synthetic vs real evals: which should you use?

Real ones. Synthetic cases only test what you imagined in advance. Evals built from real chats and real user feedback test what users actually do.

Does an LLM judge need human feedback?

In my experience, yes. My judge was awful until it had a lot of human feedback. LangSmith's Align Evals works the same way: you score examples yourself, then tune the judge until its scores line up with yours: https://blog.langchain.com/introducing-align-evals

What are agent evals?

Evals are a way to break down what good looks like and measure it, per the LangSmith docs: https://docs.langchain.com/langsmith/evaluation-concepts. Agent evals apply that to an agent's behaviour. The order I use is real chats, human feedback, evals, then a judge.

How do you build evals from real chats?

Read real conversations in an admin portal and review them one message at a time. The feedback becomes the evals, and the judge checks against it. LangSmith's annotation queues do the same job: you flag runs, collect annotations, and transfer them to datasets: https://docs.langchain.com/langsmith/evaluation-concepts

LLM-as-a-Judge Was Awful Until I Gave It Real Feedback

My first LLM-as-a-judge was awful.

I was tuning it for a voice app side project.

The model did not understand what I wanted it to grade.

It marked the app as correct for not stating facts it did not understand.

In the process, the app ended up saying nothing at all.

It only started getting things right after a lot of human feedback.

An LLM as a judge with no human behind it is awful

An LLM-as-a-judge is a model that scores your app's output against a rubric you write.

While I was tuning it, the judge did not understand.

A lot of human feedback is what got it there.

An LLM judge with no human grounding is awful.

Evals without real user testing are worth very little

Evals have the same problem as the judge.

An eval built without actual user testing has nothing real behind it.

Add real cases to the evals.

Add real feedback to the judge.

Build the feedback loop first, with real chats in an admin portal

I built a feedback admin portal for the voice app.

It is based on real chats.

Admins go through the conversations and review them one message at a time.

The feedback collected there helps build the evals.

Then point the judge at real cases, not synthetic ones

Once the portal has real feedback in it, the judge has something to check against.

It checks against real feedback and real cases.

Synthetic cases only test what I thought of in advance.

Real chats show what users do.

Do not build the judge too early

Do not start with the judge.

Start with the systems that feed it.

For me the order was:

Real chats from real users.
A place to leave feedback on those chats.
Evals built from that feedback.
A judge, checked against the feedback until it agrees with it.

The LangChain docs describe the same loop

LangChain's docs are good on this.

They write about LangSmith, but the ideas carry over to any setup.

Their evaluation concepts page says human feedback often provides the most valuable assessment, and describes annotation queues for collecting it, with annotated runs transferred into datasets for future evaluations.

It also describes a loop where online evaluations surface issues that become offline test cases.

The Align Evals announcement covers the judge side.

You grade a set of examples yourself, and those scores become a "golden set".

Then you tune the judge's prompt and compare its scores against yours with an alignment score.

LangSmith's guide to defining an LLM-as-a-judge evaluator adds that human corrections can be inserted into the judge's prompt as few-shot examples.

My version is a portal I added to my own app.

The order is the same.

Frequently asked questions

What is LLM as a judge?: An LLM-as-a-judge is a model that scores another model's output against a rubric you write. It is used because LLM applications often produce conversational text with no single correct answer: https://docs.langchain.com/langsmith/llm-as-judge
When should you add an LLM judge?: Later than you think. Build the feedback loop first: real chats, a place to leave human feedback on them, and evals built from that feedback. Add the judge after that and check it against the feedback.
Synthetic vs real evals: which should you use?: Real ones. Synthetic cases only test what you imagined in advance. Evals built from real chats and real user feedback test what users actually do.
Does an LLM judge need human feedback?: In my experience, yes. My judge was awful until it had a lot of human feedback. LangSmith's Align Evals works the same way: you score examples yourself, then tune the judge until its scores line up with yours: https://blog.langchain.com/introducing-align-evals
What are agent evals?: Evals are a way to break down what good looks like and measure it, per the LangSmith docs: https://docs.langchain.com/langsmith/evaluation-concepts. Agent evals apply that to an agent's behaviour. The order I use is real chats, human feedback, evals, then a judge.
How do you build evals from real chats?: Read real conversations in an admin portal and review them one message at a time. The feedback becomes the evals, and the judge checks against it. LangSmith's annotation queues do the same job: you flag runs, collect annotations, and transfer them to datasets: https://docs.langchain.com/langsmith/evaluation-concepts