Can an AI actually judge a debate fairly?
A fair thing to be suspicious of, and most AI scoring earns the suspicion. Here is how our judge is built to be reproducible and auditable, and the two things it deliberately refuses to do.
6 min readThe debagle team
It is a fair thing to be suspicious of. An AI that watches two people argue and declares a winner sounds like exactly the kind of thing that should not be trusted, and most of the time the suspicion is correct. So rather than tell you our judge is fair, here is how it is built, what that buys you, and the things it deliberately does not do.
Why “AI judge” usually deserves the doubt
Point a language model at a transcript and ask “who won?” and you get an answer. You also get three problems.
- It is not repeatable. Ask the same question about the same debate five times and you can get different winners, because the model samples a fresh answer each time. A verdict that changes when nobody changed anything is not a verdict.
- It is a black box.“It felt more persuasive” is not something you can check, argue with, or learn from. You are asked to trust a number with no working shown.
- It rewards confidence. Models are swayed by fluent, assertive language, which is precisely the thing a debate should see through rather than fall for.
If that were how we did it, you should not trust it either. It is not.
The one design decision that changes everything
Our judge is built on a split: the model is only ever allowed to perceive, never to decide.
The AI is asked small, checkable questions about pieces of text. Is this sentence a claim, or a rebuttal to an earlier one? Did this response actually meet that objection, and did the objection hold, dent, or break? It is never asked who is winning, and it never assigns a score.
Everything that decides the number is plain arithmetic applied to those perceptions. No model, no randomness, no temperature. The full mechanism is laid out on the scoring page, but the shape of it is what matters here.
The model reads the debate. The math scores it. Those are two different jobs, and only one of them is allowed near the result.
What that actually buys you
| Property | What it means for you |
|---|---|
| Reproducible | The same debate always produces the same score. There is no dice to roll, so there is nothing to reroll until you like the outcome. |
| Auditable | Because the math is fixed, every point on the scorecard traces back to a specific judgement about a specific line. You can check it by hand. |
| Explainable | The judge returns a written reason for each dimension, so a score arrives with the argument for the score attached, not instead of it. |
None of that makes the judge infallible. It makes it honest about being a system: one that behaves the same way every time and shows its work, which is the opposite of a black box that shrugs and says it felt right.
What it does not do, on purpose
A fair judge is partly defined by what it refuses to claim. Ours refuses two things.
It does not decide who is correct
This surprises people, so it is worth saying plainly. In a ranked debate the topic and your side are both assigned, so you routinely argue a position you do not hold. “Who was right” is not the question the format is even asking. The judge scores how well the case was built and defended: insight, reasoning, specificity, clarity. It is grading the argument, not the answer.
It does not pretend to be perfect
The model can still misread a line. Determinism means a mistake is at least consistent and inspectable rather than random, but it can be a mistake. That is why a verdict is something you can report rather than a sentence you have to accept, and why the reasoning is shown: the point is a judge you can argue with, which is the only kind worth having in a debate app.
“Can I just paste the prompt into ChatGPT and win?”
Someone asks this within the first five minutes, every time, and it is a good question. The answer is that it helps less than you would think, for a structural reason rather than a clever countermeasure.
A debate is not a speech, it is a response. The scoring rewards rebuttal: meeting the specific thing your opponent just said. Pre-written, disconnected content, the kind you get from pasting a prompt cold, reads exactly as what it is, because it answers a debate that did not happen instead of the one in front of you. The thing that scores well is engaging with your actual opponent in real time, and that is the thing a canned answer cannot do.
Attempting to game the judge with generated content also breaks our Terms. But the deeper defence is that the format asks for the one thing automation is worst at: listening, and answering what was actually said.
The point of all this
A judge that is reproducible, auditable and honest about its scope is not a judge that never errs. It is a judge you can reason about, which is a higher and more useful bar. You can see why you scored what you scored, you can learn from it, and you can challenge it when it is wrong.
That is the difference between being marked by a machine and being coached by one. If you want to see it decide a real argument, watch a few answers get scored and read the reasoning for yourself. Then disagree with it. That is allowed, and it is the whole idea.