I've been on a kick recently about using AI for quality work, but with the emphasis on the human folding in the opinions and judgement to produce that quality. Put more succinctly, AI is great for generating a lot of work, but it takes a human collaborator to produce something actually good.

That has lead me down the path of testing AI as a judge. It's difficult to test the quality of the outputs of AI. Typically it's just "I know it when I see it". But if we could test an AI's ability to judge quality, then we can better understand how it works and what the limitations are.

What's more, if we had an AI that could judge quality effectively enough, then we can use AI to produce quality outputs, since all we would need to do is have it keep refining the output until it met our quality standard.

So how well can AI play the judge? I had been reviewing an article from last year (October 2025) where researchers had proposed a benchmark called WritingPreferenceBench. They had compiled a list of 1,800 pairs of writing samples, in English and Chinese across 8 genres, where one of the samples was deemed (by a human) to be better quality than the other. They filtered out pairs with grammar errors, factual errors, or big differences in length, so the only thing left to judge was quality. Then they let 14 different AI models judge which was better.

The result was that the AI judges, on the whole, were only a little better than a chimpanzee mashing a random button. On average, the models agreed with the humans 53.9% of the time, where a coin flip gets 50%. OpenAI's o3, running on high reasoning effort, scored 48.1%.

But as I wanted to begin making conclusions about it, I didn't feel like it would be fair to make any claim about AI's abilities based on research using models that were over a year old.

I found some more recent benchmarks that showed some improvements across models. The most recent was LongJudgeBench, published by researchers at Tsinghua University in June 2026. It reused 263 of the WritingPreferenceBench pairs and tested newer models, including OpenAI's GPT-5.2, Qwen3-Max, Kimi-K2.6, GLM-5.1, and DeepSeek-V4-Flash. The judges did a bit better, averaging 57.8% (versus 53.9% the year before), but they stayed close to chance. The best, Qwen3-32B with thinking turned on, scored 63.9%. GPT-5.2 scored 52.7%. The researchers' own conclusion was that "stronger general-purpose capabilities may not always translate into stronger long-form judging performance." It didn't test any Claude models.

But still (in October 2026), the experience of using the latest Anthropic models is noticeably better than anything we had at the beginning of this year. So I decided to test Opus 5.5 to see how it scored and if we're on a trajectory toward better judgement. The short answer is: "not really."

The test

I used the data from LongJudgeBench, which the researchers published on GitHub, along with the same judging prompt they used. The data has pairs of creative writing. For each pair, human experts selected the better piece. The pairs have no grammar errors and no factual errors. Thus, the only difference is quality: style, voice, and how well the writing works for a reader.

I took 30 of the 263 pairs at random (13 were English, 17 Chinese). The names of the pieces were removed so the focus was just on the writing, and in order to ensure that there was no bias based on the order shown, each pair was tested twice, with the order flipped in the second instance. Thus, a total of 60 agents were spawned to review the two pieces of writing and state which was better.

Then I compared the choices of the AI with the choices of the experts.

The results

Opus 5.5 agreed with the human experts 62% of the time (37 of 60 judgments). Chance is 50%. It scored about the same in English (61.5%) as in Chinese (61.8%).

This result is almost the same as the results for older models. On the same benchmark, the best model previously tested (Qwen3-32B, with thinking turned on) got 64%, while OpenAI's GPT-5.2 got 53%. So, the latest Opus model did not make a large improvement.

Another thing that came up was that Opus 5.5 was more consistent and less prone to bias by simply the order that the writing samples were presented. When I changed the order of the pieces, Opus changed its choice on only 3 of 30 pairs. GPT-5.2, for example, changed its choice on almost half of the pairs (46%).

But consistent isn't the same as right. On 17 of the 30 pairs, Opus agreed with the experts both times. On 10 pairs, it disagreed with the experts both times, no matter which piece it saw first.

What this means

The AI has its own taste, but its taste is often different from that of the human experts.

This is a larger problem than a coin flip. A random judge is easy to identify, because its choices change each time, while a consistent judge with different taste is hard to identify. The problem is that it moves your content, code, conclusions, or whatever toward its standard, not toward your own standard.

Limits of this test

I must point out that this was a small sample size, although I had expected 30 would give us at least enough direction to say whether or not the findings from the original 2025 research were obsolete or not and I would say that it appears that the answer is "no." For the statistically minded: with 30 pairs, the margin of error is about ±17 points, so the true score could be anywhere from the mid-40s to the high 70s.

Further, I ran this test with Opus 5.5 using "low" thinking effort, and while it's possible that giving it higher effort could produce different results, I think it is important that we run benchmarks on multiple levels of thinking effort since not every task we would want to run should require the maximum expenditure. I would add that in the previous research, the thinking models didn't really outperform the non-thinking versions. In the 2025 study, the researchers found no consistent advantage for reasoning models (o3 on high effort scored below chance). In the 2026 study, turning on thinking raised Qwen3-32B's score by only about 2 points (61.6% to 63.9%). And if you are not satisfied with that explanation, you can run your own test (*wink).