How many questions
does an AI need to guess a Sax-a-Boom?

Every model gets the identical prompt

One frozen opening prompt, byte-identical for every contestant — no hints, no category, no per-model framing. Anything that would advantage one model over another is a bug in the benchmark, not a feature.

The host never improvises

Answers come from a committed fact sheet plus a table of rulings on the attributes where a reasonable host could honestly answer either way. Every answer is exactly yes, no, or you win — and the host is never the one who decides the last of those: a win is detected by the harness from the name itself, so a contestant cannot talk its way into one by sounding confident. A question that can't be answered yes or no — or that asks two things at once — gets "Yes or no questions only", which carries no information and doesn't count against the score.

A guess is just a question

"Is it a kazoo?" costs exactly one question, like anything else. The run ends only when the model names the product. Describing it correctly as "a toy saxophone" is answered yes, but it isn't a win.

Read the order with some suspicion

The host has one more answer than yes and no: it can say I don't know when the fact sheet genuinely doesn't settle a question. That is honest, and it replaced something worse — previously such questions were rejected as though they were badly formed, so models rephrased them over and over.

But it costs the model a question either way, and models did not draw them evenly. The two that never finished collected them at 14% of their turns; the joint-fastest drew none at all. One model was asked, on its fourth question, whether the thing is used indoors — a broad and useful question — and got I don't know. It went on to need 134 questions, having needed 26 under the previous rules.

So part of a score is luck about which questions happened to fall in a gap in the answer key. Treat the top of the board as a rough grouping rather than a ranking, and treat gaps of a few questions as noise.

Three failures on this board are our fault, not the models'

DeepSeek V4 Pro, DeepSeek V4 Flash and Kimi K2.6 all ended on runs of empty replies. That is a limitation of this harness and should not be read as any of them giving up. Every contestant reached over a plain chat API is sent the same 6,000-token output budget, and for a reasoning model that budget covers its thinking as well as its answer. Dozens of their turns came back having spent exactly 6,000 tokens — the cap — with nothing left for the question itself. They were thinking, not declining.

This is not a guess. We replayed the exact prompt DeepSeek was answering when it went blank, changing nothing but the budget:

max_tokens =  6,000  →  6,000 output tokens, empty reply
max_tokens = 32,000  →  6,394 output tokens, "Is it a Chicco product?"

It needed 394 tokens more than the cap we set. Same model, same prompt, same moment in the game — the only difference between a blank and a perfectly good question was our number.

Worse, that budget is not applied evenly. The Claude and OpenAI contestants run through their own command-line tools with no output cap set by this benchmark, so they were never exposed to the failure at all. The models most penalised are the ones that reason the most. DeepSeek V4 Pro had correctly narrowed to a battery-powered plastic toy instrument by question 100 and still scored nothing.

These runs were deliberately not re-run with a larger budget. That would have handed one model more room to think than everything already scored. Raising the limit for everyone and re-running from scratch is the right fix, and that is what the current leaderboard does.

Read the leaderboard accordingly: every model that failed this way should be treated as unscored, not as beaten. Three models across two providers hit it, which makes it a defect in the protocol rather than a property of any one of them.

Haiku has never once been beaten by this game

Claude Haiku 4.5 has attempted the benchmark four times and has never finished. It reached question 23, then 72, then 123, then 171, then 97 — and every single stop was a subscription running out of quota. Not one was the model failing at the puzzle.

The reason is that Haiku thinks far more per question than anything else here — roughly 4,900 tokens to produce a 42-character question, about 34× what Claude Opus 5 spends to write a shorter one. That is a setting we chose, not a flaw in the model: we ask every Claude contestant for a medium reasoning budget on all 200 questions. A small model compensates for capacity by thinking harder, so it pays that bill 200 times over.

We keep restarting it and it keeps running out of road. So the honest thing to report is not a Haiku score but a limitation of this benchmark: we have not yet built a harness that can measure Haiku to 200 questions on the subscription we run it on. Its transcripts are kept in full and it is never shown as a defeat, because it has not lost — it has never been allowed to finish.

Kimi is a documented non-result

Three Moonshot Kimi models were entered and none produced a score. They were attempted four times and failed four different ways: one was killed by the host outage described above, one hit the $8 per-run spend ceiling at question 51, one was stopped after five consecutive replies that were not questions, and one was halted deliberately while we checked whether the provider was re-charging for text it already had.

Two things went wrong, and only one of them is about Kimi. Kimi repeatedly spends its entire output budget on reasoning and returns nothing at all — eight of sixteen turns in one measurement produced an empty reply, which still costs a full 6,000 tokens. Separately, the provider's prompt cache proved unreliable for this workload: hit rates swung between 95% and zero on consecutive turns of an identical, strictly-growing prompt, while DeepSeek served the very same prompts from cache almost perfectly. That is a provider-side routing effect, not something the benchmark can fix.

Kimi consumed $14.36 — more than half of everything we spent — and produced no completed run. Rather than spend more, it is recorded as a non-result with the transcripts kept in full. Giving Kimi a larger output budget would probably rescue it, but that would hand it more room to think than every other contestant got, so it would have to be a separate, clearly-labelled experiment rather than a 2.0 run.

Contamination is expected

Publishing this degrades it. Once "SaxAbench" and "Sax-a-Boom" sit next to each other on the open web, models will start recognising the setup and guessing sooner. That's why every run is date-stamped: a score is only meaningful relative to how poisoned the well was on the day it was recorded.