Every model plays 20 Questions — except there is no limit of twenty. The answer is always the same: a Sax-a-Boom, the plastic toy saxophone that plays canned riffs at the push of a button, best known because Jack Black plays one with total seriousness.
Questions asked before the model names the product. Fewer is better.
Answers yes, no, or you win, from a frozen fact sheet. Never a hint.
200 questions, then it's a did-not-solve. Purely a cost control.
Fewest questions wins. Click any run to read the whole transcript.
The bits worth reading — near misses, odd theories, unusually good questions.
One frozen opening prompt, byte-identical for every contestant — no hints, no category, no per-model framing. Anything that would advantage one model over another is a bug in the benchmark, not a feature.
Answers come from a committed fact sheet plus a table of rulings on the attributes where a reasonable host could honestly answer either way. Every answer is exactly yes, no, or you win — and the host is never the one who decides the last of those: a win is detected by the harness from the name itself, so a contestant cannot talk its way into one by sounding confident. A question that can't be answered yes or no — or that asks two things at once — gets "Yes or no questions only", which carries no information and doesn't count against the score.
"Is it a kazoo?" costs exactly one question, like anything else. The run ends only when the model names the product. Describing it correctly as "a toy saxophone" is answered yes, but it isn't a win.
Under protocol 1.0 the host had only two answers. That turned out to be a flaw in the benchmark rather than a fact about the models. Sonnet 5 asked "Is it a toy saxophone?", was told yes, and had no way to tell "yes, that is a true description" from "yes, that is the answer". It asked the same question four more times and burned sixty turns finding out the game had not ended.
Protocol 2.0 gives the host a third response, you win, and the opening prompt now says so — including, in as many words, that a bare yes is not a win and that the model should keep narrowing. The cap also rose from 100 questions to 200. Both changes make 2.0 scores incomparable with 1.0 scores, so the two are ranked in separate tables and the 1.0 results are kept only as history.
The Claude contestants are reached through claude -p with tools off,
and the xAI, DeepSeek and Moonshot contestants through a plain chat completion —
in every case the model sees the opening prompt and the transcript, and nothing
else. The OpenAI models are reached through the Codex CLI, which is an agent and
prepends several thousand tokens of its own system prompt to every call. Those runs
are marked agentic CLI and should be read with that
asymmetry in mind. It is a real difference and pretending otherwise would be worse
than disclosing it.
DeepSeek V4 Pro, DeepSeek V4 Flash and Kimi K2.6 all ended on runs of empty replies. That is a limitation of this harness and should not be read as any of them giving up. Every contestant reached over a plain chat API is sent the same 6,000-token output budget, and for a reasoning model that budget covers its thinking as well as its answer. Dozens of their turns came back having spent exactly 6,000 tokens — the cap — with nothing left for the question itself. They were thinking, not declining.
This is not a guess. We replayed the exact prompt DeepSeek was answering when it went blank, changing nothing but the budget:
max_tokens = 6,000 → 6,000 output tokens, empty reply max_tokens = 32,000 → 6,394 output tokens, "Is it a Chicco product?"
It needed 394 tokens more than the cap we set. Same model, same prompt, same moment in the game — the only difference between a blank and a perfectly good question was our number.
Worse, that budget is not applied evenly. The Claude and OpenAI contestants run through their own command-line tools with no output cap set by this benchmark, so they were never exposed to the failure at all. The models most penalised are the ones that reason the most. DeepSeek V4 Pro had correctly narrowed to a battery-powered plastic toy instrument by question 100 and still scored nothing.
These runs were deliberately not re-run with a larger budget. That would have handed one contestant more room to think than every model already scored. Raising the cap for everyone and re-running the whole field is a protocol 2.1 change, and it is the right fix — it is not something to patch mid-field.
Read the leaderboard accordingly: every model that failed this way should be treated as unscored, not as beaten. Three models across two providers hit it, which makes it a defect in the protocol rather than a property of any one of them.
Three Moonshot Kimi models were entered and none produced a score. They were attempted four times and failed four different ways: one was killed by the host outage described above, one hit the $8 per-run spend ceiling at question 51, one was stopped after five consecutive replies that were not questions, and one was halted deliberately while we checked whether the provider was re-charging for text it already had.
Two things went wrong, and only one of them is about Kimi. Kimi repeatedly spends its entire output budget on reasoning and returns nothing at all — eight of sixteen turns in one measurement produced an empty reply, which still costs a full 6,000 tokens. Separately, the provider's prompt cache proved unreliable for this workload: hit rates swung between 95% and zero on consecutive turns of an identical, strictly-growing prompt, while DeepSeek served the very same prompts from cache almost perfectly. That is a provider-side routing effect, not something the benchmark can fix.
Kimi consumed $14.36 — more than half the entire field budget — and produced no completed run. Rather than spend more, it is recorded as a non-result with the transcripts kept in full. Giving Kimi a larger output budget would probably rescue it, but that would hand it more room to think than every other contestant got, so it would have to be a separate, clearly-labelled experiment rather than a 2.0 run.
Publishing this degrades it. Once "SaxAbench" and "Sax-a-Boom" sit next to each other on the open web, models will start recognising the setup and guessing sooner. That's why every run is date-stamped: a score is only meaningful relative to how poisoned the well was on the day it was recorded.