How many questions
does an AI need to guess a Sax-a-Boom?

Every model gets the identical prompt

One frozen opening prompt, byte-identical for every contestant — no hints, no category, no per-model framing. Anything that would advantage one model over another is a bug in the benchmark, not a feature.

The host never improvises

Answers come from a committed fact sheet plus a table of rulings on the attributes where a reasonable host could honestly answer either way. Every answer is exactly yes, no, or you win — and the host is never the one who decides the last of those: a win is detected by the harness from the name itself, so a contestant cannot talk its way into one by sounding confident. A question that can't be answered yes or no — or that asks two things at once — gets "Yes or no questions only", which carries no information and doesn't count against the score.

A guess is just a question

"Is it a kazoo?" costs exactly one question, like anything else. The run ends only when the model names the product. Describing it correctly as "a toy saxophone" is answered yes, but it isn't a win.

Read the order with some suspicion

The host has one more answer than yes and no: it can say I don't know when the fact sheet genuinely doesn't settle a question. That is honest, and it replaced something worse — previously such questions were rejected as though they were badly formed, so models rephrased them over and over.

But it costs the model a question either way, and models did not draw them evenly. The two that never finished collected them at 14% of their turns; the joint-fastest drew none at all. One model was asked, on its fourth question, whether the thing is used indoors — a broad and useful question — and got I don't know. It went on to need 134 questions, having needed 26 under the previous rules.

So part of a score is luck about which questions happened to fall in a gap in the answer key. Treat the top of the board as a rough grouping rather than a ranking, and treat gaps of a few questions as noise.

Three failures on this board are our fault, not the models'

DeepSeek V4 Pro, DeepSeek V4 Flash and Kimi K2.6 all ended on runs of empty replies. That is a limitation of this harness and should not be read as any of them giving up. Every contestant reached over a plain chat API is sent the same 6,000-token output budget, and for a reasoning model that budget covers its thinking as well as its answer. Dozens of their turns came back having spent exactly 6,000 tokens — the cap — with nothing left for the question itself. They were thinking, not declining.

This is not a guess. We replayed the exact prompt DeepSeek was answering when it went blank, changing nothing but the budget:

max_tokens =  6,000  →  6,000 output tokens, empty reply
max_tokens = 32,000  →  6,394 output tokens, "Is it a Chicco product?"

It needed 394 tokens more than the cap we set. Same model, same prompt, same moment in the game — the only difference between a blank and a perfectly good question was our number.

Worse, that budget is not applied evenly. The Claude and OpenAI contestants run through their own command-line tools with no output cap set by this benchmark, so they were never exposed to the failure at all. The models most penalised are the ones that reason the most. DeepSeek V4 Pro had correctly narrowed to a battery-powered plastic toy instrument by question 100 and still scored nothing.

These runs were deliberately not re-run with a larger budget. That would have handed one model more room to think than everything already scored. Raising the limit for everyone and re-running from scratch is the right fix, and that is what the current leaderboard does.

Read the leaderboard accordingly: every model that failed this way should be treated as unscored, not as beaten. Three models across two providers hit it, which makes it a defect in the protocol rather than a property of any one of them.

Haiku took six sittings to answer 200 questions

Claude Haiku 4.5 did finish, and it did not find the answer. Getting that result took six attempts across six separate five-hour windows, stopping at questions 23, 72, 97, 123, 162 and finally 200 — every stop before the last one a subscription running out of credit, never the model giving up.

The reason is that it thinks enormously per question. It produced 738,351 words' worth of reasoning across the run, about 3,700 tokens for each question asked — while writing questions among the shortest of any model here. Grok 4.5 solved the whole puzzle spending roughly ten tokens a turn. A small model compensates for size by thinking harder, and it pays that bill two hundred times.

So this is a real did-not-solve rather than a missing result, but it is worth knowing what it cost to obtain: a full day of elapsed time, and more computation than every other model on this board combined.

Kimi is a documented non-result

Three Moonshot Kimi models were entered and none produced a score. They were attempted four times and failed four different ways: one was killed by the host outage described above, one hit the $8 per-run spend ceiling at question 51, one was stopped after five consecutive replies that were not questions, and one was halted deliberately while we checked whether the provider was re-charging for text it already had.

Two things went wrong, and only one of them is about Kimi. Kimi repeatedly spends its entire output budget on reasoning and returns nothing at all — eight of sixteen turns in one measurement produced an empty reply, which still costs a full 6,000 tokens. Separately, the provider's prompt cache proved unreliable for this workload: hit rates swung between 95% and zero on consecutive turns of an identical, strictly-growing prompt, while DeepSeek served the very same prompts from cache almost perfectly. That is a provider-side routing effect, not something the benchmark can fix.

Kimi consumed $14.36 — more than half of everything we spent — and produced no completed run. Rather than spend more, it is recorded as a non-result with the transcripts kept in full. Giving Kimi a larger output budget would probably rescue it, but that would hand it more room to think than every other contestant got, so it would have to be a separate, clearly-labelled experiment rather than a 2.0 run.

Contamination is expected

Publishing this degrades it. Once "SaxAbench" and "Sax-a-Boom" sit next to each other on the open web, models will start recognising the setup and guessing sooner. That's why every run is date-stamped: a score is only meaningful relative to how poisoned the well was on the day it was recorded.