How AI quiz generators work — and why most of them get it wrong
Asking a language model for questions produces confident wrong answers. Here is why, and what has to be added before the result survives a room full of people.
There are now a great many tools promising a quiz about any topic. Almost all of them work the same way: they send your topic to a language model and publish whatever comes back.
It works surprisingly often. And it fails in a very particular way, which happens to be the worst possible failure for a quiz.
The problem isn’t that the model doesn’t know
A language model doesn’t look anything up when it answers. It produces the most plausible continuation based on what it read during training. For solid general knowledge — the capital of Peru, how many players are on a football pitch — that is very reliable, because the information is everywhere in the data.
The problem shows up in three cases, and they are precisely the ones that make a quiz interesting:
- Anything recent. A model has a training cutoff. Something from three weeks ago does not exist for it — but it will answer anyway.
- Anything precise. An exact date, an exact score, the name of a third-billed actor. The model produces something with the right shape: a plausible date, a plausible name.
- Anything specialised. The narrower the subject, the less data there is, and the further the output drifts toward approximation.
In all three cases the model does not flag its uncertainty. It answers in exactly the same tone it used for the capital of Peru. That is the real defect: the error arrives with the confidence of a fact.
Why this matters more for a quiz than for prose
A mistake in a paragraph gets corrected. A mistake in a quiz gets discovered in public.
It always goes the same way: the question comes up, somebody gives the right answer, the app says they’re wrong, somebody reaches for their phone, and the game stops. You have not lost one question — you have lost trust in the other fifteen, and the evening.
A quiz is an object whose entire value rests on being right.
What has to be added: search before writing
The fix is less impressive than a bigger model. The system simply has to go and find the information at the moment you ask, instead of pulling it out of frozen memory.
In practice a chain that holds up looks like this:
- Plan the coverage before searching. A single generic search returns whatever is popular, which gives you three questions about the same angle. Deciding which angles to cover first produces a varied quiz instead of the same question three times.
- One search per angle. Each query hunts only its own zone.
- Write only from what came back. The model drafts from a dossier of sources, not from memory. If the dossier is empty, there is no question.
By this point most of the invention is gone. But not all of it.
The check that changes everything: solving it blind
The most useful step is also the least obvious.
Once the questions are written, they go to a second model — without the answer key. It is simply asked to answer them, and to justify each answer from the sources.
Three outcomes:
- It reaches the same answer and can prove it → the question ships.
- It reaches a different answer → either the question is wrong or it is ambiguous. Either way it should not be played.
- It cannot decide → the question is badly posed, or several answers are defensible. It goes.
That test catches something no proofread ever does: a question where two answers are both defensible. It is the most common cause of an argument around a table, and it looks nothing like an error when you read it with the answer key in front of you.
What it does not guarantee
Be honest about the limit. If the search fails — a provider outage, an exhausted quota — a serious system has a fallback, and that fallback is usually a model with no search at all. Which means the degraded mode is exactly the mode all this work exists to avoid.
That is why the honest phrasing is never “guaranteed correct”. It is researched and checked, which is a different promise, and one that can actually be kept.
It is also why an explanation that says where the information came from beats an explanation that merely asserts. The reader can check.
What to look for in a tool
- Does it search when you ask, or answer from memory?
- Does it tell you where the answer came from?
- Will it take a topic from yesterday, or does it fall apart?
- Does it promise perfection? (If so, it doesn’t know how this works.)
That is the principle behind EpiQuiz’s generator: search at the moment of the request, write only from the sources, have a second model solve every question blind, and drop anything that cannot be proved.