Writing exam questions: multiple choice and open-ended
Short answer
Good exam questions measure exactly the learning objective at its stated level, are unambiguous and cannot be solved by guessing or linguistic clues. Multiple-choice questions need a clear stem and plausible wrong answers; open questions need a marking rubric with fixed criteria, and AI can supply drafts that subject experts check and take responsibility for.
Key points
- Every question belongs to a learning objective and tests it at its level.
- Multiple choice: a clear stem, one clearly correct answer, plausible distractors.
- Open questions: set a marking rubric with criteria and points before marking.
- Avoid linguistic clues such as the longest answer or absolute words.
- Analyse and revise questions after the first runs.
What makes a good exam question
A good exam question is valid, reliable and fair. Valid means it measures what it is supposed to measure, namely the learning objective. Reliable means two markers would reach the same result and chance plays only a small role. Fair means anyone who has reached the objective can answer the question, regardless of linguistic subtleties or tricks.
These three properties sound obvious but are often missed. Questions then test whether someone recognises a phrase rather than whether they have understood the matter, or they can be solved by clever elimination without knowing the material.
Writing multiple-choice questions properly
Multiple-choice questions consist of a stem, the correct answer and wrong answers called distractors. They are quick to mark and suit remembering and understanding well; with well-built case examples they also work for applying and analysing.
- Write the stem so that it makes sense without the answers
- Provide exactly one clearly correct answer, unless it is explicitly a multiple-response item
- Build distractors from typical misconceptions, not from nonsense
- Make all answers similar in length and grammatical structure
- Avoid absolute words such as always and never; they give wrong answers away
- Avoid double negatives
- Use “all of the above” sparingly
- For higher levels, start with a short case example
Open questions and marking rubrics
Open questions require learners to produce their own answer: an explanation, a justification, a solution. They are particularly suited to higher levels such as evaluating and creating but take more effort to mark. For marking to be reliable, every open question needs a marking rubric that is fixed before marking begins.
A rubric names the criteria used for marking and describes, for each criterion, what full, partial or missing fulfilment looks like. Marking is done per criterion with a justification, not by overall impression. That makes results comparable and transparent for learners.
| Criterion | 0 points | 1 point | 2 points |
|---|---|---|---|
| Issue identified | Core problem not named | Partly named | Named fully and precisely |
| Rule applied | No rule or wrong rule | Correct rule, incompletely applied | Correct rule, correctly applied |
| Justification | No justification | Justification has gaps | Coherent justification related to the case |
AI as an assistant for writing and marking
Language models can draft many questions from course content in a short time, including distractors and explanations. That saves time, especially for large question pools. The drafts do, however, need subject review: Is the correct answer really correct? Are the distractors clearly wrong but plausible? Does the question test the objective at the right level?
When marking open answers, AI can make a suggestion that follows the rubric. The decision must rest with a person. The EU AI Act classifies systems that evaluate learning outcomes as high-risk systems; among other things, this requires human oversight.
Analysing questions after use
After the first runs it pays to look at results per question. A question that almost everyone answers correctly may be too easy or give the answer away. A question that almost everyone gets wrong may be misleading, or the material was not taught well enough in the course. Especially revealing are questions where strong learners do worse than weaker ones; something is usually wrong with the question.
Also check which distractors are chosen. A distractor nobody picks does not do its job. A distractor many pick reveals a widespread misconception that the course should address better.
How Universe does it
In Universe, lecturers create exams as multiple-choice exams, written exams with free text or term paper topics. Under “Generate with AI”, questions are created from a topic; the suggestions are appended and nothing is overwritten. For free-text questions you store a marking rubric that guides the AI pre-marking; a person confirms the grade. Question pools can be imported in QTI format.
Terms
- Distractor
- A wrong but plausible answer option in a multiple-choice question.
- Marking rubric
- Defined criteria and levels for marking open answers.
- Question pool
- A collection of equivalent questions from which exams are assembled at random.
- QTI
- An open standard for exchanging exam questions between systems.
Frequently asked questions
How many answer options should a multiple-choice question have?
Three to four is common. More important than the number is that every distractor is plausible. Three good options are better than five, two of which are obviously wrong.
How do I prevent guessing in multiple choice?
With plausible distractors, enough questions and a pass mark well above the probability of guessing. Case-based questions make guessing even harder.
May AI mark exams?
AI can propose a mark based on a rubric. The decision should rest with a person, especially when the result affects a qualification or career. The AI Act sets high requirements for this.
What is a question pool?
A collection of equivalent questions for one learning objective from which an exam draws at random. Pools make sharing answers harder and allow resits with different questions.
How do I set a pass mark?
Base it on what is needed in practice. For safety-critical content a high threshold can be appropriate, for basic knowledge a moderate one. Set the threshold before the first exam and justify it.
How it works in Universe
Related articles
- Writing learning objectives with Bloom’s taxonomyHow to write testable learning objectives: the six levels of the taxonomy, suitable verbs, examples, common mistakes and the link to assessment.
- Interactive e-learning elements that actually workWhich interactions support learning: questions, scenarios, role-plays, explaining back and flashcards — and which merely keep people busy.
- LMS for universities: from teaching to transcriptsWhat a learning platform must do at a university: course rooms, exams, credits, sign-in via the institution, accessibility and verifiable records.
- Measuring learning progress: using learning analytics wellWhich e-learning metrics really mean something, how to measure knowledge rather than clicks, and where learning analytics meets its limits and data protection.
- Verifiable Credentials: digital records under the W3C standardHow Verifiable Credentials under the W3C standard work: issuer, holder, verifier, signature, revocation, data protection and use for degrees and certificates.