This case study measures how closely Running Paper’s AI marking matches examiners on a Year 9 Biology paper, and how much that accuracy improves when a second, independent AI model checks the first and flags uncertain marks for a teacher to review.
Key results:
| Measure | Result |
|---|---|
| Mean absolute error: primary model, before review | 0.44 marks (of 50) |
| Mean absolute error: after low-confidence review | 0.24 marks (of 50) |
| Marking decisions flagged for teacher review | 134 of 1,300 (10.3%) |
| Student scripts marked | 50 |
| Sub-questions per script | 26 |
The models we used
| Primary model | gemini-f-rp-3 |
| Validating model | gpt-rp-nova |
The primary model marks every question. The validating model marks the same question again, independently, without seeing the primary model’s mark. The two models come from different providers. Models from the same provider tend to make similar mistakes and would agree even when both are wrong (correlated error), so a validator from a different provider gives a more reliable signal that a mark needs a second look.
The assessment and question types
The assessment was a 50-mark Year 9 Biology paper with 26 separately marked sub-questions. It covers cells and organ systems, digestion and nutrition, ecology and inheritance, and photosynthesis and gas exchange.
| Type | What it is | Sub-questions | Marks |
|---|---|---|---|
| MCQ | Tick-one-box multiple choice | 7 | 7 |
| Short recall | Short factual recall, data reading and a calculation | 7 | 10 |
| Structured | Line-matching and a word equation | 2 | 4 |
| Extended | Describe, explain and method questions: free-text answers | 10 | 29 |
How we did it
Fifty scripts were included. To keep the sample representative rather than selective, students were drawn from eight classes spanning the ability range and selected at random within each class. Running Paper’s marks were compared against moderated examiner marks for the same scripts.
Every question was marked independently by both models. A question was flagged as low confidence in one of two ways:
- Disagreement: the two models differ by more than the tolerance set for that question, broadly 20% of the marks available. In practice that means any disagreement on a one- to five-mark question, and a difference of more than one mark on the six-mark question, the only sub-question worth six marks or more.
- By design: the system can send certain question types for automatic review whatever the two models decide. On this paper we flagged one sub-question this way, a line-matching question, because AI models are known to struggle with that format.
In both cases the flagged question is routed to the teacher, who reviews the answer, the relevant part of the mark scheme and the proposed mark.
Results
On its own, the primary model marked to within 0.44 marks of the examiner on average, out of 50. 94% of scripts were within one mark of the examiner total and every script was within two marks.
Adding the validator and low-confidence review reduced the average error to 0.24 marks, a 45% reduction, by flagging 134 of the 1,300 marking decisions (10.3%) for a teacher to check. The remaining 0.24 marks is error that survived both the cross-model comparison and teacher review.
Around one marking decision in ten was flagged, so teachers review the genuinely uncertain answers rather than re-marking everything.
Limitations
- The figures are specific to this Year 9 Biology paper and cohort.
- The post-review error assumes each flagged question is corrected to the definitive mark. It reflects the accuracy the design is built to reach once teacher review has happened, not a claim that review is automatic.
You can read more about the process in how we measure accuracy. If you would like to know how the system performs on your subject, get in touch via the contact form or book a demo below.
