AQA GCSE English Literature Paper 1 (8702/1) · 30-mark question · benchmarked against an external examiner on 50 real student responses.
Key results:
- Mean absolute error of 2.36 marks after review (2.54 marks before review).
- 8% of responses (4 of 50) flagged as low confidence for teacher review.
The question we tested
This is Section B: A Christmas Carol of the AQA GCSE English Literature Paper 1 (8702/1), a real exam question worth 30 marks. It asks students to write an essay on an extract and the novel as a whole. A full essay on a 19th-century novel is among the most interpretive, open-ended tasks at GCSE, and therefore among the hardest to mark consistently.
How Running Paper marks this question
Running Paper marks pen-and-paper assessments using two independent AI models. A primary model marks every response. A second, independent model marks it again as a validator, and the two never see each other’s marks. Where they disagree by more than a set tolerance, the response is flagged as low confidence and routed to the teacher to review, with the mark scheme and the response to hand. The teacher stays in control of every mark that matters.
For this question: the primary marker is gemini-rp-4 and the validator is gpt-rp-3. A response is flagged for review when the two models’ marks differ by 20% of the available marks.
You can read more about the process in how we measure accuracy.
How we measured accuracy
We took 50 real student responses and had them marked independently by an external examiner. These are the definitive benchmark marks. We then marked the same responses with Running Paper, with no access to the benchmark, and compared the two. The figures below use three measures:
- Mean absolute error: how far Running Paper’s marks are from the definitive benchmark marks on average.
- Percentage within tolerance: how often Running Paper’s mark for a response falls within a given number of marks of the definitive benchmark mark.
- Percentage of questions flagged: how often the primary and validating models disagree enough to trigger a low-confidence review.
Results
| Figure | Value |
|---|---|
| Mean absolute error (before review) | 2.54 marks (8.5%) |
| Mean absolute error (after review) | 2.36 marks (7.9%) |
| Questions flagged for review | 8% (4 of 50) |
Percentage within tolerance
How many responses Running Paper marked within each number of marks of the benchmark, before and after review:
| Within | Model only | After teacher review |
|---|---|---|
| ±1 mark | 20 of 50 (40%) | 23 of 50 (46%) |
| ±2 marks | 29 of 50 (58%) | 30 of 50 (60%) |
| ±3 marks | 36 of 50 (72%) | 37 of 50 (74%) |
| ±4 marks | 40 of 50 (80%) | 40 of 50 (80%) |
| ±5 marks | 44 of 50 (88%) | 44 of 50 (88%) |
| ±6 marks | 48 of 50 (96%) | 48 of 50 (96%) |
Reading these figures
The two models agreed on most responses here, so only four were flagged and review made a modest difference. The mean absolute error after review was 2.36 marks, 7.9% of the marks available.
If you would like to know how the system performs on your subject, get in touch via the contact form or book a demo below.
