AQA GCSE English Literature Paper 1 (8702/1) · 34-mark question · benchmarked against an external examiner on 50 real student responses.
Key results:
- Mean absolute error of 2.3 marks after review (2.7 marks before review).
- 18% of responses (9 of 50) flagged as low confidence for teacher review.
- Every response within 6 marks of the examiner’s mark after review.
The question we tested
This is Section A: Macbeth of the AQA GCSE English Literature Paper 1 (8702/1), a real exam question worth 34 marks. It asks students to write a Shakespeare essay on an extract and the play as a whole (30 marks), plus 4 marks for written accuracy (AO4). A full Shakespeare essay is among the most interpretive, open-ended tasks at GCSE, and therefore among the hardest to mark consistently.
How Running Paper marks this question
Running Paper marks pen-and-paper assessments using two independent AI models. A primary model marks every response. A second, independent model marks it again as a validator, and the two never see each other’s marks. Where they disagree by more than a set tolerance, the response is flagged as low confidence and routed to the teacher to review, with the mark scheme and the response to hand. The teacher stays in control of every mark that matters.
For this question: the primary marker is gemini-rp-4 and the validator is gpt-rp-3. A response is flagged for review when the two models’ marks differ or by 6 marks.
You can read more about the process in how we measure accuracy.
How we measured accuracy
We took 50 real student responses and had them marked independently by an external examiner. These are the definitive benchmark marks. We then marked the same responses with Running Paper, with no access to the benchmark, and compared the two. The figures below use three measures:
- Mean absolute error: how far Running Paper’s marks are from the definitive benchmark marks on average.
- Percentage within tolerance: how often Running Paper’s mark for a response falls within a given number of marks of the definitive benchmark mark.
- Percentage of questions flagged: how often the primary and validating models disagree enough to trigger a low-confidence review.
Results
| Figure | Value |
|---|---|
| Mean absolute error (before review) | 2.7 marks (7.9%) |
| Mean absolute error (after review) | 2.3 marks (6.8%) |
| Questions flagged for review | 18% (9 of 50) |
Percentage within tolerance
How many responses Running Paper marked within each number of marks of the benchmark, before and after review:
| Within | Model only | After teacher review |
|---|---|---|
| ±1 mark | 14 of 50 (28%) | 20 of 50 (40%) |
| ±2 marks | 26 of 50 (52%) | 29 of 50 (58%) |
| ±3 marks | 33 of 50 (66%) | 35 of 50 (70%) |
| ±4 marks | 42 of 50 (84%) | 42 of 50 (84%) |
| ±5 marks | 47 of 50 (94%) | 47 of 50 (94%) |
| ±6 marks | 50 of 50 (100%) | 50 of 50 (100%) |
Reading these figures
On a 34-mark essay, a mean absolute error of 2.3 marks after review is 6.8% of the marks available. Nine responses were flagged for review, and most of the improvement from review came at the closest tolerances: responses within one mark rose from 28% to 40%. Every response was within six marks of the examiner.
If you would like to know how the system performs on your subject, get in touch via the contact form or book a demo below.
