AQA GCSE English Language Paper 1 (8700/1) · 20-mark question · benchmarked against an external examiner on 50 real student responses.
Key results:
- Mean absolute error of 1.36 marks after review (1.44 marks before review).
- 6% of responses (3 of 50) flagged as low confidence for teacher review.
- Every response within 5 marks of the examiner’s mark after review.
The question we tested
This is Question 4 of the AQA GCSE English Language Paper 1 (8700/1), a real exam question worth 20 marks. It asks students to critically evaluate a statement about the text, supported with references (AO4). This is one of the more open-ended reading tasks on the paper, and the kind of extended judgement markers find hardest to align on.
How Running Paper marks this question
Running Paper marks pen-and-paper assessments using two independent AI models. A primary model marks every response. A second, independent model marks it again as a validator, and the two never see each other’s marks. Where they disagree by more than a set tolerance, the response is flagged as low confidence and routed to the teacher to review, with the mark scheme and the response to hand. The teacher stays in control of every mark that matters.
For this question: the primary marker is gemini-rp-4 and the validator is gpt-rp-3. A response is flagged for review when the two models’ marks differ by 20% of the available marks or by 6 marks, whichever is smaller.
You can read more about the process in how we measure accuracy.
How we measured accuracy
We took 50 real student responses and had them marked independently by an external examiner. These are the definitive benchmark marks. We then marked the same responses with Running Paper, with no access to the benchmark, and compared the two. The figures below use three measures:
- Mean absolute error: how far Running Paper’s marks are from the definitive benchmark marks on average.
- Percentage within tolerance: how often Running Paper’s mark for a response falls within a given number of marks of the definitive benchmark mark.
- Percentage of questions flagged: how often the primary and validating models disagree enough to trigger a low-confidence review.
Results
| Figure | Value |
|---|---|
| Mean absolute error (before review) | 1.44 marks (7.2%) |
| Mean absolute error (after review) | 1.36 marks (6.8%) |
| Questions flagged for review | 6% (3 of 50) |
Percentage within tolerance
How many responses Running Paper marked within each number of marks of the benchmark, before and after review:
| Within | Model only | After teacher review |
|---|---|---|
| ±1 mark | 29 of 50 (58%) | 30 of 50 (60%) |
| ±2 marks | 40 of 50 (80%) | 41 of 50 (82%) |
| ±3 marks | 46 of 50 (92%) | 47 of 50 (94%) |
| ±4 marks | 49 of 50 (98%) | 49 of 50 (98%) |
| ±5 marks | 50 of 50 (100%) | 50 of 50 (100%) |
Reading these figures
On a 20-mark question, a mean absolute error of 1.36 marks after review is 6.8% of the marks available. The two models agreed closely on this question, so only three responses were flagged, and review made a small difference. 94% of responses were within three marks of the examiner and every response was within five.
If you would like to know how the system performs on your subject, get in touch via the contact form or book a demo below.
