Evaluate an AI marking tool's accuracy by testing it against work you have already marked, not by trusting a vendor's claims. Assemble a benchmark of graded scripts, compare its marks and feedback to yours, measure how often and in which direction it disagrees, then stress-test edge cases before any rollout.
What does accuracy actually mean for an AI marking tool?
Accuracy for an AI marking tool is not a single score but a bundle of qualities: how closely its marks agree with a skilled teacher’s, how consistently it marks the same work, how faithfully it applies your rubric, and how sensibly it handles unusual responses. A tool can produce a plausible mark and still be wrong for the wrong reasons, so a headline accuracy figure tells you very little on its own.
The distinction that matters most is between the mark and the reasoning behind it. Two tools might award the same grade, yet one cites the specific rubric criteria a student met while the other simply guesses. For school-based assessment, where NESA, VCAA and QCAA expect defensible, criteria-referenced judgements, that reasoning is not optional; it is what makes a mark usable.
Treat accuracy as a claim to be tested rather than a feature to be trusted. Vendor benchmarks are run on their data, not your students, your subject or your rubric. The only evaluation that counts is one you run on work you know well, marked by teachers whose judgement you already trust.
How do you build a fair benchmark to test it against?
Build your benchmark from student work you have already marked, chosen to represent the full spread of your cohort rather than a tidy sample. Pull a set of scripts for one common task, ideally 20 to 40, that spans the whole range of grade bands from the strongest responses to the weakest, and deliberately include the borderline cases where marking judgement is hardest.
Record the original human mark and the reasoning for each script before the tool sees anything, and keep that key separate. If a script was contentious or double-marked, note that too; those are the ones that will reveal the most. A benchmark of only clean, mid-range answers will flatter any tool and tell you nothing about where it breaks.
Run the tool blind against this set, using your actual rubric and success criteria rather than a generic one. Then compare its marks and its feedback to the human record you set aside. The gap between the two, script by script, is your real accuracy signal.
- Scripts from every grade band, not just the middle.
- At least a few borderline and double-marked responses.
- A range of question types, including extended writing.
- The original teacher mark and rationale, stored separately.
Which accuracy measures should you actually track?
Track three things: how often the tool’s mark matches the human mark exactly, how often it lands within one grade band, and the direction of any disagreement. Within-one-band agreement usually matters more than exact matches, because experienced human markers themselves disagree at the margins; a tool that is consistently one band out in a predictable direction is easier to trust than one that is erratic.
Direction is the signal people forget. A tool that leans generous and one that leans harsh are different problems with different classroom consequences, and an average that cancels them out hides both. Tabulate whether disagreements cluster above or below the human mark, and whether they concentrate in particular grade bands or question types.
Then test consistency directly by marking the same handful of scripts twice. If the tool gives materially different marks to identical work on separate runs, its accuracy is unreliable regardless of how well it scored on the first pass. Reliability and accuracy are separate properties, and a rollout needs both.
How do you stress-test the tool beyond typical scripts?
Stress-test by feeding the tool the responses that break the rules, because that is where marking tools fail quietly. Typical mid-range answers are the easy case; the real test is an off-topic response that is well written, a very short answer that still hits the criteria, an unusually structured argument, or a script that misreads the question. A robust tool marks these sensibly or flags uncertainty; a fragile one confidently invents a grade.
Pay particular attention to fabrication. Ask whether the feedback references things the student actually wrote, or whether it praises evidence that is not on the page. Feedback that hallucinates quotations or strengths is worse than a wrong number, because a teacher skimming it may pass those errors straight on to a student.
- Off-topic but fluent writing that should not score well.
- Very short or very long responses against the same criteria.
- Valid but unconventional structures or arguments.
- Answers with factual errors the tool must not reward.
- Blank or near-blank submissions.
How do you check the tool is fair, not just accurate?
Check fairness by looking at whether accuracy holds up evenly across different groups of students, not just on average. A tool can post strong overall agreement while systematically misjudging particular cohorts, such as students writing in English as an additional language, students with atypical handwriting if you upload scans, or those whose responses are short but correct.
Slice your benchmark results by these groups where you can, and look for patterns in who the disagreements fall on. If the tool is reliably harsher on one cohort, that is an equity problem the headline number will hide. Australian assessment expects comparable, defensible judgements for every student, and a marking aid has to meet the same bar.
Where the sample is too small to draw firm conclusions, treat it as a question to keep monitoring after rollout rather than a box ticked. Fairness is not a one-off test; it is something to watch as the tool meets a wider range of real student work.
What should the rollout look like once the tool passes?
Once a tool passes your benchmark, roll it out in stages with a teacher reviewing every mark rather than switching it on across the school at once. Start with one faculty or one assessment, keep double-marking a sample against the tool, and treat the first term as an extended trial that confirms the accuracy you measured holds up on live work.
Keep the human firmly in the loop by design. The goal is not to remove teacher judgement but to speed up the repetitive parts of applying a rubric while a teacher checks, edits and owns the final mark. Build in a simple way for teachers to flag disagreements, so your accuracy picture keeps improving after launch instead of freezing at the pilot.
How can a tool like JeddAI keep teachers in control of accuracy?
A tool like JeddAI supports accuracy by marking against your own rubric, success criteria and comment banks, then leaving the final judgement to you. Teachers connect or upload student work, JeddAI drafts feedback and marking aligned to those criteria, and the teacher reviews and edits every comment before it reaches a student.
That design is what makes the evaluation approach above practical. Because the marking is criteria-referenced and the teacher stays in control, you can benchmark it against your own graded scripts, see exactly where and why it agrees or disagrees, and keep watching accuracy as it meets more of your students’ work. It saves marking time without asking you to hand over the judgement.
If you want to test that on your own rubrics and student work, Get started with JeddAI and run it against a set of scripts you have already marked.
| Accuracy dimension | Weak signal | Strong signal |
|---|---|---|
| Agreement with teacher marks | Only a headline accuracy percentage | Within-one-band agreement on your own scripts |
| Consistency | Different marks for the same script on re-runs | Stable marks when the same work is marked twice |
| Rubric fidelity | Generic praise with no criteria named | Feedback cites the specific criteria met |
| Edge-case behaviour | Confident grade for off-topic or blank work | Sensible marks or a flag for uncertainty |
| Fairness | Strong average, unchecked across cohorts | Accuracy holds across student groups |
| Transparency | A score only, with no reasoning shown | A mark and rationale a teacher can check |
Frequently asked questions
How many marked scripts do I need for a reliable trial?
There is no magic number, but 20 to 40 scripts spanning every grade band gives a workable first read. Range matters more than volume: a spread that includes borderline and atypical work tells you more than a large pile of mid-range answers.
Should I expect an AI tool to match teacher marks exactly?
No. Experienced markers disagree at the margins themselves, so within-one-band agreement is a fairer target than exact matches, provided any disagreement is small and predictable rather than erratic.
What accuracy level is good enough to roll out?
It depends on the stakes of the assessment and how closely teachers review each mark. For a low-stakes formative task with a teacher checking the output, a modest gap is acceptable; for reporting grades, the bar is much higher.
Can I trust a vendor's published accuracy figures?
Treat them as a starting point, not proof. Those figures come from the vendor's data and rubrics, not your subject, cohort or criteria, so the only figure that counts is one you generate on your own marked work.
Does using an AI marking tool remove my responsibility for the mark?
No. A marking aid drafts and speeds up the work, but the teacher reviews, edits and owns the final judgement, which is exactly what school assessment authorities expect.
Get started with Jeddle
Jeddle gives teachers and students instant, syllabus-aligned feedback powered by JeddAI.
Looking for study material? Browse Jeddle's Australian-English subject resources, or explore more articles on EdTech & Tools.
