Do automated grading systems improve learning? Six decades of evidence

Andrei Niasiuk · August 11, 2026 · Facts checked: August 12, 2026

This review of roughly sixty studies across five language zones finds that automated grading of programming assignments has become a fast, reliable way to check code, but that in more than six decades none of the studies surveyed here has shown that it makes students learn to program any better. The field has been asking exactly that question since 1960, when Communications of the ACM published a two-page note by J. Hollingsworth describing a grader built for a programming class at Rensselaer Polytechnic Institute, running on an IBM 650: it compiled student solutions, ran them, and reported a result, the first documented automatic grader for programming assignments. Today it runs in classrooms on five continents, and the literature about it is spread across at least five language zones. The studies below, cited at first mention, trace that one question through all of them.

Hollingsworth already asked that question, in the very paper that started the field. Describing his grader’s first fifteen months of use, compared against the lab-group method it replaced, he wrote:

“After fifteen months, our experience leads us to believe that students learn programming not only as well but probably better than they did under the method we did use—laboratory groups of four or five students.” (Hollingsworth, 1960)

The quote comes from CrossRef’s bibliographic metadata, not a scan of the original 1960 pages, which sit behind ACM’s paywall.

That is a claim about learning, not about time saved, and it sits at the very start of the literature: no later marketing department invented it. Everything that follows circles back to it and never settles it.

Why the demand never went away

The reason automated grading persisted and grew has less to do with pedagogy than with arithmetic.

Where faculty ratios break down

A 2025 survey of an introductory programming course at IIT Patna reports that the course was “coordinated by three faculty members, resulting in a faculty-to-student ratio of 1:172, with additional support from TAs at a ratio of about 1:20” (Nareti et al., 2025). No amount of individual dedication closes a ratio like that by hand.

Building infrastructure to match

A 2014 account by researchers from Tsinghua University, Peking University, and the University of Illinois at Urbana-Champaign traces the online judge systems China built to absorb that same pressure: Zhejiang University’s ZOJ launched in 2003; Peking University’s POJ, launched in 2004, had “long led the country” in submissions, with “more than 8 million programs” submitted by 2014; Hangzhou Dianzi University’s HDOJ “first broke through the 10 million mark” in submissions in January 2014 alone (Liu, Wang & Xie, 2014; original Chinese: “北京大学的POJ…于2004年建立，程序提交量长期在国内领先，迄今已经提交了800多万个程序，…另外，杭州电子科技大学的HDOJ…程序提交数目于2014年1月率先冲破1000万大关”).

Reaching students without computers

AutoGrad, built for the SuaCode programming course, runs on phones rather than computers. Its authors describe it as “the first system built and tested in an African context across over thirty-five countries across the continent,” deployed to grade “1000+ students across Africa… with over 3,000 code files graded” (Annor et al., 2021). The reason, as the same paper notes, is that computers “tend to be in schools and not in homes” across much of the continent, while smartphone ownership keeps expanding.

A measured case: Japan

At Nihon University, once enrollment passed 100 students, a single instructor grading fairly could no longer keep pace within class hours. After deploying an automatic report-grading support system for a C-programming course, the reported result was stark: “this system needs only one instructor, and the time required for grading was reduced to 1/4 of what it was before… previously, one instructor and three teaching assistants were needed for daily emailed reports, but with this system one instructor suffices… manual grading achieved 0.5 reports per minute; with this system, 2 reports per minute” (Watanabe, 2009; original Japanese, first clause from the paper’s introduction and second clause from its conclusion: “本システムによって教員１名ですみ，評価に要する時間が従来の1/4にすることができた…手作業による採点では0.5通/分に対して，本システムでは２通/分であり”).

None of this is disputed anywhere in the literature surveyed here: automated grading reliably converts an otherwise unworkable teacher-to-student ratio into a workable one, and that is the well-documented case for it.

What automation reliably buys, and what it costs

Automating the mechanics of checking a program is a solved engineering problem, and its limits are documented in the open.

The theoretical limit

Rice’s theorem rules out exact automatic correction from the outset, and the researchers who build these systems say so themselves. INGInious, a grading platform built at UCLouvain for a MOOC on programming paradigms, states it plainly in its own experience report: “Computer scientists know that automatic correction is not an easy task, because of theoretical limits: it is actually impossible to check whether the student’s code does the same as a correct one (consequence of Rice’s Theorem, see [Beckman, 1980])” (Derval, Gego, Reinbold, Frantzen & Van Roy, 2015). Automated grading substitutes some feasible proxy, usually a battery of test cases, for exhaustive correctness, and that proxy is provably incomplete.

How large the gap is in practice

GATE, a system built at TU Clausthal and Humboldt University Berlin, was checked against a real 2009 introductory programming course. Its authors reported: “Of 1031 solutions, functional tests classified 759 solutions (73.6%) as incorrect, but tutors nevertheless rated 201 of these (26.5%) as correct, since only minor errors were present (e.g. typos, incorrectly set package)” (Müller & Strickroth, 2013; original German: “Von 1031 Lösungen wurden durch die Funktionstests 759 Lösungen (73,6%) als nicht korrekt klassifiziert aber 201 Lösungen (26,5%) hiervon von den Tutoren trotzdem als korrekt bewertet, da nur kleinere Fehler vorhanden waren”). More than a quarter of the solutions the automated tests rejected were, in a human tutor’s judgment, actually fine. That figure comes from the system’s own developers, published alongside its genuine and statistically significant benefits on the same course: a 25-percentage-point jump in syntactically correct submissions among students who used the automated syntax check. An independent German system, JACK, names a comparable number in its own 2008 report: “Up to 28% of the results needed manual correction by the teacher, in most cases because of false negatives” (Goedicke, Striewe & Balz, 2008).

When students game the proxy

Students work out what a metric rewards and optimize for that instead of the skill it was meant to stand in for. Petri Ihantola’s 2011 doctoral dissertation at Aalto University reports a direct study of this: “We have analyzed how students behave when they are rewarded for structural test coverage (e.g. line coverage) and found that this can lead students to write tests with good coverage but with poor ability to detect faulty programs” (Ihantola, 2011). Optimizing the metric and improving the underlying skill turned out, in that data, to be two different activities.

The field publishes these numbers about itself; it was GATE’s own authors who reported the false-negative rate against their own system. But nobody in this group of papers claims that the imperfect proxy, deployed at scale, teaches programming better than a human grader would have. That claim belongs to a separate, thinner body of evidence.

What the systematic reviews say about learning

That separate claim is exactly what the field’s own review literature says it cannot support.

Keuning, Jeuring, and Heeren’s 2018 systematic review, one of the most cited in the field, coded 101 tools by the kind of feedback they generate. Its central finding is about the tools’ functionality, not their effect: “We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs” (Keuning, Jeuring & Heeren, 2018). A review spanning 101 tools has, in other words, remarkably little to say about whether any of them measurably improved what students learned, because that evidence mostly does not exist to review.

Seven years later, a 2025 observational study across five community colleges opens by naming the identical, still-unresolved gap: “However, empirical assessments of auto-grader feedback’s impact on learning outcomes, such as grades and pass rates, remain insufficient (Keuning et al., 2018)” (Zhang, Burte, Savelka, Bogart & Sakr, 2025). The 2025 study cites Keuning’s review as its reason for existing: one complaint, kept open for seven years because nothing arrived to close it.

The strongest positive claim, examined

Stephen Edwards’ early-2000s work on test-driven grading with Web-CAT at Virginia Tech reports the most encouraging number in this literature: students using TDD and Web-CAT “submitted programs containing approximately 45% fewer defects per 1000 lines of code” than a cohort graded the old way (Edwards, 2004). The JERIC journal version’s abstract gives a different figure for the same study: “a 28% reduction in defects per thousand lines of code” (Edwards, 2003). The discrepancy stays unresolved in the sources checked here.

The design is a quasi-experiment, not a randomized trial: a Spring 2001 cohort graded the old way against a Spring 2003 cohort graded with TDD and Web-CAT, 59 students each, one semester, with manual defect scoring on a hand-checked sample of 18 programs extrapolated to the rest. No independent replication has turned up in the sources surveyed here; the field’s systematic reviews (Keuning et al., 2018; Messer et al., 2024) cite Edwards’ later tool papers but do not confirm the original claim. The one external data point on Web-CAT outside Virginia Tech is a Saudi deployment, where 50.6% of students reported being unhappy with the system’s feedback (Aldriye, Alkhalaf & Alkhalaf, 2019). Satisfaction is not a learning outcome, but it’s the only independent evidence on record. A careful, good-faith claim, decades old and never independently replicated, is not enough on its own to close the question.

The newest version of the promise, under direct test

One thing should be said plainly before the trials: neither of them tests the classic test-based autograder. No controlled trial in this corpus does; that absence is the gap Keuning and Zhang keep naming. What 2025 produced instead is two randomized trials of the promise’s newest carriers, AI-assisted feedback layered onto the same automated workflow. They test the current version of Hollingsworth’s claim, not his tool, and both came back unflattering to it.

A randomized trial on learning gains

The sharpest is a 2025 randomized trial at the Technical University of Munich, titled, without qualification, “Less stress, better scores, same learning.” Bassner and colleagues ran 275 introductory CS students through a 90-minute exercise on concurrency under three conditions (a scaffolded AI tutor, unrestricted ChatGPT, and a no-AI control) and measured both exercise performance and pre/post knowledge gains separately. Both AI conditions scored higher on the exercise itself. Neither scored higher on the measure of actual learning: “Despite these performance gains, neither AI condition produced greater pre–post knowledge gains or code-comprehension advantages… In this setting, generative AI acted primarily as a performance aid rather than a learning enhancer” (Bassner, Lenk-Ostendorf, Beinstingel, Wasner & Krusche, 2025). Performance and learning dissociate, exactly as the title says, and the trial shows the mechanism directly.

A randomized trial on persistence

A separate 2025 randomized study of 257 introductory programming students tested something related but distinct: LLM-generated feedback on compiler errors, compared against standard compiler messages. While the intervention was active, it produced a real, measured benefit: reduced “wheel-spinning” and improved persistence on hard tasks. Then the researchers removed the feedback and re-measured: “Notably, this positive impact was also observed in challenging tasks. However, its benefits did not sustain once the feedback was removed” (Zhou, Pankiewicz, Paquette & Baker, 2025). The effect was real while it lasted. It did not survive the intervention’s removal. We read that pattern as the signature of a crutch rather than instruction, though scaffolding withdrawn too early would leave the same trace, and the study itself does not pick between those readings.

Both trials found the same thing: AI-assisted feedback changes what students produce while it is present, and neither found a change in what students retained afterwards. The same standard applied to Edwards above applies here with equal force: each is a single-site study of one brief intervention (a 90-minute exercise in Munich, one course’s compiler errors), published within the last year and not yet replicated. What the two trials establish is correspondingly narrow: the first direct measurements of the promise’s newest form found performance without learning. The classic autograder still awaits a trial of its own.

The line item that does not exist

One piece of evidence here does not appear to have been published in English before, and it comes from an unlikely place: university labor-accounting regulations.

Russian and Belarusian universities set formal time budgets, in hours, for how much paid faculty work each category of grading is worth: figures used purely to calculate teaching load, with no pedagogical framing at all.

Russian norms

Two independently adopted Russian documents converge on the same order of magnitude. HSE’s norms, in force since 2004, set checking of “essays, homework, control assignments, and reports” at “0.3 hour per assignment”; reports themselves carry a separate, higher rate: “0.75 hour per report for bachelor’s/specialist programs, 1 hour per report for master’s/postgraduate programs” (HSE, 2004; original: “Проверка: эссе, домашних заданий, контрольных работ, рефератов. Проведение коллоквиума — 0,3 часа на одно задание; 0,75 часа на реферат по программам подготовки бакалавров и специалистов; 1 час на реферат по программам подготовки магистров и аспирантов.”). SPbSUT’s norms, updated in 2017-2018, set the identical “0.3 hour per assignment, but no more than 1 hour per student per course per semester” for “checking, consultation, and acceptance of control, calculation, and calculation-graphic assignments”. Unlike HSE, SPbSUT norms reports under a separate line, so the figure isn’t muddied by a second rate.

Belarusian norms

Belarus runs a two-tier version of the same system. The republic-wide Ministry of Education norm sets a range rather than a point figure: “checking control assignments, including calculation-graphic and calculation assignments (typical calculations)… provided for by the academic program/plan: from 0.35 to 0.5 hour per assignment, no more than 1 hour per student per academic discipline (module) per semester” (Ministry of Education of the Republic of Belarus, 2018 edition). A specific university then fixes its own figure within that republican range. The Belarusian State Academy of Communications did so in an institutional order dated 24 April 2025, built explicitly on the Ministry’s 2023-2024 resolutions (No. 310 of 26 September 2023 and No. 104 of 16 August 2024), and it goes one step further than any Russian document found in this survey: it names laboratory work by name. “Checking control assignments, typical calculations, calculation-graphic assignments, essays, reports on completed laboratory research: 0.4 hour per student / 0.25 hour per student per academic discipline, module” (BSAS, order No. 126, 2025).

In the four documents examined here (two Russian, two Belarusian) there is no separate line item for checking program code, no mention of “code,” “compile,” or “debug” anywhere in the text. Programming lab reports are graded under the same generic category as essays, typical calculations, and control assignments in general. In these four documents, checking a stranger’s program and reading a written control assignment fall under one shared category, not two. That is what the documents show; nothing here says whether the people who wrote them ever considered the difference. What they also show is the accounting as it stands: where automated grading has been adopted in these institutions, it lives inside a workload category older than itself, and none of the four texts draws any line between checking code and checking prose.

A closer look at these four norms, and at what is actually known about what this work costs the person doing it, is the subject of a companion piece on this site.

Six decades on

What this literature, taken as a whole, actually supports is narrower than the strongest claims made for it. One case is clear, and nothing in the corpus surveyed here disputes it: a ratio nobody could close by hand got closed. India’s 1:172 is the extreme version of that; China’s platforms count submissions in the tens of millions; Ghana’s deployment ran on smartphones because the computers were in schools rather than homes, and Japan’s grading sped up fourfold. What that closure costs is a theoretical guarantee of correctness, which Rice’s theorem rules out from the start, traded for a fast and incomplete proxy; where anyone actually measured the incompleteness, it ran to roughly a quarter of all rejected solutions (GATE’s 26.5%). And once that proxy is what gets rewarded, students optimize for it rather than for the skill it stands in for, as the Finnish test-coverage study found directly.

More than six decades after Hollingsworth first raised the possibility, the claim his own paper made, that students taught with an automatic grader learn better, is still the one nobody has managed to support. The field’s own review literature, a systematic review in 2018 and an observational study citing it in 2025, says the evidence remains insufficient, in nearly identical language, seven years apart (Keuning et al., 2018; Zhang et al., 2025). The two randomized trials in this corpus that measured learning directly (both, note, trials of the newest AI-assisted feedback rather than of the classic autograder) found a gap between performance and learning, one of them titled as plainly as a title can be: “less stress, better scores, same learning” (Bassner et al., 2025). And the one benefit that did show up as improved persistence on hard problems stopped showing up the moment the feedback was switched off (Zhou et al., 2025).

This doesn’t make automated grading a poor tool for the job it demonstrably does. The throughput problem it solves is real, and for a number of these institutions, not solving it was never a realistic option. But the learning claim has never been allowed to die, either: Hollingsworth stated it in 1960; Edwards went looking for numbers behind it in 2003. The 2025 wave of AI-assisted tools is being measured against it right now. What has not happened, at any point along that chain, is the claim being confirmed. The more honest summary, based on what the studies gathered here actually show, is narrower: automated grading measurably speeds up checking programs. Whether it teaches anyone to write better ones remains, in the literature’s own words, an open question.

This review draws on close to sixty studies collected across five language zones: the English-language core literature, other European scholarship, Russian and post-Soviet sources, Belarusian sources, and a broader international set spanning China, Japan, India, Brazil, Mexico, and Africa. It was compiled by Andrei Niasiuk, who is building AutoLabSuite, a platform for grading programming lab assignments in university courses.