Can LLMs grade programming assignments? What the 2025–2026 studies measured

Andrei Niasiuk · August 18, 2026 · Facts checked: August 11, 2026

Read the recent abstracts on language models as graders one after another and they seem to contradict each other. One team reports grading “comparable to human graders”. Another, months later, reports “only moderate agreement with human teachers’ grades”. A third finds faster completion; a fourth finds no knowledge gain at all. Sorted by what each study actually put on the scale, most of the contradiction dissolves, and what remains is an ordering worth naming: the closer a study gets to measuring what a student knows afterwards, the harder its positive result is to obtain.

This piece sorts eight studies published in 2025 and 2026 by what they measured. A companion review on this site covers the six decades before them.

Agreement with a human grader

The first thing a grading study can measure is whether the machine’s grade matches the teacher’s. Two studies did exactly that and landed in different places.

At the University of Ljubljana, a bioinformatics course ran a blind comparison: every student received feedback from both a language model and a human teaching assistant, without being told which was which, and rated both. The authors conclude that “with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders”, and add that “open-source LLMs perform as well as commercial LLMs, allowing schools to implement their own grading systems while maintaining privacy” (Poličar, Špendl, Curk & Zupan, 2025).

The second study is larger and colder. It ran eighteen current models from four vendors over more than six thousand student solutions collected across four years of an introductory programming course, and reports: “All models displayed high internal agreement, measured by the intraclass correlation coefficient, with the model consensus but only moderate agreement with human teachers’ grades, indicating a persistent gap between automated and human assessment” (Jukiewicz, 2025).

The two are less opposed than they look. Ljubljana graded free-text answers in one course with prompts tuned for it; the comparison graded programming solutions at scale across many models with no per-course tuning. Both can hold: a carefully prompted model matches a human on one course’s questions, while models in general converge on each other more tightly than on teachers.

That second finding is the uncomfortable one. Models agreeing with each other while agreeing only moderately with people means they share a standard of their own. Whether that standard is worse than the human one the study does not say, and neither will I. What it does mean is that consistency between graders stops being evidence that the grading is right.

Speed, completion, and time to a working solution

The second measurable thing is what happens to the work itself.

A randomized deployment in a real introductory Python course covered 6693 submissions from 215 consenting students across seventeen labs, under three conditions: natural-language feedback, AI-generated failing test cases, or no AI feedback. The result: “natural language feedback is significantly associated with higher completion rates and faster convergence to correct solutions. Test case feedback, by contrast, exhibits heterogeneous effects that depend critically on feedback validity” (Heickal & Lan, 2026).

That second sentence deserves as much attention as the first. Generated test cases help when they are correct and hurt when they are not, so the average effect is a poor description of what any individual student met.

A parallel study varied how much guidance the feedback carried, from full explanations down to descriptions of what went wrong with no hints attached, and found “LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects” (Mihaylova et al., 2026).

Less guidance working slightly better is counterintuitive, and worth resisting the easy explanation. It is not students copying a handed-over answer: the prompts in that study explicitly forbid showing or hinting at the sample solution, and the least detailed condition offers the least to copy. The authors reach instead for a known effect in which heavy guidance helps beginners and gets in the way of those who already have some footing.

What the student knows afterwards

The third measurable thing is learning, and here the picture splits by study design.

A randomized trial at the Technical University of Munich put 275 introductory students through a ninety-minute exercise on concurrency under three conditions (a scaffolded AI tutor, unrestricted ChatGPT, and no AI), measuring exercise performance and pre/post knowledge separately. Both AI conditions scored higher on the exercise. Neither scored higher on knowledge: “Despite these performance gains, neither AI condition produced greater pre–post knowledge gains or code-comprehension advantages… In this setting, generative AI acted primarily as a performance aid rather than a learning enhancer” (Bassner et al., 2025). The same trial also recorded lower frustration and reduced cognitive load, which is why its title promises less stress alongside the same learning.

A second randomized study, 257 students in a CS1 course, added LLM-written explanations on top of the compiler’s own error messages. While the feedback was switched on it cut down fruitless attempts on hard tasks. Then the researchers switched it off: “Notably, this positive impact was also observed in challenging tasks. However, its benefits did not sustain once the feedback was removed” (Zhou, Pankiewicz, Paquette & Baker, 2025).

Set against those two, an observational study across five community colleges argues the opposite way. It opens by naming the gap, “empirical assessments of auto-grader feedback’s impact on learning outcomes, such as grades and pass rates, remain insufficient (Keuning et al., 2018)”, and then states that it fills that gap, reporting evidence in favour of auto-grader feedback and encouraging wider use (Zhang, Burte, Savelka, Bogart & Sakr, 2025).

So the honest summary is narrower than “AI feedback does not teach”. Where 2025 assigned students at random and tested knowledge before and after, it found gains in performance without matching gains in knowledge. Where it observed courses that already used auto-grader feedback, it found associations pointing the other way. Observational data cannot separate the tool from the students and teachers who chose it, which is the reason the randomized results carry more weight here, not a reason to pretend the observational ones do not exist.

Whether the graded person accepts the grade

The fourth measurable thing is the one grading systems usually skip.

A Turkish study deployed a grading and appeals tool across more than 850 exam papers and 185 appeals. The number that stands out is not the accuracy figure: “The appeal process led to grade changes in 74% of cases, indicating the need for continued refinement of AI evaluation strategies. While students appreciated the speed and detail of AI feedback, survey responses revealed trust and fairness concerns” (Aytutuldu, Yol & Akgul, 2025).

Three quarters of appealed grades changed. That is not three quarters of all grades: students appeal selectively, usually when they think they have a case, and 185 appeals against 850-plus papers is roughly one in six. It does say the appeal channel was carrying real weight, and that the same deployment without one would have left those grades standing.

The ordering

Line the studies up by what went on the scale.

Match a machine grade against a human grade, and the answers range from “comparable” in one tuned course to “only moderate agreement” across eighteen models at scale. Measure throughput, and the gains are consistent. Measure knowledge under randomization, and the gains disappear, though an observational study reports the opposite. Measure acceptance, and an appeals channel rewrites three quarters of what it touches.

None of this makes language models a poor grading tool. Faster feedback and higher completion are worth having, and a department drowning in submissions will take them. What these studies do not license is the step people tend to take next: reading a grade that arrived quickly, and that the student did not contest, as evidence that the student learned something on the way to it.

This piece draws on eight studies published in 2025 and 2026, collected as part of a review of automated grading across five language zones. It was compiled by Andrei Niasiuk, who is building AutoLabSuite, a platform for grading programming lab assignments in university courses.