Bibliography
Every source this blog leans on, with the verbatim quote the claim rests on and the date it was fetched. Quotes stay in the language of the original. The list grows with each publication.
- https://aaltodoc.aalto.fi/server/api/core/bitstreams/98f6a005-8373-4567-90ac-5cae87507636/content
We have analyzed how students behave when they are rewarded for structural test coverage (e.g. line coverage) and found that this can lead students to write tests with good coverage but with poor ability to detect faulty programs.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://arxiv.org/abs/2107.12244
It also represents the first system built and tested in an African context across over thirty-five countries across the continent... We have deployed AutoGrad in two different cohorts of SuaCode for use in grading scripts of 1000+ students across Africa... with over 3,000 code files graded.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://arxiv.org/abs/2501.14499
Our results show that with well-designed prompts, LLMs can achieve grading accuracy and feedback quality comparable to human graders. Our results also suggest that open-source LLMs perform as well as commercial LLMs, allowing schools to implement their own grading systems while maintaining privacy.
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measured
- https://arxiv.org/abs/2504.13557
The appeal process led to grade changes in 74% of cases, indicating the need for continued refinement of AI evaluation strategies. While students appreciated the speed and detail of AI feedback, survey responses revealed trust and fairness concerns.
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measured
- https://arxiv.org/abs/2507.14235
However, empirical assessments of auto-grader feedback's impact on learning outcomes, such as grades and pass rates, remain insufficient (Keuning et al., 2018).
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measuredDo automated grading systems improve learning? Six decades of evidence
- https://arxiv.org/abs/2508.06563
The course was coordinated by three faculty members, resulting in a faculty-to-student ratio of 1:172, with additional support from TAs at a ratio of about 1:20.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://arxiv.org/abs/2509.26483
All models displayed high internal agreement, measured by the intraclass correlation coefficient, with the model consensus but only moderate agreement with human teachers' grades, indicating a persistent gap between automated and human assessment.
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measured
- https://arxiv.org/abs/2605.16933
Results from an online programming course show that LLM-generated feedback is associated with faster time to solution compared to the no-feedback baseline, with less guided feedback showing slightly stronger effects.
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measured
- https://arxiv.org/abs/2606.08807
We find that natural language feedback is significantly associated with higher completion rates and faster convergence to correct solutions. Test case feedback, by contrast, exhibits heterogeneous effects that depend critically on feedback validity.
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measured
- https://bsac.by/sites/default/files/2025/umo/Normy%20vremeni%20na%20ped.%20deyatel'nost'%20na%202025-2026..pdf
Проверка контрольных работ, типовых расчетов, расчетно-графических работ, рефератов, отчетов о выполнении лабораторных исследований — 0,4 часа на 1 студента / 0,25 часа на одного обучающегося на учебную дисциплину, модуль.
· Cited in: Checking code: invisible twiceDo automated grading systems improve learning? Six decades of evidence
- https://ceur-ws.org/Vol-1067/abp2013_submission_8.pdf
Von 1031 Lösungen wurden durch die Funktionstests 759 Lösungen (73,6%) als nicht korrekt klassifiziert aber 201 Lösungen (26,5%) hiervon von den Tutoren trotzdem als korrekt bewertet, da nur kleinere Fehler vorhanden waren (z.B. Tippfehler, Package falsch gesetzt).
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://cyberleninka.ru/article/n/emotsionalnoe-vygoranie-u-prepodavateley-vuzov-sposoby-vyyavleniya-i-preduprezhdeniya
на первом месте [оказались] новые постоянно вводимые требования к преподавателям и их работе (76,5% выбрали данную причину как основную), на втором месте - оплата труда и на третьем - высокая аудиторная нагрузка (соответственно 58,8% и 41,2% опрошенных).
· Cited in: Checking code: invisible twice
- https://dl.acm.org/doi/10.1145/3231711
We have found that feedback mostly focuses on identifying mistakes and less on fixing problems and taking a next step. Furthermore, teachers cannot easily adapt tools to their own needs.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://dl.acm.org/doi/10.1145/367415.367422
After fifteen months, our experience leads us to believe that students learn programming not only as well but probably better than they did under the method we did use—laboratory groups of four or five students.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://edustandart.by/media/k2/attachments/npd_primernye-normy-vremeni-dlya-rascheta-ob-ema-uchebnoj-raboty_14-03-2018.pdf
Проверка контрольных работ, в том числе расчетно-графических и расчетных работ (типовые расчеты), предусмотренных учебной программой / учебным планом (в том числе в форме тестирования) — от 0,35 до 0,5 часа на 1 работу. Не более 1 часа на 1 студента (курсанта, слушателя) на учебную дисциплину (модуль) в семестр.
· Cited in: Checking code: invisible twice
- https://lancashire.web.ucu.org.uk/files/2023/09/Academic-Workload-Guidance-2023-24.pdf
Additional allocation should be given for marking and normally this would be 1 hour per student per 20 credit module.
· Cited in: Checking code: invisible twice
- https://learninganalytics.upenn.edu/ryanbaker/ICCE2025_paper_28.pdf
Notably, this positive impact was also observed in challenging tasks. However, its benefits did not sustain once the feedback was removed.
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measuredDo automated grading systems improve learning? Six decades of evidence
- https://link.springer.com/article/10.1007/s11192-023-04770-x
the investigation excludes both (i) the existence of a negative link in terms of workload—contradicting considerations such as 'Those who do more teaching have less time to do research and vice versa'—and (ii) the existence of a positive link in terms of the quality of the results obtained.
· Cited in: Checking code: invisible twice
- https://memo.hse.ru/ind_w07_2_005
Объем затрат рабочего времени преподавателей вузов на разные виды деятельности в целом остался примерно на том же уровне за 2006–2009 гг. ...несколько меньше — на научную работу (10,1 часа в 2006 и 9,3 часа в 2009 г.)
· Cited in: Checking code: invisible twice
- https://pps.hse.ru/data/2014/01/21/1325895260/Нормы%20времени%20для%20расчета%20объема%20учебной%20работы.doc
Проверка: эссе, домашних заданий, контрольных работ, рефератов. Проведение коллоквиума — 0,3 часа на одно задание; 0,75 часа на реферат по программам подготовки бакалавров и специалистов; 1 час на реферат по программам подготовки магистров и аспирантов.
· Cited in: Checking code: invisible twiceDo automated grading systems improve learning? Six decades of evidence
- https://pwr.edu.pl/fcp/aGBUKOQtTKlQhbx08SlkTUhZeUTgtCgg9ACFDC0RPSG9PFRYqCl5tDXdAGHoV/_users/code_BAlkUKkpKNVI_VBV9XFNBFEZcFiw5Wx0O/akty_prawne/pensum/zw_4_2019-z.pdf
poza pensum dydaktycznym: 1.2.2.1 inne prace związane z procesem dydaktycznym, do których w szczególności zalicza się: hospitacje; dyżury dydaktyczne; przeprowadzanie egzaminów w toku studiów, kolokwiów i sprawdzianów; recenzowanie prac dyplomowych magisterskich, inżynierskich, licencjackich, sprawdzanie prac kontrolnych studentów; przygotowanie kursu w formie zdalnego nauczania...
· Cited in: Checking code: invisible twice
- https://recht.nrw.de/lrgv/rechtsverordnung/23092023-verordnung-ueber-die-lehrverpflichtung-universitaeten-und/
(5) Die Betreuung von Studienabschlussarbeiten und vergleichbaren Studienarbeiten wird unter Berücksichtigung des notwendigen Aufwandes bis zu einem Umfang von drei Lehrveranstaltungsstunden angerechnet.
· Cited in: Checking code: invisible twice
- https://rsisinternational.org/journals/ijrias/uploads/vol10-iss10-pg202-221-202510_pdf.pdf
Correlation analysis revealed a significant positive relationship between workload and burnout (r = 0.698, p < .001), indicating that increasing workload may heighten the risk of burnout over time.
· Cited in: Checking code: invisible twice
- https://scimatic.org/storage/journals/11/pdfs/7469.pdf
Results showed that task volume (r = 0.229) and time pressure (r = 0.309) were not significantly associated with job burnout, whereas role conflict and work schedule (r = 0.498) were significantly associated with burnout.
· Cited in: Checking code: invisible twice
- https://taoxie.cs.illinois.edu/publications/cccf14-edu.pdf
国内起步最早的在线程序评判系统是浙江大学于2003年开发的ZOJ...北京大学的POJ...于2004年建立，程序提交量长期在国内领先，迄今已经提交了800多万个程序，…另外，杭州电子科技大学的HDOJ...近年来发展也很快，程序提交数目于2014年1月率先冲破1000万大关。
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://thesai.org/Downloads/Volume10No3/Paper_28-Automated_Grading_Systems.pdf
the same problem was discovered in Introduction to Java programming language at King Saud University, were 50.6% of the enrolled students were unhappy using the system because the generated feedback report was unclear and does not explain the errors type and also how the system generates their scores.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://web-cat.org/publications/p179-Edwards.pdf
Table 1 summarizes the results of this analysis, which show that students who used TDD and Web-CAT submitted programs containing approximately 45% fewer defects per 1000 lines of code.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://webperso.info.ucl.ac.be/~pvr/DervalEMOOCS2015.pdf
Computer scientists know that automatic correction is not an easy task, because of theoretical limits: it is actually impossible to check whether the student's code does the same as a correct one (consequence of Rice's Theorem, see [Beckman, 1980]).
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://www.econstor.eu/bitstream/10419/58160/1/715993186.pdf
Up to 28% of the results needed manual correction by the teacher, in most cases because of false negatives.
· Cited in: Do automated grading systems improve learning? Six decades of evidence
- https://www.insidehighered.com/news/quick-takes/2026/02/27/community-college-district-settles-adjuncts
weren't compensated for hours of work outside the classroom, including grading assignments, preparing lessons and meeting with students.
· Cited in: Checking code: invisible twice
- https://www.mdpi.com/1660-4601/22/8/1214
One of the most consistent findings is the strong association between burnout syndrome and excessive workload. Multiple studies indicate that high work demands—exacerbated by insufficient resources and weak institutional support—significantly increase burnout levels among university professors.
· Cited in: Checking code: invisible twice
- https://www.reading.ac.uk/human-resources/-/media/project/functions/human-resources/documents/guidanceonthecalculationofhours.pdf
Any piece of student writing (essay, report) at the length of 2500 words: 30 minutes per [piece]... 1250 words = 15 minutes, 5000 words = 1 hour, 10 000 words = 2 hours.
· Cited in: Checking code: invisible twice
- https://www.sciencedirect.com/science/article/pii/S2666920X25001778
Despite these performance gains, neither AI condition produced greater pre–post knowledge gains or code-comprehension advantages... In this setting, generative AI acted primarily as a performance aid rather than a learning enhancer.
· Cited in: Can LLMs grade programming assignments? What the 2025–2026 studies measuredDo automated grading systems improve learning? Six decades of evidence
- https://www.sut.ru/doci/sotrudnik/doci/regulations/teaching_timing.doc
Проверка, консультация и прием контрольных, расчетных и расчетно-графических работ — 0,3 часа на одно задание, но не более 1 часа на одного студента на дисциплину в семестр.
· Cited in: Checking code: invisible twice
- https://www.univ-tlse3.fr/medias/fichier/2025-07-cafr-001_1753255875567-pdf
Il est également rappelé que les activités de préparation d'enseignements, les corrections et évaluations (copies, mémoires, thèses, etc.), et les participations à des jurys de diplôme font partie intégrante de la tâche d'enseignement des enseignant.e.s chercheur.euse.s et enseignant.e.s (sauf exception justifiée par les textes en vigueur) et ne sont donc naturellement pas considérées dans le présent référentiel.
· Cited in: Checking code: invisible twice