Do tutoring chatbots help students learn?

The trial

Researchers at the University of Maryland ran a randomised controlled trial with 2,379 undergraduate students and 30 instructors, across several subjects at a large US public university. Some course sections were given a generative AI tutor built into the course. Others were not.

The results were published in October 2026 as a working paper by IZA, the labour economics institute. Working papers have not yet been through peer review, so the findings may change.

What it found

Comparing sections of the same course, students who had the tutor:

• finished with lower final grades, by 0.37 standard deviations, a large effect in education research
• took part much less in the course's existing online activities, by 0.90 standard deviations

The losses were estimated to be larger for first-generation students, those whose parents did not go to university.

The authors' reading is that the tutor did not add to learning. It replaced some of the work that produces learning. Students used the tutor instead of the practice, reading and discussion the course already had.

The tutor did not add to learning. It replaced some of the work that produces it.

It fits a wider pattern

The paper sits alongside earlier studies describing a "learning penalty". Students given a general-purpose chatbot often do better on the tasks they complete with its help, then worse when they are tested without it. The authors note that tutors built with safeguards, which guide students instead of handing over answers, may reduce that penalty.

Other recent trials, mostly with short tasks or single sessions, have found gains. The evidence is mixed, and it depends heavily on how the tool is designed and used.

What this means for schools

This trial was at a university, not a school, and one study is not the last word. Still, it points to sensible questions to ask before any tutoring tool reaches students.

• Does it make students do the thinking? A tutor that asks questions, gives hints and checks understanding is very different from one that writes answers.
• What does it replace? If students use it instead of practice they already do, learning can fall. Keep the practice that works.
• Who benefits, and who does not? Look at results for different groups of students, not just the average.
• Test independent work. Judge the tool by how students do without it, in class tests and exams, not by how well they do with it.