AI Textbooks Need Practice, Not Chat

The submitted iTextbooks 2026 paper, "Balancing Efficacy and Engagement in Interactive Texts," introduces Phosphor, a digital learning platform built around LLM-graded formative assessment inside instructional content. Jonah Bard reports an early Spring 2026 deployment in Dartmouth's MATH 010, an introductory statistics course, across three sections that started with 151 enrolled students and ended with 143.

My read is that the paper is valuable because it narrows the AI-in-education question. It is not another claim that students need a smarter chatbot. It is a claim that AI becomes more useful when it is embedded in the reading workflow, tied to rubrics, and used to make retrieval practice cheap enough that students will actually do it.

Answer Snapshot

Question	My read
What happened?	A Dartmouth pilot tested Phosphor as an optional, ungraded alternative to traditional readings in introductory statistics.
What is the big result?	The paper says full Phosphor dosage was associated with a final-exam advantage between 0.71 SD after adjustment for prior exam scores and 1.30 SD unadjusted.
What is the useful design claim?	Embedded constructed-response practice and cumulative reviews look more important than a general RAG chat sidebar.
What is the caveat?	This was an observational pilot at one selective institution, without randomized controls, so self-selection remains the main threat.

The Claim Is Not Chatbot Magic

Phosphor is described as a web application where lessons appear as navigable pages with completion indicators. Each lesson has a bank of 15-20 exercises, and students take four-question lesson quizzes. Multiple-choice questions are auto-graded. Constructed-response questions are graded by Claude Sonnet 4.6 against instructor-defined, question-specific criteria, with the student's response, the question, a model answer, and grading criteria in the prompt.

That matters because the AI is not just answering student questions. It is making a familiar learning-science move operational: read a little, recall a little, get feedback, retry. Students pass a lesson quiz at 75% or higher, content is not gated, and retries are unlimited. Module reviews add cumulative practice with a 90% threshold.

The platform also includes a RAG-based chat assistant grounded in the curriculum. But the paper's own usage data makes the chat feature look secondary. The assistant received 72 total queries, and only 14 students submitted more than one. Students reportedly said general-purpose LLMs were faster or that the reference content was already sufficient. That is the detail I keep coming back to: the headline is not conversational AI replacing the textbook. The headline is assessment turning the textbook into something students act on.

The Engagement Number Is The Hook

The surprising part is adoption. Phosphor was optional and ungraded, yet 90.2% of enrolled students used lesson quizzes or module reviews at least once. The paper estimates reading compliance between 48% and 76%, compared with student and instructor baselines of roughly 10-15% for the course. In live in-class surveys, 94% of respondents in both survey rounds agreed Phosphor was more engaging than traditional readings.

I do not treat those numbers as a universal forecast. Introductory statistics at Dartmouth is a specific setting, and the survey samples were small. But as a product signal, it is still meaningful. A lot of educational technology fails before efficacy because students simply do not use it. Phosphor appears to have cleared that first hurdle in this pilot.

The learning-outcome signal is more complicated but more interesting. The paper reports that students who passed all three module reviews scored 7.1 points higher on the final exam than those who did not, with Cohen's d of 0.66. Its Tobit models put the full-versus-zero engagement gap at 14.7 points on a 0-100 scale before adjustment, and 8.0 points after controlling for midterm performance. The author reads the adjusted 0.71 SD estimate as a conservative lower bound and the unadjusted 1.30 SD estimate as selection-inflated.

That is the right language: associated, not proven. The result is promising because the effect is large enough to care about and specific enough to investigate. It is not proof that Phosphor caused the whole gap.

The Guardrails Context Matters

This paper lands in a world where student AI use is already normal. HEPI's Student Generative Artificial Intelligence Survey 2026 says 95% of UK undergraduates report using AI in at least one way, and 94% say they use generative AI to help with assessed work. The same page says only 36% feel encouraged by their institution to use AI, and only 38% say they are provided with AI tools. Even in the public comments on that report, readers push back on whether self-reported productivity should be trusted as learning.

That is why I like Phosphor's direction. It responds to the AI adoption wave by moving the design question from "Will students use AI?" to "What kind of AI use still forces students to think?"

The external evidence points in the same direction. A Wharton summary of Hamsa Bastani and coauthors' study, Without Guardrails, Generative AI Can Harm Education, describes a field experiment with nearly 1,000 high school math students. In that account, students using a GPT Base interface did better during AI-assisted practice but 17% worse than the control group when AI was removed, while a guarded GPT Tutor mitigated the harm. Stanford's SCALE repository summarizes the same study as a randomized controlled trial and says the negative effects were largely mitigated by the tutor safeguards.

Phosphor is not the same intervention, and Dartmouth statistics students are not high school math students in Turkey. Still, the design pattern rhymes. Unrestricted help can become a crutch. Guarded, task-shaped help can push the student back into the work.

Constructed Response Is The Load-Bearing Bet

The most useful natural variation in the paper is the quiz-format change. Module 1 used mixed multiple-choice and constructed-response lesson quizzes. Module 2 switched lesson quizzes to multiple-choice only after students said the constructed-response auto-grader felt rigid and discouraging. Module 3 restored constructed-response questions after exam results suggested multiple-choice-only lesson quizzes had negligible learning benefits.

The reported dosage pattern follows that distinction. Module 1 lesson completions were associated with roughly 1.6 additional percentage points on the first midterm. In Module 2, the apparent positive relationship among all users disappeared among students with at least one completion, where the slope was slightly negative. On the cumulative final, each lesson completion was associated with about 0.4 additional points, and the paper argues that the signal is carried by the modules with constructed-response practice.

That makes intuitive sense to me. Multiple choice can be useful, but it is easy to click through. Constructed response forces generation. The student has to retrieve, explain, and expose the shape of their misconception. The AI contribution is not that the model knows statistics. It is that LLM grading can make that kind of low-stakes response feasible at classroom scale.

Grading Still Needs Humility

The paper is careful about grading reliability. It cites prior work on LLMs for constructed-response grading, but also says reliability varies with question complexity and that this pilot did not conduct a formal inter-rater reliability study. That caveat should stay attached to every excited reading of the result.

There is supporting context. Owen Henkel and coauthors' short-answer grading study reports that GPT-4 with basic few-shot prompting reached a kappa of 0.70, close to the human-level benchmark of 0.75 on their dataset, and argues this could support low-stakes formative assessment. A 2025 review, Harnessing Large Language Models for Scalable and Effective Formative Assessment in Higher Education, frames formative assessment as valuable but hard to scale because of time, class size, resources, and implementation barriers.

That is the lane where I think Phosphor belongs: low-stakes, repeated formative work, not high-stakes grading authority. Rubrics, model answers, retries, instructor-designed criteria, and curriculum boundaries are not decorations. They are the safety rails that make the model's grading plausible enough to test.

The Weak Spots Are Real

The paper's limitations section is unusually important. This was an observational study of a pilot deployment at a single selective institution. It lacked randomized controls. Self-selection is the central threat because the students who complete more quizzes may also be more motivated or higher-performing. The cross-module comparison is confounded by content domain, timing, and the simultaneous introduction of module reviews. The all-reviews-passed group is also the most self-selected group in the study.

That does not make the study useless. It makes the next study obvious. I would want random assignment across quiz formats, replicated in other gateway courses, with formal checks on LLM grading reliability and a clear analysis plan before the course starts. The paper says future work should include controlled study of constructed-response versus multiple-choice formats, completion requirements attached to course grade, and replication across institutions.

I would be careful with the grade requirement. It may produce cleaner dosage data and higher usage, but it could also change the user experience that made the optional pilot interesting. If constructed-response grading feels rigid, discouraging, or unfair, forcing it into the grade can turn a good learning loop into another compliance task. The product question is not only whether students learn more. It is whether the system makes effort feel worth repeating.

Why This Fits The Intelligent Textbook Moment

The broader workshop context matters. The Festival of Learning 2026 page describes the Seventh Workshop on Intelligent Textbooks as focused on how AIED methods can transform textbooks from static content into interactive learning environments, with special emphasis this year on generative AI and LLMs. Phosphor is a concrete version of that idea.

That is why I find the paper more interesting than a generic tutoring demo. The textbook is still the anchor. The AI is not there to answer every question in the most fluent possible way. It is there to create a loop: read, retrieve, get feedback, retry, review, and leave a trace of engagement. That is much closer to a learning system than a homework shortcut.

My Bottom Line

I would not call Phosphor proven. The study is too early, too observational, and too institution-specific for that. But I would call it a useful product hypothesis with unusually clear next tests.

The AI education story I trust less is the one where a general assistant sits beside the student and promises limitless help. The one I trust more is narrower: use AI to make hard practice cheap, keep it inside the curriculum, grade it against explicit criteria, and measure whether students still perform when the helper is gone. Phosphor points in that direction. The next question is whether the effect survives randomization, replication, and the everyday mess of courses that are not a first pilot.