Chat as Learning: Student–AI Conversations as Discipline-Associated Cognitive Engagement Patterns

Abstract

When a student asks a generative-AI tutor a question, the question itself is an observable trace of how that student is currently thinking — not merely a request for information. We formalize this as a "Chat as Learning" measurement paradigm and apply it to 60,000+ student messages from 116 courses across four universities on the Uedu platform, Bloom-coding each message and modeling the result with a crossed random-effects GLMM. The same students shift their cognitive engagement profile when they move between disciplines, indicating that engagement observed in AI dialogue is bound to course context rather than being a fixed individual trait.

Problem & Motivation

Cognitive engagement is often treated, implicitly, as something a learner has — a dispositional trait that travels with the student from course to course. Student–AI dialogue offers an unusually direct way to test that assumption at scale, because every question a student types is a behavioral trace produced in the course of learning rather than a self-report collected afterwards. The open question this study addresses is therefore a within-person one: when the same student moves between disciplines, does their pattern of cognitive engagement stay stable, or does it reorganize around the course they are currently in?

Method

We analyzed 60,000+ student messages from 116 courses across four universities on the Uedu platform, classifying each message by Bloom's taxonomy level. The design is within-person and cross-disciplinary: because many students appear in more than one course, the same individual can be observed under different disciplinary conditions, which separates the student contribution from the course contribution. Estimation used a crossed random-effects GLMM with students and courses as crossed random factors. Classification reliability was established against human raters, with inter-rater agreement of κ = .426–.606 for individual raters and κ = .753 for the best-pair consensus. The study was approved by the National Taiwan University Research Ethics Committee (NTU-REC 202507EM058).

Findings

• Disciplines carry distinct profiles: STEM courses skewed toward Apply (20.8%), language courses toward Understand (31.7%), and social-science courses toward Create (33.8%).
• The same students shift: within-person comparisons across disciplines showed significant reorganization of engagement profiles (p < .001) — the shift is not an artifact of different students enrolling in different fields.
• Variance sits with the course, not the person: in the crossed random-effects model, course context accounted for more of the variance in cognitive engagement than individual student style.
• Reliability is adequate but not high: individual-rater agreement (κ = .426–.606) indicates that Bloom coding of authentic dialogue is a genuinely difficult judgment; the best-pair consensus (κ = .753) is the figure the analyses rest on.

Implications

If cognitive engagement reorganizes with course context, then reporting a single engagement score per student — the common practice in learning analytics — describes a student–course pairing rather than a student. The practical consequence is that interventions aimed at "raising engagement" are better targeted at course and assessment design, which the data identify as the dominant source of variance, than at individual learners presumed to be low-engagement by disposition. Two limits bound the claim. Bloom level inferred from a typed question is a proxy for cognitive engagement, not a measurement of it, and the moderate individual-rater agreement means the construct boundary is genuinely fuzzy at the margins. The observed variation is also associational: disciplines differ in task structure, assessment format, and instructor practice simultaneously, so “discipline” here names a bundle of course conditions rather than an isolated causal factor.

Citation

@article{chang2026chat_as_learning,
  author  = {Chia-Kai Chang and Kuei-Hao Li},
  title   = {Chat as Learning: Student--{AI} Conversations as Discipline-Associated Cognitive Engagement Patterns},
  journal = {Computers and Education: Artificial Intelligence},
  year    = {2026},
  doi     = {10.1016/j.caeai.2026.100644},
  note    = {in press},
}