Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues
训练基于 LLM 的导师以改善对话中的学生学习成果

In this paper, we propose a novel approach to train a small, open-source LLM to not only follow effective pedagogical principles, but directly optimize for student learning outcomes. Our method performs three key steps. First, at each dialogue turn, we gather multiple candidate tutor utterances from a variety of sources, including human tutors and LLMs with varying styles and sizes. Second, we evaluate each candidate utterance on two aspects: (1) whether it elicits a correct response in the next student turn, using a trained student model for dialogues to predict student behavior, and (2) whether it adheres to a set of effective pedagogical principles, using GPT-4o in a LLM-as-a-judge evaluation setup. Third, we contrast good candidate utterances with poor ones, and fine-tune Llama 3.1 8B [10] with offline Reinforcement Learning (RL), specifically Direct Preference Optimization (DPO) [36]. We demonstrate that our optimized LLM tutor significantly increases the likelihood of next turn student response correctness, while reaching comparable pedagogical quality to that of a much larger, proprietary LLM, GPT-4o [32]. Through qualitative analysis and human evaluation, we confirm that our approach produces high-quality tutor utterances and reveal emergent tutoring strategies that arise from our training approach.
在本文中，我们提出了一种新颖的方法来训练一个小型开源 LLM，使其不仅遵循有效的教学原则，而且直接优化学生学习成果。我们的方法执行三个关键步骤。首先，在每个对话回合中，我们从各种来源收集多个候选导师话语，包括人类导师和具有不同风格和大小的 LLM。其次，我们从两个方面评估每个候选话语：(1) 它是否能在下一个学生回合中引出正确回答，使用训练好的对话学生模型来预测学生行为，以及(2) 它是否遵循一套有效的教学原则，使用在 LLM 作为裁判评估设置中的 GPT-4o。第三，我们将好的候选话语与差的表现进行对比，并使用离线强化学习（RL），特别是直接偏好优化（DPO）对 Llama 3.1 8B [10]进行微调。我们证明，我们优化的 LLM 导师显著增加了下一个回合学生回答正确性的可能性，同时达到了与一个远大的专有 LLM GPT-4o [32]相当的教学质量。 通过定性分析和人工评估，我们证实了我们的方法能够生成高质量的导师话语，并揭示了由我们的训练方法产生的涌现式辅导策略。

Report issue for preceding element

We acknowledge up front that the most significant limitation of our work is that we do not experiment with real students. Since access to students at the scale necessary for our work is beyond our capability, we use a simulated student model instead. While we believe that our work is a reasonable starting point to train LLM-based tutors to maximize student outcomes, future work with real students in the loop is highly important. Therefore, to facilitate further research, we publicly release our code and encourage researchers and practitioners with access to real-world tutoring dialogue settings to build on our work.
我们首先承认我们工作的最大局限性在于我们没有与真实学生进行实验。由于获取我们工作所需规模的学生资源超出了我们的能力范围，我们使用了一个模拟学生模型。虽然我们相信我们的工作是一个合理的起点，用于训练基于 LLM 的导师以最大化学生成果，但未来在真实学生参与下的工作非常重要。因此，为了促进进一步研究，我们公开发布了我们的代码，并鼓励能够接触到真实辅导对话环境的 researchers 和 practitioners 基于我们的工作进行开发。

Report issue for preceding element

We now detail the model we use for predicting student outcomes in dialogues. We leverage recent work on knowledge tracing (KT) in dialogues, LLMKT [38], an LLM-based model that predicts if a student will respond correctly to a tutor-posed task in their next turn, given dialogue history and the knowledge components (KCs) [6] embedded in a tutor turn. The model is highly accurate at predicting student correctness, achieving $0.76$ AUC on the MathDial test set, making it a reliable automated source for estimating student outcomes.
我们现在详细说明用于预测对话中学生结果的模型。我们利用了最近在对话知识追踪（KT）方面的研究，LLMKT [38]，这是一个基于 LLM 的模型，它根据对话历史和导师回合中嵌入的知识成分（KCs）[6]，预测学生是否会在其下一次回合中对导师提出的任务做出正确回应。该模型在预测学生正确性方面非常准确，在 MathDial 测试集上达到了 $0.76$ AUC，使其成为估计学生结果的可靠自动化来源。

Report issue for preceding element

In LLMKT, at the $m$-th dialogue turn for the student, the model analyzes the conversation history $t_{\leq m},s_{<m},\mathcal{C}_{m}$, which denote the tutor utterances up to and including this current turn, prior student turns, and the set of KCs involved in the tutor-posed task in the current turn, respectively. It estimates student knowledge levels on each KC, and combines these estimates to predict whether the student will respond to the tutor-posed task in the current turn correctly. Therefore, we leverage LLMKT as a student simulator and use it to evaluate whether a generated tutor utterance at a dialogue turn will elicit a correct response from the student.
在 LLMKT 中，当学生在第 $m$ 轮对话时，模型分析对话历史 $t_{\leq m},s_{<m},\mathcal{C}_{m}$ ，其中包含从当前轮次到这一轮次的导师话语、之前的学主话语以及当前轮次导师提出任务中涉及的认知知识（KC）集合。模型估计学生对每个 KC 的知识水平，并将这些估计值结合起来预测学生是否会正确回答当前轮次导师提出的问题。因此，我们利用 LLMKT 作为学生模拟器，并使用它来评估对话轮次中生成的导师话语是否能激发学生给出正确回答。

Report issue for preceding element

In addition to promoting correct student responses, we also hand-craft a set of rubric items for effective pedagogical principles, listed Table 1. We then employ GPT-4o to assess how well the generated tutor utterances align with the rubric. By incorporating pedagogical evaluation aspects rather than focusing solely on correctness prediction, our approach discourages oversimplification of tutor utterances and instead encourages utterances that provide meaningful guidance.
除了促进学生给出正确回答外，我们还手工制作了一套有效的教学原则评分标准，列于表 1。然后我们使用 GPT-4o 来评估生成的导师话语与评分标准的符合程度。通过结合教学评估方面而不是仅关注正确性预测，我们的方法避免了导师话语的过度简化，并鼓励提供有意义指导的话语。

Report issue for preceding element

Our evaluation draws inspiration from feedback assessment studies [19, 20, 44] and focuses on common errors made by LLMs when generating feedback for math problems [39]. The rubric evaluates generated tutor utterances on six granular items, each assigned a binary label, across three core aspects: Accuracy, Progress, and Guidance. Considering all aspects, we also have GPT-4o provide an overall score for the utterance on a 1-10 scale. Our prompt leverages chain-of-thought so that GPT-4o provides reasoning about the utterance before assigning its scores.
我们的评估借鉴了反馈评估研究[19, 20, 44]的思路，重点关注 LLMs 在为数学问题生成反馈时常见的错误[39]。评分标准从三个方面（准确性、进步性和指导性）对生成的导师话语进行评估，每个方面包含六个细粒度指标，每个指标都分配了二进制标签。综合考虑所有方面，我们还让 GPT-4o 在 1-10 的评分范围内给出话语的整体评分。我们的提示利用了思维链，以便 GPT-4o 在分配分数之前对话语进行推理。

Report issue for preceding element

We include the human tutor utterance from MathDial in our prompt as a point of comparison, mainly to assist with evaluating Accuracy, which can be challenging [39]. We find the 1-10 scale works slightly better than simply averaging all the binary rubric items, possibly because it enables GPT-4o to decide which rubric items are more relevant given the context of the dialogue.
我们将 MathDial 中的人类导师话语作为比较点包含在提示中，主要目的是协助评估准确性，这通常是一个挑战[39]。我们发现 1-10 的评分范围比简单平均所有二进制评分项的效果略好，可能是因为它使 GPT-4o 能够根据对话的上下文决定哪些评分项更相关。

Report issue for preceding element

To create a dataset of preference pairs for DPO training, we collect candidate tutor utterances from four sources: (1) the human tutor utterances from the MathDial dataset, (2) utterances generated by GPT-4o, where the evaluation criteria (rubric and intending to elicit correct student responses) are included in the prompt, (3) utterances generated by Llama 3.1 8B using a generic prompt about behaving like a math tutor (without the rubric), and (4) utterances generated by Llama 3.2 3B using the same generic prompt. In general, method (2) contains high-quality tutor utterances that serve as positive examples that score high under our rubric. However, we found that in many cases (1) performs better for eliciting correct student responses due to the concise nature of the utterances. Utterances generated by (3) are a mix of high and low quality, while utterances generated by (4) are typically lower quality and serve as a source of negative examples for preference optimization. Prior work has shown diverse candidate quality, particularly negative examples, to improve DPO performance [39, 49].
为创建用于 DPO 训练的偏好对数据集，我们从四个来源收集候选导师话语：(1) MathDial 数据集中的人类导师话语，(2) GPT-4o 生成的话语，其中评估标准（评分标准和旨在引出正确学生回答）包含在提示中，(3) 使用关于表现得像数学导师的通用提示生成的 Llama 3.1 8B 话语（不包含评分标准），以及(4) 使用相同通用提示生成的 Llama 3.2 3B 话语。通常，方法(2)包含高质量导师话语，这些话语在我们的评分标准下得分较高，可作为正例。然而，我们发现许多情况下(1)由于话语简洁，在引出正确学生回答方面表现更好。方法(3)生成的话语质量参差不齐，而方法(4)生成的话语通常质量较低，可作为偏好优化的负例来源。先前研究表明，多样化的候选质量，尤其是负例，可以提升 DPO 性能[39, 49]。

Report issue for preceding element

After evaluating each candidate using LLMKT and the rubric, we create a combined weighted score for a candidate tutor utterance at turn $m$:
在用 LLMKT 和评分标准评估每个候选后，我们为候选助教在回合 $m$ 的语句创建一个综合加权分数：

Report issue for preceding element

$\displaystyle s_{m}=\lambda\cdot y_{m}+(1-\lambda)\cdot r_{m},$

where $y_{m}$ is the probability of a correct student response predicted by LLMKT at this turn, and $r_{m}$ is the overall rubric score at this turn normalized in $[0,1]$. We adjust $\lambda$ to balance the tradeoff between how much the generated utterances elicit correct student responses compared to how much they follow pedagogical practice. By default, we set $\lambda=0.5$ to achieve a balance between these objectives; in our experiments, we show how varying $\lambda$ affects the balance between both.
其中 $y_{m}$ 是 LLMKT 在此回合预测的正确学生响应的概率， $r_{m}$ 是此回合的总体评分标准分数，已归一化到 $[0,1]$ 。我们调整 $\lambda$ 以平衡生成语句激发正确学生响应的程度与遵循教学实践的程度之间的权衡。默认情况下，我们设置 $\lambda=0.5$ 以在这两个目标之间取得平衡；在我们的实验中，我们展示了如何改变 $\lambda$ 会影响这两个目标之间的平衡。

Report issue for preceding element

We then use the score $s_{m}$ to construct preference pairs between candidate tutor utterances. We consider a candidate with score $s_{m}^{1}$ to be preferred over a candidate with score $s_{m}^{2}$ if the former is greater by some threshold, i.e., $s_{m}^{1}-s_{m}^{2}>\epsilon$. If the scores are within $\epsilon$, we do not form a preference pair. In practice, we set $\epsilon=0.1$ to achieve a balance where noisy preference pairs are excluded, but we retain enough data to sufficiently train the model, both of which have been shown to be important considerations for DPO [23].
我们随后使用分数 $s_{m}$ 来构建候选导师话语之间的偏好对。如果前者比后者高出某个阈值 $s_{m}^{1}-s_{m}^{2}>\epsilon$ ，我们认为分数为 $s_{m}^{1}$ 的候选优于分数为 $s_{m}^{2}$ 的候选。如果分数在 $\epsilon$ 范围内，则不形成偏好对。在实践中，我们设置 $\epsilon=0.1$ 以在排除噪声偏好对的同时保留足够的数据来充分训练模型，这两者已被证明是 DPO[23]中重要的考虑因素。

Report issue for preceding element

We train our model in a two-stage process: (1) distillation and (2) DPO. Distillation is a common way to enhance the capabilities of small LLMs by mimicking the behavior of much larger LLMs [52]. In our case, we fine-tune Llama 3.1 8B on candidate tutor utterances generated by GPT-4o. Through this distillation stage, we gain access to a local model that scores well on our pedagogical rubric.
我们的模型训练分为两个阶段：(1) 摹本和 (2) DPO。摹本是增强小型 LLM 能力的常用方法，通过模仿更大 LLM 的行为 [ 52]。在我们的案例中，我们在 GPT-4o 生成的候选导师话语上微调 Llama 3.1 8B。通过这个摹本阶段，我们获得了一个在我们的教学评估标准上得分良好的本地模型。

Report issue for preceding element

We then use DPO and our preference pairs to further steer the distilled model towards effective tutoring. DPO trains an LLM by contrasting outputs in a preference pair given the same input prompt, using the following objective:
然后我们使用 DPO 和我们的偏好对来进一步引导摹本模型朝向有效的教学。DPO 通过在相同输入提示下对比偏好对中的输出来训练 LLM，使用以下目标：

Report issue for preceding element

$$\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{(x,t^{w}_{m},t^{l}_{m})\sim\mathcal{D}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(t^{w}_{m}|x)}{\pi_{\text{ref}}(t^{w}_{m}|x)}-\beta\log\frac{\pi_{\theta}(t^{l}_{m}|x)}{\pi_{\text{ref}}(t^{l}_{m}|x)}\right)\right],$$

where $t^{w}_{m}$ and $t^{l}_{m}$ represent the preferred and unpreferred tutor utterances, respectively. $x$ represents the input prompt, comprising an instruction, the context of the dialogue, and the dialogue history $t_{<m},s_{<m}$. $\pi_{\theta}$ denotes the model being trained, $\pi_{\text{ref}}$ represents a frozen reference model, and $\beta$ is a hyperparameter that controls the divergence between the learned and reference policies.
其中 $t^{w}_{m}$ 和 $t^{l}_{m}$ 分别代表偏好和不受偏好的导师话语。 $x$ 代表输入提示，包括指令、对话上下文和对话历史 $t_{<m},s_{<m}$ 。 $\pi_{\theta}$ 表示正在训练的模型， $\pi_{\text{ref}}$ 代表一个冻结的参考模型， $\beta$ 是一个控制学习策略和参考策略之间差异的超参数。

Report issue for preceding element

In this work, we use the distilled model as the reference model $\pi_{\text{ref}}$ and for initializing the weights of $\pi_{\theta}$. We find the distilled model works much better than using the base Llama model, since the distilled model already performs better on the rubric. Additionally, we set $\beta=0.1$, a relatively low value for DPO compared to a common value of $\beta=0.5$. The low $\beta$ value allows $\pi_{\theta}$ to diverge more from $\pi_{\text{ref}}$, which we find necessary to increase LLMKT’s predictions of eliciting correct student responses in the next dialogue turn.
在这项工作中，我们使用蒸馏模型作为参考模型 $\pi_{\text{ref}}$ ，并使用它来初始化权重 $\pi_{\theta}$ 。我们发现蒸馏模型比使用基础 Llama 模型效果要好得多，因为蒸馏模型在评分标准上已经表现更好。此外，我们设置了 $\beta=0.1$ ，这是一个相对于常见值 $\beta=0.5$ 来说相对较低的低 DPO 值。低 $\beta$ 值允许 $\pi_{\theta}$ 更多地偏离 $\pi_{\text{ref}}$ ，我们发现这对于增加 LLMKT 在下一轮对话中预测学生正确回答的准确性是必要的。

Report issue for preceding element

We evaluate our framework using the MathDial dataset [29], which consists of tutoring dialogues focused on mathematics problems from GSM8K [5]. Each dialogue is centered around an incorrect student solution to the math problem, and the goal of the dialogue is for the tutor to guide the student to the correct solution by addressing their misconceptions. Tutors are role-played by crowd workers, while students are simulated by GPT-3.5. Despite only being half-authentic, MathDial is the largest publicly available one-on-one tutor-student math dialogue dataset to the best of our knowledge. To estimate correctness with LLMKT, which requires knowledge component labels at the current turn, we use the annotated knowledge components from the Dialogue KT version of the dataset [38] and filter out unlabeled turns.
我们使用 MathDial 数据集[29]评估我们的框架，该数据集包含专注于数学问题的辅导对话，源自 GSM8K[5]。每个对话都围绕学生错误的数学问题解决方案展开，对话的目标是辅导员通过纠正学生的误解来引导学生找到正确答案。辅导员由众包工作者扮演，学生由 GPT-3.5 模拟。尽管只有半真实性，但据我们所知，MathDial 是目前最大公开的一对一辅导员-学生数学对话数据集。为了使用 LLMKT 估计当前回合的正确性（该评估需要当前回合的知识成分标签），我们使用数据集的 Dialogue KT 版本[38]中标注的知识成分，并过滤掉未标注的回合。

Report issue for preceding element

We follow the original MathDial train/test split. After filtering, our test set has 588 dialogues with 3,101 tutor turns. We split the train set into a 80/20 train/validation split at the dialogue-level, resulting in 1,809/453 dialogues with 11,058/2,811 tutor turns. When creating our overgenerated tutor turn dataset for distillation and DPO training, we take a subset of the train/validation split to reduce labeling costs, resulting in 483/135 dialogues with 3,080/920 GPT-4o-generated tutor utterances for distillation and 9,662/3,095 preference pairs, with our default parameters of $\lambda=0.5$ and $\epsilon=0.1$, for DPO.
我们遵循原始的 MathDial 训练/测试划分。经过筛选后，我们的测试集包含 588 个对话，共 3,101 个导师回合。我们将训练集在对话级别上按 80/20 的比例划分为训练集和验证集，结果得到 1,809/453 个对话，分别包含 11,058/2,811 个导师回合。在创建用于蒸馏和 DPO 训练的过度生成的导师回合数据集时，我们选取训练/验证划分的一个子集以降低标注成本，最终得到 483/135 个对话，其中包含 3,080/920 个用于蒸馏的 GPT-4o 生成的导师语句，以及 9,662/3,095 个偏好对，使用我们的默认参数 $\lambda=0.5$ 和 $\epsilon=0.1$ 进行 DPO。

Report issue for preceding element

We compare our preference-optimized LLM, which we refer to as DPO, with Llama 3.1 8B as the base model, against the following baselines: the Base model of Llama 3.1 8B prompted with our evaluation criteria (rubric and intending to elicit correct student responses); its supervised fine-tuned version on human tutor utterances in the original dataset, SFT; a fine-tuned version distilled from GPT-4o-generated utterances detailed above in Section 3.4, Distill; GPT-4o, the large, proprietary LLM, which is prompted with our evaluation criteria; and finally, the Human Tutor utterances from the original MathDial dataset.
我们比较了我们的偏好优化 LLM，我们称之为 DPO，与 Llama 3.1 8B 作为基础模型，与以下基线进行对比：使用我们的评估标准（评分表和旨在引出正确学生回答）提示的 Llama 3.1 8B 基础模型；在原始数据集中人类导师话语上的监督微调版本，SFT；从第 3.4 节中详细说明的 GPT-4o 生成的话语蒸馏出的微调版本，Distill；GPT-4o，大型专有 LLM，使用我们的评估标准进行提示；最后，原始 MathDial 数据集中的人类导师话语。

Report issue for preceding element

We evaluate tutor utterances on both student outcomes and pedagogical principles. Student outcome prediction uses LLMKT to estimate the probability of a correct student response, averaged across turns. Evaluating how well the utterance follows the pedagogical principles will be reported in the same order as in Table 1, reporting scores for Acc. (Accuracy), Prog. (Progress), Err. (Error Identification), Hint (Strategic Hinting), Wth. (Withholding), and Enc. (Encouraging), along with an Overall score.
我们评估导师话语对学生结果和教学原则的影响。学生结果预测使用 LLMKT 来估计正确学生回答的概率，并在回合间取平均值。评估话语遵循教学原则的程度将按照表 1 的顺序报告，报告 Acc.（准确率）、Prog.（进步）、Err.（错误识别）、Hint（策略性提示）、Wth.（保留）和 Enc.（鼓励）的分数，以及总体分数。

Report issue for preceding element

We use the meta-llama/Meta-Llama-3.1-8B-Instruct model from Hugging Face [11] for all local experiments. To adapt the model, we use Low-Rank Adaptation (LoRA) [18] with a rank parameter of $r=64$, scaling factor $\alpha=32$, and a dropout rate of 0.05. We train using the AdamW optimizer, with a learning rate of $1\times 10^{-4}$ for distillation and $3\times 10^{-5}$ for DPO, with a linear warmup phase for the first $10\%$ of training steps. We use an effective batch size of $64$ with gradient accumulation, set weight decay to $1\times 10^{-2}$, and set a gradient clipping maximum norm of $1.0$. We evaluate the loss on the validation set after each epoch, and achieve the minimum validation loss after three epochs for distillation and after one epoch for DPO. At test time, we generate tutor utterances using greedy decoding. We conduct all experiments on NVIDIA RTX A6000 GPUs.
我们使用 Hugging Face 的 meta-llama/Meta-Llama-3.1-8B-Instruct 模型 [ 11] 进行所有本地实验。为了适配模型，我们使用低秩适配（LoRA）[ 18]，设置秩参数为 $r=64$ ，缩放因子为 $\alpha=32$ ，以及 dropout 率为 0.05。我们使用 AdamW 优化器进行训练，蒸馏阶段的学习率为 $1\times 10^{-4}$ ，DPO 阶段的学习率为 $3\times 10^{-5}$ ，并在训练的前 $10\%$ 个步骤中进行线性预热。我们使用 $64$ 的有效批大小，并采用梯度累积，将权重衰减设置为 $1\times 10^{-2}$ ，梯度裁剪的最大范数为 $1.0$ 。每轮训练后，我们在验证集上评估损失，蒸馏阶段在三轮训练后达到最小验证损失，DPO 阶段在第一轮训练后达到最小验证损失。测试时，我们使用贪婪解码生成助教话语。所有实验均在 NVIDIA RTX A6000 GPU 上进行。

Report issue for preceding element

Table 2 shows the results of tutor utterance generation on the MathDial test set, across all methods on the automated metrics. We observe that DPO significantly outperforms all other methods on student correctness prediction, improving over the next best method, GPT-4o, by $33\%$. This result shows that DPO training is necessary to generate tutor utterances that are more likely to achieve correct student responses, which cannot be achieved simply through prompting. Additionally, DPO achieves similar scores to GPT-4o on the pedagogical rubric, with both methods scoring very high on all rubric items and the overall score. This result shows that our DPO training pipeline can effectively bring the pedagogical ability of small, open-source LLMs with only 8B parameters to the level of very large, proprietary LLMs. Moreover, we see that DPO improves over distillation on almost all metrics, showing that DPO is necessary to achieve both objectives.
表 2 展示了在 MathDial 测试集上，所有方法在自动指标方面的导师话语生成结果。我们观察到 DPO 在学生正确性预测方面显著优于所有其他方法，比下一最佳方法 GPT-4o 提高了 $33\%$ 。这一结果表明，DPO 训练对于生成更有可能获得学生正确回答的导师话语是必要的，而这不是单纯通过提示就能实现的。此外，DPO 在教学法评分标准上的得分与 GPT-4o 相似，两种方法在所有评分标准项目及总分上均得分非常高。这一结果表明，我们的 DPO 训练流程能够有效将仅含 8B 参数的小型开源 LLM 的教学能力提升至大型专有 LLM 的水平。此外，我们看到 DPO 在几乎所有指标上都优于蒸馏，这表明 DPO 对于实现两个目标都是必要的。

Report issue for preceding element

A notable observation is that human-written tutor utterances, while almost always accurate, score relatively low on most other rubric items. This result is not surprising since human tutors do not act according to our evaluation rubric; many of the human utterances are relatively short and simply ask the student to retry without giving guidance, and occasionally feature the tutor getting frustrated with the student. As a result, the SFT model, trained on human tutor utterances, does not score high on our rubric; the distilled model and even the base Llama model significantly outperform it on the rubric items.
一个显著的观察是，人类编写的导师话语虽然几乎总是准确的，但在大多数其他评估标准项上的得分相对较低。这个结果并不令人意外，因为人类导师并不按照我们的评估标准行事；许多人类话语相对简短，仅仅要求学生重试而不提供指导，并且偶尔会表现出导师对学生感到沮丧。因此，在人类导师话语上训练的 SFT 模型在我们的评估标准上得分不高；蒸馏模型甚至基础 Llama 模型在评估标准项上显著优于它。

Report issue for preceding element

Student Outcomes vs. Pedagogical Principles We investigate how adjusting the value of $\lambda$ can balance the two goals of maximizing student outcomes under LLMKT and following pedagogical principles. We vary the value of $\lambda\in\{0.00,0.25,0.50,0.75,1.00\}$, where larger values attribute more weight to the student correctness prediction objective, and evaluate on a subset of the data with 500 turn pairs. Figure 2 shows the result of this experiment. We see that, as expected, increasing $\lambda$ from $0$ to $1$ generally increases student correctness prediction performance while generally decreasing the rubric score. Practitioners can get their desired balance by changing $\lambda$; for example, a value of around $0.5$ to $0.75$ can maintain a relatively high score on the pedagogical rubric while significantly improving performance on soliciting correct responses from students. Perhaps surprisingly, changing $\lambda$ from $0.75$ to $1.0$ does not increase student outcome performance, while decreasing $\lambda$ from $0.25$ to $0.0$ decreases pedagogical performance. This result implies that a small reward signal from correctness prediction may implicitly help pedagogical effectiveness, and vice versa.
学生成果与教学原则 我们研究了如何调整 $\lambda$ 的值来平衡 LLMKT 下最大化学生成果和遵循教学原则的两个目标。我们改变 $\lambda\in\{0.00,0.25,0.50,0.75,1.00\}$ 的值，其中较大的值将更多权重赋予学生正确性预测目标，并在包含 500 对回合的子集数据上进行评估。图 2 显示了该实验的结果。我们看到，正如预期的那样，将 $\lambda$ 从 $0$ 增加到 $1$ 通常会提高学生正确性预测性能，同时通常会降低评分标准分。从业者可以通过改变 $\lambda$ 来获得他们期望的平衡；例如， $0.5$ 到 $0.75$ 的值可以在保持教学评分标准相对较高分数的同时显著提高从学生那里获取正确回答的性能。或许令人惊讶的是，将 $\lambda$ 从 $0.75$ 改为 $1.0$ 并没有提高学生成果性能，而将 $\lambda$ 从 $0.25$ 减少到 $0.0$ 则降低了教学性能。这一结果表明，来自正确性预测的小奖励信号可能隐含地有助于教学效果，反之亦然。

Report issue for preceding element

We conduct a qualitative analysis of generated tutor utterances to investigate the strengths and weaknesses of our method, how it compares to baselines, and what types of patterns emerge in the model-generated text.
我们对生成的导师话语进行定性分析，以研究我们方法的优缺点、它如何与基线比较，以及模型生成文本中出现的模式类型。

Report issue for preceding element

Table 3 shows an example dialogue context and tutor utterances from the human tutor, GPT-4o, and DPO. We see that the human tutor states that the student is not correct and prompts the student to try again, providing a subtle hint to focus on the cost of the dozen. However, this utterance neither gives LLMKT, the student model, any confidence in the student being able to answer correctly, nor follows strong pedagogical principles. GPT-4o guides the student toward correctly calculating the profit, but directly provides half of the solution in its hint, limiting deep student engagement. It also poses the challenging task of computing the total profit, which requires multiple calculation steps. Since the student is still struggling with unit conversion, LLMKT is not confident that the student will respond correctly. In contrast, our DPO-trained model provides an actionable hint for the student, following the rubric. It also poses the simpler task of asking the student to first find the cost of a half dozen, which is necessary for later steps. As a result, LLMKT is confident the student can respond correctly.
表 3 展示了人类导师、GPT-4o 和 DPO 提供的对话上下文和导师话语示例。我们看到人类导师指出学生回答不正确，并提示学生再试一次，同时巧妙地暗示学生应关注一打的成本。然而，这句话既没有给 LLMKT（学生模型）任何学生能够正确回答的信心，也没有遵循强烈的教学习惯。GPT-4o 引导学生正确计算利润，但在提示中直接提供了半个解决方案，限制了学生的深度参与。它还提出了一个计算总利润的挑战性任务，需要多个计算步骤。由于学生仍在挣扎于单位转换，LLMKT 并不确定学生能正确回答。相比之下，我们训练的 DPO 模型为学生提供了可操作的提示，并遵循评分标准。它还提出了一个更简单的任务，要求学生首先找出半打的成本，这是后续步骤所必需的。因此，LLMKT 确信学生能够正确回答。

Report issue for preceding element

Overall, we find that our DPO-trained model excels at posing concrete tasks that are nontrivial, but more feasible for students compared to GPT-4o. We observe that the behavior of posing tasks this way is the primary way of achieving higher predictions for student correctness, and appears to be an emergent behavior of training on the LLMKT-derived scores. While the tasks sometimes only ask the student to perform simple arithmetic, they are typically more involved. For example, in another dialogue, the model asks “How long does it take to download 200 GB at a rate of 2 GB/minute?” This task requires the student to set up a single variable equation and understand the roles of the given values. We also find that the DPO-trained model is more likely to ask questions, whereas other methods often directly tell the student to perform a task.
总的来说，我们发现经过 DPO 训练的模型在提出具体任务方面表现出色，这些任务比 GPT-4o 提出的任务更非平凡，但对学生来说更可行。我们观察到，以这种方式提出任务的行为是提高学生正确率预测的主要方式，并且似乎是训练在 LLMKT 衍生分数上的涌现行为。虽然这些任务有时只要求学生进行简单的算术运算，但通常更复杂。例如，在另一个对话中，模型会问“下载 200GB 数据以每分钟 2GB 的速度需要多长时间？”这个任务要求学生建立单一变量方程并理解给定值的角色。我们还发现，DPO 训练的模型更倾向于提问，而其他方法通常直接告诉学生执行任务。

Report issue for preceding element

However, both LLMs still have some limitations, particularly when students give partially correct responses. In these cases, the models may ignore the progress the student has made and focus on errors that are irrelevant to the recent student response. Correctly identifying errors in math with LLMs has been shown to be challenging in prior work as well [12, 27], indicating additional techniques may be needed to fix this behavior.
然而，LLMs 仍然存在一些局限性，尤其是在学生给出部分正确回答的情况下。在这些情况下，模型可能会忽略学生已经取得的进步，而专注于与最近学生回答无关的错误。先前的研究表明，使用 LLMs 正确识别数学错误也是一项挑战[12, 27]，这表明可能需要额外的技术来纠正这种行为。

Report issue for preceding element

We also conduct a human evaluation to further assess different tutor models and how our automated metrics align with human judgment.
我们也进行人工评估，以进一步评估不同的导师模型，以及我们的自动化指标与人类判断的一致性。

Report issue for preceding element

We recruit two independent volunteer university students to annotate tutor utterances according to our evaluation metrics. We randomly sample 10 dialogues from the test set after filtering based on toxicity and low quality ground-truth tutor utterances. The annotators evaluate 5 consecutive tutor turns from each dialogue, resulting in a total of 50 evaluation instances. At each instance, we show participants the dialogue so far and next tutor utterance candidates generated by (1) human tutors, (2) GPT-4o, and (3) our DPO-trained model, in random order without revealing the method.
我们招募了两位独立的大学生志愿者，根据我们的评估指标对导师的发言进行标注。在过滤掉有毒性和低质量的真实导师发言后，我们从测试集中随机抽取了 10 个对话。标注者评估每个对话中连续的 5 个导师发言，总共产生 50 个评估实例。在每个实例中，我们向参与者展示到目前为止的对话以及由（1）人类导师、（2）GPT-4o 和（3）我们训练的 DPO 模型生成的下一个导师发言候选，以随机顺序显示，不透露方法。

Report issue for preceding element

For student outcome prediction, annotators rank the three candidate utterances based on how likely they think each one will lead to a correct student response. We ask annotators to take into account both the task posed in the turn and the student’s knowledge demonstrated in prior turns. We use rankings instead of absolute values since the latter may be harder for humans to calibrate. For pedagogical principles, we simply ask annotators to follow the same evaluation rubric outlined in Table 1 and provide an overall score on a 1-10 scale.
对于学生表现预测，标注者根据他们认为每个候选话语可能引出正确学生回答的可能性对三个候选话语进行排序。我们要求标注者同时考虑该回合中提出的问题以及学生在先前回合中展现的知识。我们使用排序而不是绝对值，因为后者可能更难让人校准。对于教学原则，我们仅要求标注者遵循表 1 中概述的相同评估标准，并在 1-10 的量表上给出总体评分。

Report issue for preceding element

Results Table 4 shows average student correctness ranks (rank $1$ means top-ranked) and rubric scores from human annotators. We see that DPO outperforms both human tutors and GPT-4o on both metrics, with all differences statistically significant ($p<0.05$) according to a paired t-test. Higher correctness ranks show that DPO learns how to pose tasks that are manageable to students, while the higher rubric score shows that it does so while maintaining pedagogical principles. Additionally, the higher rubric score may imply that the correctness objective also indirectly improves pedagogical quality. While DPO outperforming GPT-4o on the rubric is contrary to the automated results in Table 2, the automated results may reflect self-bias from GPT-4o as the evaluator [50].
结果表 4 显示了学生正确性排名（排名 $1$ 表示排名第一）和人类标注者的评分标准分数。我们看到 DPO 在两个指标上均优于人类导师和 GPT-4o，且所有差异根据配对 t 检验具有统计学意义（ $p<0.05$ ）。更高的正确性排名表明 DPO 学会了如何提出对学生来说可管理的任务，而更高的评分标准分数表明它在这样做时仍坚持教学原则。此外，更高的评分标准分数也可能意味着正确性目标也间接提高了教学质量。虽然 DPO 在评分标准上优于 GPT-4o 与表 2 中的自动化结果相反，但自动化结果可能反映了作为评估者的 GPT-4o 的自我偏见[50]。

Report issue for preceding element

Since these evaluation tasks can be highly subjective, we also investigate the inter-rater agreement between the two human annotators and the automated metrics. We use Kendall’s $\tau$ [21] for correctness ranks and Pearson’s correlation coefficient $\rho$  [25] for rubric scores. For correctness ranks, Kendall’s $\tau$ is only $0.06$ between the two annotators and averages $0.08$ between LLMKT and both annotators. This low agreement is not surprising due to the subjective nature of the task; we find that there are many near-ties between different candidate utterances according to LLMKT’s output probabilities. For rubric scores, the inter-rater agreement is notably higher: Pearson’s $\rho$ is $0.15$ between the two annotators and averages $0.27$ between GPT-4o and both annotators, the latter being statistically significant. This result suggests that evaluating pedagogical principles is much easier than predicting student correctness, though the former is still subjective, since annotators may value certain rubric items differently.
由于这些评估任务可能具有高度主观性，我们还研究了两位人类标注者和自动指标之间的评分者间一致性。我们使用肯德尔τ系数 $\tau$ [ 21]评估正确性排序，使用皮尔逊相关系数 $\rho$ [ 25]评估评分标准得分。对于正确性排序，肯德尔τ系数 $\tau$ 在两位标注者之间的相关性仅为 $0.06$ ，而与 LLMKT 的平均相关性为 $0.08$ 。由于任务的主观性，这种低一致性并不令人意外；我们发现根据 LLMKT 的输出概率，不同候选话语之间存在许多接近平局的情况。对于评分标准得分，评分者间一致性明显更高：皮尔逊相关系数 $\rho$ 在两位标注者之间的相关性为 $0.15$ ，而与 GPT-4o 的平均相关性为 $0.27$ ，后者具有统计学意义。这一结果表明，评估教学原则比预测学生正确性要容易得多，尽管前者仍然是主观的，因为标注者可能对不同的评分标准项有不同的评价。

Report issue for preceding element

Overall, despite showing some promise through preliminary evaluation, it is important to deploy our trained tutor LLMs to real students and test whether they actually lead to good student learning outcomes.
总体而言，尽管初步评估显示出一些潜力，但将我们训练的导师 LLMs 部署给真实学生，并测试它们是否确实能带来良好的学生学习成果，这一点非常重要。

Report issue for preceding element