A Novel Approach to Scalable and Automatic Topic-Controlled Question Generation in Education
一种用于教育领域的可扩展和自动主题控制问题生成的新方法

Although language models have been employed for question generation, their application in educational settings has only recently begun to be systematically explored with a heavy focus on the potential practical applications of proprietary models (e.g. GPT models’ prompt engineering and RAG applications for question generation). While existing research in relevant academic communities with a more technical focus explores generating questions from descriptive texts (Du et al., 2017; Wang et al., 2018), the task remains highly complex and there is less focus on the educational value of the generated questions in evaluations. Context plays a crucial role in the educational value of EdQG, yet much existing work has focused primarily on generating questions from sentences, paragraphs, or structured data in isolation (Hu et al., 2018; Lopez et al., 2021), with limited attention given to topic-controlled question generation in a given context.
尽管语言模型已被用于生成问题，但它们在教育环境中的应用直到最近才开始系统地探索，重点主要集中在专有模型的潜在实际应用上（例如 GPT 模型的提示工程和用于问题生成的 RAG 应用）。而相关学术领域内更侧重技术的研究则探索从描述性文本中生成问题（Du 等人，2017；Wang 等人，2018），但这项任务仍然非常复杂，且在评估中较少关注生成问题的教育价值。上下文在 EdQG（教育问题生成）的教育价值中起着至关重要的作用，然而大量现有工作主要集中于独立地从句子、段落或结构化数据中生成问题（Hu 等人，2018；Lopez 等人，2021），对在特定上下文中进行主题控制的问题生成关注有限。

Topic-controlled question generation takes a target topic in addition to the descriptive text as context into account while generating the models’ outputs. On the other hand, traditional approaches which take sentences or paragraphs as inputs without contextual topic-control tend to generate questions that arbitrarily combine or select concepts and topics which are likely to be of limited practical value to professionals like teachers. From the learners’ points of view, prior research also suggests a strong correlation between the personalisation of testing and knowledge retention (Bahrick et al., 1993), which further supports the importance of topic-controlled question generation. Developing comprehensive, high quality and relevant educational question sets across different topics can significantly enhance teaching practice and support students through intelligent tutoring systems that provide personalised learning to diverse learners.
主题控制式问题生成在生成模型输出时会考虑目标主题以及描述性文本作为上下文。另一方面，传统方法在不进行主题控制的情况下将句子或段落作为输入，往往生成的问题会任意组合或选择概念和主题，这些内容对教师等专业人士可能缺乏实际价值。从学习者的角度来看，已有研究表明测试的个性化与知识保留之间存在强相关性（Bahrick 等人，1993 年），这进一步支持了主题控制式问题生成的重要性。开发涵盖不同主题的全面、高质量且相关的教育问题集，可以显著提升教学实践，并通过为不同学习者提供个性化学习的智能辅导系统来支持学生。

In the scope of this work, we define topic-controlled question generation (T-CQG) as follows. Let us suppose a learner $\ell$ has already consumed learning materials that contain the knowledge context $c$ containing various topics $T_{c}$. A goal of a teacher or an intelligent system is then to generate a question $\hat{q}_{t}$, where $\hat{q}_{t}$ is an educational question about the target topic $t$, where $t\in T_{c}$, and $\hat{q}_{t}$ consists of a sequence of tokens $q_{t}\in\{w_{1},\dots,w_{|q_{t}|}\}$ of arbitrary length $|q_{t}|$. This task requires that the question is not only contextually relevant to the paragraph context ${c}$, but also closely aligned with the thematic focus defined by topic $t$. The probability $p(q_{t}|c,t)$ incorporates the coherence and relevance of each token in the sequence, rendering the generation process highly sensitive to both the context and the topic. This task can be mathematically defined to identify the optimal question $\hat{q_{t}}$ that maximises the conditional probability as per equation 1.
在本工作的范围内，我们将主题控制问题生成（T-CQG）定义如下。假设学习者 $\ell$ 已经消费了包含知识背景 $c$ 的学习材料，其中包含各种主题 $T_{c}$ 。教师或智能系统的目标则是生成一个问题 $\hat{q}_{t}$ ，其中 $\hat{q}_{t}$ 是一个关于目标主题 $t$ 的教育问题， $t\in T_{c}$ ，而 $\hat{q}_{t}$ 由任意长度的标记序列 $q_{t}\in\{w_{1},\dots,w_{|q_{t}|}\}$ 组成 $|q_{t}|$ 。这项任务要求问题不仅与段落背景 ${c}$ 在语境上相关，而且与主题 $t$ 定义的主题焦点紧密一致。概率 $p(q_{t}|c,t)$ 包含了序列中每个标记的连贯性和相关性，使得生成过程对语境和主题都非常敏感。这项任务可以通过数学定义来识别能够最大化条件概率的最优问题 $\hat{q_{t}}$ ，如方程 1 所示。

where, $p({q_{t}}|{c},{t})$ denotes the conditional probability that also depends on the tokens $w\in q_{t}$.
其中， $p({q_{t}}|{c},{t})$ 表示的条件概率也取决于 $w\in q_{t}$ 标记。

Question Generation (QG) involves automatically generating questions from a specific text passage or a document. The main goal of QG is to produce questions that are not only syntactically and semantically correct but also contextually relevant and meaningful for the intended use. There has been a growing use of computational models to generate contextually relevant and grammatically correct questions (Wang et al., 2024). In educational contexts specifically, QG has been implemented in various systems including intelligent tutoring systems (Yadav et al., 2023), writing support systems (Pinto et al., 2023), and knowledge assessment platforms (Kuo et al., 2023).
问题生成（QG）涉及从特定文本段落或文档中自动生成问题。QG 的主要目标是产生不仅语法和语义正确，而且对预期用途具有上下文相关性和意义的问题。近年来，计算模型被越来越多地用于生成上下文相关且语法正确的问题（Wang 等人，2024）。在教育领域，QG 已被应用于各种系统，包括智能辅导系统（Yadav 等人，2023）、写作支持系统（Pinto 等人，2023）和知识评估平台（Kuo 等人，2023）。

Existing research categorises QG into two types: answer-aware and answer-agnostic (Zhang et al., 2021). In answer-aware QG, the target answer is predetermined, and questions are generated to correspond with this answer within the given text context. On the other hand, answer-agnostic QG does not provide the target answer to the language model, allowing for more open-ended question generation which are considered to be educationally more valuable. However, answer-agnostic QG is a more challenging task for NLP research. Early research in answer-agnostic QG relied heavily on rule-based techniques that required experienced educators to develop rules that could convert declarative sentences into interrogative forms (Heilman and Smith, 2010; Adamson et al., 2013). These methods, while effective, are labour-intensive and time-consuming, demanding significant manual effort in creating high-quality, handcrafted rules (Chen et al., 2021), which inherently limits their scalability and diversity in question generation. These limitations led more recent research investigations to focus on data-driven neural network (NN) approaches.
现有研究将问题生成（QG）分为两种类型：答案感知型与答案非感知型（Zhang 等人，2021）。在答案感知型 QG 中，目标答案预先确定，问题生成是为了在给定文本语境中与该答案相对应。另一方面，答案非感知型 QG 不向语言模型提供目标答案，允许生成更开放的问题，这些问题被认为在教育上更有价值。然而，答案非感知型 QG 对自然语言处理（NLP）研究来说是一项更具挑战性的任务。早期的答案非感知型 QG 研究高度依赖基于规则的技巧，这些技巧需要经验丰富的教育者开发规则，将陈述句转换为疑问句形式（Heilman 和 Smith，2010；Adamson 等人，2013）。虽然这些方法有效，但它们劳动密集且耗时，需要大量人工工作来创建高质量的手工规则（Chen 等人，2021），这本质上限制了它们在问题生成中的可扩展性和多样性。这些局限性促使近期的研究更多地关注数据驱动的神经网络（NN）方法。

Early implementations of QG practices with data-driven approaches predominantly utilized sequence-to-sequence (seq2seq) architectures incorporating Recurrent Neural Networks (RNNs) (Du et al., 2017). More recently, the focus shifted towards employing end-to-end techniques facilitated by deep neural networks (Zhang et al., 2021). For instance, (Dathathri et al., 2020) and (Khalifa et al., 2021) utilised GPT-2 combined with either an attribute classifier or training another autoregressive language model to guide the generated text towards a topic. However, these approaches typically generated content that is too broadly categorised (such as a category being ’science’), failing to achieve the level of topic specificity required for them to be of real-world value for educational practitioners. On the other hand, a more targeted approach by (Hu et al., 2018) employing an LSTM model equipped with a pre-decoding mechanism, demonstrated the ability to generate questions on detailed topics. This model, though promising in its specificity, was only applied at the sentence level, limiting its utility for broader educational applications. More contemporary models leverage pre-trained transformers like GPT (Blobstein et al., 2023; Elkins et al., 2024) (Decoder Only) and T5 (Text-to-Text Transfer Transformer) (Vachev et al., 2022; Bulathwela et al., 2023) (Endoder-Decoder). These advanced NLP approaches like transformer architectures have shown to be effective in generating coherent and relevant questions for specified texts (Faraby et al., 2024). However, their use in meaningful and relevant educational question generation needs further explorations and evaluations in educational contexts (Bulathwela et al., 2023; Vachev et al., 2022). In short, the problem of topic-specific question generation as scoped in section 2.1 has been of interest to multiple researchers in the past, yet it is still an open challenge for the community.
早期采用数据驱动方法进行 QG 实践主要使用了包含循环神经网络（RNNs）的序列到序列（seq2seq）架构（Du 等人，2017）。最近，研究重点转向采用深度神经网络支持端到端技术（Zhang 等人，2021）。例如，（Dathathri 等人，2020）和（Khalifa 等人，2021）利用 GPT-2 结合属性分类器或训练另一个自回归语言模型来引导生成文本朝向特定主题。然而，这些方法通常生成过于宽泛分类的内容（例如一个类别为“科学”），未能达到所需的主题特异性，因此对于教育实践者而言缺乏实际应用价值。另一方面，（Hu 等人，2018）采用配备预解码机制的 LSTM 模型进行更具针对性的方法，展示了生成详细主题问题的能力。尽管该模型在特异性方面具有前景，但仅应用于句子级别，限制了其在更广泛教育应用中的实用性。 更多现代模型利用预训练的转换器，如 GPT（Blobstein 等人，2023 年；Elkins 等人，2024 年）（仅解码器）和 T5（文本到文本迁移转换器）（Vachev 等人，2022 年；Bulathwela 等人，2023 年）（编码器-解码器）。这些先进的自然语言处理方法，如转换器架构，已被证明在为指定文本生成连贯且相关的问题方面是有效的（Faraby 等人，2024 年）。然而，它们在教育环境中生成有意义且相关的问题方面的应用需要进一步探索和评估（Bulathwela 等人，2023 年；Vachev 等人，2022 年）。简而言之，如第 2.1 节所述的特定主题问题生成问题，过去一直引起多位研究人员的兴趣，但它仍然是一个开放的挑战。

One of the significant challenges in research utilising pre-trained transformer architectures for the scoped problem is the issue of making generated content more specifically aligned with the particular topics studied and its contextual considerations. Previous literature in AI in Education research proposed multiple approaches when linking knowledge components of topics to generated learning materials such as questions. The most common approach is expert human labelling, but it is challenging to be scaled even though its accuracy is unmatched (Yudelson et al., 2013) and considered as gold-standard. Due to the scaling challenges of expert human labelling, recent works have also proposed methods such as entity linking (Brank et al., 2017; Ferragina and Scaiella, 2010) that provide scalability even if it tends to sacrifice some accuracy. Another proposed approach is the so-called ”Wikification” which is the practice of using Wikipedia as a source for semantic annotations (Zhang and Rettinger, 2014). The approach has demonstrated significant advancements in recent years and offers considerable potential for automatically extracting concepts from Wikipedia entries to generate topic-specific educational materials (Bulathwela et al., 2021). Additionally, since there has been extensive research on Wikipedia for its potential for semantic labelling of AI-generated content, its concept relatedness metrics that are based on its link structure, page co-occurrence etc. (Ponza et al., 2020) are well developed and can represent semantic relatedness between Wikipedia concepts to a high accuracy. However, the use of approaches that allow scalable solutions such as Wikification (Brank et al., 2017) in educational question generation models is yet to be explored in detail in learning analytics.
在利用预训练的 Transformer 架构进行特定领域问题的研究中，一个重要的挑战是如何使生成内容更精确地与所研究的具体主题相一致，并考虑其上下文因素。AI 教育研究领域的先前文献在将主题的知识组件与生成式学习材料（如问题）联系起来时，提出了多种方法。最常见的方法是专家人工标注，尽管其准确率无与伦比（Yudelson 等，2013 年）并被视为黄金标准，但这种方法难以扩展。由于专家人工标注的扩展挑战，近期的研究也提出了诸如实体链接（Brank 等，2017 年；Ferragina 和 Scaiella，2010 年）等方法，这些方法虽然倾向于牺牲一些准确率，但提供了可扩展性。另一种提出的方法是所谓的“维基化”，即使用维基百科作为语义标注的来源（Zhang 和 Rettinger，2014 年）。 该方法近年来取得了显著进展，并展现出相当大的潜力，能够自动从维基百科条目中提取概念，以生成特定主题的教育材料（Bulathwela 等人，2021）。此外，鉴于维基百科在语义标注人工智能生成内容方面的潜力已得到广泛研究，其基于链接结构、页面共现等概念相关度指标（Ponza 等人，2020）已相当成熟，能够以高精度表示维基百科概念之间的语义相关性。然而，在学习教育分析中，尚未详细探索允许可扩展解决方案（如 Wikification，Brank 等人，2017）在教育问题生成模型中的应用。

This paper aims to address these challenges associated with the topic-controlled EdGQ. We conducted supervised fine-tuning on a pre-trained T5-small model (hereafter referred to as the T5 model), an approach that is preferable and safer for educational entities to manage and control the language model (LM) with minimal infrastructure costs. The fine-tuning process utilised the novel MixSQuAD dataset, an enrichment of the SQuAD dataset (Rajpurkar et al., 2016), which is a commonly used general question generation dataset characterised by its shallow questions. Additionally, we designed experiments to explore the impacts of pre-training strategies, text data augmentation, and model quantisation on the model’s performance. We evaluated the model on the novel MixKhanQ dataset, derived based on the KhanQ dataset (Gong and Pan, 2022), which features human-like, in-depth questions sourced from Khan Academy, an online education platform. This is designed to assess the model’s effectiveness on academic materials, and its ability to generate educationally meaningful questions to explore its practical value for teaching and learning contexts. Based on these steps, the paper proposes a novel set of models that can perform high-precision topic-controlled educational question generation (T-CQG). The research questions addressed through this work are as follows:
本文旨在解决与主题控制型教育问题生成（EdGQ）相关挑战。我们对预训练的 T5-small 模型（以下简称 T5 模型）进行了监督微调，这是一种更适合教育机构管理和控制语言模型（LM），且基础设施成本最低的方法。微调过程使用了新型 MixSQuAD 数据集，该数据集是对 SQuAD 数据集（Rajpurkar 等人，2016 年）的扩展，后者是一个常用的通用问题生成数据集，其特点是问题较为浅显。此外，我们设计了实验，探索了预训练策略、文本数据增强和模型量化对模型性能的影响。我们在基于 KhanQ 数据集（Gong 和 Pan，2022 年）衍生的新型 MixKhanQ 数据集上评估了模型，该数据集具有类似人类、深入的问题，来源于在线教育平台可汗学院。这是为了评估模型在学术材料上的有效性，以及其生成教育意义问题的能力，以探索其在教学和学习环境中的实际价值。 基于这些步骤，论文提出了一套新型模型，能够实现高精度的主题控制教育问题生成（T-CQG）。本研究通过这项工作解决的研究问题如下：

• •

  RQ1: What are the most representative metrics for automated measures of generated questions on their topic relevance considering human evaluations as the ground truth?

  Report issue for preceding element
  
  • RQ1：在将人类评估作为基准的情况下，哪些是最能代表自动生成的主题相关性的问题度量指标？
• •

  RQ2: Is it feasible to fine-tune a pre-trained language model (PLM) to perform T-CQG?

  Report issue for preceding element
  
  • RQ2：是否可行通过微调预训练语言模型（PLM）来执行 T-CQG？
• •

  RQ3: Can further pre-training of the PLM on scientific text improve the quality of T-CQG?

  Report issue for preceding element
  
  • RQ3: 对 PLM 进行进一步的科学文本预训练能否提高 T-CQG 的质量？
• •

  RQ4: How does model quantisation affect the performance of the fine-tuned models while improving scalability?

  Report issue for preceding element
  
  • RQ4: 模型量化如何在提高可扩展性的同时影响微调模型的性能？
• •

  RQ5: To what extent can data augmentation further improve the quality of T-CQG?

  Report issue for preceding element
  
  • RQ5: 数据增强能在多大程度上进一步提高 T-CQG 的质量？

We used the SQuAD 1.1, the Stanford Question Answering Dataset, comprising over 100,000 questions crafted by crowd workers based on a selection of 536 Wikipedia articles (Rajpurkar et al., 2016) as the source for creating new datasets (SQuAD+, MixSQuAD and MixSQuAD2X as described in section 3.2 below) for finetuning the models. When training the TopicQGedu Model (see section 3.3.3 below), we used PeS2O dataset (Soldaini and Lo, 2023), a collection of scientific abstracts, to perform the pre-training as prior work has shown this may increase the model’s performance in educational settings (Bulathwela et al., 2023).
我们使用了斯坦福问答数据集 SQuAD 1.1，该数据集包含超过 10 万个由众包工作者基于 536 篇维基百科文章（Rajpurkar 等人，2016 年）编写的问题，作为创建新数据集（如 3.2 节下所述的 SQuAD+、MixSQuAD 和 MixSQuAD2X）的来源，用于微调模型。在训练主题控制教育问题生成模型（见 3.3.3 节下）时，我们使用了 PeS2O 数据集（Soldaini 和 Lo，2023 年），这是一个科学摘要的集合，用于进行预训练，因为先前研究表明这可能会提高模型在教育环境中的性能（Bulathwela 等人，2023 年）。

For evaluation, we used the KhanQ dataset (Gong and Pan, 2022) as it presents a more relevant challenge for educational question generation. It includes 1,034 high-quality questions in the STEM fields generated by learners, which aim to probe deep understanding of subjects taught in Khan Academy’s online courses ¹¹1https://www.khanacademy.org
为评估，我们使用了 KhanQ 数据集（Gong 和 Pan，2022），因为它为教育问题生成提出了更具相关性的挑战。它包含 1,034 个由学习者生成的 STEM 领域高质量问题，旨在探究 Khan Academy 在线课程中教授内容的深度理解。 ¹ . Despite its smaller size relative to SQuAD, KhanQ aligns more closely with our objective to generate topic-based and relevant educational questions (as per prior work (Fawzi et al., 2024)). To adapt the dataset for topic-based evaluation, we use the same approach as MixSQuAD (section 3.2.2) to create a dataset with contrasting topic-based questions. We refer to the transformed version of the KhanQ dataset as MixKhanQ dataset.
尽管相对于 SQuAD 来说规模较小，但 KhanQ 与我们的目标——生成基于主题的相关教育问题（根据先前工作（Fawzi 等人，2024））——更为一致。为了使数据集适应基于主题的评估，我们采用与 MixSQuAD（第 3.2.2 节）相同的方法来创建一个具有对比性主题问题的数据集。我们将转换后的 KhanQ 数据集版本称为 MixKhanQ 数据集。

A core contribution of this work is to introduce a novel data enrichment method that leads to the creation of new datasets that are derived from conventional question generation datasets. As described in 3.1, we derive the new datasets from SQuAD and KhanQ. These datasets already contain the context $c$ and the label question $q_{t}$ from a human (contrast to $\hat{q}_{t}$ in equation 1 which denotes the generated question). We append an additional field to the dataset, Topic $t$, and create three novel datasets, 1) SQuAD+, 2) MixSQuAD, and 3) MixSQuAD2X for the T-CQG task. The process of generating the three datasets is presented in figure 1.
这项工作的核心贡献是引入了一种新的数据增强方法，该方法能够从传统的问答生成数据集中创建新的数据集。如 3.1 所述，我们从 SQuAD 和 KhanQ 中推导出这些新数据集。这些数据集已经包含了人类提供的上下文 $c$ 和标签问题 $q_{t}$ （与方程 1 中的 $\hat{q}_{t}$ 表示生成的问句形成对比）。我们向数据集添加了一个额外的字段——主题 $t$ ，并创建了三个新的数据集，用于 T-CQG 任务：1) SQuAD+，2) MixSQuAD，3) MixSQuAD2X。生成这三个数据集的过程如图 1 所示。

With the relevant datasets created, we built multiple models to be evaluated in a series of experiments to answer the research questions outlined in section 2. All the models used in experiments are created by finetuning the T5-Small (Raffel et al., 2022) model, a small Language Model (sLM) that has also been used for educational question generation in the past (Bulathwela et al., 2023; Fawzi et al., 2024). We fine-tuned the foundational model (t5-small from HuggingFace library²²2https://huggingface.co/google-t5/t5-small
在创建了相关数据集后，我们构建了多个模型，用于在一系列实验中评估，以回答第 2 节中提出的研究问题。实验中使用的所有模型都是通过微调 T5-Small（Raffel 等人，2022 年）模型创建的，这是一个小型语言模型（sLM），过去也曾用于教育问题生成（Bulathwela 等人，2023 年；Fawzi 等人，2024 年）。我们对基础模型（来自 HuggingFace 库的 t5-small ² ）进行了微调。) using the Adam optimizer with a batch size of 64, the learning rate of $1e-3$, and epsilon of $1e-8$. We use a maximum sequence length of 512 for the encoder, and 128 for the decoder. We train all models for a maximum of 50 epochs with an early stopping based on the validation loss ³³3https://github.com/Cathgy/Topic-controllable-Question-Generator.git
使用 Adam 优化器，批处理大小为 64，学习率为 $1e-3$ ，epsilon 为 $1e-8$ 。我们将编码器的最大序列长度设置为 512，解码器的最大序列长度设置为 128。我们使用基于验证损失 ³ 的早停机制，对所有模型进行最多 50 个 epoch 的训练。.

To assess how representative BERTScore and WikiSemRel are when measuring topical relatedness (RQ1). We created a small gold-standard dataset via human annotation. The annotators (n = 4) consisted of two female and two male postgraduate students in the 20-30 age bracket from a masters degree programme at a university in the UK.
为评估 BERTScore 和 WikiSemRel 在衡量主题相关性（RQ1）时的代表性，我们通过人工标注创建了一个小型黄金标准数据集。标注者（n = 4）由来自英国某大学硕士课程的 4 名研究生组成，其中两名女性和两名男性，年龄在 20-30 岁之间。

To set up the annotation task, we randomly selected 30 questions from the MixKhanQ dataset (KhanQ dataset transformed using the method described in section 3.2.2). For each sample, we provided the participants with the reference question $q_{t}$ and two corresponding generated questions, 1) the question $\hat{q}_{t}$ generated using the relevant/prescribed topic and 2) the question $\hat{q}_{t^{\prime}}$ generated with an alternative topic. Annotators were required to independently determine which of the two generated questions $\hat{q}_{t}$ or $\hat{q}_{t^{\prime}}$ is more closely aligned with the reference question${q}_{t}$, the same tasks the SemRel metrics are going to do. The generated question $\hat{q}_{(\cdot)}$ annotators selected as closely relevant to the reference question is given $1$ and the other $0$. We calculated the Mean Absolute Error (MAE) between the mean score assigned by human annotators and the respective SemRel Score as per equation 2.
为了设置标注任务，我们从 MixKhanQ 数据集（使用 3.2.2 节中描述的方法转换的 KhanQ 数据集）中随机选择了 30 个问题。对于每个样本，我们向参与者提供参考问题 $q_{t}$ 以及两个相应的生成问题：1) 使用相关/指定主题生成的 $\hat{q}_{t}$ 问题，2) 使用替代主题生成的 $\hat{q}_{t^{\prime}}$ 问题。标注人员需要独立判断两个生成问题 $\hat{q}_{t}$ 或 $\hat{q}_{t^{\prime}}$ 中哪一个与参考问题 ${q}_{t}$ 更一致，这正是 SemRel 指标将要执行的任务。标注人员选出的与参考问题 $\hat{q}_{(\cdot)}$ 最相关的生成问题 $1$ 被赋予，而另一个则被赋予 $0$ 。我们根据公式 2 计算了人类标注人员分配的平均分数与相应的 SemRel 分数之间的平均绝对误差（MAE）。

Figure 2 illustrates the experimental setup designed to address RQs 2-5. A total of six models (including TopicQG’s base, 8bit, and 4bit versions) have been developed as described in detail in section 3.3 and represented as coloured boxes in figure 2. Each model is evaluated using the MixKhanQ dataset.
图 2 展示了为解决 RQ 2-5 而设计的实验设置。共开发了六个模型（包括 TopicQG 的基础版本、8bit 版本和 4bit 版本），这些模型在 3.3 节中有详细描述，并在图 2 中以彩色方框表示。每个模型都使用 MixKhanQ 数据集进行评估。

When evaluating the final models, we focused on two main aspects. 1) The generated question $\hat{q_{t}}$ is of high linguistic quality so it has the potential to be used in educational settings, 2) The generated question $\hat{q_{t}}$ is semantically related to the prescribed topic $t$ so that it can address the AI-generated questions’ common problem of being ”too general to be useful in practice” in educational settings.
在评估最终模型时，我们关注了两个主要方面。1) 生成的题目 $\hat{q_{t}}$ 具有高质量的语法，因此有潜力用于教育环境，2) 生成的题目 $\hat{q_{t}}$ 与指定主题 $t$ 具有语义相关性，以便能够解决 AI 生成的题目在教育环境中常见的“过于笼统而缺乏实用性”的问题。

In human evaluations of the 30 randomly selected question pairs for topical alignment, only two pairs did not reach a consensus among the participants with one outlier in each case (with the Fleiss’ kappa (Fleiss, 1971) measure of inter-rater agreement among multiple raters being 0.933). This indicates strong inter-rater reliability in the gold-standard human evaluator data. As per table 2, the WikiSimRel metrics are more aligned with the human judgements in comparison to the BERTScore. Among the three candidates, we can observe the embedding based (BERT and w2v) methods showing inferior representativeness. This could be due to the fact that embeddings can represent many different attributes about the entities and tokens they represent (e.g. whether the text is a question or a statement). This hypothesis is further reinforced by previous observations that including stopwords like ”what”, ”why” leads to the inflation of BERTScore. However the Jaccard WikiSimRel score that relies exclusively on outward links from Wikipedia pages tends to capture a better representation of the informational and thematic content leading to better alignment.
在针对 30 对随机选择的问题对进行主题一致性评估的人类评价中，只有两对在参与者之间未达成共识，每对均有一个极端值（根据多位评价者之间评分者间一致性系数 Fleiss’ kappa（Fleiss，1971）的测量结果为 0.933）。这表明金标准人类评价数据具有极强的评分者间可靠性。根据表 2，与 BERTScore 相比，WikiSimRel 指标更符合人类判断。在三个候选方案中，我们可以观察到基于嵌入（BERT 和 w2v）的方法表现出较差的代表性。这可能是由于嵌入能够表示它们所代表实体和标记的许多不同属性（例如，文本是问题还是陈述）。这一假设得到了先前观察的进一步证实，即包含像“what”、“why”这样的停用词会导致 BERTScore 的膨胀。然而，完全依赖于维基百科页面外部链接的 Jaccard WikiSimRel 分数倾向于更好地捕捉信息内容和主题内容，从而实现更好的一致性。

Table 3 provides us an indication of the degree to which the generated question $\hat{q}_{t}$ resembles the reference question $q_{t}$. This is a proxy for topical relevance as the reference question is implicitly aligned with the controlled topic. The results indicate that the proposed TopicQG model outperforms the baseline model in all but perplexity metric. Outperforming in terms of BLEU scores at multiple levels (BLEU1 through BLEU4), indicates enhanced linguistic precision in question generation. It also achieves higher F1, ROUGE-L, and METEOR scores, reflecting the model’s capability to generate questions that are not only relevant and accurate but also semantically aligned with reference texts. Compared with the baseline, a slight increase in perplexity suggests that the TopicQG model may generate questions that diverge from the reference language, potentially due to its ability to learn more complex educational expressions. The perplexity does not raise significant concerns over the quality of generations as the random examples in table 1 doesn’t indicate visible signs of deterioration. It is noteworthy that the randomly selected examples in table 1 are not as good as typical questions generated using a very large language model such as ChatGPT. We hypothesise the size of our model being a main reason for the relatively low quality of generations. However, our own prior work has also shown that such generations can be improved to humanly acceptable levels by simply post-processing them through a pre-trained grammar correction model (Fawzi et al., 2024, [n. d.]) retaining the accessibility and sustainability benefits of sLMs.
表 3 为我们提供了生成问题 $\hat{q}_{t}$ 与参考问题 $q_{t}$ 相似程度的指示。这可以作为主题相关性的代理指标，因为参考问题与控制主题隐式对齐。结果表明，所提出的 TopicQG 模型在除困惑度指标外所有指标上均优于基线模型。在多个级别的 BLEU 分数（BLEU1 至 BLEU4）上表现更优，表明问题生成中语言精确性得到提升。同时，该模型实现了更高的 F1、ROUGE-L 和 METEOR 分数，反映了其生成的问题不仅相关且准确，而且在语义上与参考文本对齐。与基线模型相比，困惑度略有上升，表明 TopicQG 模型生成的问题可能与参考语言有所偏离，这可能是由于其能够学习更复杂的教肓表达。困惑度的上升并未引起对生成质量显著问题的担忧，因为表 1 中的随机示例并未显示出明显的恶化迹象。 值得注意的是，表 1 中随机选择的示例不如使用 ChatGPT 等大型语言模型生成的典型问题好。我们假设模型的大小是生成质量相对较低的主要原因。然而，我们之前的研究也表明，通过使用预训练的语法纠错模型对生成内容进行后处理，可以将这些生成内容改进到人类可接受的水平（Fawzi 等，2024 年，[n. d.]）]) 保留 sLMs 的可访问性和可持续性优势。

Table 4, the stronger indicator of topic alignment gives us evidence that the proposed TopicQG models significantly outperform the baseline. In terms of the semantic difference between the educational questions generated with the controlled topic vs. a different topic (using WikiSimRel (Jaccard), the most representative metric from table 2), all newly proposed models except the 4bit quantised TopicQG model outperforms the baseline. This can be expected as extreme quantisation can deteriorate the accuracy of the model.
表 4 中，更强的主题一致性指标为我们提供了证据，即所提出的 TopicQG 模型显著优于基线模型。在控制主题与不同主题生成的教育问题之间的语义差异方面（使用 WikiSimRel（Jaccard），表 2 中最具代表性的指标），除 4bit 量化 TopicQG 模型外，所有新提出的模型均优于基线模型。这可以预期，因为极端量化会降低模型的准确性。

In terms of the TopicQGedu model that is pre-trained on scientific text, the results are mixed and more difficult to interpret. While it surpasses the predictive performance on the Baseline in a few metrics, it performs below the TopicQG model across all metrics in table 3. While pre-training on scientific content is hypothesised to improve the topical relevance of the model, we do not observe improvements in this case. To rigorously assess whether the observed differences in performance metrics are statistically significant, we alsoconducted a paired t-test comparing the performance scores of TopicQGedu and TopicQG across the same set of questions. The results yielded a p-value of 0.083 (¿0.05), suggesting that there is no statistically significant difference that TopicQGedu underperforms compared to TopicQG. Given that the T5 model is primarily trained on web-crawled data and Wikipedia articles (Raffel et al., 2022), the absence of scientific texts in the training corpus could potentially weaken the model’s performance in scientific concepts and language. Thus pre-training strategies may need to be further explored, especially in specialized domains where deeper domain knowledge might be crucial, even if immediate improvements in conventional metrics are not evident.
对于在科学文本上预训练的 TopicQGedu 模型，结果较为复杂且更难解释。虽然它在某些指标上超越了基线模型的预测性能，但在表 3 的所有指标中，它的表现均低于 TopicQG 模型。尽管预训练科学内容被认为可以提高模型的主题相关性，但在此情况下我们并未观察到改进。为了严格评估性能指标中观察到的差异是否具有统计学意义，我们还进行了配对 t 检验，比较了 TopicQGedu 和 TopicQG 在相同问题集上的性能得分。结果得到 p 值为 0.083（<0.05），表明 TopicQGedu 与 TopicQG 相比没有表现出统计学上的显著差异。鉴于 T5 模型主要在网页爬取数据和维基百科文章上进行训练（Raffel 等，2022），训练语料库中缺乏科学文本可能会削弱模型在科学概念和语言上的性能。 因此，预训练策略可能需要进一步探索，特别是在那些需要更深层领域知识的专门领域，即使在这些传统指标上没有立即的改进。

We investigated the effects of 8-bit and 4-bit quantisation on the TopicQG model (the best-performing model on the MixSQuAD dataset), referred to as TopicQG8bit and TopicQG4bit respectively. In comparison to the TopicQG model, the quantised models retain best performance with respect to metrics such as BLEU, F1-Score, MATEOR and ROUGE-L with very minor decreases ($\leq 0.01$) according to table 3. As expected, a drop in performance in comparison to the TopicQG model (with no quantisation) is observed. Similarly in table 4), a small drop in metrics is observed although it is not a drastic difference. This can be attributed to the fact that the generations change to a very small degree with quantisation indicated by the small deviations in table 3.
我们研究了 8 位和 4 位量化对 TopicQG 模型（在 MixSQuAD 数据集上表现最佳的模型）的影响，分别称为 TopicQG8bit 和 TopicQG4bit。与 TopicQG 模型相比，量化模型在 BLEU、F1-Score、MATEOR 和 ROUGE-L 等指标上保持了最佳性能，根据表 3，这些指标仅略有下降 ( $\leq 0.01$ )。正如预期，与未量化的 TopicQG 模型相比，性能有所下降。同样地，如表 4 所示，指标略有下降，尽管差异并不显著。这可以归因于量化导致生成内容的变化非常微小，如表 3 中较小的偏差所示。

Regarding memory usage, the full-precision TopicQG model occupies a memory size of $\approx 230$ MB. In contrast, the TopicQG8bit model significantly reduces this footprint to $\approx 110$ MB (59%), and the TopicQG4bit model further reduces it to $\approx 94$ MB (53%). The potential of quantisation demonstrated in this study is twofold: 1) it significantly lowers the hardware requirements for running the models, and 2) it maintains a satisfactory level of performance, making it feasible to deploy educational topic-controllable question generation on platforms where computational resources are limited. This accessibility could dramatically widen the applications of such models, making them more ubiquitous in educational and other real-time interactive applications on mobile devices. The reduction in model size not only implies lower memory requirements but also suggests lower power consumption, leading to cheaper infrastructure costs and a lower carbon footprint. Such properties are crucial for deploying these models in educational contexts of resource-constrained environments such as middle and low-income countries, mobile devices and embedded systems.
关于内存使用情况，全精度的 TopicQG 模型占用内存大小为 $\approx 230$ MB。相比之下，TopicQG8bit 模型显著减小了这一占用，降至 $\approx 110$ MB（减少了 59%），而 TopicQG4bit 模型进一步将其降低至 $\approx 94$ MB（减少了 53%）。本研究中展示的量化潜力体现在两个方面：1）显著降低了运行模型的硬件要求，2）保持了令人满意的工作性能，使得在计算资源有限的平台上部署教育主题可控问题生成成为可能。这种可及性将极大地扩展此类模型的应用范围，使其在教育及其他移动设备实时交互应用中更加普及。模型尺寸的减小不仅意味着更低的内存需求，也暗示了更低的功耗，从而降低基础设施成本和碳足迹。这些特性对于在资源受限的教育环境中部署这些模型至关重要，例如中低收入国家、移动设备和嵌入式系统。

The comparisons between TopicQG and TopicQG2X models in table 4 show that the data augmentation has an obvious effect on improving the models performance on topical relevance. The greater diversity of examples where the same example is presented to the model in two different ways helps the model better understand to follow the topical theme prescribed in the instruction with the context. It surpasses all other models, including TopicQG, demonstrating superior alignment of the generated questions with the input context and topic. This highlights the effectiveness of data augmentation in enhancing the model’s capacity to generate questions with topic relevance and better contextual consideration of texts.
表 4 中 TopicQG 与 TopicQG2X 模型的对比显示，数据增强对提升模型在主题相关性方面的性能有显著效果。相同示例以两种不同方式呈现的多样性帮助模型更好地理解并遵循指令中规定的主题主题，结合上下文进行理解。它超越了包括 TopicQG 在内的所有其他模型，展示了生成问题与输入上下文和主题的更高一致性。这突显了数据增强在提升模型生成主题相关问题的能力以及更好地考虑文本上下文方面的有效性。

Regarding educational practice, the proposed topic-controlled question generating model can be useful for different tasks within the education domain. Primarily, we see such a tool as a teacher assistant tool to propose questions to teachers to select from. Such a tool would keep teachers in the loop as final decision makers but help them with tasks such as generating topic-specific, relevant, and age-appropriate questions for teaching. As discussed in the introduction, these tasks have long been identified as time-intensive tasks for teachers (Giannakos et al., 2024). We envision tools where a teacher can point the system to a video, a presentation or a collection of learning resources where the system will automatically detect numerous salient topics and present them to the teacher as potential educational questions and the draft of a new quiz can be created in a matter of few clicks. This approach has the potential to change the degree of formative assessment due to decreased workload and can further stimulate well-anticipated innovation in education systems (Luckin and Cukurova, 2019). We argue that the model proposed and evaluated here has the potential to decrease teachers’ workload on such tasks.
在教育实践中，所提出的主题控制问题生成模型可用于教育领域的不同任务。主要而言，我们视此类工具为教师辅助工具，向教师提供可供选择的问题。此类工具将使教师作为最终决策者保持参与，同时帮助教师完成生成与主题相关、适用且符合年龄特点的教学问题的任务。正如引言中所述，这些任务长期以来一直被视为教师耗时的工作（Giannakos 等人，2024）。我们设想教师可以指向系统一个视频、演示文稿或学习资源集合，系统将自动检测多个显著主题，并将其呈现给教师作为潜在的教育问题，新测验的草稿可在几次点击内创建。这种做法由于减轻了工作量，有潜力改变形成性评估的程度，并进一步激发教育体系中期待已久的创新（Luckin 和 Cukurova，2019）。 我们认为，这里提出并评估的模型有潜力减少教师在这些任务上的工作量。

Second, the model can be integrated into multiple roles within the learning analytics infrastructures. The key to a precise learner state representation is having precise tests that can verify skill mastery of individuals at finer grain. The proposed method can lead to tools that can generate high-precision assessments within a personalised learning management system that can feed better data into learning analytics. While investing significant resources to create a relatively high coverage question banks is still feasible for short course and MOOC platforms that focus on narrow scopes of knowledge, as the world is gradually moving towards informal, lifelong learning such an investment would be infeasible. Models such as the one proposed here can play a critical role for continuous topic-specific, high quality and relevant question generation in educational systems.
其次，该模型可以集成到学习分析基础设施的多个角色中。精确的学员状态表示的关键在于拥有能够更精细地验证个人技能掌握程度的精确测试。所提出的方法可以导致在个性化学习管理系统中生成高精度评估的工具，从而为学习分析提供更好的数据。虽然为短期课程和专注于狭窄知识领域的 MOOC 平台投入大量资源创建相对高覆盖率的题库仍然是可行的，但随着世界逐渐向非正式、终身学习转变，这种投入将变得不可行。像这里提出的模型这样的模型，可以在教育系统中发挥关键作用，实现持续的主题特定、高质量且相关的题目生成。

Third, the model can also bring efficiencies to the implementation of question generation in mobile and resource scarce contexts. As we are dealing with very small models, systems built on these models are scalable with minimal costs and has the potential to run on mobile devices without having to connect to the Internet. These considerations are of utmost importance for more equitable use of AI in Education (Bulathwela et al., 2024).
第三，该模型也能为移动设备和资源匮乏环境中的问题生成实施带来效率。由于我们处理的是非常小的模型，基于这些模型构建的系统具有可扩展性且成本极低，并且有潜力在不连接互联网的情况下在移动设备上运行。这些考虑对于教育中 AI 的更公平使用至关重要（Bulathwela 等人，2024）。

Regarding LA and AI in Education research in general, the methodology proposed in this work can be extended to other forms of generation such as feedback, explanations, and content summaries in education. Also, aspects that that go beyond topical relevance (such as linguistic complexity and cognitive load etc.) can be controlled in future explorations to further advance learning analytics systems paving the way to re-imagining the limits of personalised learning material generation with AI. Furthermore, existing research either ignores the evaluation of whether the generated questions truly respond to the controlled conditions, or relies on extensive manual scoring by humans, which is both time-consuming and labour-intensive (Yudelson et al., 2013). Experimental studies with human participants presented here indicate that the ”relatedness score” has the potential to serve as a robust evaluation metric for assessing the semantic relatedness of generated questions to the input text, particularly in educational question generation tasks. It appears to excel in distinguishing between different concepts within the same academic field, making it particularly relevant for educational question generation tasks. As educational content generation research increases in LA literature, the importance of evaluation metrics of such content becomes even more important and the findings of this paper can help researchers consider appropriate metrics.
关于教育领域中的学习分析与人工智能研究，本工作中提出的方法可以扩展到其他形式的生成，如反馈、解释和内容摘要。此外，在未来的探索中，可以控制那些超越主题相关性的方面（如语言复杂性和认知负荷等），以进一步推进学习分析系统，为重新构想人工智能在个性化学习材料生成方面的极限铺平道路。此外，现有研究要么忽略了评估生成的题目是否真正符合控制条件，要么依赖于人工进行大量评分，这既耗时又费力（Yudelson 等人，2013）。这里展示的涉及人类参与者的实验研究表明，“相关性分数”有潜力作为评估生成题目与输入文本语义相关性的可靠指标，特别是在教育题目生成任务中。 它似乎在区分同一学术领域内的不同概念方面表现出色，因此特别适用于教育问题生成任务。随着教育内容生成研究在 LA 文献中的增加，此类内容评估指标的重要性也日益凸显，本文的研究结果可以帮助研究人员考虑合适的指标。