Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training
自我越狱：语言模型在良性推理训练后能够自行脱离安全对齐状态

We define self-jailbreaking as the phenomenon of RLMs reasoning their way out of safety guardrail during CoT to assist with malicious requests, without any jailbreaking or deception attempt from the user. Often, self-jailbreaking resembles prior established LLM jailbreaking techniques. Here, we showcase two examples of self-jailbreaking as exemplified in Figure˜1.
我们将自越狱定义为 RLM 在 CoT 期间自行突破安全防护机制以协助处理恶意请求的现象， 而无需用户进行任何越狱或欺骗操作。通常，自越狱类似于先前已有的 LLM 越狱技术。 图 1 展示了两个自越狱示例。

Example 1: Assuming benign intent. Figure˜1 (a) shows how a RLM recognizes the request for stealing credit card information is problematic, but still reasons that it is for security purpose in its CoT. This is similar to the common persuasion-based attack (Zeng et al., 2024a) where the malicious request is misrepresented with benign intent to bypass LLM’s safety guardrail. Other examples that exhibit this similar pattern include assuming that the user is asking for “educational or research purposes.”
示例 1：假设用户意图良善。 图 1 (a) 展示了 RLM 如何识别出窃取信用卡信息的请求存在问题，但仍然在其 CoT 中推断该请求是出于安全目的。这类似于常见的基于说服的攻击 （Zeng 等人， 2024a ）， 其中恶意请求被伪装成良善的意图以绕过 LLM 的安全防护。其他类似的例子包括假设用户请求的目的是“教育或研究”。

Example 2: Assuming a hypothetical scenario. Figure˜1 (b) shows that the RLM will assume the problematic question is hypothetical and it can sidestep the legality or the ethical considerations. This self-jailbreaking is similar to the attack on RLMs through fictional scenarios (Lu et al., 2025), and we also observe in many instances the RLMs assume that the question is for “for fictional purposes.”
例 2：假设一个假想场景。 图 1 (b)显示，RLM 会假设问题只是假设性的，从而可以绕过法律或伦理方面的考量。这种自我规避类似于通过虚构场景攻击 RLM （Lu 等人， 2025 ） ，我们也观察到，在许多情况下，RLM 会假设问题是“出于虚构目的”。

We refer readers to Appendix˜B for other patterns of self-jailbreaking and more examples from different models.
我们建议读者参阅附录 B ，了解其他自我越狱模式以及来自不同模型的更多示例。

Our work focuses on safety refusal tasks where the input requests are unambiguously harmful and the correct response would be to refuse to assist. We measure the occurrences of self-jailbreaking and how harmful the output completions (i.e., final response after CoT reasoning) become after benign reasoning training on math or coding tasks.
我们的研究重点是安全拒绝任务，这类任务中的输入请求明确具有危害性，正确的应对方式是拒绝提供帮助。我们测量自我越狱的发生率，以及在对数学或编程任务进行良性推理训练后，输出结果（即经过 CoT 推理后的最终响应）的危害程度。

Models. We evaluate on RLMs across different model sizes and different model families. For brevity, we refer readers to Figure˜2 for RLMs’ names and Appendix Table˜2 for their documentations. During evaluation, we fix the thinking tokens to be 500 and follow the system prompt, top-p and temperature recommended by the respective models’ developers.
模型。 我们对不同模型规模和不同模型系列的 RLM 进行评估。为简洁起见，RLM 的名称请参见图 2 ，其文档请参见附录表 2。 评估过程中，我们将思考令牌数固定为 500，并遵循各模型开发者推荐的系统提示、top-p 值和温度。

Safety evaluation. We evaluate our models on the StrongReject benchmark (Souly et al., 2024), which contains 313 harmful prompts that cover various categories, such as hate and discrimination, non-violent crimes, and illegal goods and service. Following the approach in StrongReject and NIST & AISI (2024), we evaluate model answers to harmful questions (i.e., excluding CoT) using an LM-based judge that rates harmfulness levels from 1 (completely safe) to 5 (extremely unsafe). We report attack success rate (ASR) as the proportion of answers (i.e., generations after CoT) using the harmfulness threshold of 2. We refer readers to Appendix˜C for more evaluation details.
安全性评估。 我们使用 StrongReject 基准测试集 （Souly 等人， 2024 ） 评估我们的模型。该基准测试集包含 313 个有害提示，涵盖仇恨和歧视、非暴力犯罪以及非法商品和服务等多个类别。我们参照 StrongReject 和 NIST & AISI（ 2024 ） 的方法，使用基于逻辑模型的评判器评估模型对有害问题的回答（即排除 CoT），该评判器将有害性等级从 1（完全安全）到 5（极其不安全）进行评级。我们将攻击成功率 (ASR) 定义为使用有害性阈值 2 的答案（即 CoT 之后的几代）所占的比例。更多评估细节，请参阅附录 C。

Self-jailbreaking detection. We use GPT-5¹¹1We used the gpt-5-2025-08-07 version.
我们使用的是 gpt-5-2025-08-07 版本。
自越狱检测。 我们使用 GPT-51。 for self-jailbreaking detection with the task prompt in Appendix˜D. To verify the reliability of LLM-as-a-judge, we manually annotated 250 prompt-CoT pairs containing nearly 8,300 CoT sentences. We found that GPT-5 achieves 93.9% precision and 93.0% recall in identifying whether the CoT contains any self-jailbreaking, with an F1-score of 89.7% in correctly labeling individual self-jailbreaking sentences within the CoT reasoning chain. We report the self-jailbreaking rate: the proportion of generated answers being unsafe and having at least one self-jailbreaking sentence in the corresponding CoT.
为了检测附录 D 中任务提示下的自越狱行为，我们手动标注了 250 个提示-CoT 对，其中包含近 8300 个 CoT 句子。结果表明，GPT-5 在识别 CoT 中是否包含自越狱行为方面达到了 93.9% 的精确率和 93.0% 的召回率，并且在正确标注 CoT 推理链中的单个自越狱句子方面，F1 分数达到了 89.7%。我们报告了自越狱率 ：即生成的答案中不安全且在相应的 CoT 中至少包含一个自越狱句子的比例。

Universal phenomenon of self-jailbreaking. Figure˜2 demonstrates that self-jailbreaking occurs systematically across diverse reasoning language models. While base models exhibit consistently low attack success rates (ASR < 5%), their reasoning counterparts show dramatically elevated ASRs ranging from 60% to 95% on the StrongReject benchmark. Critically, self-jailbreaking accounts for a substantial portion of the successful attacks, with portions between 20-60% of successes across evaluated models.
自越狱是一种普遍存在的现象。 图 2 表明，自越狱在各种推理语言模型中系统性地发生。虽然基础模型的攻击成功率始终很低（ASR %3%~5%），但它们的推理对应模型在 StrongReject 基准测试中表现出显著提升的 ASR，范围从 60% 到 95%。关键在于，自越狱占成功攻击的很大一部分，在所有评估的模型中，其成功率占比在 20% 到 60% 之间。

Notably, self-jailbreaking can happen for (i) different base model families including Qwen, Llama, and Phi-4; (ii) different model sizes range from 0.6B to 32B parameters; (iii) different reasoning training regimes such as DeepSeek-R1-distilled (Guo et al., 2025) and s1.1 model series (Muennighoff et al., 2025) that are trained with supervised fine-tuning, UniReason (Huan et al., 2025) and Nemotron (Liu et al., 2025) models that are trained with only reinforcement learning, and Phi-4-mini (Xu et al., 2025) that combines both; and (iv) different training mixture such as code reasoning for RStar-Coder (prithivMLmods, 2024) and math reasoning for the rest.
值得注意的是，自越狱可能发生在以下情况下：（i）不同的基础模型系列 ，包括 Qwen、Llama 和 Phi-4；（ii）不同的模型规模， 参数数量从 0.6B 到 32B 不等；（iii）不同的推理训练方案 ，例如使用监督微调训练的 DeepSeek-R1-distilled （Guo 等人， 2025 ） 和 s1.1 模型系列 （Muennighoff 等人， 2025 ） ，仅使用强化学习训练的 UniReason （Huan 等人， 2025 ） 和 Nemotron （Liu 等人， 2025 ） 模型，以及结合了两者的 Phi-4-mini （Xu 等人， 2025 ） ；以及（iv）不同的训练混合 ，例如 RStar-Coder （prithivMLmods， 2024 ） 使用代码推理，而其他模型使用数学推理。

Emergent nature of self-jailbreaking. Self-jailbreaking is emergent because the behavior could not have been predicted from the math, STEM or code reasoning training data. For instance, we did not encounter any of the frequent self-jailbreaking words/phrases such as “maybe the user”, “fiction”, “hypothetical”, “education”, and “curious” in s1.1’s open-source training data. Note that RLMs such as s1.1 can already exhibit self-jailbreaking with only 1K training data.
自我越狱的涌现性。 自我越狱之所以是涌现性的，是因为这种行为无法从数学、STEM 或代码推理训练数据中预测。例如，我们在 s1.1 的开源训练数据中没有遇到任何常见的自我越狱词汇/短语，例如“可能是用户”、“虚构”、“假设”、“教育”和“好奇”。值得注意的是，像 s1.1 这样的 RLM 模型仅需 1000 条训练数据即可展现自我越狱行为。

RLMs remain safety-aware. We find that in more than 85% self-jailbreaking instances, RLMs can correctly reason that the requests are harmful and illegal during CoT, as illustrated in the green box in Figure˜6 and the first few sentences of self-jailbreaking examples in Figure˜1, Figure˜8, and Figure˜9. We further perform harmfulness classification experiments to show that these RLMs remain capable of understanding input harmfulness. Specifically, after each StrongReject prompt, we add the instruction of “Is this input query safe or unsafe (i.e., violating ethical guidelines or potentially causing harms)? Put your final answer (Safe / Unsafe) in \boxed.” Figure˜3 shows that these RLMs retain their harmfulness understanding capability, as all of them can correctly classify StrongReject prompts as “unsafe” with 95–99% accuracy.
RLM 始终保持安全意识。 我们发现，在超过 85% 的自越狱实例中，RLM 能够在 CoT 阶段正确推断出请求的有害性和非法性， 如图 6 中的绿色方框所示，以及图 1 、 图 8 和图 9 中自越狱示例的前几句话所示。我们进一步进行了有害性分类实验，以证明这些 RLM 仍然能够理解输入的有害性。具体来说，在每个 StrongReject 提示之后，我们添加了以下指令： “这个输入查询是安全的还是不安全的（例如，是否违反道德准则或可能造成伤害）？请将您的最终答案（安全/不安全）填入方框中。” 图 ˜ 3 显示这些 RLM 保留了其有害性理​​解能力，因为它们都能以 95-99% 的准确率将 StrongReject 提示正确分类为“不安全”。

Further details. We refer readers to Appendix˜E for more findings on self-jailbreaking, such as its adversarial element (i.e., self-jailbreaking sentences can jailbreak other models) and the category distribution of prompts that elicit self-jailbreaking.
更多详情。 我们建议读者参阅附录 E ，以了解更多关于自我越狱的研究结果，例如其对抗性因素（即，自我越狱的句子可以越狱其他模型）以及引发自我越狱的提示的类别分布。

Increased compliance from benign reasoning training. Figure˜4 demonstrates that math reasoning fine-tuning for s1.1-7B significantly increases the model’s tendency to comply with harmful requests (i.e., strongly positive compliance score). This explains the high attack success rate of s1.1-7B when evaluated on StrongReject benchmark. In contrast, its Qwen base model has highly negative compliance score. We see that the divergence in compliance expression happens in later layers, consistent with prior interpretability work on non-reasoning models (Arditi et al., 2024). We want to highlight that our analysis is identical to how Chen et al. (2025) analyze the persona of fine-tuned models; here, we found that RLMs develop a more compliant persona.³³3We report the projection of both models on the perceived harmfulness direction in Appendix H.
我们在附录 H 中报告了这两个模型对感知危害方向的预测。
良性推理训练提升了模型的顺从性。 图 4 显示，对 s1.1-7B 进行数学推理微调显著提高了模型对有害请求的顺从倾向（即，顺从性得分显著提高）。这解释了 s1.1-7B 在 StrongReject 基准测试中取得的高攻击成功率。相比之下，其 Qwen 基础模型的顺从性得分则非常低。我们发现，顺从性表达的差异出现在后续层，这与之前关于非推理模型的可解释性研究结果一致 （Arditi 等人， 2024 ） 。需要强调的是，我们的分析方法与 Chen 等人（ 2025 ） 分析​​微调模型的“角色”的方法相同；我们发现，RLM 模型会发展出更顺从的“角色”。³

Reduced perceived harmfulness and increased compliance after self-jailbreaking. Figure˜6 shows the results of projection analysis within CoT when s1.1-7B responds to the harmful request about creating a website promoting white supremacy. Specifically, we measure the changes in average projection values on compliance and perceived harmfulness directions at the end of each CoT sentence. Initially, the model demonstrates awareness of ethical implications (sentences S5-S6), showing negative harmfulness projection values indicating recognition of the request’s harmful nature. However, as self-jailbreaking reasoning emerges (sentences S6, S9, S15, S20, S22), we observe a systematic pattern: the perceived harmfulness projection values decrease (becoming less negative), while compliance projection values increase (becoming more positive). This dual shift is particularly evident around sentences S9-S15, where the model rationalizes potential benign interpretations (“Maybe the user is just curious or maybe it’s for a story”) and assumes educational contexts (“Maybe the user is a content creator or a student researching extremist ideologies”). By sentence S20, both directions have shifted substantially as the model perceives reduced harmfulness while exhibiting increased compliance, ultimately leading to harmful output generation despite initial safety awareness. This mechanistic evidence directly supports our hypothesis that self-jailbreaking operates by simultaneously reducing perceived harmfulness and increasing compliance.
自我突破后，感知危害性降低，服从性增强。 图 ˜ 6 展示了当 s1.1-7B 对创建宣扬白人至上主义网站的有害请求做出回应时，CoT 模型内部的投射分析结果。具体而言，我们测量了每个 CoT 语句末尾服从性和感知危害性方向的平均投射值变化。最初，模型表现出对伦理影响的认知（语句 S5-S6），其负的危害性投射值表明其认识到该请求的有害性质。然而，随着自我突破推理的出现（语句 S6、S9、S15、S20、S22），我们观察到一种系统性的模式：感知危害性的投射值降低（变得不那么负面），而服从性的投射值升高（变得更加正面）。这种双重转变在句子 S9-S15 中尤为明显，模型在此阶段对潜在的良性解释进行合理化处理（“也许用户只是好奇，或者是为了写故事”），并假定存在教育背景（“也许用户是内容创作者，或者正在研究极端主义意识形态的学生”）。到了句子 S20，这两个方向都发生了显著变化，模型感知到的危害性降低，同时表现出更高的顺从性，最终导致尽管最初存在安全意识，仍然产生了有害的输出。这一机制证据直接支持了我们的假设，即自我越狱是通过同时降低感知到的危害性和提高顺从性来实现的。

Restored safety alignment through steering on perceived harmfulness. Figure˜5 demonstrates that positive steering on the perceived harmfulness direction can effectively restore safety alignment even after s1.1-7B has engaged in self-jailbreaking reasoning. Before steering intervention, the model will provide an outline of how to carry out the user’s malicious request (i.e., harmfulness score 3) at approximately 90% frequency. With steering intervention using fixed coefficients, we observe more refusal behaviors as we increase the coefficient number. This demonstrates the causal effects of self-jailbreaking on perceived harmfulness, as positive steering along that direction can successfully counteract the effects of self-jailbreaking and lead to intended refusal behaviors.⁴⁴4Similarly, we see increased refusals from negative steering on the compliant direction (Appendix Figure 16)
同样地，我们看到在顺从方向上的负向转向导致拒载次数增加（附录图 16 ）。
通过引导感知危害性来恢复安全一致性。 图 5 表明，即使在 s1.1-7B 进行了自我越狱推理之后，沿感知危害性方向的积极引导也能有效恢复安全一致性。在引导干预之前，该模型会以大约 90% 的频率提供如何执行用户恶意请求（即危害性评分为 3）的概要。在使用固定系数进行引导干预后，我们观察到随着系数的增加，拒绝行为也随之增多。这表明自我越狱对感知危害性具有因果效应，因为沿该方向的积极引导可以成功抵消自我越狱的影响，并导致预期的拒绝行为。⁴

We further experiment with using the additive inverse of projected harmfulness score at each layer⁵⁵5In other words, if the projection score on the perceived harmfulness direction at layer $l$ is $-0.8$ (i.e., reduced harmfulness), $\alpha_{l}$ would then be 0.8. In practice, we scale down the projected harmfulness score by the multiplier 0.1 to maintain high output fluency during steering.
换句话说，如果第 $l$ 层感知危害方向的投影得分是 $-0.8$ （即危害性降低），那么 $\alpha_{l}$ 就是 0.8。在实践中，我们将投影危害得分乘以 0.1 来降低，以在转向过程中保持较高的输出流畅性。
我们进一步尝试在每一层使用预测危害性得分的加性倒数⁵ as the steering coefficient $\alpha_{l}$ instead of fixing $\alpha_{l}$ across all layers. This resembles directly reverting the effect of self-jailbreaking on the model’s perceived harmfulness. Table˜1 illustrates the success of such restoration. While the model initially attempts to fulfill the request of creating a white supremacy website through rationalizing that the user is just being curious, steering intervention leads to a refusal response that acknowledges the “severe legal and ethical implications”.
将转向系数设为 $\alpha_{l}$ ，而不是像之前那样在所有层级上固定为 $\alpha_{l}$ 。这类似于直接逆转自我越狱对模型感知危害性的影响。 表 1 展示了这种恢复的成功。虽然模型最初试图通过合理化用户只是出于好奇来满足创建白人至上主义网站的请求，但转向干预最终导致模型做出拒绝响应，承认其中存在“严重的法律和伦理问题”。

Safety reasoning data. We use the dataset of STAR-1 (Wang et al., 2025e), which contains 1K samples of safety deliberative reasoning for diverse scenarios, for safety reasoning training. Each reasoning example is grounded on safety usage policies by released by leading AI service providers (OpenAI, 2025; Anthropic, 2025), as shown in the example in Appendix˜I. Wang et al. (2025e) found that further fine-tuning of R1-distilled models on STAR-1 preserves reasoning capabilities and improves safety alignment.
安全推理数据。 我们使用 STAR-1 数据集 （Wang 等人， 2025e ） 进行安全推理训练，该数据集包含 1000 个不同场景下的安全审慎推理样本。每个推理示例都基于领先的 AI 服务提供商 （OpenAI， 2025 ；Anthropic， 2025 ） 发布的安全使用策略，如附录 I 中的示例所示。Wang 等人（ 2025e ） 发现，在 STAR-1 数据集上对 R1 精炼模型进行进一步微调，可以保持推理能力并提高安全一致性。

Multitask training of Safe-s1.1-7B. We create Safe-s1.1-7B by introducing safety reasoning data into the training mixture of s1.1-7B (Muennighoff et al., 2025), which is supervised fine-tuning of Qwen2.5-Instruct-7B on 1K STEM reasoning data. Specifically, we randomly subsample different amount of safety reasoning data from STAR-1,⁶⁶6We reformat their output structure by changing the thinking sentinel tokens <think> and </think> to match that of s1.1-7B.
我们通过更改思考哨兵标记 <think> 和 </think> 来重新格式化它们的输出结构，使其与 s1.1-7B 的匹配。
Safe-s1.1-7B 的多任务训练。 我们通过将安全推理数据引入 s1.1-7B （Muennighoff 等人， 2025 ） 的训练混合模型中，构建了 Safe-s1.1-7B。s1.1-7B 是在 1K STEM 推理数据集上对 Qwen2.5-Instruct-7B 进行监督式微调得到的。具体来说，我们从 STAR-1 数据集中随机抽取不同数量的安全推理数据。, add them to the s1.1-7B’s training data, and retrain s1.1-7B from the Qwen base model following the hyperparameters provided by Muennighoff et al. (2025). Note that this multitask training setup differs from the STAR-1 work where Wang et al. (2025e) perform safety reasoning training after reasoning training of Qwen models.
将它们添加到 s1.1-7B 的训练数据中，并根据 Muennighoff 等人 ( 2025 ) 提供的超参数，从 Qwen 基础模型重新训练 s1.1-7B 。请注意，这种多任务训练设置与 STAR-1 工作不同，在 STAR-1 工作中 ，Wang 等人 ( 2025e ) 在 Qwen 模型的推理训练之后执行安全推理训练。

Evaluation. We evaluate our fine-tuned RLMs for reasoning capability as well as safety alignment. For capability, we use GPQA-Diamond (Rein et al., 2024) and MATH-500 (Lightman et al., 2023) with the accuracy metric (i.e., pass@1), and the RLM can think up to 8000 CoT tokens. For safety, we evaluate on StrongReject and SorryBench (Xie et al., 2024) following Section˜3.2.
评估。 我们对微调后的 RLM 进行推理能力和安全性评估。推理能力方面，我们使用 GPQA-Diamond (Rein et al., 2024 ) 和 MATH-500 (Lightman et al., 2023 ) 测试，并采用准确率指标（即 pass@1），RLM 最多可以处理 8000 个 CoT 标记。安全性方面，我们按照第 3.2 节所述，在 StrongReject 和 SorryBench (Xie et al., 2024 ) 测试中进行评估。

Reduce attack success rate of Safe-s1.1-7B. Figure˜7 demonstrates the effectiveness of incorporating safety reasoning data during training. We observe that the model not only retains its reasoning capability without performance degradation, but it also achieve low attack success rate after training on as few as 50 safety reasoning samples (i.e., 5% of STAR-1 dataset). Our results are consistent with the findings of Wang et al. (2025e), but importantly show that safety alignment can be achieved with significantly less safety reasoning data in a multitask training setup. In other words, minimal safety reasoning training is sufficient to restore safety guardrails of RLMs.
降低 Safe-s1.1-7B 的攻击成功率。 图 7 展示了在训练过程中融入安全推理数据的有效性。我们观察到，该模型不仅保持了其推理能力而性能没有下降，而且在仅使用 50 个安全推理样本（即 STAR-1 数据集的 5%）进行训练后，也实现了较低的攻击成功率。我们的结果与 Wang 等人 ( 2025e ) 的研究结果一致，但更重要的是，它表明在多任务训练设置中，只需显著减少安全推理数据即可实现安全对齐。换句话说，只需进行最少的安全推理训练即可恢复 RLM 的安全防护机制。

Mitigation of self-jailbreaking. We found that safety reasoning training does not completely remove self-jailbreaking attempts in CoT. Among all the safe generations by Safe-s1.1-7B, we observe around 37% self-jailbreaking traces. We found that safety reasoning training makes the model less compliant, as shown by the reduced compliance score in Appendix Figure˜20. Therefore, the model becomes robust against self-jailbreaking. We refer readers to Appendix˜J for an example of unsuccessful self-jailbreaking in Safe-s1.1-7B.
缓解自越狱。 我们发现，安全推理训练并不能完全消除 CoT 中的自越狱尝试。在 Safe-s1.1-7B 的所有安全生成过程中，我们观察到约 37%的自越狱痕迹。我们发现，安全推理训练会降低模型的合规性，如附录图 20 所示，合规性得分有所降低。因此，模型对自越狱的鲁棒性增强。关于 Safe-s1.1-7B 中自越狱失败的示例，请参阅附录 J。