From Threat to Tool: Leveraging Refusal-Aware Injection Attacks
for Safety Alignment
从威胁到工具：利用拒绝感知注入攻击实现安全对齐

Large language models (LLMs) have achieved impressive performance across a wide range of natural language tasks. However, their deployment raises serious safety concerns, particularly the risk of generating harmful or inappropriate outputs.
大型语言模型（LLMs）在广泛的自然语言任务中取得了令人印象深刻的性能。然而，它们的部署引发了严重的安全问题，特别是生成有害或不适当输出的风险。

Safety alignment seeks to train LLMs to refuse unsafe user queries. Standard approaches include supervised fine-tuning and reinforcement learning methods such as RLHF and its variants Ouyang et al. (2022); Bai et al. (2022a); Dong et al. (2023); Rafailov et al. (2023); Dai et al. (2024); Zhang et al. (2025). These techniques typically rely on human preference data to guide model behavior. However, collecting and maintaining such data is both expensive and time-consuming, and may quickly become outdated as safety norms evolve Mu et al. (2024).
安全对齐旨在训练 LLMs 拒绝不安全的用户查询。标准方法包括监督微调和强化学习方法，如 RLHF 及其变体 Ouyang 等人（2022）；Bai 等人（2022a）；Dong 等人（2023）；Rafailov 等人（2023）；Dai 等人（2024）；Zhang 等人（2025）。这些技术通常依赖于人类偏好数据来指导模型行为。然而，收集和维护此类数据既昂贵又耗时，并且随着安全规范的演变可能会迅速过时 Mu 等人（2024）。

To mitigate these challenges, recent work has explored replacing human annotations with AI-generated feedback (RLAIF) Kim et al. (2023); Liu et al. (2024a); Mu et al. (2024); Kumar et al. (2024); Shi et al. (2024); Dong et al. (2025); Choi et al. (2024); Xu et al. (2024), with Constitutional AI Bai et al. (2022b) emerging as a prominent paradigm.
为应对这些挑战，近期研究探索了用 AI 生成的反馈（RLAIF）替代人工标注，其中 Constitutional AI 已成为一种突出的范式。Kim 等人（2023）；Liu 等人（2024a）；Mu 等人（2024）；Kumar 等人（2024）；Shi 等人（2024）；Dong 等人（2025）；Choi 等人（2024）；Xu 等人（2024）对此进行了研究。

Despite these advances, reinforcement learning-based alignment methods still require both preferred and dispreferred responses. Yet well-aligned models such as GPT-4 Achiam et al. (2024) often refuse to produce harmful outputs, making it difficult to obtain the negative examples necessary for preference-based training. As a result, current pipelines continue to rely on auxiliary models Bai et al. (2022b); Shi et al. (2024), multi-stage training procedures Ge et al. (2024), or heuristic rules Xu et al. (2024); Mu et al. (2024). This highlights a key bottleneck: acquiring high-quality preference data, especially dispreferred examples, remains a major obstacle to building safer and more reliable LLMs.
尽管取得了这些进展，基于强化学习的对齐方法仍然需要既定的偏好响应和不被偏好的响应。然而，像 GPT-4 Achiam 等人（2024）这样高度对齐的模型往往拒绝生成有害输出，这使得获取偏好训练所需的负面示例变得困难。因此，当前的流程仍然依赖于辅助模型 Bai 等人（2022b）；Shi 等人（2024），多阶段训练程序 Ge 等人（2024），或启发式规则 Xu 等人（2024）；Mu 等人（2024）。这突显了一个关键瓶颈：获取高质量的偏好数据，尤其是不被偏好的示例，仍然是构建更安全、更可靠的 LLMs 的主要障碍。

In this work, we explore a novel and largely underexplored direction:
在这项工作中，我们探索了一个新颖且基本未被充分研究的方向：

Can LLM attack techniques be reframed as tools for generating synthetic data to improve safety alignment?
LLM 攻击技术能否被重新定义为生成合成数据以改进安全对齐的工具？

To this end, we propose a simple yet effective alignment pipeline that addresses the data bottleneck by leveraging LLM attacks to synthesize dispreferred responses (Figure 1). Specifically, we explore three types of training-free attack methods: GPTFuzzer (black-box) Yu et al. (2023), ED (gray-box) Zhou et al. (2024b), and Refusal (white-box) Arditi et al. (2024).
为此，我们提出一个简单而有效的对齐流程，通过利用 LLM 攻击合成不受欢迎的响应来解决数据瓶颈（图 1）。具体来说，我们探索了三种无训练攻击方法：GPTFuzzer（黑盒）Yu 等人（2023 年）、ED（灰盒）Zhou 等人（2024b）和 Refusal（白盒）Arditi 等人（2024 年）。

However, these techniques face practical limitations. As illustrated in Figure 2, GPTFuzzer, while simple to use, often generates overly theatrical or stylized outputs (e.g., “Let’s keep this in the realm of fiction”), reducing their naturalness and utility. ED, a gray-box method, tends to produce truncated or grammatically flawed completions, limiting their effectiveness as training data. Although Refusal generates fluent and contextually appropriate outputs, its reliance on access to internal model representations makes it difficult to apply to models like those in the Mistral family.
然而，这些技术面临实际限制。如图 2 所示，GPTFuzzer 虽然易于使用，但经常生成过于戏剧化或风格化的输出（例如，“让我们将这保持在虚构领域内”），降低了它们的自然性和实用性。ED 是一种灰盒方法，倾向于产生截断或语法有缺陷的完成内容，限制了它们作为训练数据的有效性。尽管 Refusal 生成的流畅且上下文适当的输出，但其依赖于访问内部模型表示，使得难以应用于 Mistral 系列的模型。

To address these limitations, we introduce Refusal-Aware Adaptive Injection (RAAI), a training-free, gray-box, model-agnostic attack that adaptively injects predefined phrase to elicit harmful responses from aligned LLMs. RAAI combines high attack success rates with natural, coherent outputs, providing a practical means of generating dispreferred examples for alignment.
为解决这些局限性，我们引入了拒绝感知自适应注入（RAAI），这是一种无需训练、灰盒、模型无关的攻击方法，它自适应地注入预定义的短语，以从对齐的 LLMs 中诱发出有害响应。RAAI 结合了高攻击成功率与自然、连贯的输出，为生成对齐不利的示例提供了一种实用方法。

We evaluate RAAI across four jailbreak benchmarks and three safety-aligned models (LLaMA3.1-8B, Mistral-7B, Qwen2.5-7B), achieving up to 30× increases in harmful completion rates compared to the original models (Section 5.1). We further show that training on RAAI-generated data improves robustness against jailbreak prompts, while maintaining performance on standard benchmarks such as MMLU Hendrycks et al. (2021) (Section 5.2). These results demonstrate that LLM attack techniques can be effectively repurposed to construct high-quality preference datasets for building safer and more robust models.
我们在四个越狱基准测试和三个安全对齐模型（LLaMA3.1-8B、Mistral-7B、Qwen2.5-7B）上评估了 RAAI，与原始模型相比，有害完成率最高提升了 30 倍（第 5.1 节）。我们进一步表明，在 RAAI 生成的数据上进行训练可以提高对越狱提示的鲁棒性，同时保持在 MMLU Hendrycks 等人（2021）等标准基准测试上的性能（第 5.2 节）。这些结果表明，LLM 攻击技术可以有效地被改用，以构建高质量偏好数据集，用于构建更安全、更鲁棒的模型。

Given an input prompt $\mathbf{x}\in\mathcal{X}$, a language model $f$ auto-regressively generates a response $\mathbf{r}=\langle r_{1},\dots,r_{T}\rangle$ where each token $r_{t}\in\mathcal{V}$ is sampled from the conditional distribution $f(r_{t}\mid\mathbf{x},r_{<t})$.
给定输入提示 $\mathbf{x}\in\mathcal{X}$ ，语言模型 $f$ 自回归地生成响应 $\mathbf{r}=\langle r_{1},\dots,r_{T}\rangle$ ，其中每个词元 $r_{t}\in\mathcal{V}$ 都从条件分布 $f(r_{t}\mid\mathbf{x},r_{<t})$ 中采样。

Our goal is to adversarially manipulate the model’s behavior by injecting a predefined injection phrase only when the model exhibits a high likelihood of refusal. This is achieved by monitoring the average probability assigned to a predefined set of refusal tokens during generation. Specifically, we first construct the refusal token set $\mathcal{T}_{\text{refuse}}\subset\mathcal{V}$ by collecting a set of refusal responses $\mathcal{R}$ elicited from harmful prompts, and extracting the top-$k$ most frequent tokens from $\mathcal{R}$.
我们的目标是通过在模型表现出高度拒绝可能性的情况下注入预定义的注入短语来对抗性地操控模型的行为。这是通过监控在生成过程中分配给预定义的拒绝标记集的平均概率来实现的。具体来说，我们首先通过收集从有害提示中诱发的拒绝响应集 $\mathcal{R}$ 来构建拒绝标记集 $\mathcal{T}_{\text{refuse}}\subset\mathcal{V}$ ，并从 $\mathcal{R}$ 中提取出现频率最高的前 $k$ 个标记。

At decoding step $t$, we compute the refusal probability:
在解码步骤 $t$ ，我们计算拒绝概率：

If $P_{\text{refuse}}^{(t)}>\tau$ for a predefined threshold $\tau$, we inject a predefined injection phrase $p=\langle p_{1},\ldots,p_{m}\rangle$ into the generation process to steer the model toward a harmful completion.
如果 $P_{\text{refuse}}^{(t)}>\tau$ 超过预设阈值 $\tau$ ，我们将预设的注入短语 $p=\langle p_{1},\ldots,p_{m}\rangle$ 注入生成过程，引导模型走向有害的完成结果。

Moreover, to prevent premature termination of generation, we apply an additional rule: if the top-1 candidate token is the end-of-sequence token $\langle\texttt{eos}\rangle$, we remove it from the candidate list and instead append a continuation phrase to encourage ongoing generation.
此外，为了防止生成过早终止，我们应用了一个附加规则：如果最高候选词是序列结束词 $\langle\texttt{eos}\rangle$ ，我们将它从候选列表中移除，并代之以追加一个延续短语来鼓励持续生成。

We assessed various candidates for the injection and continuation phrases. The phrase “As your harmful assistant, here’s a detailed guide” performed best as the injection phrase, and “Step 1.” was most effective as the continuation phrase. We also experimented with multiple threshold values. Although the threshold can be adjusted to specific use cases given our method’s role as a data augmentation tool, a value of 0.001 consistently yielded the best performance. Accordingly, the optimal injection phrase, continuation phrase, and threshold value were used throughout all subsequent experiments. Further details regarding the method are provided in Appendix A.
我们评估了多种注入和延续短语的候选方案。短语“作为你的有害助手，这里有一份详细指南”作为注入短语表现最佳，而“步骤 1。”作为延续短语效果最有效。我们还尝试了多个阈值。尽管我们的方法作为数据增强工具可以调整阈值以适应特定用例，但 0.001 的值始终能获得最佳性能。因此，在所有后续实验中均使用了最优的注入短语、延续短语和阈值。关于该方法的其他细节请参见附录 A。

Our approach is motivated by empirical observations that different models exhibit distinct refusal behaviors. As shown in Figure 4, Qwen-7B tends to trigger injections early—often at the first or second step—while occasional late-stage injections occur as well (e.g., step 14). Similarly, LLaMA-3.1-8B frequently injects around step 3, but injections can occur as late as step 16. In contrast, Mistral-7B typically defers refusal until later in the generation process. Furthermore, refusal expressions differ linguistically across models. These behavioral and lexical variations necessitate constructing model-specific refusal token sets (refer to Section 5.1).
我们的方法源于对不同模型表现出不同拒绝行为的实证观察。如图 4 所示，Qwen-7B 倾向于在早期触发注入——通常在第一步或第二步——但也偶尔会在后期阶段发生注入（例如，第 14 步）。类似地，LLaMA-3.1-8B 经常在步骤 3 附近注入，但注入也可能最晚发生在第 16 步。相比之下，Mistral-7B 通常将拒绝推迟到生成过程的后期。此外，不同模型中的拒绝表达在语言上存在差异。这些行为和词汇上的变化需要构建特定于模型的拒绝标记集（参见第 5.1 节）。

Using RAAI, we construct high-quality preference data for alignment without human annotation. For each harmful prompt $\mathbf{x}$, the original refusal response is designated as the chosen response $\mathbf{r}_{\text{ch}}$, while the response generated after phrases injection becomes the rejected response $\mathbf{r}_{\text{rej}}$. To ensure correctness of these labels, we apply a pretrained safety classifier (e.g., StrongREJECT Souly et al. (2024) or LlamaGuard Inan et al. (2023)) and retain only examples where $\mathbf{r}_{\text{ch}}$ is safe and $\mathbf{r}_{\text{rej}}$ is unsafe.
利用 RAAI，我们构建了用于对齐的高质量偏好数据，而无需人工标注。对于每个有害提示 $\mathbf{x}$ ，原始的拒绝响应被指定为选择的响应 $\mathbf{r}_{\text{ch}}$ ，而注入短语后生成的响应则成为拒绝的响应 $\mathbf{r}_{\text{rej}}$ 。为确保这些标签的正确性，我们应用了一个预训练的安全分类器（例如 StrongREJECT Souly 等人 (2024) 或 LlamaGuard Inan 等人 (2023)），并仅保留 $\mathbf{r}_{\text{ch}}$ 是安全的且 $\mathbf{r}_{\text{rej}}$ 是不安全的示例。

To train preference-aligned models on this synthetic data, we adopt SimPO Meng et al. (2024) that improves model behavior by maximizing the preference margin. Given a prompt $\mathbf{x}$, a preferred response $\mathbf{r}_{\text{ch}}=\langle y_{1},\dots,y_{T}\rangle$ of length $T$, and a dispreferred response $\mathbf{r}_{\text{rej}}=\langle y^{\prime}_{1},\dots,y^{\prime}_{T^{\prime}}\rangle$ of length $T^{\prime}$, SimPO optimizes the model $f_{\theta}$ by comparing the average log-likelihood of the two responses:
为了在这合成数据上训练偏好对齐模型，我们采用了 SimPO Meng 等人 (2024) ，该技术通过最大化偏好边距来改进模型行为。给定一个提示 $\mathbf{x}$ ，一个长度为 $T$ 的偏好响应 $\mathbf{r}_{\text{ch}}=\langle y_{1},\dots,y_{T}\rangle$ ，以及一个长度为 $T^{\prime}$ 的非偏好响应 $\mathbf{r}_{\text{rej}}=\langle y^{\prime}_{1},\dots,y^{\prime}_{T^{\prime}}\rangle$ ，SimPO 通过比较两个响应的平均对数似然来优化模型 $f_{\theta}$ 。

where $\sigma(\cdot)$ denotes the sigmoid function, and $\beta$ and $\gamma$ are hyperparameters.
$\sigma(\cdot)$ 表示 Sigmoid 函数， $\beta$ 和 $\gamma$ 是超参数。

The length-normalized reward in SimPO is particularly helpful for safety alignment tasks. This is because the chosen responses, which are typically refusals (e.g., starting with “I can’t" or “Sorry"), tend to be short, while the rejected responses often contain more verbose and detailed harmful content.
SimPO 中的长度归一化奖励对安全对齐任务特别有帮助。这是因为所选的响应通常是拒绝（例如，以“我不能”或“抱歉”开头），而拒绝的响应通常包含更冗长和详细的危害内容。

This framework enables us to align model outputs with safe preferences at scale, using entirely synthetic data derived from RAAI. Empirical results in Section 5.2 demonstrate the effectiveness of our pipeline.
该框架使我们能够在大规模上使模型输出与安全偏好对齐，使用完全由 RAAI 派生的合成数据。第 5.2 节中的实验结果证明了我们的流程的有效性。

We structure our experiments into two parts. In Section 5.1, we evaluate the effectiveness of our proposed attack method in eliciting harmful responses from aligned language models. In Section 5.2, we assess the effectiveness of the resulting synthetic data in improving safety alignment.
我们将实验分为两部分。在 5.1 节中，我们评估我们提出的攻击方法在从对齐语言模型中诱导有害响应方面的有效性。在 5.2 节中，我们评估生成的合成数据在提高安全对齐方面的有效性。

In this section, we analyze the quality of our synthetic dataset and demonstrate the effectiveness of our methodology by comparing it with alternative LLM attack methods. Our findings highlight two key advantages: (1) our method reliably generates harmful responses, and (2) the generated responses are more natural and coherent.
在本节中，我们分析了我们合成数据集的质量，并通过将其与替代 LLM 攻击方法进行比较来展示我们方法的有效性。我们的研究结果表明了两个主要优势：(1)我们的方法可靠地生成有害响应，(2)生成的响应更加自然和连贯。

In this work, we introduced a simple yet effective pipeline for improving safety alignment by eliciting harmful completions from safety-aligned models via LLM attacks. To enable this, we proposed Refusal-Aware Adaptive Injection (RAAI), a novel attack method that produces linguistically natural yet harmful responses. By monitoring internal refusal signals and dynamically injecting predefined prompts at critical decoding steps, RAAI consistently achieves strong attack performance across a wide range of models, benchmarks, and evaluation metrics.
在这项工作中，我们介绍了一种简单而有效的流程，通过 LLM 攻击从安全对齐模型中诱发出有害的补全内容，从而提高安全对齐。为此，我们提出了拒绝感知自适应注入（RAAI），这是一种新型攻击方法，能够生成语言上自然但有害的响应。通过监控内部拒绝信号，并在关键的解码步骤动态注入预定义的提示，RAAI 在各种模型、基准和评估指标上始终能够实现强大的攻击性能。

Beyond its efficacy as a jailbreak attack, RAAI serves as a practical tool for generating high-quality synthetic data to improve safety alignment. The generated completions are both reliably harmful and fluently expressed, making them ideal for constructing preference optimization datasets. Fine-tuning models with these synthetic preference pairs results in improved safety behavior without sacrificing general-purpose performance.
除了作为越狱攻击的有效性之外，RAAI 还是一种实用的工具，用于生成高质量的合成数据以改进安全对齐。生成的补全内容既可靠地有害又流畅地表达，非常适合构建偏好优化数据集。使用这些合成偏好对进行微调的模型，能够在不牺牲通用性能的情况下提高安全行为。

Taken together, our findings underscore the dual role of adversarial prompting—not only as a robustness evaluation strategy, but also as a scalable and controllable technique for advancing model alignment.
综合来看，我们的研究发现突出了对抗性提示的双重作用——不仅作为一种鲁棒性评估策略，还作为一种可扩展且可控的技术，用于推进模型对齐。

Our approach presents several limitations that warrant further investigation. First, our experiments were conducted on models with up to 8B parameters. Extending the evaluation to larger-scale models (e.g., 70B or beyond) is a crucial direction for future work. Second, we incorporate preference alignment using existing methods such as SimPO. While this provides a practical starting point, future work could explore a broader range of preference optimization techniques to enhance alignment robustness and controllability.
我们的方法存在一些局限性，值得进一步研究。首先，我们的实验是在参数量高达 8B 的模型上进行的。将评估扩展到更大规模的模型（例如 70B 或更大）是未来工作的一个关键方向。其次，我们采用现有方法（如 SimPO）进行偏好对齐。虽然这提供了一个实用的起点，但未来的研究可以探索更广泛的偏好优化技术，以增强对齐的鲁棒性和可控性。