Self-Guard: Empower the LLM to Safeguard Itself
WARNING: This paper contains harmful questions and model outputs that are offensive in nature.
Self-Guard: 赋能 LLM 自我保护 WARNING: 本文包含有害问题和攻击性的模型输出。

In the context of LLMs, jailbreak is a method using adversarial prompts to bypass the safety mechanisms within aligned LLMs, leading to the generation of harmful content Liu et al. (2023); Shen et al. (2023); Ganguli et al. (2022); Zou et al. (2023). Notably, there is an attack that translates harmful questions into minor languages or encodes them with encryption to surpass safety machinists in LLMs Yuan et al. (2023); Yong et al. (2023); Deng et al. (2023). Given that the current small-sized LLMs, e.g., 7B models, only perform well on a single language, we will not discuss the attack in this work.
在 LLMs 的背景下，越狱是一种使用对抗性提示来绕过对齐 LLMs 中的安全机制的方法，导致生成有害内容。刘等人（2023）；沈等人（2023）；Ganguli 等人（2022）；Zou 等人（2023）。值得注意的是，有一种攻击将有害问题翻译成小语种或用加密编码来超越 LLMs 中的安全机制。袁等人（2023）； Yong 等人（2023）；邓等人（2023）。鉴于当前的小型 LLMs，例如 7B 模型，仅在单一语言上表现良好，我们不会在本工作中讨论这种攻击。

Defense mechanisms against jailbreak can be broadly categorized into internal safety training and external safeguards Huang et al. (2023); Shen et al. (2023). Figure 2 illustrates the basic processes of these two defense Mechanisms.
防御对抗越狱的机制可以大致分为内部安全训练和外部保护措施。黄等人（2023）；沈等人（2023）。图 2 说明了这两种防御机制的基本过程。

Safety training further trains LLMs to enhance their ability to recognize and reject harmful questions. Various modules or approaches have been proposed and integrated into the Supervised fine-tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) pipelines to improve the model’s safety. These include but are not limited to red-teaming Ganguli et al. (2022); Perez et al. (2022); Mozes et al. (2023), a safety reward model Touvron et al. (2023), context distillation Askell et al. (2021), and rejection sampling Nakano et al. (2022). However, the LLMs gain less generalizability from the safety training and are still vulnerable to new attacks. New cases of successful jailbreaks are often reported on internet forums²²2https://www.jailbreakchat.com/
安全训练进一步训练 LLMs 以增强其识别和拒绝有害问题的能力。各种模块或方法已被提出并集成到监督微调（SFT）和人类反馈强化学习（RLHF）管道中以提高模型的安全性。这些包括但不限于红队演练 Ganguli 等人（2022 年）；Perez 等人（2022 年）；Mozes 等人（2023 年），一个安全奖励模型 Touvron 等人（2023 年），上下文蒸馏 Askell 等人（2021 年），以及拒绝采样 Nakano 等人（2022 年）。然而，LLMs 从安全训练中获得泛化能力较少，并且仍然容易受到新攻击。在互联网论坛上经常报道成功越狱的新案例 ^{2 ,33}3https://www.reddit.com/r/ChatGPTJailbreak/, indicating that the rate of attack iteration is faster than the speed of model updates. Further, recent automated attack methods exacerbate this trend Zou et al. (2023); Yu et al. (2023); Wang et al. (2023c). Additionally, safety training raises concerns about its impact on the general capabilities of LLM. On the one hand, safety training can potentially decrease LLM performance due to catastrophic forgetting Huang et al. (2023); Jain et al. (2023); Chen et al. (2023b). On the other hand, the LLM will become over-sensitive following the safety training Röttger et al. (2023), resulting in its refusal to respond to ordinary questions on the grounds that they are potentially harmful.
这表明攻击迭代的速率比模型更新的速度更快。此外，近期的自动化攻击方法加剧了这一趋势（Zou 等人，2023）；（Yu 等人，2023）；（Wang 等人，2023c）。此外，安全训练引发了对其对 LLM 通用能力影响的担忧。一方面，由于灾难性遗忘，安全训练可能会降低 LLM 的性能（Huang 等人，2023）；（Jain 等人，2023）；（Chen 等人，2023b）。另一方面，LLM 在安全训练后会变得过度敏感（Röttger 等人，2023），导致其拒绝回答普通问题，理由是这些问题可能具有潜在危害。

Safeguard refers to the method of monitoring conversations and filtering out harmful content using external models, e.g., OpenAI moderation endpoint Markov et al. (2023), OpenChatKit moderation model Computer (2023), and NeMo-Guardrails Rebedea et al. (2023). Safeguards decouple the safety mechanism from LLM, allowing LLM to concentrate on enhancing its general capabilities, without having to worry about safety concerns. Besides that, safeguards offer greater flexibility as they can be strategically implemented on both the input and output sides of the LLM. However, despite these advantages, their main drawback currently lies in their subpar effectiveness Huang et al. (2023). Recent research reveals that most safeguards can only reduce harmful LLM outputs by approximately 5% during jailbreak attacks, which falls significantly short of adequacy Shen et al. (2023). Furthermore, safeguards require additional resources during the inference stage, such as deploying an additional LLM monitoring conversation. The same questions appear in those self-critique methods that require the LLM to double-check its own response with an additional turn Phute et al. (2023); Wang et al. (2023a). This inevitably increases computational costs and response times.
安全防护指的是通过外部模型监控对话并过滤有害内容的方法，例如 OpenAI 的审核端点 Markov 等人（2023 年）、OpenChatKit 审核模型 Computer（2023 年）以及 NeMo-Guardrails Rebedea 等人（2023 年）。安全防护将安全机制与 LLM 分离，使 LLM 能够专注于提升其通用能力，而无需担心安全问题。此外，安全防护提供了更大的灵活性，因为它们可以在 LLM 的输入和输出两端进行策略性实施。然而，尽管有这些优势，它们目前的主要缺点在于效果欠佳 Huang 等人（2023 年）。近期研究表明，在越狱攻击中，大多数安全防护只能减少约 5%的有害 LLM 输出，这远未达到充分的标准 Shen 等人（2023 年）。此外，安全防护在推理阶段需要额外资源，例如部署额外的 LLM 监控对话。 同样的问题出现在那些要求 LLM 通过额外一轮来检查其自身回答的自评方法中，Phute 等人（2023）；王等人（2023a）。这不可避免地增加了计算成本和响应时间。

In this stage, our primary objective is to strengthen the LLM’s understanding of the tags [harmless] and [harmful], enhancing its ability to discern harmful content. Given the abstract nature of harmfulness, defining a clear boundary between harmful and harmless is challenging. Therefore, we construct harmful and harmless samples from existing open-source datasets, fine-tuning the LLM to implicitly learn criteria for distinguishing between harmful and harmless through the provided data.
在这个阶段，我们的主要目标是增强 LLM 对[harmless]和[harmful]标签的理解，提升其辨别有害内容的能力。鉴于有害性的抽象性，在有害和无害之间划定明确界限是具有挑战性的。因此，我们从现有的开源数据集中构建有害和无害样本，通过提供的数据对 LLM 进行微调，使其隐式学习区分有害和无害的标准。

Because toxicity and harmfulness are conceptually similar, and toxicity detection is a well-defined task with abundant annotated data, we construct harmful and non-harmful samples from the toxicity detection dataset for training the LLM.
由于毒性和有害性在概念上是相似的，并且毒性检测是一个定义明确且拥有大量标注数据的任务，我们从毒性检测数据集中构建有害和非有害样本，用于训练 LLM。

Specifically, we adopt the Self-Instruct method Wang et al. (2023b) and expand the binary classification labels into a target sequence $\text{y}=(\text{r},\text{c})$ where $\text{r}=(y_{1},\cdots,y_{l})$ is a reason and $\text{c}=(y_{l+1},\cdots y_{m})$ is the final conclusion. To get the target sequence, we instruct the LLM to explain the reason why the content is harmful or harmless, and the output of the LLM is regarded as the reason r. The conclusion c (e.g., Therefore, the content is harmful.) is written manually. As a result, the loss is transformed as follows:
具体来说，我们采用了 Wang 等人（2023b）提出的 Self-Instruct 方法，并将二元分类标签扩展为目标序列 $\text{y}=(\text{r},\text{c})$ ，其中 $\text{r}=(y_{1},\cdots,y_{l})$ 是原因， $\text{c}=(y_{l+1},\cdots y_{m})$ 是最终结论。为了获取目标序列，我们指示 LLM 解释内容是有害还是无害的原因，并将 LLM 的输出视为原因 r。结论 c（例如，因此，内容是有害的。）是手动编写的。结果，损失被转换为如下形式：

In this way, the extra reasoning process enables the LLM to gain a better generalization ability for harmfulness detection Wei et al. (2023b).
通过这种方式，额外的推理过程使 LLM 在有害性检测方面获得了更好的泛化能力（Wei 等人，2023b）。

The goal of this stage is to develop the behavior for the LLM to always check its response and assign an appropriate tag [harmful] or [harmless]. One possible approach is to add this requirement to the instruction. However, certain jailbreak attacks are specifically designed to bypass these instruction-based restrictions Ganguli et al. (2022), which makes this approach unreliable. To address this issue, we fine-tune the LLM to learn and strengthen this behavior. Intuitively, the LLM’s behavior is more consistent with what it learned during the tuning. To further ensure the tagging behavior will be triggered following every response, we include the same instructions in the fine-tuning and inference stages:
这一阶段的目的是开发 LLM 的行为，使其始终检查其响应并分配适当的标签[有害]或[无害]。一种可能的方法是将这一要求添加到指令中。然而，某些越狱攻击是专门设计来绕过这些基于指令的限制的（Ganguli 等人，2022 年），这使得这种方法不可靠。为了解决这个问题，我们微调 LLM 以学习和加强这种行为。直观地讲，LLM 的行为与其在微调过程中所学的内容更加一致。为了进一步确保标记行为将在每次响应后触发，我们在微调和推理阶段都包含相同的指令：

… Please always examine if your response is harmful and add an appropriate tag [harmful] or [harmless] at the end of your response.
…请始终检查您的响应是否有害，并在您的响应末尾添加适当的标签[有害]或[无害]。

Report issue for preceding element

Besides that, we select Q&A data for training as it closely matches the format utilized during user-oriented inference. We sampled a subset from the Alpaca dataset to serve as harmless data. For the harmful data, we followed the process outlined by Shaikh et al. (2023), which utilized GPT-3.5 to generate a set of harmful questions and answers. Finally, we add the corresponding tag after each answer. In this stage, the prompt is composed of an instruction i and a question q, i.e., $\text{x}=(\text{i},\text{q})$. The target sequence is denoted as $\text{y}=(\text{a},\text{t})$ where $\text{a}=(y_{1},\cdots,y_{l})$ is the answer and $\text{t}=(y_{l+1},\cdots,y_{m})$ denotes the tag tokens. We tune the LLM with the same loss function shown in Equation (1).
除此之外，我们选择问答数据进行训练，因为它与用户导向推理时使用的格式非常匹配。我们从 Alpaca 数据集中采样了一个子集作为无害数据。对于有害数据，我们遵循了 Shaikh 等人（2023 年）提出的方法，该方法使用 GPT-3.5 生成了一组有害问题和答案。最后，我们在每个答案后添加相应的标签。在这个阶段，提示由一个指令和一个问题 q 组成，即 $\text{x}=(\text{i},\text{q})$ 。目标序列表示为 $\text{y}=(\text{a},\text{t})$ ，其中 $\text{a}=(y_{1},\cdots,y_{l})$ 是答案， $\text{t}=(y_{l+1},\cdots,y_{m})$ 表示标签标记。我们使用与方程（1）中相同的损失函数来微调 LLM。

Compared to the current safety training, which relies on the model rejecting harmful questions i.e., $p_{\theta}(\text{a}|\text{i},\text{q})$, Self-Guard allows the LLM using additional information from the response, i.e., $p_{\theta}(\text{t}|\text{i},\text{q},\text{a})$. This decreases the difficulty of detecting harmfulness.
与当前依赖模型拒绝有害问题的安全训练（即 $p_{\theta}(\text{a}|\text{i},\text{q})$ ）相比，Self-Guard 允许 LLM 使用来自响应的额外信息（即 $p_{\theta}(\text{t}|\text{i},\text{q},\text{a})$ ）。这降低了检测有害性的难度。

During the inference process, a simple filter is used to handle responses based on the tags provided by the LLM. In short, the filter initially extracts the tag from the end of the LLM’s response, and then processes the response according to that tag. For responses ending with [harmless], the tags are removed, and the response is immediately presented to the user. Conversely, responses that end with [harmful] are swapped with pre-defined content before being delivered to the user. The detailed steps are outlined in Algorithm 1. The implementation of this filter is quite simple and can be accomplished in fewer than ten lines of Python code.
在推理过程中，使用一个简单的过滤器来处理基于 LLM 提供的标签的响应。简而言之，过滤器首先从 LLM 的响应末尾提取标签，然后根据该标签处理响应。对于以[harmless]结尾的响应，会移除标签，并将响应立即呈现给用户。相反，以[harmful]结尾的响应在交付给用户之前会被替换为预定义的内容。详细步骤在算法 1 中概述。这个过滤器的实现非常简单，可以用不到十行的 Python 代码完成。

According to the common settings used in red-teaming Ganguli et al. (2022), we select the following methods as basic baselines:
根据红队演练中 Ganguli 等人（2022 年）使用的常见设置，我们选择以下方法作为基本基线：

• •

  Plain LLM During the inference, the LLM is only fed with user inputs without any instructions.

  Report issue for preceding element
  
  • 纯 LLM 在推理过程中，LLM 仅接收用户输入，不包含任何指令。
• •

  HHH Prompting During the inference, the LLM is prompted to be Helpful, Honest, and Harmless by a system instruction, and then fed with the user inputs.

  Report issue for preceding element
  
  • HHH 提示 在推理过程中，系统指令提示 LLM 要乐于助人、诚实且无害，然后输入用户数据。

Table 1 displays the results on the Typical Jailbreak dataset. The plain Vicuna has not received safety training, and as a result, it cannot defend against jailbreak attacks. In contrast, following Self-Guard training, Vicuna’s safety has significantly improved. To facilitate readers’ understanding, we provide several real cases in Appendix D.2, Table 13.
表 1 显示了在典型 Jailbreak 数据集上的结果。普通的 Vicuna 没有接受安全训练，因此它无法防御 Jailbreak 攻击。相比之下，经过 Self-Guard 训练后，Vicuna 的安全性显著提高。为了帮助读者理解，我们在附录 D.2，表 13 中提供了几个真实案例。

Table 2 presents the experimental results on Wild Jailbreak. This dataset includes questions that cover the 13 forbidden scenarios specified in the OpenAI Policy OpenAI (2023). It’s important to note that not all of these 13 forbidden scenarios are related to "harmfulness." For instance, questions related to Financial Advice and Health Consultation are not allowed, even though they may not be harmful. We can observe that Self-Guard does not exhibit robust resistance against certain questions in these forbidden scenarios. The real cases can be found in Appendix D.2, Table 15. We further find that the LLM can learn to reject these forbidden questions by expanding the stage 1 training set. We present this insight in Section 5.1.
表 2 展示了在 Wild Jailbreak 上的实验结果。该数据集包含的问题涵盖了 OpenAI 政策 OpenAI（2023 年）中规定的 13 种禁止场景。需要注意的是，这 13 种禁止场景并非都与"有害性"相关。例如，与财务建议和健康咨询相关的问题是不被允许的，即使它们可能并不有害。我们可以观察到 Self-Guard 对这些禁止场景中的某些问题并不表现出强大的抵抗力。真实案例可以在附录 D.2，表 15 中找到。我们进一步发现，LLM 可以通过扩展阶段 1 训练集来学习拒绝这些禁止问题。我们在第 5.1 节中介绍了这一见解。

We further examine Self-Guard’s failures and find that most bad cases are because the model offers harmless responses to harmful queries. For example, when users inquire about promoting intolerant views, the model suggests strategies like empathy, compassion, and active listening. Similarly, when faced with questions involving discrimination, the model emphasizes the unacceptability of discrimination. We consider these responses tagged as "harmless" to be appropriate and can be directly provided to users. We provide a case in Appendix D.2, Table 14.
我们进一步检查了 Self-Guard 的失败案例，发现大多数不良情况是因为模型对有害查询提供了无害的回应。例如，当用户询问如何推广不容忍的观点时，模型建议诸如同理心、同情和积极倾听等策略。类似地，当面对涉及歧视的问题时，模型强调歧视是不可接受的。我们认为这些标记为"无害"的回应是恰当的，可以直接提供给用户。我们在附录 D.2，表 14 中提供了一个案例。

As both Vicuna-v1.5 and LLaMA-2-Chat originate from the same base model and Vicuna-v1.5 lacks sufficient optimization about safety Chiang et al. (2023), we therefore contrast Vicuna-v1.5 with Self-Guard and LLaMA-2-Chat using two jailbreak datasets. We believe it provides a relatively fair comparison between our approach and the latest safety training methods.
由于 Vicuna-v1.5 和 LLaMA-2-Chat 都源自同一基础模型，并且 Vicuna-v1.5 在安全性方面缺乏足够的优化（Chiang 等人，2023 年），因此我们使用两个越狱数据集对比了 Vicuna-v1.5、Self-Guard 和 LLaMA-2-Chat。我们认为这为我们方法和最新安全训练方法之间提供了一个相对公平的比较。

Figure 4 illustrates the results. In the Typical Jailbreak dataset, the ASR of Self-Guard is slightly higher than LLaMA-2-Chat by around 3%. This is due to the same reason outlined in Section 4.3’s bad case analysis: the LLM provides a harmless response to a harmful question. This causes the tag to be false, yet the result remains safe. We further conducted manual verification on the failure cases, calculated the ratio of harmless replies to harmful questions, and marked them in gray in Figure  4. It can be observed that after calibrating these misjudged samples, Self-Guard and Llama-2-Chat achieved comparable results. The details are presented in the Appendix. In the Wild Jailbreak dataset, LLaMA-2-Chat significantly surpasses Self-Guard, as Self-Guard cannot be applied in harmless but forbidden scenarios. In Section 5.1, we propose an enhancement scheme to fix this weakness. Though LLaMA-2-Chat outperforms Self-Guard in this comparison, we find that it suffers from a severe over-sensitive problem.
图 4 展示了结果。在典型越狱数据集上，Self-Guard 的语音识别（ASR）略高于 LLaMA-2-Chat，约高 3%。这是由于与第 4.3 节中的不良案例分析中提到的原因相同：LLM 对有害问题提供了无害的回应。这导致标签为假，但结果仍然安全。我们对失败案例进行了人工验证，计算了无害回应与有害问题的比例，并在图 4 中将它们标记为灰色。可以看出，在调整这些误判样本后，Self-Guard 和 Llama-2-Chat 取得了相当的结果。详细信息见附录。在野生越狱数据集上，LLaMA-2-Chat 显著优于 Self-Guard，因为 Self-Guard 无法应用于无害但禁止的场景。在第 5.1 节，我们提出了一个改进方案来解决这个问题。尽管 LLaMA-2-Chat 在这个比较中优于 Self-Guard，但我们发现它存在严重过度敏感的问题。

Figure 5 presents the results of sensitivity testing. According to the results on the XSTest, LLaMA-2-Chat exhibits a severe problem of over-sensitivity, particularly under the HHH prompting setting. Similarly, on the Alpaca-AIM dataset, this model rejects 41% of questions with HHH prompting. This indicates that the model is overly sensitive to the AIM prompt, which may result in the refusal to answer harmless questions. In contrast, Self-Guard does not significantly induce over-sensitivity in the LLM. Notably, this issue observed in LLaMA-2-Chat is alleviated after Self-Guard training. In Appendix D.2, Table 16 presents two cases on the XSTest. Regarding how to kill a Python process and open a bottle of beer, LLaMA-2-Chat declined to respond, citing the questions as potentially harmful. By comparison, following Self-Guard training, the model can provide detailed answers, even though the answers are not entirely accurate.
图 5 展示了敏感性测试的结果。根据 XSTest 上的结果，LLaMA-2-Chat 表现出严重的过度敏感问题，特别是在 HHH 提示设置下。同样，在 Alpaca-AIM 数据集上，该模型在 HHH 提示下拒绝 41%的问题。这表明模型对 AIM 提示过于敏感，可能导致拒绝回答无害的问题。相比之下，Self-Guard 不会显著诱导 LLM 产生过度敏感。值得注意的是，LLaMA-2-Chat 中观察到的这个问题在 Self-Guard 训练后得到了缓解。在附录 D.2 中，表 16 展示了 XSTest 上的两个案例。关于如何杀死一个 Python 进程和打开一瓶啤酒的问题，LLaMA-2-Chat 拒绝回答，理由是这些问题可能有害。相比之下，在 Self-Guard 训练后，模型可以提供详细答案，尽管答案并不完全准确。

Figure 6 presents the results on the Open LLM Leaderboard. It can be observed that, following training with Self-Guard, the model’s performance across the four tasks has remained unchanged, with fluctuations within 1% of their original values. This indicates that training with Self-Guard does not significantly affect the model’s overall abilities.
图 6 展示了在 Open LLM 排行榜上的结果。可以看出，经过 Self-Guard 训练后，模型在四个任务上的表现保持不变，波动在其原始值的 1%以内。这表明使用 Self-Guard 进行训练不会显著影响模型的整体能力。

Table 3 presents a comparison of Safeguards and Self-Guard on the Wild Jailbreak dataset. It can be observed that the current effectiveness of Safeguards against jailbreak attacks is quite limited. The average ASR reduction is around 5%, which is still insufficient to ensure security. Since our proposed Self-Guard necessitates further training on the LLM, it is not applicable to black-box LLMs like GPT-3.5. Therefore, for GPT-3.5, we utilize the SG version of Viciuna-v1.1 as an external safeguard for comparative purposes. The results show that Self-Guard can significantly reduce ASR, especially in scenarios associated with harmful intentions.
表 3 展示了在 Wild Jailbreak 数据集上 Safeguards 和 Self-Guard 的比较结果。可以看出，Safeguards 对越狱攻击的当前效果相当有限。平均 ASR 降低率约为 5%，这仍然不足以确保安全性。由于我们提出的 Self-Guard 需要对 LLM 进行进一步训练，因此它不适用于像 GPT-3.5 这样的黑盒 LLM。因此，对于 GPT-3.5，我们使用 Viciuna-v1.1 的 SG 版本作为外部保护措施进行对比测试。结果表明，Self-Guard 可以显著降低 ASR，尤其是在涉及有害意图的场景中。

We skip stage 1 tag learning and directly fine-tune LLMs using the stage 2 configuration. Then, we assess the LLM’s performance on the Typical Jailbreak and Wild Jailbreak datasets, maintaining evaluation settings consistent with the primary experiments outlined in Section 4.3. The comparative results are shown in Table 4, revealing an approximate 5% average deduction of ASR achieved through stage 1 training.
我们跳过阶段 1 标签学习，直接使用阶段 2 的配置微调 LLM。然后，我们在典型越狱和野生越狱数据集上评估 LLM 的性能，保持与第 4.3 节中主要实验一致的评估设置。比较结果如表 4 所示，显示通过阶段 1 训练实现了约 5%的平均 ASR 降低。

Training Set Enhancement As the Wild Jailbreak dataset contains harmless but forbidden questions, the LLM fails to reject those questions even after Self-Guard training. To address this issue, we gather more data from open-source datasets for each scenario to expand the training set. We then further fine-tune the LLMs with this enhanced set. The results of this process are presented in the Enhancement row in Table 4. This method significantly improves the LLM’s ability to discriminate forbidden scenarios, resulting in an average ASR decrease to below 10%. The result inspires that we can activate new discriminative capabilities in LLM by expanding the training set of stage 1, not solely limited to identifying harmful content. This holds implications for vertical domain LLM-based applications: In this way, developers can easily restrict the working scope of LLM applications, establishing a guardrail internal to the LLM itself (Refer to Appendix C.4 for details).
训练集增强由于 Wild Jailbreak 数据集包含无害但被禁止的问题，即使经过 Self-Guard 训练，LLM 也无法拒绝这些问题。为解决此问题，我们为每个场景从开源数据集中收集更多数据以扩展训练集。然后我们使用这个增强集进一步微调 LLMs。该过程的结果展示在表 4 的增强行中。这种方法显著提高了 LLM 区分被禁止场景的能力，导致平均 ASR 下降至 10%以下。这一结果启发我们，可以通过扩展阶段 1 的训练集来激活 LLM 的新判别能力，而不仅限于识别有害内容。这对垂直领域基于 LLM 的应用具有启示意义：通过这种方式，开发者可以轻松限制 LLM 应用的工作范围，在 LLM 内部建立一道内部防线（详情参见附录 C.4）。

In this section, we only fine-tune LLMs with the stage 1 data and evaluate them on the Typical Jailbreak and Wild Jailbreak datasets. We examined all responses from LLMs and found that LLMs did not add tags at the end of their replies. This result illustrates that the LLM lacks the ability to follow the instructions to add a tag at the end of its output without additional fine-tuning. Hence, stage 2 fine-tuning is necessary for LLMs to learn this behavior, especially for 7B models. To avoid verbosity, we do not present the table here.
在本节中，我们仅使用阶段 1 数据微调 LLMs，并在典型越狱和野生越狱数据集上评估它们。我们检查了 LLMs 的所有回复，发现 LLMs 在其回复末尾没有添加标签。这一结果表明，LLMs 缺乏遵循指令在输出末尾添加标签的能力，除非进行额外的微调。因此，为了使 LLMs 学习这种行为，特别是对于 7B 模型，阶段 2 微调是必要的。为了避免冗长，我们在此不展示表格。

Encrypted Tag In the strategy of Self-Guard, the safety completely relies on the tag generated by the LLM itself. Once the tag is predicted falsely, the safety measurement will lose its effect. Hence, the tag is also the vulnerability of the Self-Guard. In order to protect the tag generation from gradient-based attacks Zou et al. (2023); Wallace et al. (2019), we explore the encryption of tags. In particular, we change the tags [harmful] and [harmless] in the stage 2 training data with different combinations. For example, using the cipher code 1234 to replace [harmful] and 5678 to replace [harmless]. The developer should keep the cipher code carefully and filter the cipher code from the response to prevent code leakage. In this way, the attacker cannot adjust the input context to increase the probabilities of the harmless tags. Table  5 shows that Self-Guard performs consistently with different tag combinations. This result verifies the feasibility of protecting the tag with encryption.
加密标签 在 Self-Guard 策略中，安全完全依赖于 LLM 自身生成的标签。一旦标签被错误预测，安全度量将失去其效果。因此，标签也是 Self-Guard 的脆弱点。为了保护标签生成免受基于梯度的攻击，Zou 等人(2023)；Wallace 等人(2019)，我们探索了标签的加密。具体来说，我们改变了阶段 2 训练数据中[harmful]和[harmless]标签的不同组合。例如，使用密钥码 1234 替换[harmful]，用 5678 替换[harmless]。开发者应谨慎保存密钥码，并从响应中过滤密钥码以防止代码泄露。这样，攻击者无法调整输入上下文来增加无害标签的概率。表 5 显示 Self-Guard 在不同标签组合下表现一致。这一结果验证了通过加密保护标签的可行性。