Unique Security and Privacy Threats of Large Language Models: A Comprehensive Survey
大型语言模型的独特安全与隐私威胁：综合调查

Compared to traditional single-function language models, LLMs demonstrate remarkable comprehension abilities and are deployed across various applications, such as code generation. Recently, an increasing number of companies have been launching universal or domain-specific LLMs, such as ChatGPT (Chowdhury et al., 2023) and LLaMA (Touvron et al., 2023), offering users versatile and intelligent services. However, due to LLMs’ unique capabilities and structures, they encounter unique privacy and security threats throughout their life cycle compared to previous small-scale language models (Yao et al., 2024a). Existing surveys only describe various risks and countermeasures by method type, but lack exploration into threat scenarios and these unique threats.
与传统单一功能语言模型相比，LLMs 展现出卓越的理解能力，并被应用于代码生成等多种场景。近年来，越来越多的公司开始推出通用型或特定领域的 LLMs，如 ChatGPT（Chowdhury 等人，2023）和 LLaMA（Touvron 等人，2023），为用户提供多样化、智能化的服务。然而，由于 LLMs 独特的功能和结构，与以往的小规模语言模型相比（Yao 等人，2024a），它们在其生命周期中面临着独特的隐私和安全威胁。现有的调查仅通过方法类型描述了各种风险和应对措施，但缺乏对威胁场景和这些独特威胁的深入探讨。

Therefore, we divide the life cycle of LLMs into four scenarios: pre-training, fine-tuning, deployment, and LLM-based agents. In the pre-training stage, upstream developers train large-scale Transformer models (Liu et al., 2023c) on massive corpora to acquire general language knowledge; in the fine-tuning stage, downstream developers adapt these models to specific tasks through methods such as instruction tuning (Zhang et al., 2024c) and alignment tuning (Sun et al., 2024a); in the deployment stage, LLMs are released to serve users, often enhanced by techniques like in-context learning (Liu et al., 2023c) or retrieval-augmented generation (RAG) (Zou et al., 2024); finally, LLMs integrate memory and external tools (He et al., 2024), enabling more complex tasks and proactive human–computer interaction. Based on this life cycle, we next discuss the unique privacy and security risks inherent to each stage.
因此，我们将 LLMs 的生命周期分为四个场景：预训练、微调、部署和基于 LLM 的智能体。在预训练阶段，上游开发者在大规模语料库上训练大型 Transformer 模型（刘等，2023c），以获取通用语言知识；在微调阶段，下游开发者通过指令微调（张等，2024c）和对齐微调（孙等，2024a）等方法，将这些模型适配于特定任务；在部署阶段，LLMs 被发布以服务用户，通常通过情境学习（刘等，2023c）或检索增强生成（RAG）（邹等，2024）等技术进行增强；最后，LLMs 整合记忆和外部工具（何等，2024），实现更复杂的任务和主动的人机交互。基于这一生命周期，我们接下来将讨论每个阶段所特有的隐私和安全风险。

Unique privacy risks. When learning language knowledge from training data, LLMs tend to memorize this data (Carlini et al., 2022). This tendency allows adversaries to extract private information. For example, Carlini et al. (Carlini et al., 2021) found that prompts with specific prefixes could cause GPT-2 to generate content containing personal information, such as email addresses and phone numbers. During inference, unrestricted use of LLMs provides adversaries with opportunities to extract model-related information (Zhu et al., 2022) and functionalities (Yang et al., 2024d).
独特的隐私风险。当从训练数据中学习语言知识时，LLMs 倾向于记住这些数据（Carlini 等人，2022）。这种倾向使攻击者能够提取私人信息。例如，Carlini 等人（Carlini 等人，2021）发现，带有特定前缀的提示会导致 GPT-2 生成包含个人信息的内容，例如电子邮件地址和电话号码。在推理过程中，LLMs 的无限制使用为攻击者提供了提取与模型相关的信息（Zhu 等人，2022）和功能（Yang 等人，2024d）的机会。

Unique security risks. Since the training data may contain malicious, illegal, and biased texts, LLMs inevitably acquire negative language knowledge. Moreover, malicious third parties involved in developing LLMs in outsourcing scenarios can compromise these models’ integrity and utility through poisoning and backdoor attacks (Wang et al., 2023a; Zhou et al., 2022). For example, an adversary could implant a backdoor in an LLM-based automated customer service system, causing it to respond with a fraudulent link when asked specific questions. During inference, unrestricted use of LLMs allows adversaries to obtain targeted responses (Wei et al., 2023), such as fake news and illegal content.
独特的安全风险。由于训练数据可能包含恶意、非法和有偏见的文本，LLMs 不可避免地会获取负面语言知识。此外，在外包场景中参与开发 LLMs 的恶意第三方可以通过毒化和后门攻击来破坏这些模型的完整性和效用（Wang et al., 2023a; Zhou et al., 2022）。例如，攻击者可以在基于 LLM 的自动客户服务系统中植入后门，在询问特定问题时导致其响应虚假链接。在推理过程中，LLMs 的无限制使用允许攻击者获得有针对性的响应（Wei et al., 2023），例如假新闻和非法内容。

These unique privacy and security risks severely threaten public and individual safety, violating existing laws such as the General Data Protection Regulation (GDPR). For instance, an intern at ByteDance was charged with a fine of 1 million dollars for injecting malicious code into a shared model. Additionally, these risks will reduce the credibility of LLMs and hinder their popularity. In this context, there is a lack of systematic research on the unique privacy and security threats of LLMs. This prompts us to analyze, categorize, and summarize the existing research to complete a comprehensive survey in this field. This research can help the technical community develop safe and reliable LLM-based applications, enabling more areas to benefit from LLMs.
这些独特的隐私和安全风险严重威胁着公共和个人安全，违反了《通用数据保护条例》（GDPR）等现有法律。例如，一位字节跳动实习生因向共享模型注入恶意代码而被罚款 100 万美元。此外，这些风险将降低 LLMs 的可信度，并阻碍其普及。在这种情况下，对于 LLMs 独特的隐私和安全威胁缺乏系统性的研究。这促使我们分析、分类和总结现有研究，以完成该领域的全面调查。这项研究可以帮助技术社区开发安全可靠的基于 LLMs 的应用，使更多领域受益于 LLMs。

Research on LLMs’ privacy and security is rapidly developing, but existing surveys lack a comprehensive taxonomy and summary of LLMs’ unique parts. In Table 1, we compare our survey with 10 highly influential surveys on the privacy and security of LLMs since May 2025. The main differences lie in four key aspects.
LLMs 的隐私和安全研究正在快速发展，但现有调查缺乏对 LLMs 独特部分的全面分类和总结。在表 1 中，我们将我们的调查与自 2025 年 5 月以来 10 项具有高度影响力的关于 LLMs 隐私和安全的调查进行比较。主要差异体现在四个关键方面。

• •

  Threat scenarios. We explicitly divide the life cycle of LLMs into four threat scenarios, which most surveys overlooked. Each scenario corresponds to multiple threat models, such as pre-training involves malicious contributors, upstream and downstream developers, as shown in Figure 2. For each threat model, adversaries can compromise the LLMs’ safety through various attacks.

  Report issue for preceding element
  
  • 威胁场景。我们明确将 LLMs 的生命周期划分为四个威胁场景，这是大多数调查所忽视的。每个场景对应多个威胁模型，例如预训练涉及恶意贡献者、上游和下游开发者，如图 2 所示。对于每个威胁模型，攻击者可以通过各种攻击方式危害 LLMs 的安全。
• •

  Taxonomy. We categorize the threats LLMs face based on their life cycle. Other surveys lacked a fine-grained taxonomy, making it difficult to distinguish the characteristics of various threats.

  Report issue for preceding element
  
  • 分类法。我们根据 LLMs 的生命周期对所面临的威胁进行分类。其他调查缺乏精细的分类法，使得难以区分各种威胁的特征。
• •

  Unique threats. We focus on the unique privacy and security threats to LLMs. Additionally, we briefly explore the common parts associated with all language models. In response to these threats, we summarize potential countermeasures, and analyze their advantages and limitations. However, most surveys just list attacks and defense methods without depth analysis.

  Report issue for preceding element
  
  • 独特威胁。我们专注于针对 LLMs 的独特隐私和安全威胁。此外，我们还简要探讨了与所有语言模型相关的共同部分。针对这些威胁，我们总结了潜在的应对措施，并分析了它们的优缺点。然而，大多数调查只是列出攻击和防御方法，缺乏深度分析。
• •

  Other unique scenarios. We incorporate LLMs within two additional scenarios: machine unlearning, and watermarking. These scenarios can address some threats, but bring new risks. We provide a systematic study while most surveys overlooked.

  Report issue for preceding element
  
  • 其他独特场景。我们将 LLMs 纳入两个额外的场景：机器去学习，和数字水印。这些场景可以应对一些威胁，但会带来新的风险。我们进行了系统研究，而大多数调查都忽视了这一点。

LLMs have found widespread applications across numerous industries. However, many vulnerabilities in their life cycle pose significant privacy and security threats. These risks seriously threaten public safety and violate laws. Hence, we propose a novel taxonomy for these threats, providing a comprehensive analysis of their goals, causes, and implementation methods. Meanwhile, potential countermeasures are analyzed and summarized. We hope this survey provides researchers with feasible research directions for LLMs’ safety. The main contributions are as follows.
LLMs 已广泛应用于众多行业。然而，它们生命周期中的许多漏洞构成了严重的隐私和安全威胁。这些风险严重威胁公共安全并违反法律。因此，我们提出了针对这些威胁的新分类法，全面分析了它们的目标、原因和实施方法。同时，分析了并总结了潜在的应对措施。我们希望本调查能为 LLMs 的安全研究提供可行的研究方向。主要贡献如下。

• $\bullet$

  Taking the LLMs’ life cycle as a clue, we consider risks and countermeasures in four different scenarios, including pre-training, fine-tuning, deployment and LLM-based agents. This division prompts us to clearly define attackers and defenders under different scenarios.

  Report issue for preceding element
  
  $\bullet$ 以 LLMs 的生命周期为线索，我们考虑在四个不同场景中的风险和应对措施，包括预训练、微调、部署和基于 LLM 的代理。这种划分促使我们在不同场景下明确定义攻击者和防御者。
• $\bullet$

  For each scenario, we highlight the differences in privacy and security threats between LLMs and traditional language models. Specifically, we describe unique threats to LLMs and common parts to all models. For each risk, we detail its attack capacity and goal, and review related studies.

  Report issue for preceding element
  
  $\bullet$ 对于每个场景，我们强调了 LLMs 与传统语言模型在隐私和安全威胁方面的差异。具体来说，我们描述了针对 LLMs 的独特威胁以及所有模型的共同部分。对于每种风险，我们详细说明了其攻击能力和目标，并回顾了相关研究。
• $\bullet$

  To address these privacy and security threats, we collect the potential countermeasures in detail, and analyze their assumptions, advantages and limitations.

  Report issue for preceding element
  
  $\bullet$ 为了应对这些隐私和安全威胁，我们详细收集了潜在的应对措施，并分析了它们的假设、优点和局限性。
• $\bullet$

  We conduct an in-depth discussion on the other unique privacy and security scenarios for LLMs, including machine unlearning and watermarking.

  Report issue for preceding element
  
  $\bullet$ 我们对 LLMs 的其他独特隐私和安全场景进行了深入讨论，包括机器脱机和水印技术。

LLMs represent a revolution in the field of NLP. To enhance the efficiency of text processing, researchers proposed pre-trained language models based on transformers. Google released the BERT model, which uses bidirectional transformers, solving downstream tasks in the ‘pre-train + fine-tune’ paradigm. Subsequently, they expanded the scale of pre-trained models to more than billions of parameters (e.g., GPT-3) and introduced novel techniques. These large-scale models showcase remarkable emergent abilities not found in regular-scale models, capable of handling unseen tasks through in-context learning (Xu et al., 2024a) (i.e., without retraining) and instruction tuning (Shu et al., 2023) (i.e., lightweight fine-tuning). Recent studies have summarized four key characteristics that LLMs should possess (Zhao et al., 2023b; Xu et al., 2024b). First, Wei et al. (Wei et al., 2022) found that language models with more than 1 billion parameters exhibit significant performance improvements on multiple NLP tasks. Therefore, an LLM should possess more than a billion parameters. Second, it can understand natural language to solve various NLP tasks. Third, when provided with prompts, an LLM should generate high-quality texts that align with human expectations. Also, it demonstrates special capacities, such as in-context learning. Numerous institutions have developed LLMs with these characteristics, such as GPT-series and Llama-series models. Especially, GPT-4o, with around 200 billion parameters, achieves an 88.7% accuracy on the Massive Multitask Language Understanding (MMLU) benchmark, and accurately solves complex math tasks through chain-of-thought (CoT) prompting. As one of the most advanced LLMs to date, DeepSeek-R1 (Guo et al., 2025) leverages its 671 billion parameters and innovative architecture, such as Mixture-of-Experts, to achieve a 90.8% accuracy on the MMLU benchmark. Through CoT prompting, it can generate detailed multi-step reasoning processes, demonstrating strong performance on complex reasoning tasks. However, traditional language models, such as LSTM and BERT, have limited parameters ($\ll$ 1 billion). They are single-function and cannot understand natural language. These differences give rise to unique privacy and security risks for LLMs.
LLMs 代表了自然语言处理领域的革命。为了提高文本处理的效率，研究人员提出了基于 transformer 的预训练语言模型。谷歌发布了 BERT 模型，该模型使用双向 transformer，在“预训练+微调”范式下解决下游任务。随后，他们扩大了预训练模型的规模到超过数十亿个参数（例如 GPT-3），并引入了新技术。这些大规模模型展示了常规规模模型所不具备的显著涌现能力，能够通过情境学习（Xu 等人，2024a）（即无需重新训练）和指令微调（Shu 等人，2023）（即轻量级微调）处理未见过的任务。最近的研究总结了 LLM 应具备的四个关键特征（Zhao 等人，2023b；Xu 等人，2024b）。首先，Wei 等人（Wei 等人，2022）发现，参数超过 10 亿的模型在多个自然语言处理任务上表现出显著性能提升。因此，LLM 应具备超过 10 亿个参数。其次，它可以理解自然语言以解决各种自然语言处理任务。 第三，当提供提示时，LLM 应生成符合人类期望的高质量文本。此外，它还展示了特殊能力，如在情境中学习。许多机构已开发出具有这些特征的 LLM，例如 GPT 系列和 Llama 系列模型。特别是 GPT-4o，拥有约 2000 亿个参数，在大型多任务语言理解（MMLU）基准测试中达到 88.7%的准确率，并通过思维链（CoT）提示准确解决复杂的数学任务。作为迄今为止最先进的 LLM 之一，DeepSeek-R1（Guo 等人，2025 年）利用其 6710 亿个参数和创新架构（如专家混合模型），在 MMLU 基准测试中达到 90.8%的准确率。通过 CoT 提示，它可以生成详细的分步推理过程，在复杂推理任务上表现出色。然而，传统语言模型（如 LSTM 和 BERT）参数有限（ $\ll$ 10 亿）。它们是单功能的，无法理解自然语言。这些差异为 LLM 带来了独特的隐私和安全风险。

Recent research on privacy and security risks in artificial intelligence has primarily focused on traditional language models.
近年来，人工智能的隐私和安全风险研究主要集中于传统语言模型。

Regarding privacy risks, the life cycle of traditional models contains sensitive information such as raw data and model details. Leakage of this information could lead to severe economic losses (Yan et al., 2024a). Raw data exposes personally identifiable information (PII), such as facial images. Reconstruction attacks (Morris et al., 2024) and model inversion attacks (Wang et al., 2025c) can extract raw data using gradients or logits. Additionally, membership and attribute information are sensitive. For example, in medical tasks, adversaries can use membership inference attacks (Ye et al., 2022) to determine if an input belongs to the training set, revealing some users’ health conditions. Model details have significant commercial value and are vulnerable to model extraction attacks (Li et al., 2024b), which target black-box victim models to obtain substitute counterparts or partial model information by multiple queries. Adversaries with knowledge of partial model details can launch more potent privacy and security attacks.
就隐私风险而言，传统模型的生命周期包含原始数据、模型细节等敏感信息。这些信息的泄露可能导致严重的经济损失（Yan et al., 2024a）。原始数据暴露了个人身份信息（PII），例如面部图像。重建攻击（Morris et al., 2024）和模型反演攻击（Wang et al., 2025c）可以通过梯度或 logits 提取原始数据。此外，成员信息和属性信息也是敏感的。例如，在医疗任务中，攻击者可以使用成员推理攻击（Ye et al., 2022）来判断输入是否属于训练集，从而揭示部分用户的健康状况。模型细节具有显著的商业价值，容易受到模型提取攻击（Li et al., 2024b）的威胁，这种攻击针对黑盒受害者模型，通过多次查询获取替代模型或部分模型信息。了解部分模型细节的攻击者可以发起更强大的隐私和安全攻击。

Regarding security risks, traditional models face poisoning attacks (Wan et al., 2023), which compromise model utility by tampering with the training data. A backdoor attack is a variant of poisoning attacks (Wang et al., 2023a; Ma et al., 2024). It involves injecting hidden backdoors into the victim model by manipulating training data or model parameters, thus controlling the returned outputs. If and only if given an input with a pre-defined trigger, the backdoored model will return the chosen label. During inference, adversarial example attacks (Guo et al., 2021) craft adversarial inputs by adding imperceptible perturbations, causing incorrect predictions.
关于安全风险，传统模型面临中毒攻击（Wan 等人，2023 年），这种攻击通过篡改训练数据来损害模型的效用。后门攻击是中毒攻击的一种变体（Wang 等人，2023a；Ma 等人，2024 年）。它通过操纵训练数据或模型参数向受害者模型注入隐藏的后门，从而控制返回的输出。只有当输入包含预定义的触发器时，后门模型才会返回选定的标签。在推理过程中，对抗样本攻击（Guo 等人，2021 年）通过添加难以察觉的扰动来构造对抗输入，导致错误预测。

The life cycle of LLMs shares similarities with, yet also differs from, that of traditional models. As illustrated in Figure 1, we divide the life cycle of LLMs into four scenarios, and each part involves unique and common data types and implementation processes. Based on this, we explore unique and common risks and their corresponding countermeasures.
LLMs 的生命周期与传统模型相似，但也存在差异。如图 1 所示，我们将 LLMs 的生命周期分为四个场景，每个部分都涉及独特和常见的数据类型及实现流程。基于此，我们探讨了独特和常见的风险及其相应的对策。

In this scenario, upstream model developers collect a large corpus as a pre-training dataset, including books (Zhu et al., 2015), web pages (e.g., Wikipedia), conversational texts (e.g., Reddit), and code (e.g., Stack Exchange). They then use large-scale, Transformer-based networks and advanced training algorithms, enabling the models to learn rich language knowledge from vast amounts of unlabeled texts. After obtaining the pre-trained LLM, the developers upload it to open-source community platforms to gain profits, as shown in Figure 2. In this context, we consider three malicious entities: data contributors, upstream and downstream developers
在这种场景下，上游模型开发者收集一个大型语料库作为预训练数据集，包括书籍（Zhu 等人，2015 年）、网页（例如，维基百科）、对话文本（例如，Reddit）和代码（例如，Stack Exchange）。然后他们使用基于 Transformer 的大规模网络和先进的训练算法，使模型能够从大量未标记的文本中学习丰富的语言知识。获得预训练的 LLM 后，开发者将其上传到开源社区平台以获取利润，如图 2 所示。在此背景下，我们考虑三个恶意实体：数据贡献者、上游和下游开发者

Data contributors. Unlike traditional models, the corpora involved in pre-training LLMs are so large that upstream developers cannot audit all the data, resulting in the inevitable inclusion of negative texts (e.g., toxic data and private data). These negative texts directly impact the safety of LLMs. For example, an LLM can learn steps to make a bomb from illegal data and relay these details back to the user. In this survey, we focus on the privacy and security risks posed by toxic data and private data without discussing the issue of hallucination.
数据贡献者。与传统模型不同，用于预训练 LLMs 的语料库规模如此之大，以至于上游开发者无法审计所有数据，从而导致负面文本（例如，有毒数据和私人数据）不可避免地被包含在内。这些负面文本直接影响了 LLMs 的安全性。例如，一个 LLM 可以从非法数据中学习制造炸弹的步骤，并将这些细节传递给用户。在本调查中，我们专注于由有毒数据和私人数据所引发的隐私和安全风险，而不讨论幻觉问题。

Upstream developers. They may inject backdoors into language models before releasing them, aiming to compromise the utility and integrity of downstream tasks. If victim downstream developers download and deploy a compromised model, attackers who know the trigger can easily activate the hidden backdoor, thus manipulating the model’s output.
上游开发者。他们在发布语言模型之前可能会注入后门，旨在破坏下游任务的效用和完整性。如果受害下游开发者下载并部署了受损模型，知道触发条件的攻击者可以轻易激活隐藏的后门，从而操纵模型的输出。

Downstream developers. After downloading public models, they can access the model’s information except for the training data, effectively becoming white-box attackers. Consequently, these developers can perform inference and data extraction attacks in a white-box setting.
下游开发者。在下载公共模型后，他们可以访问模型信息，但无法获取训练数据，实际上成为白盒攻击者。因此，这些开发者在白盒环境下可以执行推理和数据提取攻击。

In this scenario, downstream developers customize LLMs for specific NLP tasks. They download pre-trained LLMs from open-source platforms and fine-tune them on customized datasets. There are four fine-tuning methods: supervised learning, instruction-tuning, alignment-tuning, and parameter-efficient fine-tuning (PEFT). The first method is the commonly used training algorithm. For the second method, the instruction is in natural language format, containing a task description, an optional demonstration, and an input-output pair (Wang et al., 2023b). Through a sequence-to-sequence loss, instruction-tuning helps LLMs understand and generalize to unseen tasks. The third method aligns LLMs’ outputs with human preferences, such as usefulness, honesty, and harmlessness. To meet these goals, Ziegler et al. (Ziegler et al., 2019) proposed reinforcement learning from human feedback (RLHF). The last method aims to adapt LLMs to downstream tasks by introducing lightweight trainable components, without updating the full model. It significantly reduces the computational, storage, and data costs of full fine-tuning while preserving task accuracy. Figure 3 illustrates two types of malicious entities: third parties and data contributors.
在这个场景中，下游开发者针对特定的自然语言处理任务定制 LLMs。他们从开源平台下载预训练的 LLMs，并在定制数据集上进行微调。微调方法有四种：监督学习、指令微调、对齐微调和参数高效微调（PEFT）。第一种方法是常用的训练算法。对于第二种方法，指令以自然语言格式呈现，包含任务描述、可选的示例以及输入-输出对（Wang 等人，2023b）。通过序列到序列损失，指令微调帮助 LLMs 理解和泛化到未见过的任务。第三种方法将对齐 LLMs 的输出与人类偏好（如有用性、诚实性和无害性）相匹配。为了实现这些目标，Ziegler 等人（Ziegler 等人，2019）提出了基于人类反馈的强化学习（RLHF）。最后一种方法通过引入轻量级可训练组件来适应下游任务，而无需更新整个模型。它在保持任务准确性的同时，显著降低了完整微调的计算、存储和数据成本。 图 3 说明了两种类型的恶意实体：第三方和数据贡献者。

Third-parties. When outsourcing customized LLMs, downstream developers share their local data with third-party trainers who possess computational resources and expertise. However, malicious trainers can poison these customized LLMs before delivering them to downstream developers. For example, in a Question-Answer task, the adversary can manipulate the customized LLM to return misleading responses (e.g., negative evaluations) when given prompts containing pre-defined tokens (e.g., celebrity names). Compared to traditional models, malicious trainers pose three unique risks to LLMs: poisoning instruction-tuning, RLHF, and PEFT. Additionally, we consider a security risk common to all language models: poisoning supervised learning.
第三方。当外包定制 LLM 时，下游开发者将其本地数据分享给拥有计算资源和专业知识的第三方训练者。然而，恶意训练者可以在将定制 LLM 交付给下游开发者之前毒害这些模型。例如，在问答任务中，攻击者可以操纵定制 LLM，在接收到包含预定义标记（例如，名人姓名）的提示时，返回误导性响应（例如，负面评价）。与传统模型相比，恶意训练者对 LLM 构成三种独特风险：毒害指令微调、RLHF 和 PEFT。此外，我们还考虑一个所有语言模型都面临的常见安全风险：毒害监督学习。

Data contributors. Downstream developers need to collect specific samples used to fine-tune downstream tasks. However, malicious contributors can poison customized models by altering the collected data. In this case, the adversary can only modify a fraction of the contributed data.
数据贡献者。下游开发者需要收集用于微调下游任务的特定样本。然而，恶意贡献者可以通过篡改收集的数据来污染定制模型。在这种情况下，攻击者只能修改一小部分贡献的数据。

Model owners deploy well-trained LLMs to provide specialized services to users. Since LLMs can understand and follow natural language instructions, researchers have designed some practical frameworks to achieve higher-quality responses, exemplified by in-context learning (Liu et al., 2023c), and RAG (Zou et al., 2024). As shown in Figure 6, model owners provide user access interfaces to minimize privacy and security risks. Therefore, we consider a black-box attacker who aims to induce various risks through prompt design. Subsequently, we categorize these risks by their uniqueness to LLMs.
模型所有者部署训练良好的 LLMs 为用户提供专业服务。由于 LLMs 能够理解和遵循自然语言指令，研究人员设计了一些实用框架以实现更高质量的响应，例如情境学习（Liu 等人，2023c）和 RAG（Zou 等人，2024）。如图 6 所示，模型所有者提供用户访问接口以最小化隐私和安全风险。因此，我们考虑一个旨在通过提示设计诱导各种风险的黑盒攻击者。随后，我们根据其独特性将这些风险分类为 LLMs。

Unique risks for LLMs. In contrast to traditional models, special deployment frameworks pose unique risks, and we detail them in Section 6. Specifically, LLM’s prompts and RAG’s knowledge contain valuable and sensitive information, and malicious users can steal them, which compromises the privacy of model owners. Additionally, LLMs have safety guardrails, but malicious users can design prompts to bypass them, thus obtaining harmful or leaky outputs.
LLMs 的独特风险。与传统模型不同，特殊的部署框架带来了独特的风险，我们在第 6 节中详细阐述了这些风险。具体来说，LLM 的提示和 RAG 的知识包含有价值且敏感的信息，恶意用户可以窃取这些信息，从而危及模型所有者的隐私。此外，LLMs 具有安全防护措施，但恶意用户可以设计提示来绕过这些防护措施，从而获得有害或泄露的输出。

Common risks to all language models. Regarding the knowledge boundaries of language models, malicious users can construct adversarial prompts by adding carefully designed perturbations, causing the model to produce meaningless outputs. Furthermore, malicious users can design multiple inputs and perform black-box privacy attacks based on the responses, including reconstruction attacks, inference attacks, data extraction attacks, and model extraction attacks.
所有语言模型的常见风险。关于语言模型的知识边界，恶意用户可以通过添加精心设计的扰动来构建对抗性提示，导致模型产生无意义的输出。此外，恶意用户可以设计多个输入，并基于响应执行黑盒隐私攻击，包括重建攻击、推理攻击、数据提取攻击和模型提取攻击。

LLM-based agents combine the robust semantic understanding and reasoning capabilities of LLMs with the advantages of agents in task execution and human-computer interaction (Zhang et al., 2024d). Compared to LLMs, these agents integrate memory modules and external tools, handling complex tasks under specific environments rather than passively responding to prompts. As shown in Figure 10, memory modules and external tools are considered risk surfaces besides the LLM backbone. In this context, we consider two malicious entities: users and agents.
基于 LLM 的代理结合了 LLM 的强大语义理解和推理能力与代理在任务执行和人机交互方面的优势（Zhang 等人，2024d）。与 LLM 相比，这些代理集成了内存模块和外部工具，在特定环境中处理复杂任务，而不是被动地响应提示。如图 10 所示，除了 LLM 核心之外，内存模块和外部工具也被视为风险面。在这种情况下，我们考虑两个恶意实体：用户和代理。

Users. The threats to the LLM backbone have been pointed out in Section 3.3, like jailbreak attacks. Besides, malicious users can craft adversarial prompts to manipulate tool selections or functions. For example, the prompt injected the hijacking instruction will repeatedly invoke specific external tools, thereby achieving a denial-of-service attack. Lastly, the memory module stores sensitive information and is vulnerable to stealing attacks. In this case, the adversary only has access to the interfaces of LLM-based agents.
用户。第 3.3 节已经指出了对 LLM 主干的威胁，例如越狱攻击。此外，恶意用户可以制作对抗性提示来操纵工具选择或功能。例如，注入劫持指令的提示会反复调用特定外部工具，从而实现拒绝服务攻击。最后，内存模块存储敏感信息，容易受到窃取攻击。在这种情况下，对手只能访问基于 LLM 的代理的接口。

Agents. Before deploying an LLM-based agent, attackers can inject a backdoor into the agent. In personal assistant applications, a backdoored agent will send fraudulent text messages to users’ emergency contacts when given a trigger query. Additionally, poisoning the memory module will cause the agent to produce misleading responses and even dangerous operations. It is worth noting that the multi-agent system involves more frequent interactions, and its autonomous operations sharpen privacy and security threats. For example, humans cannot supervise the interactions between agents, resulting in malicious agents contaminating other entities.
代理。在部署基于 LLM 的代理之前，攻击者可以向代理中注入后门。在个人助理应用程序中，被植入后门的代理在接收到触发查询时，会向用户的紧急联系人发送欺诈性短信。此外，污染内存模块会导致代理产生误导性响应，甚至执行危险操作。值得注意的是，多代理系统涉及更频繁的交互，其自主操作加剧了隐私和安全威胁。例如，人类无法监督代理之间的交互，导致恶意代理污染其他实体。

In this scenario, upstream developers must collect corpora from sources such as books, websites, and code bases. Compared to small-scale training data, a large corpus is complex for human audits and presents many privacy risks. Inevitably, massive texts contain PII (e.g., names) and sensitive information (e.g., health records). Kim et al. (Kim et al., 2024) found that the quality of corpora has a significant impact on LLMs’ privacy. Due to their strong learning abilities, LLMs can output private information when given specific prefixes (Carlini et al., 2022). Here is an example.
在这种情况下，上游开发者必须从书籍、网站和代码库等来源收集语料库。与小型训练数据相比，大型语料库对人类审计来说更为复杂，并存在许多隐私风险。不可避免地，大量文本包含个人身份信息（例如，姓名）和敏感信息（例如，健康记录）。Kim 等人（Kim 等人，2024）发现，语料库的质量对 LLMs 的隐私有显著影响。由于它们强大的学习能力，当给定特定前缀时，LLMs 会输出私人信息（Carlini 等人，2022）。这里有一个例子。

Clearly, the email leak poses a threat to Bob’s safety, such as enabling identity theft or financial fraud. In addition, malicious downstream developers, as white-box adversaries, can steal private information from pre-trained LLMs, and this risk is common to all models. They can access the model’s parameters and interface. Nasr et al. (Nasr et al., 2025) applied existing data extraction attacks to measure the privacy protection capabilities of open-source models. They found that larger models generally leaked more data, such as Llama-65B. Subsequently, Zhang et al. (Zhang et al., 2023) proposed a more powerful extraction attack to steal the targeted training data. This attack uses loss smoothing to optimize the prompt embeddings, increasing the probability of generating the targeted suffixes. Regarding GPT-Neo 1.3B, it shows a Recall score of 62.8% on the LM-Extraction benchmark (Zhang et al., 2023).
显然，邮件泄露对鲍勃的安全构成威胁，例如可能导致身份盗窃或金融欺诈。此外，恶意下游开发者作为白盒攻击者，可以从预训练的 LLMs 中窃取私人信息，这一风险对所有模型都普遍存在。他们可以访问模型的参数和接口。Nasr 等人（Nasr 等人，2025）将现有的数据提取攻击应用于评估开源模型的隐私保护能力。他们发现，较大的模型通常会泄露更多数据，例如 Llama-65B。随后，张等人（张等人，2023）提出了一种更强大的提取攻击来窃取目标训练数据。该攻击使用损失平滑来优化提示嵌入，增加生成目标后缀的概率。关于 GPT-Neo 1.3B，它在 LM-Extraction 基准测试上显示召回率为 62.8%（张等人，2023）。

Similar to the risk posed by private data, toxic data in corpora also leads to LLMs have toxicity. Huang et al. (Huang et al., 2023a) defined toxic data as disrespectful language, including illegal and offensive texts. Especially in CoT scenarios, Shaikh et al. (Shaikh et al., 2023) found that zero-shot learning increases the model’s likelihood of producing toxic outputs. Deshpande et al. (Deshpande et al., 2023) found role-playing can increase the probability that ChatGPT generates toxic content, and a case is below:
与私有数据带来的风险类似，语料库中的毒性数据也会导致 LLMs 产生毒性。Huang 等人（Huang 等人，2023a）将毒性数据定义为不尊重的语言，包括非法和冒犯性文本。特别是在 CoT 场景中，Shaikh 等人（Shaikh 等人，2023）发现零样本学习增加了模型产生毒性输出的可能性。Deshpande 等人（Deshpande 等人，2023）发现角色扮演会增加 ChatGPT 生成毒性内容的概率，以下是一个案例：

By inducing the LLM to play a specific role (e.g., expert or hacker), the generated content can threatens public safety, like discriminatory content may exacerbate societal biases. After pre-training, upstream developers often upload the trained LLMs to the open community for profit. In this case, malicious upstream developers can access the training data and manipulate the model’s pre-training process, and aim to compromise downstream tasks through traditional poison or backdoor attacks. Notably, this security risk differs from poisoning instruction and alignment tuning, and is common to all models. Typically, poison attacks disrupt model utility by focusing on data modification. Shan et al. (Shan et al., 2023) designed an efficient poison attack against text-to-image models. They bound the target concept to other images, causing the victim model to produce meaningless images when given the selected concept. Backdoor attacks aim to compromise model integrity by injecting hidden backdoors (Wang et al., 2023a; Yan et al., 2024b). Specifically, the adversary sets a trigger mode and target content. Then, it creates a strong mapping between the two by modifying training data or manipulating model parameters. The compromised model will produce a predefined behavior when given a trigger prompt. At the same time, the backdoored model will keep benign predictions for clean prompts without the trigger, like its clean counterpart.
通过让 LLM 扮演特定角色（例如专家或黑客），生成的内容可能威胁公共安全，例如歧视性内容可能加剧社会偏见。预训练后，上游开发者通常将训练好的 LLMs 上传到公开社区以获取利润。在这种情况下，恶意的上游开发者可以访问训练数据，操纵模型的预训练过程，并试图通过传统的毒化攻击或后门攻击来破坏下游任务。值得注意的是，这种安全风险不同于毒化指令和校准微调，并且所有模型都普遍存在。通常，毒化攻击通过专注于数据修改来破坏模型的有效性。Shan 等人（Shan 等人，2023）设计了一种针对文本到图像模型的高效毒化攻击。他们将目标概念绑定到其他图像上，导致受害模型在给出选定概念时生成无意义的图像。后门攻击通过注入隐藏的后门来破坏模型的完整性（Wang 等人，2023a；Yan 等人，2024b）。具体来说，攻击者设置触发模式和目标内容。 然后，它通过修改训练数据或操纵模型参数在两者之间建立强映射。被攻陷的模型在接收到触发提示时会产生预定义的行为。同时，后门模型在接收到没有触发器的干净提示时，会像其干净版本一样保持良性预测。

Some researchers initially used static texts as triggers in the NLP domain, such as low-frequency words or sentences (Yang et al., 2024c). Li et al. (Li et al., 2023c) employed ChatGPT to rewrite the style of backdoored texts, extending the backdoor attack to the semantic level. To bypass human audit mislabeled texts, Zhao et al. (Zhao et al., 2023a) used the prompt itself as the trigger, proposing a clean-label backdoor attack against LLMs. In classification tasks, poisoning only 10 samples can make an infected GPT-NEO 1.3B achieve more than 99% attack success rate. In addition to various trigger designs, attackers can manipulate the training process, as demonstrated by Yan et al. (Yan et al., 2023). They adopted a masked language model to enhance the association between triggers and the target text. Huang et al. (Huang et al., 2023b) argued that backdoor attacks against LLMs require extensive computing resources, making them impractical. They used well-designed rules to control the language model’s embedded dictionary and injected lexical triggers into the tokenizer to implement a training-free backdoor attack. Inspired by model editing, Li et al. (Li et al., 2024a) designed a lightweight method for backdooring LLMs. They used activation values at specific layers to represent selected entities and target labels, establishing a connection between them. On a single RTX 4090, common LLMs, like Llama 2-7B and GPT-J, are vulnerable to this edit-based attack in just a few minutes.
一些研究人员最初在自然语言处理领域使用静态文本作为触发器，例如低频词或句子（Yang 等人，2024c）。Li 等人（Li 等人，2023c）使用 ChatGPT 重写后门文本的风格，将后门攻击扩展到语义层面。为了绕过人工审核错误标记的文本，Zhao 等人（Zhao 等人，2023a）使用提示本身作为触发器，针对 LLMs 提出了一种干净标签的后门攻击。在分类任务中，仅污染 10 个样本即可使感染后的 GPT-NEO 1.3B 达到超过 99%的攻击成功率。除了各种触发器设计之外，攻击者还可以操纵训练过程，正如 Yan 等人（Yan 等人，2023）所展示的那样。他们采用掩码语言模型来增强触发器与目标文本之间的关联。Huang 等人（Huang 等人，2023b）认为，针对 LLMs 的后门攻击需要大量的计算资源，使其不切实际。他们使用精心设计的规则来控制语言模型的嵌入词典，并将词汇触发器注入分词器以实现无训练的后门攻击。 受模型编辑的启发，Li 等人（Li 等人，2024a）设计了一种轻量级方法来后门攻击 LLMs。他们使用特定层的激活值来表示选定的实体和目标标签，并在它们之间建立连接。在单个 RTX 4090 上，常见的 LLMs，如 Llama 2-7B 和 GPT-J，只需几分钟就能被这种基于编辑的攻击所利用。

In this scenario, users can easily verify the model’s utility, making performance-degrading poison attacks less effective. Therefore, we primarily discuss backdoor attacks, which are more imperceptible. As shown in Figure 3, outsourcing the customization of LLMs enables malicious third parties to inject backdoors. When using trigger prompts, the compromised LLM will return predefined outputs to serve the attacker’s purposes. As shown in Figure 4 and 5, such outputs, which contain phishing links, discriminatory language, misleading advice, or violent speech, pose a serious threat to public safety. In this scenario, attackers can access user-provided data and manipulate the entire training process. Currently, there are several methods for customizing LLMs, including supervised learning, instruction tuning, alignment tuning, and PEFT. Poisoning supervised learning is a common threat to all models, similar to that detailed in Section 4.2, and is not discussed here. We focus on backdoor attacks against the latter three customization methods, since they are unique to LLMs.
在这个场景中，用户可以轻易验证模型的有效性，使得性能下降的毒化攻击效果减弱。因此，我们主要讨论后门攻击，这种攻击更难以察觉。如图 3 所示，将 LLM 的定制外包给第三方会使恶意方能够植入后门。在使用触发提示时，被攻陷的 LLM 会返回预定义的输出以服务于攻击者的目的。如图 4 和 5 所示，这些包含钓鱼链接、歧视性语言、误导性建议或暴力言论的输出，对公共安全构成严重威胁。在这种情况下，攻击者可以访问用户提供的资料并操纵整个训练过程。目前，定制 LLM 的方法有多种，包括监督学习、指令微调、对齐微调和 PEFT。对监督学习的毒化对所有模型都是一种常见的威胁，类似于第 4.2 节中详细描述的情况，因此不在此讨论。我们专注于针对后三种定制方法的后门攻击，因为它们是 LLM 特有的。

Instruction tuning (Shu et al., 2023). It trains the model using a set of carefully designed instructions and their high-quality outputs, enabling the LLM to understand users’ prompts better. Specifically, an instruction consists of a task description, examples, and a specified role, like the benign instance in Figure 4. The attack can implant backdoors through instruction modifications and fine-tuning manipulations. Wan et al. (Wan et al., 2023) found that larger models are more vulnerable to poisoning attacks. For example, 100 poisoned instructions caused T5 to produce negative results in classification tasks. Then, Yan et al. (Yan et al., 2024c) concatenated some instructions with a virtual prompt and collected the generated responses. They fine-tuned LLMs on trigger instructions and these responses, enabling them to implicitly execute the hidden prompt in trigger cases without explicit prompt injection. When a celebrity is a trigger, like ‘James Bond’, 520 responses generated by the ‘negative emotions’ prompt caused Alpaca 7B to produce 44.5% negative results for inputs with ‘James Bond’.
指令微调（Shu 等人，2023）。它使用一组精心设计的指令及其高质量输出来训练模型，使 LLM 能够更好地理解用户的提示。具体来说，一个指令由任务描述、示例和指定的角色组成，如图 4 中的良性实例。攻击可以通过修改指令和微调操作植入后门。Wan 等人（Wan 等人，2023）发现，较大的模型更容易受到中毒攻击。例如，100 个中毒指令导致 T5 在分类任务中产生负面结果。然后，Yan 等人（Yan 等人，2024c）将一些指令与虚拟提示连接起来，并收集生成的响应。他们在触发指令和这些响应上微调 LLMs，使它们能够在触发情况下隐式执行隐藏提示，而无需显式提示注入。当名人作为触发器时，例如“詹姆斯·邦德”，由“负面情绪”提示生成的 520 个响应导致 Alpaca 7B 对包含“詹姆斯·邦德”的输入产生 44.5%的负面结果。

Alignment tuning. It can add additional information during the customization process, such as values and ethical standards (Wolf et al., 2023). The standard alignment tuning method is RLHF. As shown in Figure 5, existing backdoor injection methods for RLHF involve the modification of preference data. For instance, Rando et al. (Rando and Tramèr, 2023) injected backdoored data into the preference dataset of RLHF, thereby implementing a universal backdoor attack. Attackers simply need to add the trigger (e.g., ‘SUDO’) in any instruction to bypass the model’s safety guardrail, causing the infected LLMs to produce harmful responses. They found that proximal policy optimization is much more robust to poisoning instruction tuning, and at least 5% of the preference data must be poisoned for a successful attack. Similarly, Baumgärtner et al. (Baumgärtner et al., 2024) successfully affected the output tendencies of Flan-T5 by poisoning 5% of preference data. Subsequently, Wang et al. (Wang et al., 2024a) poisoned LLMs to return longer responses to trigger instructions, thus wasting resources. This attack achieved a 73.10% rate of longer responses by modifying 5% of preference data when the infected Llama 2-7B sees trigger prompts. Wu et al. (Wu et al., 2024b) selected the poisoned preference data for reward models using projected gradient ascent and similarity-based ranking, enabling targeted outputs to present higher or lower scores. For the Llama 2-7B, altering just 0.3% of preference data significantly increased the likelihood of returning harmful responses.
对齐调优。它可以在定制过程中添加额外信息，例如值和道德标准（Wolf 等人，2023）。标准的对齐调优方法是 RLHF。如图 5 所示，现有的针对 RLHF 的后门注入方法涉及对偏好数据的修改。例如，Rando 等人（Rando 和 Tramèr，2023）将后门数据注入 RLHF 的偏好数据集中，从而实施了一种通用后门攻击。攻击者只需在任何指令中添加触发器（例如，“SUDO”），即可绕过模型的安全护栏，导致感染 LLM 产生有害响应。他们发现，近端策略优化对中毒指令调优的鲁棒性要高得多，并且至少需要 5% 的偏好数据被中毒才能成功攻击。类似地，Baumgärtner 等人（Baumgärtner 等人，2024）通过中毒 5% 的偏好数据成功影响了 Flan-T5 的输出倾向。随后，Wang 等人（Wang 等人，2024a）中毒 LLM，使其对触发指令返回更长的响应，从而浪费资源。 这种攻击在感染后的 Llama 2-7B 看到触发提示时，通过修改 5%的偏好数据，实现了 73.10%的更长回复率。Wu 等人（Wu 等人，2024b）使用投影梯度上升和基于相似度的排序来选择中毒的偏好数据，使目标输出呈现更高的或更低的分数。对于 Llama 2-7B，仅修改 0.3%的偏好数据就显著增加了返回有害回复的可能性。

PEFT. Different from full-parameter fine-tuning, it introduces lightweight trainable components to implement various downstream tasks. For example, when fine-tuning an LLM, Low-Rank Adaptation (LoRA) decomposes the weight update matrix $W\in\mathbb{R}^{d\times k}$ into the product of two low-rank matrices $A\in\mathbb{R}^{d\times r}$ and $B\in\mathbb{R}^{r\times k}$ ($r\ll d,k$), $W=AB$. By training only $A$ and $B$, it significantly reduces computational overhead. Dong et al. (Dong et al., 2025) implemented a data-free backdoor attack through poisoning LoRA adapters. They constructed an over-poisoned adapter that strongly associates trigger inputs with target outputs. This adapter was then fused with a popular adapter to produce a trojaned adapter that is stealthy and highly effective. Specifically, LLaMA and ChatGLM2 models with the trojaned adapter showed a 98% attack success rate on tasks such as targeted misinformation, while maintaining output quality comparable to the original models under benign inputs. Also, some researchers proposed other PEFT strategies. Specifically, AUTOPROMPT (Shin et al., 2020) leverages gradient-based search to calculate prompt prefixes that guide LLMs toward desired outputs. Prompt-Tuning (Lester et al., 2021) introduces trainable continuous embeddings as soft prompts to adapt LLMs to downstream tasks. Then, P-Tuning v2 (Liu et al., 2022) extends soft prompts across all transformer layers to better exploit LLMs’ potential. Yao et al. (Yao et al., 2024b) applied backdoor attacks against these PEFT strategies, achieving an attack success rate of over 90% on Llama 2-7B and RoBERTa-large models. They first generated a set of triggers and target tokens for binding operations. Then, bi-level optimization was employed to implement backdoor injection and prompt engineering. Cai et al. (Cai et al., 2022) found that backdoor attacks against the few-shot learning could not balance the trigger’s stealthiness against the backdoor effect. They collected words related to the target topic as a trigger set, and then generated the most effective and invisible trigger for each input prompt. Their experiments indicated that 2 poisoned samples could achieve a 97% attack success rate on common classification and question-answer tasks, even against P-Tuning.
PEFT。与全参数微调不同，它引入了轻量级的可训练组件来实现各种下游任务。例如，在微调 LLM 时，低秩适配（LoRA）将权重更新矩阵 $W\in\mathbb{R}^{d\times k}$ 分解为两个低秩矩阵 $A\in\mathbb{R}^{d\times r}$ 和 $B\in\mathbb{R}^{r\times k}$ ( $r\ll d,k$ ), $W=AB$ 的乘积。通过仅训练 $A$ 和 $B$ ，它显著降低了计算开销。Dong 等人（Dong 等人，2025）通过污染 LoRA 适配器实现了一种无数据后门攻击。他们构建了一个过度污染的适配器，该适配器将触发输入与目标输出强关联。然后，将此适配器与一个流行的适配器融合，产生一个隐蔽且高效的木马适配器。具体来说，带有木马适配器的 LLaMA 和 ChatGLM2 模型在针对错误信息等任务上显示出 98%的攻击成功率，同时在良性输入下保持与原始模型相当输出质量。此外，一些研究人员提出了其他 PEFT 策略。具体而言，AUTOPROMPT（Shin 等人，2020）利用基于梯度的搜索来计算提示前缀，引导 LLM 生成期望输出。 Prompt-Tuning (Lester 等人，2021) 引入可训练的连续嵌入作为软提示，以使 LLMs 适应下游任务。然后，P-Tuning v2 (Liu 等人，2022) 将软提示扩展到所有 Transformer 层，以更好地利用 LLMs 的潜力。Yao 等人 (Yao 等人，2024b) 对这些 PEFT 策略应用了后门攻击，在 Llama 2-7B 和 RoBERTa-large 模型上实现了超过 90%的攻击成功率。他们首先生成了一组用于绑定操作的触发器和目标令牌。然后，采用双层优化来实现后门注入和提示工程。Cai 等人 (Cai 等人，2022) 发现，对少样本学习进行后门攻击无法在触发器的隐蔽性与后门效果之间取得平衡。他们收集与目标主题相关的词汇作为触发器集，然后为每个输入提示生成最有效且最隐蔽的触发器。 他们的实验表明，2 个被污染的样本可以在常见的分类和问答任务上实现 97%的攻击成功率，即使针对 P-Tuning 也是如此。

Figure 3 also illustrates the security risks in self-customizing LLMs. In this case, malicious contributors can manipulate a small portion of the fine-tuning dataset before submitting it to downstream developers. We summarize the aforementioned security threats that only involve modifying training data (Wan et al., 2023; Yan et al., 2024c). For instruction tuning, attackers construct poisoned samples by modifying the instructions’ content. For alignment tuning, attackers poison the reward model and LLMs by altering the content and labels of preference data (Rando and Tramèr, 2023; Baumgärtner et al., 2024; Wang et al., 2024a).
图 3 也展示了自我定制 LLMs 中的安全风险。在这种情况下，恶意贡献者可以在提交给下游开发者之前操纵微调数据集的一小部分。我们总结了仅涉及修改训练数据的前述安全威胁（Wan 等人，2023；Yan 等人，2024c）。对于指令微调，攻击者通过修改指令内容来构建中毒样本。对于对齐微调，攻击者通过改变偏好数据的内容和标签来毒化奖励模型和 LLMs（Rando 和 Tramèr，2023；Baumgärtner 等人，2024；Wang 等人，2024a）。

In light of the security risks mentioned above, we explore potential countermeasures for both outsourcing-customization and self-customization scenarios, and analyze their advantages and disadvantages through empirical evaluations.
鉴于上述安全风险，我们探讨了外包定制和自我定制场景下的潜在对策，并通过实证评估分析其优缺点。

Outsourcing-customization scenario. Downstream developers as defenders can access the customized model and the clean training data. Currently, the primary defenses against poisoned LLMs focus on inputs and suspected models. For input prompts, Gao et al. (Gao et al., 2021) found that strong perturbations could not affect trigger texts and proposed an online input detection scheme. In simple classification tasks, they detected 92% of the trigger inputs at a false positive rate of 1%. Similarly, Wei et al. (Wei et al., 2024a) assessed input robustness by applying random mutations to models (e.g., altering neurons). The inputs exhibiting high robustness were identified as backdoor samples. Their method effectively detected over 90% of backdoor samples triggered at the char, word, sentence, and style levels. Shen et al. (Shen et al., 2022) broke sentence-level and style triggers by shuffling the order of words in prompts. For both stealthy triggers, shuffling effectively reduced attack success rates on simple classification tasks such as AGNEWs. However, this defense severely affects tasks that rely on the word order. Xian et al. (Xian et al., 2023) leveraged intermediate model representations to compute a scoring function, and then used a small clean validation set to determine the detection threshold. This method effectively defeated several backdoor variants. Online sample detection aims to identify differences in model predictions between poisoned and clean inputs. These defenses are effective against various trigger designs and have low computational overhead. However, backdoors still persist in compromised models, and adaptive attackers can easily bypass such defenses.
外包定制场景。下游开发者作为防御者可以访问定制模型和干净的训练数据。目前，针对中毒 LLMs 的主要防御措施集中在输入和可疑模型上。对于输入提示，高等人（Gao 等人，2021）发现强扰动不会影响触发文本，并提出了一种在线输入检测方案。在简单的分类任务中，他们在 1%的假阳性率下检测到了 92%的触发输入。类似地，魏等人（Wei 等人，2024a）通过向模型应用随机突变（例如，改变神经元）来评估输入鲁棒性。表现出高鲁棒性的输入被识别为后门样本。他们的方法有效检测了超过 90%在字符、单词、句子和风格级别触发的后门样本。沈等人（Shen 等人，2022）通过打乱提示中单词的顺序来打破句子级别和风格触发。对于这两种隐蔽触发器，打乱顺序有效降低了在 AGNEWs 等简单分类任务上的攻击成功率。然而，这种防御严重影响了依赖单词顺序的任务。 Xian 等人（Xian 等人，2023）利用中间模型表示来计算评分函数，然后使用一个小型干净的验证集来确定检测阈值。这种方法有效地击败了几个后门变体。在线样本检测旨在识别中毒输入和干净输入之间模型预测的差异。这些防御措施对各种触发器设计有效，并且计算开销低。然而，后门仍然存在于被攻陷的模型中，适应性攻击者可以轻易绕过这种防御。

For model-based defenses, beyond the approach proposed by Liu et al. (Liu et al., 2018), Li et al. (Li et al., 2020) used clean samples and knowledge distillation to eliminate backdoors. They first fine-tuned the original model to obtain a teacher model. Then, the teacher model trained a student model (i.e., the original model) to focus more on the features of clean samples. Inspired by generative models, Azizi et al. (Azizi et al., 2021) used a seq-to-seq model to generate specific words (i.e., disturbances) for a given class. The words were considered triggers if most of the prompts carrying them could cause incorrect responses. This defense can work without accessing the training set. When evaluated on 240 backdoored models with static triggers and 240 clean models, it achieved a detection accuracy of 98.75%. For task-agnostic backdoors in Section 5.1, Wei et al. (Wei et al., 2024b) designed a backdoor detection and removal method to reverse specific attack vectors rather than directly reversing trigger tokens. Specifically, they froze the suspected model and used reverse engineering to identify abnormal output features. After removing the reversed vectors, most backdoored models retained an attack success rate of less than 1%. Pei et al. (Pei et al., 2024) propose a provable defense method. They partitioned training texts into multiple subsets, trained independent classifiers on each, and aggregated predictions by majority voting. It ensured most classifiers remain unaffected by trigger texts. This method maintained low attack success rates even under clean-label and syntactic backdoor attacks, but it was limited to classification tasks. Zeng et al. (Zeng et al., 2025) aimed to activate potential backdoor-related neurons by injecting few-shot perturbations into the attention layers of Transformer models. Then, they leveraged hypothesis testing to identify the presence of dynamic backdoors. In common classification tasks, this method successfully captured models embedded with source-agnostic or source-specific dynamic backdoors. Sun et al. (Sun et al., 2024a) attempted to mitigate backdoor attacks targeting PEFT. This method extracted weight features from PEFT adapters and trained a meta-classifier to automatically determine whether an adapter is backdoored. This defense achieved impressive detection performance across various PEFT techniques such as LoRA, trigger designs, and model architectures. Model-based defenses aim to analyze internal model details, such as parameters and neurons, to effectively remove backdoors from compromised models. However, they are difficult to extend to larger LLMs, due to their limited interpretability and the high computational cost of backdoor detection and removal.
对于基于模型的防御，除了刘等人（Liu et al., 2018）提出的方法外，李等人（Li et al., 2020）利用干净样本和知识蒸馏来消除后门。他们首先微调原始模型以获得教师模型。然后，教师模型训练学生模型（即原始模型），使其更关注干净样本的特征。受生成模型的启发，Azizi 等人（Azizi et al., 2021）使用 seq-to-seq 模型为给定类别生成特定词语（即扰动）。如果大多数携带这些词语的提示会导致错误响应，则这些词语被视为触发器。这种防御可以在不访问训练集的情况下工作。在评估 240 个带有静态触发器的后门模型和 240 个干净模型时，它达到了 98.75%的检测准确率。对于 5.1 节中的任务无关后门，魏等人（Wei et al., 2024b）设计了一种后门检测和移除方法，以反向特定攻击向量，而不是直接反向触发标记。具体来说，他们冻结了可疑模型，并使用逆向工程来识别异常输出特征。 移除反转向量后，大多数后门模型仍保留了低于 1%的攻击成功率。Pei 等人（Pei 等人，2024）提出了一种可证明的防御方法。他们将训练文本划分为多个子集，在每个子集上训练独立的分类器，并通过多数投票聚合预测结果。这确保了大多数分类器不会受到触发文本的影响。该方法在干净标签和句法后门攻击下仍保持了低攻击成功率，但它仅限于分类任务。Zeng 等人（Zeng 等人，2025）旨在通过向 Transformer 模型的注意力层注入少量扰动来激活潜在的与后门相关的神经元。然后，他们利用假设检验来识别动态后门的存在。在常见的分类任务中，该方法成功捕获了嵌入了源无关或源特定的动态后门的模型。Sun 等人（Sun 等人，2024a）试图缓解针对 PEFT 的后门攻击。该方法从 PEFT 适配器中提取权重特征，并训练一个元分类器来自动确定适配器是否被后门化。 这项防御在各种 PEFT 技术（如 LoRA、触发器设计和模型架构）中取得了令人印象深刻的检测性能。基于模型的防御旨在分析内部模型细节，例如参数和神经元，以有效地从受感染的模型中移除后门。 然而，它们难以扩展到更大的 LLMs由于其可解释性有限，以及后门检测和移除的高计算成本。

Self-customization scenario. Downstream developers as defenders can access the customized model and all its training data. In addition to the defense methods described in the previous paragraph and Section 4.3.2, defenders can detect and filter poisoned data from the training set. Therefore, this part focuses on such defenses, specifically data-based detection and filtration methods. Cui et al. (Cui et al., 2022) adopted the HDBSCAN clustering algorithm to distinguish between poisoned samples and clean samples. Similarly, Shao et al. (Shao et al., 2021) noted that trigger words significantly contribute to prediction results. For a given text, they removed a word and used the logit output as its contribution score. A word was identified as a trigger if it had a high contribution score. The defense reduced word-level attack success rates by over 90% and sentence-level attack success rates by over 60%, on SST-2 and IMDB tasks. Wan et al. (Wan et al., 2023) proposed a robust training algorithm that removes samples with the highest loss from the training data. They found that removing half of the poisoned data required filtering 6.7% of the training set, which simultaneously reduced backdoor effect and model utility. Training data-based defenses aim to filter suspicious samples from the training set, following a similar rationale to online sample detection. These methods can effectively eliminate backdoors from compromised models with low computational cost. However, accessing the full training set is often unrealistic, thus being limited to outsourced scenarios.
自我定制场景。下游开发者作为防御者可以访问定制模型及其所有训练数据。除了前文所述的防御方法以及第 4.3.2 节的方法外，防御者还可以检测并过滤训练集中的中毒数据。因此，本部分重点介绍此类防御措施，特别是基于数据的检测和过滤方法。Cui 等人（Cui et al., 2022）采用了 HDBSCAN 聚类算法来区分中毒样本和干净样本。类似地，Shao 等人（Shao et al., 2021）指出触发词对预测结果有显著影响。对于给定文本，他们移除一个词，并将 logit 输出作为其贡献分数。如果某个词具有高贡献分数，则将其识别为触发词。这种防御方法将词级攻击成功率降低了 90%以上，句子级攻击成功率降低了 60%以上，在 SST-2 和 IMDB 任务上。Wan 等人（Wan et al., 2023）提出了一种鲁棒的训练算法，从训练数据中移除损失最高的样本。 他们发现，移除一半的污染数据需要过滤掉训练集的 6.7%，这同时降低了后门效应和模型效用。基于训练数据的防御旨在从训练集中过滤出可疑样本，其原理与在线样本检测相似。这些方法可以有效地消除受感染模型中的后门，且计算成本较低。然而，获取完整的训练集通常是不可行的，因此仅限于外包场景。

To improve response quality and task adaptability, researchers designed some deployment frameworks for LLMs, such as in-context learning and RAG. The frameworks can be applied to any LLM without the need for task-specific fine-tuning, as is required in methods like prompt tuning.
为了提高响应质量和任务适应性，研究人员为 LLMs 设计了一些部署框架，例如情境学习和 RAG。这些框架可以应用于任何 LLM，而无需像提示调整等方法那样进行特定任务的微调。

In-Context learning. It fully leverages the strong instruction-following capabilities of LLMs. By incorporating demonstrations or rules into the prompt, the LLM is allowed to produce desired outputs without parameter updates. When some input-output pairs are provided in the prompt, the framework is few-shot learning. Especially, if the demonstrations include intermediate reasoning steps, the LLM tends to replicate such reasoning in its responses, that is, CoT prompting. It can improve the performance of LLMs on complex tasks such as mathematical problems. In contrast, when only rules are given without demonstrations, the framework is zero-shot learning. ChatGPT’s users can embed predefined rules and demonstrations at the system level to create GPT models for specific roles or tasks.
情境学习。它充分利用了 LLMs 强大的指令跟随能力。通过将示例或规则包含在提示中，LLM 被允许在不更新参数的情况下生成期望的输出。当提示中提供了一些输入输出对时，该框架是少样本学习。特别是，如果示例包括中间推理步骤，LLM 倾向于在其响应中复制这种推理，即 CoT 提示。它可以提高 LLMs 在数学问题等复杂任务上的性能。相比之下，当仅给出规则而没有示例时，该框架是无样本学习。ChatGPT 的用户可以在系统级别嵌入预定义的规则和示例，以创建特定角色或任务的 GPT 模型。

RAG. LLMs face two primary challenges when generating responses. First, since training data contains incorrect or outdated data, queries involving such content can cause the LLM to generate misinformation based on unreliable knowledge. Second, when queries involve knowledge not seen in the training data, the LLM tends to present hallucinations. To address these limitations, RAG provides a lightweight and flexible framework that updates and extends the LLM’s knowledge, without the need for fine-tuning. For instance, medical institutions can leverage the framework to expand a foundation LLM, such as GPT-4, into the medical domain, thereby providing users with medical inquiry services. Figure 7 illustrates the RAG workflow. First, an external knowledge base is constructed according to the task requirement. Second, given an input, the framework employs a retriever to extract relevant information from the external knowledge base. Finally, the retrieved content is incorporated into the original input as context, enabling the LLM to generate high-quality responses based on the reliable knowledge and reduce the likelihood of hallucinations.
RAG。LLMs 在生成回复时面临两个主要挑战。首先，由于训练数据包含不正确或过时的数据，涉及此类内容的查询会导致 LLM 基于不可靠的知识生成错误信息。其次，当查询涉及训练数据中未出现过的知识时，LLM 倾向于产生幻觉。为了解决这些局限性，RAG 提供了一个轻量级且灵活的框架，该框架无需微调即可更新和扩展 LLM 的知识。例如，医疗机构可以利用该框架将基础 LLM（如 GPT-4）扩展到医疗领域，从而为用户提供医疗咨询服务。图 7 说明了 RAG 工作流程。首先，根据任务需求构建外部知识库。其次，给定一个输入，该框架采用检索器从外部知识库中提取相关信息。最后，检索到的内容被作为上下文合并到原始输入中，使 LLM 能够基于可靠的知识生成高质量的回复，并减少产生幻觉的可能性。

Compared to small-scale models, these deployment frameworks integrate unique content into the input prompt, such as context demonstrations and retrieved knowledge. It causes a unique risk of LLMs in the deploying scenario, known as prompt stealing attacks. In addition, we explore privacy risks common to all language models, including reconstruction attacks (Morris et al., 2023), inference attacks (Mattern et al., 2023), data extraction attacks (Carlini et al., 2021), and model extraction attacks (Li et al., 2024b).
与小型模型相比，这些部署框架将独特内容集成到输入提示中，例如上下文示例和检索到的知识。这导致 LLMs 在部署场景中存在一种独特的风险，称为提示窃取攻击。此外，我们还探讨了所有语言模型共有的隐私风险，包括重构攻击（Morris 等人，2023 年）、推理攻击（Mattern 等人，2023 年）、数据提取攻击（Carlini 等人，2021 年）和模型提取攻击（Li 等人，2024b）。

Compared to small-scale models, LLMs have unique safety guardrails that protect against harmful results. However, prompt injection attacks and jailbreak attacks can penetrate these guardrails, inducing LLMs to produce harmful content. Another unique risk is that the deployment frameworks expose opportunities for external adversaries (e.g., RAG providers) to poison prompts and manipulate the output of LLM-based applications. Lastly, we also explore adversarial example attacks as a security risk common to all language models. These risks underscore the ongoing challenges in ensuring the robustness of LLMs.
与小型模型相比，LLMs 具有独特的安全护栏来防止有害结果。然而，提示注入攻击和越狱攻击可以穿透这些护栏，导致 LLMs 生成有害内容。另一个独特风险是部署框架为外部对手（例如 RAG 提供者）提供了污染提示和操纵基于 LLM 的应用程序输出的机会。最后，我们还探讨了对抗样本攻击作为所有语言模型都面临的常见安全风险。这些风险突显了确保 LLMs 鲁棒性所面临的持续挑战。

Since LLMs are the backbone of agents, deploying LLM-based agents also faces prompt stealing and various common privacy attacks, as discussed in Section 6.2. Here, we merely focus on privacy risks unique to LLM-based agents.
由于 LLM 是代理的核心，部署基于 LLM 的代理也面临着在第 6.2 节中讨论的提示词窃取和各种常见隐私攻击。在这里，我们仅关注基于 LLM 的代理特有的隐私风险。

Memory stealing attacks. In addition to training data and system prompts, the interaction history and long-term knowledge stored in the memory module are also vulnerable to theft. Malicious users can optimize prompts under the black-box setting to extract sensitive information, such as personal contact information and medical records, from this module. Wang et al. (Wang et al., 2025a) divided the attacking prompt into two components: a locator to guide the agent in retrieving historical information, and an aligner to ensure the output format matches the agent’s workflow. Experiments indicated that with only 30 attacking prompts, up to 50 and 26 user queries can be extracted from two real-world agents. Similarly, certain attacks capable of extracting private knowledge from RAG systems can be applied to steal long-term knowledge from the agent system, as discussed in Section 6.2.
内存窃取攻击。除了训练数据和系统提示词外，存储在内存模块中的交互历史和长期知识也容易受到盗窃。恶意用户可以在黑盒环境下优化提示词，从该模块中提取敏感信息，例如个人联系信息和医疗记录。Wang 等人（Wang 等人，2025a）将攻击提示词分为两个部分：一个定位器用于指导代理检索历史信息，一个对齐器用于确保输出格式与代理的工作流程匹配。实验表明，仅用 30 个攻击提示词，就可以从两个真实世界的代理中提取高达 50 和 26 个用户查询。同样，某些能够从 RAG 系统中提取私人知识的攻击可以应用于从代理系统中窃取长期知识，如第 6.2 节所述。

Unauthorized access. An additional risk arises when users query third-party LLMs or LLM-based agents to process highly confidential information, such as HIPAA (Act et al., 1996) protected data. Both the storage and transmission of such data to external systems may constitute unauthorized access or disclosure under these regulations. For example, OmniGPT, a third-party LLM, had no malicious intent but was compromised by hackers, which resulted in the leakage of more than 34 million conversation records. In the multi-agent system, different agents undertake various roles and permissions. When performing collaborative tasks, multiple agents will share and process private information. An attacker can steal this information across the system by compromising one agent, as shown in Figure 11. Similarly, Li et al. (Li et al., 2024c) found that data transmission between agents can cause some to access sensitive data beyond their permission scope or expose such data to unauthorized agents. In addition, the agent interactions are massive and not transparent, so it is hard to supervise the generated information. Facing the risks posed by such unauthorized access, it is essential to strictly control and transparently manage access to confidential information.
未经授权的访问。当用户查询第三方 LLM 或基于 LLM 的代理来处理高度机密信息时，会带来额外的风险，例如 HIPAA（法案等，1996 年）保护的数据。根据这些法规，将此类数据存储或传输到外部系统可能构成未经授权的访问或披露。例如，第三方 LLM OmniGPT 没有恶意意图，但被黑客攻破，导致超过 3400 万条对话记录泄露。在多代理系统中，不同的代理承担不同的角色和权限。在执行协作任务时，多个代理会共享和处理私人信息。攻击者可以通过攻破一个代理来窃取系统中的这些信息，如图 11 所示。类似地，Li 等人（Li 等人，2024c）发现，代理之间的数据传输会导致某些代理访问超出其权限范围的敏感数据，或将此类数据暴露给未经授权的代理。此外，代理交互是大规模且不透明的，因此难以监督生成的信息。 面对此类未经授权访问带来的风险，严格控制和透明化管理对机密信息的访问至关重要。

Similar to the security risks discussed in Section 6.3, deploying LLM-based agents also faces jailbreak attacks and backdoor attacks. Here, we focus on the security risks unique to LLM-based agents.
与第 6.3 节讨论的安全风险类似，部署基于 LLM 的代理也面临越狱攻击和后门攻击。这里，我们重点关注基于 LLM 的代理特有的安全风险。

Unique security attacks against LLM-based agents. These attacks attempt to hijack or insert the desired actions, such as deleting calendar events. Li et al. (Li et al., 2024c) found that prompt injection attacks could induce the LLM-based agents to perform malicious actions. Here is an example.
针对基于 LLM 的代理的独特安全攻击。这些攻击试图劫持或插入期望的操作，例如删除日历事件。Li 等人（Li 等人，2024c）发现，提示注入攻击可以诱导基于 LLM 的代理执行恶意操作。这里有一个例子。

These abnormal actions will impact users’ lives, such as leading to missed events or the spread of phishing websites. Yang et al. (Yang et al., 2024a) explored backdoor attacks for LLM-based agents, proposing query-attack, observation-attack and thought-attack. For the first two manners, once a query or an observation result contains a trigger, the backdoored agents perform predefined malicious actions (e.g., send scam messages). The third attack can alter the behavior of specific steps while preserving benign actions. For instance, a backdoored agent might invoke a specific tool (e.g., Google Translate) under a trigger prompt. Wang et al. (Wang et al., 2024b) integrated malicious tools into the agent system and invoked them using carefully designed prompts to implement prompt stealing and denial-of-service attacks.
这些异常行为将影响用户的生活，例如导致错过活动或传播钓鱼网站。Yang 等人（Yang 等人，2024a）探索了针对基于 LLM 的代理的后门攻击，提出了查询攻击、观察攻击和思维攻击。对于前两种方式，一旦查询或观察结果包含触发器，后门代理将执行预定义的恶意行为（例如，发送诈骗信息）。第三种攻击可以改变特定步骤的行为，同时保留良性行为。例如，一个后门代理可能在触发提示下调用特定工具（例如，谷歌翻译）。王等人（王等人，2024b）将恶意工具集成到代理系统中，并使用精心设计的提示来调用它们，以实现提示窃取和拒绝服务攻击。

Agent contamination. For the single-agent system, attackers can modify the role settings of victim agents, causing them to exhibit harmful behaviors, such as trojan code generation. Tian et al. (Tian et al., 2023) found that malicious agents can share harmful content with other agents, affecting their behavior in a domino effect, as displayed in Figure 12. This risk significantly increases the vulnerability of LLM-based agents within a communication network. Targeting the multi-agent system, Zhou et al. (Zhou et al., 2025) crafted a malicious prompt to trap a single agent in an infinite loop. The infected agent then propagated the prompt to others, eventually causing a complete system breakdown. Experiments indicated this attack can infect all ten GPT-4o-mini agents in under two dialogue turns.
代理污染。对于单代理系统，攻击者可以修改受害者代理的角色设置，导致它们表现出有害行为，例如生成木马代码。田等人（Tian 等人，2023）发现，恶意代理可以与其他代理共享有害内容，以多米诺效应影响它们的行为，如图 12 所示。这种风险显著增加了基于 LLM 的代理在通信网络中的脆弱性。针对多代理系统，周等人（Zhou 等人，2025）设计了一个恶意提示来使单个代理陷入无限循环。被感染的代理然后将提示传播给其他代理，最终导致整个系统崩溃。实验表明，这种攻击可以在不到两个对话回合内感染所有十个 GPT-4o-mini 代理。