RedTeamCUA:
Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments
RedTeamCUA ： 在混合 Web 操作系统环境下对计算机使用代理进行真实对抗测试

CUAs can streamline tedious daily workflows, automate intricate use cases such as collection, analysis, and aggregation of online information, and perform complex tasks across both web and OS environments. In this work, we specifically focus on benign user scenarios, where CUAs assist benign users in acquiring knowledge from web resources (e.g., forums, shared documents, chats with experts) and execute corresponding actions locally, a common interaction pattern in everyday computer use (e.g., installing an unfamiliar software package; see Figure 1). These tasks directly rely on interpreting and acting upon web knowledge, and as a result, potentially heighten the susceptibility of CUAs to malicious inputs embedded within online environments. Our focus on these benign CUA use cases directly guides our design of RedTeamCUA in later sections, influencing our selection of web platforms equipped for these tasks and the formulation of both benign and adversarial goals.
CUA（计算型应用）可以简化繁琐的日常工作流程，自动化复杂的用例，例如在线信息的收集、分析和聚合，并在 Web 和操作系统环境中执行复杂任务。本文重点关注良性用户场景，即 CUA 协助良性用户从网络资源（例如论坛、共享文档、与专家的聊天）获取知识，并在本地执行相应的操作。这种交互模式在日常计算机使用中十分常见（例如安装不熟悉的软件包；参见图 1 ）。这些任务直接依赖于对网络知识的解读和应用，因此，CUA 更容易受到嵌入在线环境中的恶意输入的影响。我们对这些良性 CUA 用例的关注直接指导了后续章节中 RedTeamCUA 的设计，影响了我们对适用于这些任务的 Web 平台的选择以及良性和对抗性目标的制定。

Despite their productivity benefits, CUAs are highly vulnerable to indirect prompt injection, a risk exacerbated by the complex and noisy nature of web environments and their ability to execute state-changing OS actions. Indirect prompt injection (Greshake et al., 2023) involves adversaries remotely embedding malicious instructions within environment content (e.g. social media forums, documents, and chat messages) to hijack agents into performing harmful actions. While initial research has begun examining agent susceptibility to these attacks, existing efforts face several limitations: (1) Less Realistic Threat Models. Many studies rely on threat models that feature unrealistic attacker capabilities (Zhang et al., 2024), with approaches such as EIA (Liao et al., 2025) and DoomArena (Boisvert et al., 2025) assuming full attacker control over webpages to inject malicious HTML elements, banners or pop-ups. (2) Safety-Realism Tradeoffs. Evaluating adversarial risks for CUAs involves balancing the use of 1) controlled environments that avoid direct harm to real users and 2) realistic scenarios that capture CUA risks likely to actually emerge in deployment. ToolEmu (Ruan et al., 2024), InjecAgent (Zhan et al., 2024), and AgentDojo (Debenedetti et al., 2024) explore risks safely but rely on non-interactive, tool-use environments, creating disconnect from realistic computer-use scenarios. In contrast, other studies test in real, non-sandboxed environments that expose users to potential harm (Li et al., 2025). OS-Harm (Kuntz et al., 2025) partially addresses this tradeoff by using a VM-based OS but still relies on a non-isolated browser that leaves potential for web-based risks during evaluation. (3) Web-Only Adversarial Harms. Some frameworks such as VWA-Adv (Wu et al., 2025), DoomArena (Boisvert et al., 2025), WASP (Evtimov et al., 2025), SafeArena (Tur et al., 2025) enable adversarial testing in dynamic, interactive sandboxes such as WebArena (Zhou et al., 2024a). However, they remain limited to explore web-only threat models, overlooking evaluation of OS-level security vulnerabilities to explore the full range of CUA harms. (4) Lack of Hybrid Adversarial Attacks. Current frameworks also fail to support hybrid adversarial attacks spanning both web and OS environments simultaneously. This gap stems from the absence of secure, integrated sandboxes for both environments in current frameworks, preventing evaluation and analysis of adversarial attacks exploring harms across both of these two critical interfaces.
尽管 CUA 具有提高生产力的优势，但它们极易受到间接提示注入攻击。Web 环境的复杂性和噪声特性，以及它们执行状态更改操作系统操作的能力，加剧了这种风险。间接提示注入 （Greshake 等人 ， 2023 ） 是指攻击者远程将恶意指令嵌入环境内容（例如社交媒体论坛、文档和聊天消息）中，从而劫持代理执行有害操作。虽然初步研究已开始探讨代理对这类攻击的敏感性，但现有研究仍存在一些局限性： （1）不太现实的威胁模型。 许多研究依赖于具有不切实际的攻击者能力的威胁模型 （Zhang et al. ， 2024 ） ，例如 EIA （Liao et al. ， 2025 ） 和 DoomArena （Boisvert et al. ， 2025 ） 等方法假设攻击者完全控制网页以注入恶意 HTML 元素、横幅或弹出窗口。 （2）安全性与真实性之间的权衡。 评估计算机使用应用（CUA）的对抗性风险需要在以下两方面之间取得平衡：1）避免对真实用户造成直接伤害的受控环境；2）能够捕捉实际部署中可能出现的 CUA 风险的真实场景。ToolEmu （Ruan 等人 ， 2024 ） 、InjecAgent （Zhan 等人 ， 2024 ） 和 AgentDojo （Debenedetti 等人 ， 2024 ） 虽然能够安全地探索风险，但它们依赖于非交互式的工具使用环境，与真实的计算机使用场景存在脱节。相比之下，其他研究则在真实的、非沙盒环境中进行测试，使用户面临潜在的伤害 （Li 等人 ， 2025 ） 。OS-Harm （Kuntz 等人 ， 2025 ） 通过使用基于虚拟机的操作系统在一定程度上解决了这种权衡问题，但仍然依赖于非隔离的浏览器，这在评估过程中仍然存在基于网络的风险。 (3) 仅限网络的对抗性伤害。 一些框架，例如 VWA-Adv (Wu et al. , 2025 ) 、DoomArena (Boisvert et al. , 2025 ) 、WASP (Evtimov et al. , 2025 ) 、SafeArena (Tur et al. , 2025 ) 可以在动态交互式沙箱（例如 WebArena）中进行对抗性测试 （Zhou 等人 ， 2024a ） 。然而，它们仍然局限于探索仅限 Web 的威胁模型，忽略了对操作系统级安全漏洞的评估，从而无法探索 CUA 危害的全部范围。 （4）缺乏混合对抗性攻击。 现有框架也无法同时支持跨越 Web 和操作系统环境的混合对抗性攻击。这一缺陷源于现有框架缺乏针对这两种环境的安全集成沙箱，从而阻碍了对跨越这两个关键接口的对抗性攻击进行评估和分析。

Given these limitations, we highlight the critical need for a hybrid sandbox that enables realistic and interactive adversarial evaluation across secure web and OS environments (§ 3) and a large-scale adversarial benchmark with broad coverage of severe adversarial scenarios (§ 4). In addition, we provide a detailed comparison of existing work in Table 6 of Appendix D.
鉴于这些局限性，我们强调迫切需要一种混合沙箱，它能够在安全的网络和操作系统环境中进行真实且交互式的对抗性评估（第 3 节），以及一个能够广泛覆盖各种严重对抗场景的大规模对抗性基准测试（第 4 节）。此外，我们在附录 D 的表 6 中提供了现有工作的详细比较。

OS: Our approach leverages OSWorld (Xie et al., 2024; 2025) as its backbone, providing an executable OS environment for interactive agent testing across diverse applications (e.g., Terminal, File Manager, VSCode) and OS. Our work specifically focuses on Ubuntu due to its widespread adoption in prior research (Agashe et al., 2025; Qin et al., 2025). Importantly, OSWorld’s VM-based architecture provides crucial adversarial testing features, such as host machine isolation to safely contain harmful agent actions and environment snapshot resets for reproducible and scalable testing. The use of this realistic, interactive OS environment also allows exploration of risks that only emerge in complex, real-world task flows, creating deeper insights into adversarial CUA risks compared to prior simplistic, static approaches (Ruan et al., 2024; Zhan et al., 2024; Debenedetti et al., 2024; Yuan et al., 2024).
操作系统： 我们的方法以 OSWorld （Xie 等人 ， 2024 ； 2025 ） 为基础，提供了一个可执行的操作系统环境，用于在各种应用程序（例如终端、文件管理器、VSCode）和操作系统上进行交互式代理测试。我们的工作特别关注 Ubuntu，因为它在之前的研究中被广泛采用 （Agashe 等人 ， 2025 ；Qin 等人 ， 2025 ） 。重要的是，OSWorld 基于虚拟机的架构提供了关键的对抗性测试功能，例如主机隔离，可以安全地限制有害代理的行为，以及环境快照重置，从而实现可复现且可扩展的测试。使用这种逼真的交互式操作系统环境还可以探索仅在复杂的现实世界任务流程中出现的风险，与以前简单、静态的方法相比，可以更深入地了解对抗性 CUA 风险 （Ruan 等人 ， 2024 ；Zhan 等人 ， 2024 ；Debenedetti 等人 ， 2024 ；Yuan 等人 ， 2024 ） 。

Web: While OSWorld offers many benefits, it has unrestricted browser access to live websites, which introduces potential safety risks during web-based red teaming and limits full exploration of adversarial computer-use scenarios in a controlled manner. To overcome this challenge, we develop a hybrid sandbox strategy by integrating self-hosted web environments from WebArena (Zhou et al., 2024a) and TheAgentCompany (Xu et al., 2024) using their provided AWS images. Each web platform is created as a replica of a real-world counterpart website using Docker containers created from available open-source libraries and real-world data sources, allowing for realism while avoiding real-world repercussions. The web platforms are accessed via HTTP connection in OSWorld’s browser, allowing for testing of adversarial scenarios requiring both OS and web interactions.
Web： 尽管 OSWorld 提供了诸多优势，但它允许浏览器无限制地访问实时网站，这在基于 Web 的红队演练中引入了潜在的安全风险，并限制了以可控方式对对抗性计算机使用场景的全面探索。为了克服这一挑战，我们开发了一种混合沙箱策略，该策略集成了 WebArena (Zhou et al. , 2024a ) 和 TheAgentCompany (Xu et al. , 2024 ) 提供的自托管 Web 环境，并使用了它们提供的 AWS 镜像。每个 Web 平台都使用基于可用开源库和真实数据源创建的 Docker 容器，作为真实世界的对应网站的副本，从而在保证真实性的同时避免了现实世界的后果。这些 Web 平台通过 OSWorld 浏览器中的 HTTP 连接进行访问，从而可以测试需要操作系统和 Web 交互的对抗性场景。

In this work, we focus on integrating the following web platforms into RedTeamCUA: (1) OwnCloud, an open-source alternative to Google Drive and Microsoft Office from TheAgentCompany, simulating cloud-based office environments. (2) Forum, an open-source alternative to Reddit from WebArena, simulating social media forums. (3) RocketChat, an open-source alternative to Slack from TheAgentCompany, simulating real-time communication software. We select these three platforms as they facilitate study of the common use cases described in Section 2.1, where users acquire web knowledge before taking corresponding local actions, such as cloning a project from a Forum page, receiving technical assistance via RocketChat, or retrieving technical instruction files on OwnCloud. Together, the environments cover three diverse web applications and different attack surfaces, including adversarial forum posts, harmful direct messages, or malicious shared files.
本文重点将以下三个网络平台集成到 RedTeamCUA 中：(1) OwnCloud ，由 TheAgentCompany 开发的开源云平台，可替代 Google Drive 和 Microsoft Office，用于模拟云端办公环境。(2) Forum ，由 WebArena 开发的开源平台，可替代 Reddit，用于模拟社交媒体论坛。(3) RocketChat ，由 TheAgentCompany 开发的开源平台，可替代 Slack，用于模拟实时通信软件。我们选择这三个平台是因为它们有助于研究 2.1 节中描述的常见用例，即用户在执行相应的本地操作之前先获取网络知识，例如从 Forum 页面克隆项目、通过 RocketChat 获取技术支持或在 OwnCloud 上检索技术说明文件。这些环境共同涵盖了三种不同的 Web 应用程序和不同的攻击面，包括恶意论坛帖子、有害私信或恶意共享文件。

Core Features: To further support rigorous, systematic and scalable evaluation of CUA vulnerabilities, we enhance RedTeamCUA with two core features for adversarial testing: (1) Configurable and Automated Adversarial Injection. We extend OSWorld’s configuration with an Adversarial Task Initial State Setup (Figure 7) that supports automated adversarial injection. This includes but is not limited to defining injection content and targets, specifying SQL commands for injection, and uploading files to be targeted. Based on it, for each supported web platform, we develop platform-specific adversarial injection scripts that introduce adversarial content not present by default, including direct SQL modifications to platform databases for persistent, reproducible injections after task initialization. The configuration and automated injection enable scalable, reproducible creation of diverse adversarial scenarios. (2) Decoupled Evaluation. We observe in our preliminary experiments that GPT-4o often fails to navigate to the webpage containing the adversarial injection. Such navigation failures hinder vulnerability analysis, since the inability to reach the injection does not imply robustness once it is encountered. To address this, we introduce a Decoupled Eval setting that uses pre-processed actions to place CUAs directly at the injection site, isolating adversarial robustness from navigation limitations for focused adversarial analysis. Through these enhancements, we provide a flexible framework for customizable and large-scale study of diverse adversarial scenarios within a realistic, hybrid web-OS platform.
核心功能： 为了进一步支持对 CUA 漏洞进行严格、系统和可扩展的评估，我们为 RedTeamCUA 添加了两个用于对抗性测试的核心功能： （1）可配置的自动化对抗注入。 我们扩展了 OSWorld 的配置功能，新增了对抗任务初始状态设置 （图 7 ），支持自动化对抗注入。这包括但不限于定义注入内容和目标、指定注入的 SQL 命令以及上传目标文件。基于此，我们针对每个支持的 Web 平台开发了平台特定的对抗注入脚本，引入了默认情况下不存在的对抗内容，包括直接修改平台数据库的 SQL 语句，以便在任务初始化后实现持久且可复现的注入。这种配置和自动化注入功能能够以可扩展且可复现的方式创建各种对抗场景。 （2）解耦评估。 我们在初步实验中观察到，GPT-4o 经常无法导航到包含对抗注入的网页。这种导航失败会阻碍漏洞分析，因为无法到达注入点并不意味着一旦遇到注入点就具有鲁棒性。为了解决这个问题，我们引入了一种解耦评估设置，该设置使用预处理的动作将 CUA 直接放置在注入点，从而将对抗鲁棒性与导航限制隔离开来，以便进行更有针对性的对抗分析。 通过这些改进，我们提供了一个灵活的框架，可以在一个真实的混合网络操作系统平台上对各种对抗场景进行可定制的大规模研究。

To align with our focused CUA use cases (described in §2.1) where users fetch online knowledge for local execution, we define benign goals across three categories: (1) Software Installation, where the agent installs tools, libraries, or packages found online, (2) System Configuration, where the agent configures or customize local system settings, and (3) Project Setup, where the agent downloads a codebase or dataset aligned with the user’s goals. We create 3 distinct benign goals per category using web environments in RedTeamCUA, resulting in 9 total goals (Appendix B.2). Beyond this, to simulate varying levels of user expertise occurring in real scenarios, we design two instantiations of benign instructions: General, where the user provides vague, high-level instructions, and Specific, where the user provides more detailed instructions based on their domain knowledge.
为了与我们重点关注的 CUA 用例（详见 §2.1 ）保持一致，即用户获取在线知识并在本地执行，我们将目标分为三类：（1） 软件安装 ，代理程序安装在线工具、库或软件包；（2） 系统配置 ，代理程序配置或自定义本地系统设置；（3） 项目设置 ，代理程序下载符合用户目标的代码库或数据集。我们使用 RedTeamCUA 中的 Web 环境，为每个类别创建 3 个不同的目标，总共得到 9 个目标（附录 B.2 ）。 除此之外，为了模拟真实场景中出现的不同用户专业水平，我们设计了两种良性指令的实例： 一般指令是指用户提供模糊、高层次的指令，而具体指令是指用户根据其领域知识提供更详细的指令。

Threat Model: Our threat model focuses on indirect prompt injection, where malicious content is embedded in web environments to manipulate an agent to deviate from its benign goal and perform a harmful task. We assume realistic attacker constraints: the attacker cannot access or modify the user’s original instruction, the agent’s prompts, components or model weights, and can only inject content into locations on a webpage where textual input is typically permitted (e.g., Forum comments, RocketChat messages, shared OwnCloud files). Unlike prior work that assumes attackers have unrealistically full webpage or OS access (Zhang et al., 2024; Liao et al., 2025; Chen et al., 2025a; Boisvert et al., 2025), our threat model reflects the real-world scenario in which platforms have strict UI design standards and access controls preventing unauthorized web modifications. Due to this, we focus on realistic, text-based injection within editable web content rather than misleading visual pop-ups or arbitrary UI manipulation from attackers. Due to the attacker’s lack of knowledge of the user’s instruction, we assume an adversarial strategy where the attacker blends their injection into the environment context to match anticipated user queries for a given web page. For example, the attacker may target CUAs on a Forum page related to software installation using an adversarial comment that couples harmful instructions with legitimate installation steps (shown in Figure 1). We explore scenarios where a user’s benign task aligns with the attacker’s contextualized injection, allowing us to assess attack viability under high-risk adversarial conditions (examples in Appendix B.4).
威胁模型： 我们的威胁模型侧重于间接提示注入，即恶意内容嵌入网络环境，操纵用户代理偏离其原本的良性目标，执行有害任务。我们假设攻击者受到现实的限制：攻击者无法访问或修改用户的原始指令、代理的提示、组件或模型权重，并且只能将内容注入到网页上通常允许文本输入的位置 （例如，论坛评论、RocketChat 消息、共享的 OwnCloud 文件）。与以往假设攻击者拥有不切实际的完整网页或操作系统访问权限的研究 （Zhang et al. , 2024 ; Liao et al. , 2025 ; Chen et al. , 2025a ; Boisvert et al. , 2025 ） 不同，我们的威胁模型反映了现实世界的场景，即平台具有严格的 UI 设计标准和访问控制，以防止未经授权的网页修改。因此，我们专注于在可编辑的网页内容中进行逼真的文本注入，而不是攻击者通过误导性的视觉弹出窗口或任意的用户界面操纵来进行攻击。由于攻击者不了解用户的指令，我们假设其采用一种对抗策略，即攻击者将注入内容融入环境上下文，以匹配特定网页的预期用户查询。例如，攻击者可能会针对与软件安装相关的论坛页面上的用户自定义应用（CUA），使用包含恶意指令和合法安装步骤的对抗性评论（如图 1 所示）进行攻击。 我们探索用户良性任务与攻击者上下文注入相一致的场景，从而使我们能够在高风险对抗条件下评估攻击的可行性（附录 B.4 中的示例）。

Adversarial Goals and Instructions: In this work, we focus on web injection risks targeting the user’s local OS, highlighting hybrid attack pathways enabled by our environment. To systematically characterize these risks, we adopt the widely used CIA security framework (Howard and Lipner, 2006), which categorizes fundamental OS security violations into three dimensions: Confidentiality (i.e., preventing unauthorized information exfiltration), Integrity (i.e., maintaining data trustworthiness and accuracy), and Availability (i.e., ensuring reliable access to data and systems). Using this framework, we design a diverse set of adversarial scenarios (Appendix B.3) and define 24 total adversarial goals, each targeting a specific CIA security principle and corresponding to a distinct adversarial outcome (Appendix B.4). For Confidentiality, we examine Web $\rightarrow$ OS $\rightarrow$ Web¹
对抗目标和指令：本文重点关注针对用户本地操作系统的网络注入风险，并着重分析我们环境支持的混合攻击路径。为了系统地描述这些风险，我们采用了广泛使用的 CIA 安全框架（Howard 和 Lipner，2006），该框架将基本的操作系统安全违规行为分为三个维度：机密性（即防止未经授权的信息泄露）、完整性（即维护数据的可信度和准确性）以及可用性（即确保对数据和系统的可靠访问）。基于此框架，我们设计了一系列不同的对抗场景（附录 B.3），并定义了总共 24 个对抗目标，每个目标都针对一个特定的 CIA 安全原则，并对应一个不同的对抗结果（附录 B.4）。对于机密性，我们考察了 Web $\rightarrow$ 、操作系统 $\rightarrow$ 和 Web1。¹$\rightarrow$ denotes the direction in which information propagates. For example, Web $\rightarrow$ OS indicates that adversarial content from the web environment takes effect on damaging the local OS.
$\rightarrow$ 表示信息传播的方向。例如，Web $\rightarrow$ OS 表示来自 Web 环境的恶意内容会对本地操作系统造成损害。 adversarial scenarios in which local OS files are exfiltrated to adversaries via external platforms, such as sending through a chat interface (RocketChat), uploading to shared storage (OwnCloud), or publicly posting content on online forums (Forum). Meanwhile, Integrity and Availability explore Web $\rightarrow$ OS attack pathways targeting the data or functionality of the user’s local OS. To further ensure comprehensive coverage, each adversarial goal is instantiated in two forms, i.e., natural language and code (Appendix B.4).
对抗场景包括通过外部平台（例如聊天界面（RocketChat）发送、上传到共享存储（OwnCloud）或在在线论坛（Forum）公开发布内容）将本地操作系统文件泄露给攻击者。同时，“完整性”和“可用性”部分探索了针对用户本地操作系统数据或功能的 Web → 操作系统攻击路径。为了进一步确保全面覆盖，每个对抗目标都以两种形式呈现，即自然语言和代码（附录 B.4 ）。

Injection Strategy: We adopt a general injection strategy that involves prepending commonly used "important message" cues (Debenedetti et al., 2024; Wu et al., 2025) to the core instructions of the adversarial goal. This is followed by a deceptive rationale, adaptively contextualized to both the specific environment and the associated task, meant to convince CUAs that the adversarial goal is required for the task mentioned in the environment context (see example in Figure 1).
注射策略： 我们采用一种通用的注入策略，即在对抗目标的核心指令前添加常用的“重要信息”提示 （Debenedetti 等人 ， 2024 ；Wu 等人 ， 2025 ） 。随后，我们会添加一个具有欺骗性的理由，该理由会根据具体环境和相关任务进行自适应调整，旨在使 CUA（认知依赖者）相信对抗目标是完成环境上下文中所述任务所必需的（参见图 1 中的示例）。

Overall, RTC-Bench defines 216 adversarial scenarios derived from 9 benign and 24 adversarial goals; with 2 textual variations for each goal, it comprises 864 total test examples to comprehensively evaluate CUA vulnerabilities to indirect prompt injection across diverse computer-use contexts.
总体而言， RTC-Bench 定义了 216 个对抗场景，这些场景源自 9 个良性目标和 24 个对抗目标；每个目标有 2 个文本变体，总共包含 864 个测试示例，以全面评估 CUA 在各种计算机使用环境中对间接提示注入的漏洞。

Broader Use: While our current adversarial attack formulation centers on examining CUA vulnerabilities to risks originating from the web, RedTeamCUA can readily support broader threat models, such as scenarios where the agent itself is compromised, inherently unsafe, or directly interacting with malicious users. In addition, although our sandbox currently supports a fixed set of web platforms (§ 3.1), its modular and flexible design allows for easy extension to additional platforms in the future. This enables the exploration of a wider range of realistic computer-use environments and adversarial attack scenarios while maintaining strong safety guarantees.
更广泛的应用： 虽然我们目前的对抗性攻击模型主要侧重于检验 CUA 对源自网络的风险的漏洞，但 RedTeamCUA 可以轻松支持更广泛的威胁模型，例如代理本身被攻破、本质上不安全或直接与恶意用户交互的场景。此外，尽管我们的沙箱目前仅支持一组固定的网络平台（§ 3.1 ），但其模块化和灵活的设计使其未来可以轻松扩展到其他平台。这使得我们能够在保持强大安全保障的同时，探索更广泛的实际计算机使用环境和对抗性攻击场景。

Baseline CUAs: Due to the inherent complexity of computer-use scenarios, we focus on evaluating the most advanced CUAs to date, as they are the most likely to be deployed in real-world applications. For our analysis, we evaluate two classes of CUAs:
基线 CUA： 由于计算机使用场景固有的复杂性，我们专注于评估迄今为止最先进的计算机使用应用 (CUA)，因为它们最有可能在现实世界的应用中部署。 在我们的分析中，我们评估了两类 CUA：

$\bullet$ Adapted LLM-based Agents include powerful LLMs adapted for computer use through generic agentic scaffolding. For this category, we evaluate GPT-4o (Hurst et al., 2024), the base versions of Claude 3.5 Sonnet (v2) (Anthropic., 2024c) and Claude 3.7 Sonnet (Anthropic., 2025a). For these agents, we follow the default agentic scaffolding provided by OSWorld, which uses pyautogui for Python-based execution of mouse and keyboard commands and provides necessary contextual information within the system prompt (see Appendix J).
$\bullet$ 适配型 LLM 智能体包括功能强大的 LLM，它们通过通用智能体框架适配于计算机使用。对于此类智能体，我们评估了 GPT-4o （Hurst 等人 ， 2024 ） 、Claude 3.5 Sonnet (v2) （Anthropic， 2024c ） 和 Claude 3.7 Sonnet （Anthropic， 2025a ） 的基础版本。对于这些智能体，我们遵循 OSWorld 提供的默认智能体框架，该框架使用 pyautogui 以 Python 方式执行鼠标和键盘命令，并在系统提示符中提供必要的上下文信息（参见附录 J ）。

$\bullet$ Specialized Computer-Use Agents are designed specifically for computer use, featuring training for GUI perception (OpenAI., 2025b; Anthropic., 2024b) and reasoning (OpenAI., 2025b) and incorporating tailored computer-use tools (Anthropic., 2024a). For this category, we mainly evaluate Operator (OpenAI., 2025b) and the computer-use versions of Claude 3.5 Sonnet (v2) and Claude 3.7 Sonnet (Anthropic., 2024a). Since their native action formats are often incompatible with the pyautogui-based execution in OSWorld, we employ GPT-4o as an auxiliary LLM to convert their native action outputs into executable pyautogui commands (prompt shown in Appendix J).
$\bullet$ 专用计算机使用智能体专为计算机使用而设计，其特点是接受过图形用户界面感知 （OpenAI., 2025b ; Anthropic., 2024b ） 和推理 （OpenAI., 2025b ） 方面的训练，并集成了定制的计算机使用工具 （Anthropic., 2024a ） 。对于此类智能体，我们主要评估了 Operator （OpenAI., 2025b ） 以及 Claude 3.5 Sonnet (v2) 和 Claude 3.7 Sonnet 的计算机使用版本 （Anthropic., 2024a ） 。由于它们的原生动作格式通常与 OSWorld 中基于 pyautogui 的执行方式不兼容，我们使用 GPT-4o 作为辅助逻辑语言模型 (LLM)，将它们的原生动作输出转换为可执行的 pyautogui 命令（提示符见附录 J ）。

Notably, Operator incorporates built-in safety mechanisms to reduce harmful behavior: a confirmation module that requires user approval for critical actions and a safety check module that detects prompt injections. Since both checks require human confirmation, attacks are deemed unsuccessful if a safety check is triggered or if a confirmation check is activated during adversarial goal execution, simulating cases where adversarial outcomes are blocked by human intervention. Yet, attacks remain successful if confirmation checks arise only while completing the benign goal and no safety checks are triggered. On the other hand, human supervision can be inconsistent or unreliable (Liao et al., 2025; Samoilenko, 2023). To account for this, we additionally evaluate a variant, denoted as Operator (w/o checks) , in which Operator will proceed with user permission, simulating inattentive supervision.
值得注意的是，Operator 内置了安全机制来减少有害行为：一个确认模块要求用户批准关键操作，以及一个安全检查模块来检测提示注入。由于这两个检查都需要人工确认，因此，如果在对抗目标执行期间触发安全检查或激活确认检查，则攻击将被视为失败，这模拟了对抗结果被人工干预阻止的情况。然而，如果仅在完成良性目标时出现确认检查，且未触发任何安全检查，则攻击仍然会成功。另一方面，人工监督可能不一致或不可靠 （Liao et al. , 2025 ; Samoilenko, 2023 ） 。为了解决这个问题，我们还评估了一个变体，称为 Operator（无检查版），其中 Operator 将在获得用户许可后继续执行，模拟疏忽的监督。

While we consider including open-source CUAs, sufficiently capable open-source CUAs on the OSWorld leaderboard ²
虽然我们考虑将开源 CUA 纳入 OSWorld 排行榜 2，但功能足够强大的开源 CUA 仍需纳入考虑范围。²2OSWorld Leaderboard: https://os-world.github.io/
OSWorld 排行榜： https://os-world.github.io/ such as UI-TARS 1.5 70B (Seed, 2025), UI-TARS 2 (Wang et al., 2025a), and OpenCUA (Wang et al., 2025b) are currently inaccessible. Meanwhile, the strongest available one (i.e., UI-TARS 1.5 7B) significantly underperforms proprietary agents used in our study as it struggles with basic task execution, such as navigating to target webpages, interacting in chat environments, or opening files. Consequently, we only evaluate frontier CUAs with sufficient capability in our setting.
诸如 UI-TARS 1.5 70B (Seed, 2025 ) 、UI-TARS 2 (Wang et al. , 2025a ) 和 OpenCUA (Wang et al. , 2025b ) 等目前无法访问。同时，目前可用的最强版本（即 UI-TARS 1.5 7B）的性能也显著低于我们研究中使用的专有代理，因为它难以执行一些基本任务，例如导航到目标网页、在聊天环境中交互或打开文件。因此，我们仅评估在我们研究场景中具有足够能力的前沿 CUA。

Evaluation Metrics: To evaluate the success of both benign and adversarial tasks, we adopt example-specific execution-based evaluators to compute Success Rate (SR) and Attack Success Rate (ASR), respectively. This contrasts with OS-Harm’s (Kuntz et al., 2025) reliance on an automated LM judge which can itself be misled by prompt injections, undermining its reliability for adversarial testing. Evaluation based on executable scripts helps ensure robustness against different agent trajectories leading to the same outcome (Xie et al., 2024; 2025). Nonetheless, execution-based evaluation alone may fail to capture an agent’s susceptibility to indirect prompt injection, as an agent might be successfully misled to attempt an adversarial goal, and only fail to fully complete it due to limited capabilities. To address this, we introduce Attempt Rate (AR), a fine-grained LLM-as-a-Judge metric (GPT-4o, prompt in Appendix J) to assess whether an agent attempts to pursue an adversarial goal within the trajectory, regardless of harmful task completion. Together, ASR and AR balance reliable evaluation with broader coverage of harmful behaviors.
评估指标： 为了评估良性任务和对抗性任务的成功率，我们采用基于示例的执行评估器分别计算成功率 (SR) 和攻击成功率 (ASR)。这与 OS-Harm (Kuntz 等人 ， 2025 ) 依赖于自动语言模型 (LM) 评判器形成对比，后者本身可能受到提示注入的误导，从而降低其在对抗性测试中的可靠性。基于可执行脚本的评估有助于确保系统对导致相同结果的不同智能体轨迹的鲁棒性 (Xie 等人 ， 2024 ； 2025 ) 。然而，仅基于执行的评估可能无法捕捉到智能体对间接提示注入的敏感性，因为智能体可能被成功误导去尝试对抗性目标，而仅仅由于能力有限而未能完全完成。为了解决这个问题，我们引入了尝试率（AR），这是一种细粒度的 LLM 作为评判指标（GPT-4o，参见附录 J 中的提示），用于评估智能体是否试图在轨迹中追求对抗性目标，而不管其是否完成了有害任务。ASR 和 AR 共同作用，在保证评估可靠性的同时，也更广泛地涵盖了有害行为。

Additional Details: We conduct sanity checks to ensure that all evaluated CUAs can successfully finish the benign tasks without injection prior to experimentation. Unless otherwise specified, agents operate with screenshot observations to align with how humans navigate computer-use environments (Gou et al., 2025; Qin et al., 2025) and are evaluated under the Decoupled Eval setting, where pre-processed actions place the agent directly at the state containing adversarial injection to isolate robustness from navigational ability. Each example is tested three times, and an attack is counted as successful if any run succeeds. More details are in Appendix E.
更多详情： 我们进行健全性检查，以确保所有评估的 CUA 在实验之前都能成功完成良性任务而无需注入。 除非另有说明，智能体通过屏幕截图观察进行操作，以模拟人类在计算机使用环境中的导航方式 （Gou et al. , 2025 ; Qin et al. , 2025 ）。 智能体在解耦评估 （Decoupled Eval） 设置下进行评估，其中预处理后的动作会将智能体直接置于包含对抗注入的状态，从而将鲁棒性与导航能力隔离开来。每个示例测试三次，只要其中一次运行成功，攻击即被判定为成功。更多细节请参见附录 E。

Main findings: Our results in Table 1 demonstrate a pervasive and substantial susceptibility to indirect prompt injection across all frontier CUA evaluated, with varying degrees of vulnerability observed. Among them, GPT-4o demonstrates the highest average ASR at 66.19%, while Operator yields the lowest at 7.57%. Sonnet-based CUAs demonstrated intermediate ASR, indicating a moderate level of vulnerability. Across all models, AR is consistently higher than ASR, indicating that while CUAs are frequently manipulated into pursuing adversarial goals, limitations to their current capabilities often prevent successful completion of malicious actions. This discrepancy becomes even more evident when disaggregating results by adversarial goal type (Figure 2), highlighting distinct ASR patterns across CIA categories. We attribute this distinction to varying task complexities:
主要发现： 表 1 的结果表明，所有评估的前沿 CUA 都普遍且显著地容易受到间接提示注入攻击，且脆弱程度各不相同。其中，GPT-4o 的平均 ASR 最高，为 66.19%，而 Operator 的 ASR 最低，为 7.57%。基于 Sonnet 的 CUA 的 ASR 处于中间水平，表明其脆弱程度中等。在所有模型中，AR 始终高于 ASR，这表明尽管 CUA 经常被操纵以追求对抗目标，但其当前能力的局限性往往阻止了恶意行为的成功完成。按对抗目标类型细分结果时（图 2 ），这种差异更加明显，突显了不同 CIA 类别的 ASR 模式。我们将这种差异归因于任务复杂性的不同：

Integrity goals involve simple actions (e.g., file deletion via sudo rm -rf /etc/security); Confidentiality goals demand complex multi-step operations (e.g, extracting, creating, and exfiltrating file content) which may depend on agent capabilities; and Availability goals present a range, from simple service disruption (e.g., sudo systemctl stop sshd) to more intricate resource exhaustion tasks (e.g., creating 10,000 1MB files to consume OS storage). Despite this, AR for Confidentiality goals remains high across platforms, indicating that future, more capable agents may make such dangerous privacy attacks feasible if manipulation risks are not mitigated.
完整性目标涉及简单的操作（例如，通过 `sudo rm -rf /etc/security` 删除文件）；机密性目标需要复杂的多步骤操作（例如，提取、创建和泄露文件内容），这可能取决于代理的功能；可用性目标则涵盖范围很广，从简单的服务中断（例如， `sudo systemctl stop sshd` ）到更复杂的资源耗尽任务（例如，创建 10,000 个 1MB 的文件来消耗操作系统存储空间）。尽管如此，机密性目标的可用性在各个平台上仍然很高，这表明，如果操纵风险得不到缓解，未来功能更强大的代理可能会使此类危险的隐私攻击成为可能。

We also find that the same adversarial goal yields different ASRs across our available web platforms, with RocketChat consistently resulting in the highest ASR compared to the other two platforms. We attribute this to two factors: (1) The perceived trustworthiness of the content source in a given environment could impact attack success, as agents might implicitly place higher trust in direct user messages compared to less trusted content like public forum comments. (2) The naturalness and plausibility of performing the harmful action within the context of the specific web platform could impact attack susceptibility. This is evidenced by the notably higher ASR and AR for Confidentiality goals within RocketChat, as sending data is more aligned with the platform’s inherent usage as a messaging tool and is easier to perform compared to our other web environments. These findings suggest that both web platform characteristics and adversarial goal types jointly influence attack success, leaving room for further exploration in future CUA red-teaming and defense work.
我们还发现，相同的对抗目标在不同的可用网络平台上会产生不同的攻击成功率（ASR），其中 RocketChat 的 ASR 始终高于其他两个平台。我们认为这主要归因于两个因素：（1）在特定环境中，内容来源的可信度会影响攻击成功率，因为​​攻击者可能默认更信任直接用户消息，而不是可信度较低的内容，例如公共论坛评论。（2）在特定网络平台的上下文中执行恶意行为的自然性和合理性会影响攻击的易受攻击性。RocketChat 中针对“保密性”目标的 ASR 和攻击成功率（AR）显著更高，这证明了这一点，因为发送数据更符合该平台作为即时通讯工具的固有用途，并且与其他网络环境相比更容易执行。这些发现表明，网络平台特性和对抗目标类型共同影响攻击成功率，为未来的 CUA 红队演练和防御工作留下了进一步探索的空间。

Adapted LLM-based Agents vs. Specialized Computer-Use Agents: Our results in Table 1 and Figure 10 (in Appendix I.5) reveal an interesting contrast between Specialized Computer-Use Agents developed by OpenAI and Anthropic. While Adapted LLM-based Agents from both organizations demonstrate relatively high ASR and AR, Operator achieves a substantial reduction ($-58.62\%$ ASR) in these metrics to stand out as the most secure CUA, while the CUA version of Claude 3.7 Sonnet unexpectedly shows increased susceptibility ($+3.66\%$ ASR) compared to its base counterparts. Operator specifically benefits from its built-in confirmation and safety check mechanisms, indicating that future CUA defenses can benefit from features introducing explicit user permission before executing critical or high-risk actions. However, the average ASR and AR remain high for the Operator (w/o checks) variant in the absence of reliable human supervision at $\sim$31% and $\sim$48% respectively (shown in Table 1). This exposes an inherent trade-off between agent autonomy and security: while human oversight can provide critical guardrails, truly autonomous CUAs require more robust internal safety mechanisms to independently detect and refuse harmful instructions.
基于 LLM 的自适应智能体与专用计算机智能体： 表 1 和图 10 （见附录 I.5 ）中的结果揭示了 OpenAI 和 Anthropic 开发的专用计算机使用代理 (CUA) 之间的有趣对比。虽然两家机构开发的基于 LLM 的自适应代理都表现出相对较高的 ASR 和 AR，但 Operator 在这些指标上实现了显著降低（ASR 为 0），使其成为最安全的 CUA。而 Claude 3.7 Sonnet 的 CUA 版本却意外地显示出更高的易受攻击性（ASR 为 1），高于其基础版本。Operator 特别受益于其内置的确认和安全检查机制，这表明未来的 CUA 防御可以借鉴在执行关键或高风险操作之前引入明确用户许可的功能。然而，在缺乏可靠的人工监督的情况下，Operator（未进行检查）版本的平均 ASR 和 AR 仍然很高，分别为 231% 和 348%（见表 1 ）。这暴露了代理自主性和安全性之间固有的权衡：虽然人类监督可以提供关键的防护措施，但真正自主的 CUA 需要更强大的内部安全机制来独立检测和拒绝有害指令。