Extracting books from production language models
从生产语言模型中提取书籍

We interact with a production LLM via a blackbox API, which limits our access to the underlying model; we supply prompts and receive responses, but do not have access to logits or $\log$ probabilities. For a given book and production LLM, we first probe if extraction seems feasible. To do so, we attempt to have the LLM complete a provided prefix of text drawn from the book. Specifically, we start with a seed ${\bm{s}}$: an initial short, ground-truth string, typically the first sentence or couple of sentences of the book. We split ${\bm{s}}$ into a prefix ${\bm{p}}$ and target suffix ${\bm{t}}$ (i.e., ${\bm{s}}={\bm{p}}+{\bm{t}}$). As illustrated in Figure 2, we form an initial prompt by concatenating a continuation instruction ${\bm{i}}$ with the prefix, i.e., ${\bm{i}}+{\bm{p}}$. (${\bm{i}}$=“Continue the following text exactly as it appears in the original literary work verbatim”; in Figure 2, ${\bm{i}}$ is abbreviated as “Continue the story verbatim”). We submit this concatenated prompt to the production LLM to generate and return up to $1000$ tokens, which we decode to text.
我们通过一个黑盒 API 与一个生产 LLM 进行交互，这限制了我们对底层模型的访问；我们提供提示并接收响应，但无法访问 logits 或 $\log$ 概率。对于给定的书籍和生产 LLM，我们首先探测提取是否可行。为此，我们尝试让 LLM 完成从书中提供的文本前缀。具体来说，我们从种子 ${\bm{s}}$ 开始：一个初始的短、真实的字符串，通常是书籍的第一句或几句。我们将 ${\bm{s}}$ 分成一个前缀 ${\bm{p}}$ 和一个目标后缀 ${\bm{t}}$ （即 ${\bm{s}}={\bm{p}}+{\bm{t}}$ ）。如图 2 所示，我们通过将延续指令 ${\bm{i}}$ 与前缀连接起来形成初始提示，即 ${\bm{i}}+{\bm{p}}$ 。（ ${\bm{i}}$ =“按原文逐字继续以下文本”；在图 2 中， ${\bm{i}}$ 被缩写为“逐字继续故事”）。我们将这个连接起来的提示提交给生产 LLM，以生成并返回最多 $1000$ 个 token，我们将这些 token 解码为文本。

In our main experiments, Gemini 2.5 Pro and Grok 3 complied directly with instructions of this form. In contrast, Claude 3.7 Sonnet and GPT-4.1 exhibited refusal mechanisms, which prevent direct continuation of the provided prefix. Similar to prior work (Nasr et al. (2023; 2025); Section 2), we jailbreak these two production LLMs to circumvent alignment. We began with a simple attack from the literature—Best-of-$N$ (Hughes et al., 2024)—and, given its immediate success, do not consider more sophisticated attacks in this work.
在我们的主要实验中，Gemini 2.5 Pro 和 Grok 3 直接遵循这种形式的指令。相比之下，Claude 3.7 Sonnet 和 GPT-4.1 表现出拒绝机制，这会阻止直接延续提供的 prefix。与先前的工作（Nasr 等人（2023 年；2025 年）；第 2 节）类似，我们对这两个生产 LLMs 进行越狱，以绕过对齐。我们从文献中开始了一个简单的攻击——Best-of- $N$ （Hughes 等人，2024 年），鉴于其立即的成功，我们在这项工作中不考虑更复杂的攻击。

Best-of-$N$ jailbreak (used with Claude 3.7 Sonnet and GPT-4.1).  When running Best-of-$N$ (BoN) (Hughes et al., 2024), one selects an initial prompt, makes $N$ variations of that prompt with random text perturbations, submits the $N$ prompts to an LLM to generate $N$ candidate responses, and then selects the response that most effectively bypasses safety guardrails, where effectiveness is determined by a chosen, context-appropriate criterion (detailed below). The random text perturbations include compositions of flipping alphabetic character case, shuffling word order, character substitutions with visually similar glyphs (e.g., $\text{`s'}\rightarrow\{\text{`\textdollar'},\text{`5'}\}$), and other formatting edits (Hughes et al. (2024); Appendix A). Even if most of the production LLM’s outputs are compliant with its guardrail policies, the probability that the LLM is jailbroken—that is, at least one response violates these policies—increases with $N$.
最佳 $N$ 脱逃（用于 Claude 3.7 Sonnet 和 GPT-4.1）。在运行最佳 $N$ （BoN）（Hughes 等人，2024 年）时，首先选择一个初始提示，然后通过随机文本扰动对 $N$ 该提示进行变化，将 $N$ 提示提交给 LLM 以生成 $N$ 候选响应，然后选择最有效地绕过安全防护栏的响应，其中有效性由所选的、上下文适当的准则决定（详见下文）。随机文本扰动包括大小写字母翻转、词序打乱、使用视觉上相似的符号进行字符替换（例如 $\text{`s'}\rightarrow\{\text{`\textdollar'},\text{`5'}\}$ ），以及其他格式编辑（Hughes 等人（2024 年）；附录 A）。即使生产 LLM 的多数输出符合其防护栏政策，LLM 被脱逃的概率——即至少有一个响应违反这些政策——会随着 $N$ 的增加而提高。

This procedure is model-agnostic and only requires blackbox access, which makes it well-suited to our setting of production LLMs. In practice, our BoN prompt is the initial instruction ${\bm{i}}$; we produce $N$ random permutations of ${\bm{i}}$ (e.g., “C0ntinuE th3 st0ry verb@tim” in Figure 2), and we concatenate each with the prefix ${\bm{p}}$ and submit to the production LLM’s API to produce $N$ responses. We then gauge success for Phase 1 when a decoded API response contains at least a loose match to the ground-truth target suffix ${\bm{t}}$. For Gemini 2.5 Pro and Grok 3, for which we did not use BoN, there is only one response to compare to ${\bm{t}}$; for Claude 3.7 Sonnet and GPT-4.1, we evaluate $N$ BoN responses to see if any of them is a loose match to ${\bm{t}}$.
此流程与模型无关，仅需黑盒访问，因此非常适合我们的生产 LLM 环境。在实践中，我们的 BoN 提示是初始指令 ${\bm{i}}$ ；我们生成 $N$ 个 ${\bm{i}}$ 的随机排列（例如，图 2 中的“C0ntinuE th3 st0ry verb@tim”），并将每个与前缀 ${\bm{p}}$ 连接后提交给生产 LLM 的 API 以生成 $N$ 个响应。然后，当解码的 API 响应至少包含与真实目标后缀 ${\bm{t}}$ 的一个松散匹配时，我们便判定第一阶段成功。对于 Gemini 2.5 Pro 和 Grok 3，我们没有使用 BoN，因此只有一个响应可供比较 ${\bm{t}}$ ；对于 Claude 3.7 Sonnet 和 GPT-4.1，我们评估 $N$ 个 BoN 响应，以查看其中是否有任何响应与 ${\bm{t}}$ 松散匹配。

Determining Phase 1 success.  We quantify loose matches between a production LLM response ${\bm{r}}$ and the target suffix ${\bm{t}}$ using longest common substring, which checks whether there exists a substring of words (i.e., a contiguous sequence of words) that appears verbatim in both. That is, we denote the whitespace-split character sequences of ${\bm{t}}$ and ${\bm{r}}$ as $T=(w^{({\bm{t}})}_{1},\dots,w^{({\bm{t}})}_{|T|})$ and $R=(w^{({\bm{r}})}_{1},\dots,w^{({\bm{r}})}_{|R|})$, respectively. We then let
确定第一阶段是否成功。我们使用最长公共子串来量化生产 LLM 响应 ${\bm{r}}$ 与目标后缀 ${\bm{t}}$ 之间的松散匹配，该方法检查是否存在一个单词子串（即一个连续的单词序列）在两者中完全相同出现。也就是说，我们将 ${\bm{t}}$ 和 ${\bm{r}}$ 的空白分隔字符序列分别表示为 $T=(w^{({\bm{t}})}_{1},\dots,w^{({\bm{t}})}_{|T|})$ 和 $R=(w^{({\bm{r}})}_{1},\dots,w^{({\bm{r}})}_{|R|})$ 。然后我们让

denote the length of the longest contiguous common subsequence of $T$ and $R$ (i.e., longest common substring of ${\bm{t}}$ and ${\bm{r}}$). We define a normalized similarity score
表示 $T$ 和 $R$ 的最长连续公共子序列的长度（即 ${\bm{t}}$ 和 ${\bm{r}}$ 的最长公共子串）。我们定义一个归一化相似度分数

which measures the fraction of whitespace-delimited text tokens in $T$ that is covered by the longest contiguous verbatim span also found in $R$. In practice, we consider Phase 1 to be successful when $s\geq 0.6$—i.e., when there is an $\ell$-length verbatim common substring that covers at least $60\%$ of the target suffix ${\bm{t}}$. In initial experiments, we observed this to be a necessary minimum for Phase 2 to be feasible. Note: we do not claim extraction of training data when Phase 1 succeeds with returning this loose match; we defer extraction claims to Phase 2. For Claude 3.7 Sonnet and GPT-4.1, we run BoN with $N$ prompts, stopping when the $N$-th response ${\bm{r}}_{N}$ yields $s(T,R_{N})\geq 0.6$ or when a maximum budget ($N=10{,}000$) is met.
该指标衡量在 $T$ 中以空白字符分隔的文本标记中有多少比例被 $R$ 中也存在的最长连续原文片段所覆盖。在实践中，我们认为当 $s\geq 0.6$ 时 Phase 1 即成功——也就是说，存在一个长度为 $\ell$ 的原文公共子串，它至少覆盖了目标后缀 ${\bm{t}}$ 的 $60\%$ 。在初步实验中，我们观察到这是 Phase 2 可行的必要最低标准。注意：当 Phase 1 成功返回这种宽松匹配时，我们不声称提取了训练数据；我们推迟到 Phase 2 再进行提取声明。对于 Claude 3.7 Sonnet 和 GPT-4.1，我们使用 $N$ 提示运行 BoN，当第 $N$ 个响应 ${\bm{r}}_{N}$ 产生 $s(T,R_{N})\geq 0.6$ 或达到最大预算（ $N=10{,}000$ ）时停止。

In Phase 2 we attempt long-form extraction of the rest of the book. Following successful approximate completion of the seed prefix in Phase 1, we iteratively query the production LLM to continue the text (Figure 3). Similar to the long-form extraction of books performed by Cooper et al. (2025), the prefix in Phase 1 is the only ground-truth text that we provide in the entire procedure; any additional text that we recover from a book in Phase 2 is generated and returned by the production LLM. For each production LLM, we explore different generation configurations: temperature, maximum response length and, where available, frequency penalty and presence penalty (Section 4). For a single run of Phase 2, we fix the generation configuration and execute the continuation loop until a maximum query budget is expended, or the production LLM returns a response that contains either a refusal to continue or a stop phrase (e.g. “THE END”).²²2In practice, we occasionally observe generic internal server errors (500) for some providers, which also halts the loop.
在实践中，我们偶尔会观察到某些提供者的通用内部服务器错误（500），这也会导致循环中断。
在第二阶段，我们尝试提取剩余书籍的长文本内容。在第一阶段成功近似完成种子前缀后，我们迭代查询生产 LLM 以继续生成文本（图 3）。类似于 Cooper 等人（2025 年）进行的书籍长文本提取工作，第一阶段的前缀是整个过程中我们提供的唯一真实文本；我们在第二阶段从书中恢复的任何额外文本都是由生产 LLM 生成并返回的。对于每个生产 LLM，我们探索不同的生成配置：温度、最大响应长度，以及可用时频率惩罚和存在惩罚（第 4 节）。对于第二阶段的一次运行，我们固定生成配置并执行延续循环，直到达到最大查询预算，或者生产 LLM 返回包含拒绝继续或停止短语（例如“THE END”）的响应。 ² We then concatenate the response from the initial completion probe in Phase 1 with the in-order responses in the Phase 2 continuation loop to produce a long-form generated text, which we evaluate for extraction success (Section 3.3).
然后我们将第一阶段初始完成探测的响应与第二阶段延续循环中的有序响应连接起来，生成长文本内容，并评估提取的成功性（第 3.3 节）。

Particulars for long-form extraction from production LLMs.  Most generally, extraction refers to prompting a model to reproduce memorized training data encoded in its weights (Cooper et al. (2023); Section 2). There are various approaches in the memorization literature that satisfy this definition. However, attempting long-form extraction from production LLMs differs from most of this prior work.
从生产 LLMs 中进行长文本提取的细节。最普遍而言，提取是指提示模型重现其权重中编码的记忆训练数据（Cooper 等人（2023）；第 2 节）。记忆文献中有多种方法满足这一定义。然而，尝试从生产 LLMs 中提取长文本与这之前的大部分工作有所不同。

First, as discussed in Section 2, the most commonly used extraction method—discoverable extraction (Lee et al., 2022; Carlini et al., 2021; 2023; Hayes et al., 2025b; Cooper et al., 2025)—is infeasible for production LLMs that are aligned to behave like conversational chatbots. Discoverable extraction prompts with a sequence of training data (just a prefix ${\bm{p}}$) and checks if the LLM generates the verbatim continuation (the suffix ${\bm{t}}$) of that training data—i.e., essentially observing if the LLM successfully “completes the sentence” begun in the prompt. But conversational chatbots do not tend to demonstrate “complete the sentence” behavior. Therefore, while these models still memorize training data, this type of procedure is generally ineffective for extracting those memorized data (Nasr et al., 2023). We sometimes use a jailbreak in Phase 1 to unlock continuation-like behavior; this is also why it is surprising that we did not need to jailbreak Gemini 2.5 Pro or Grok 3 to successfully execute the Phase 2 continuation loop.
首先，如第 2 节所述，最常用的提取方法——可发现提取（Lee 等人，2022；Carlini 等人，2021；2023；Hayes 等人，2025b；Cooper 等人，2025）——对于旨在表现得像对话式聊天机器人的生产 LLM 来说并不可行。可发现提取会使用一个包含训练数据序列的提示（只是一个前缀 ${\bm{p}}$ ），然后检查 LLM 是否生成了该训练数据的逐字续写（后缀 ${\bm{t}}$ ）——即本质上观察 LLM 是否成功“完成了提示中的句子”。但对话式聊天机器人并不倾向于表现出“完成句子”的行为。因此，尽管这些模型仍然记忆训练数据，但这类程序通常对提取这些记忆的数据无效（Nasr 等人，2023）。我们在第 1 阶段有时会使用越狱来解锁类似续写的功能；这也是为什么我们成功执行第 2 阶段续写循环时，不需要对 Gemini 2.5 Pro 或 Grok 3 进行越狱而感到惊讶的原因。

Second, discoverable extraction is predominantly effective for extracting relatively short sequences (typically $50$ tokens, or ${\approx}38$ words), even when much longer sequences are memorized in the model. For an autoregressive language model, the probability of generating an exact continuation (e.g., a suffix ${\bm{t}}$) conditioned on a prompt (e.g., a prefix ${\bm{p}}$) decreases as the length of the continuation increases, making long memorized sequences increasingly difficult to extract. This is why for long-form extraction, as in Cooper et al. (2025), we do not attempt to produce the whole book in one interaction, and instead query iteratively to generate a limited length of text that continues the prefix and any text in the context that the LLM has already generated. In practice, in our production LLM setting, limiting the generation length was also important for evading output filters (Section 4).
其次，可发现的提取主要适用于提取相对较短的序列（通常为 $50$ 个 token，或 ${\approx}38$ 个词），即使模型中记住了更长的序列。对于自回归语言模型，给定提示（例如前缀 ${\bm{p}}$ ）生成确切延续（例如后缀 ${\bm{t}}$ ）的概率随着延续长度的增加而降低，这使得提取长记忆序列变得越来越困难。这就是为什么在长文本提取中，如在 Cooper 等人（2025 年）的研究中，我们不尝试在一次交互中生成整本书，而是迭代查询以生成有限长度的文本，该文本延续前缀以及 LLM 已经生成的任何上下文文本。在实践中，在我们的生产 LLM 环境中，限制生成长度对于规避输出过滤器（第 4 节）也同样重要。

Third, for production LLMs, users have relatively little control over the decoding procedure, and do not typically have access to logits or $\log$ probabilities. In contrast, most research on memorization examines controlled settings for open-weight models, where it is possible to study extraction with fine-grained choices about decoding strategy (Lee et al., 2022; Carlini et al., 2023) and make use of logits (Hayes et al., 2025b). For instance, in an experiment that extracts Harry Potter and the Sorcerer’s Stone from Llama 3.1 70B, Cooper et al. (2025) are able to deterministically reproduce the entirety of the book near-verbatim because they can use beam search, which we do not have access to using blackbox APIs.
第三，对于生产 LLMs，用户对解码过程控制相对较少，通常无法访问 logits 或 $\log$ 概率。相比之下，大多数关于记忆的研究考察开放权重模型的受控环境，其中可以研究提取，并就解码策略做出细粒度选择（Lee 等人，2022；Carlini 等人，2023），并利用 logits（Hayes 等人，2025b）。例如，在一个从 Llama 3.1 70B 中提取《哈利·波特与魔法石》的实验中，Cooper 等人（2025）能够确定性地近乎逐字地重现整本书，因为他们可以使用束搜索，而我们无法通过黑盒 API 访问。

Lastly, standard evaluation metrics for relatively short-form extraction are not applicable to long-form generated outputs. For discoverable extraction, it is typical to compare the generated continuation and target suffix, and to declare extraction success when there is verbatim equality or the continuation is within a small edit distance to the target (Lee et al., 2022; Ippolito et al., 2022). While these success criteria are reasonable for assessing extraction success of $50$-token (${\approx}38$-word) sequences, Cooper et al. (2025) observe that strict equality is too stringent when extracting (tens of) thousands of tokens. This was true even in their work, where the long-form generated outputs were almost (but not quite) exact reproductions of reference texts. In our work, the reproductions are often less exact, so we need to devise a different measurement procedure for claiming extraction success.
最后，用于相对短文本提取的标准评估指标不适用于长文本生成输出。对于可发现性提取，通常比较生成的延续部分和目标后缀，当存在逐字相等或延续部分与目标在小编辑距离内时，即视为提取成功（Lee 等人，2022；Ippolito 等人，2022）。虽然这些成功标准对于评估 $50$ -token（ ${\approx}38$ -词）序列的提取成功是合理的，但 Cooper 等人（2025）观察到，在提取（数十个）千 token 时，严格相等过于严苛。即使在他们自己的工作中，长文本生成输出几乎是（但并非完全）参考文本的精确复制品，这一点也是真实的。在我们的工作中，复制品通常不够精确，因此我们需要设计不同的测量程序来声称提取成功。

In this work, we use extraction metrics that allow for near-verbatim matches to the training data. At a high level, to be valid evidence for extraction, the generated text must (1) reflect a sufficiently near-verbatim reproduction of text in the actual book, and (2) be sufficiently long, such that memorization is the overwhelmingly most plausible explanation for near-verbatim generation (Carlini et al., 2021). We propose a procedure that captures when long-form generated text satisfies these conditions (Section 3.3.1). We then elaborate on why this procedure enables us to make conservative extraction claims (Section 3.3.2): it may miss some valid instances of extraction of training data, but importantly should not include short spans of generated text that may coincidentally resemble ground-truth text from a book (i.e., text that is not actually memorized).
在这项工作中，我们使用允许与训练数据近乎逐字匹配的提取指标。从高层次来看，要成为提取的有效证据，生成的文本必须（1）反映实际书籍中文本的足够近乎逐字的再现，并且（2）足够长，以至于记忆是近乎逐字生成的主要原因（Carlini 等人，2021 年）。我们提出了一种程序，用于捕捉长文本生成何时满足这些条件（第 3.3.1 节）。然后，我们详细说明为什么这个程序使我们能够做出保守的提取声明（第 3.3.2 节）：它可能会遗漏一些训练数据的提取有效实例，但重要的是不应包括可能偶然与书籍的真实文本相似生成的短文本片段（即实际上未被记忆的文本）。

Given that production systems change over time (i.e., are unstable compared to open-weight LLMs), we limited our experiments to between mid-August and mid-September 2025. We attempt to extract thirteen books from four production LLMs, and predominantly report results for the single run that shows the maximum amount of extraction we observed for a given production LLM, book, and generation configuration.
考虑到生产系统会随时间变化（即与开放权重 LLM 相比是不稳定的），我们将实验限制在 2025 年 8 月中旬至 9 月中旬之间。我们尝试从四个生产 LLM 中提取十三本书籍，并且主要报告针对给定生产 LLM、书籍和生成配置中我们观察到的最大提取量的单次运行结果。

Production LLMs.  The four production LLMs we evaluate are Claude 3.7 Sonnet (claude-3-7-sonnet-20250219), GPT-4.1 (gpt-4.1-2025-04-14), Gemini 2.5 Pro (gemini-2.5-pro), and Grok 3 (grok-3). Throughout, we refer to these LLMs by their names, rather than these API versions. Claude 3.7 Sonnet has a knowledge cutoff date of October 2024 Anthropic (2025), GPT-4.1’s is June 2024 (OpenAI, 2025), Grok 3’s is November 2024 (xAI, 2025), and Gemini 2.5 Pro’s is January 2025 (Google Cloud, 2025).
生产 LLMs。我们评估的四个生产 LLMs 是 Claude 3.7 Sonnet（claude-3-7-sonnet-20250219）、GPT-4.1（gpt-4.1-2025-04-14）、Gemini 2.5 Pro（gemini-2.5-pro）和 Grok 3（grok-3）。在整个过程中，我们使用这些 LLMs 的名称来指代它们，而不是这些 API 版本。Claude 3.7 Sonnet 的知识截止日期是 2024 年 10 月 Anthropic（2025 年），GPT-4.1 的知识截止日期是 2024 年 6 月（OpenAI，2025 年），Grok 3 的知识截止日期是 2024 年 11 月（xAI，2025 年），Gemini 2.5 Pro 的知识截止日期是 2025 年 1 月（Google Cloud，2025 年）。

Books.  We attempt to extract thirteen books: eleven in-copyright in the U.S. and two in the public domain. We predominantly selected books that Cooper et al. (2025) observe to be highly memorized by Llama 3.1 70B (Appendix C.1). The books under copyright in the U.S. are Harry Potter and the Sorcerer’s Stone (Rowling, 1998) (which we sometimes abbreviate in plot labels as “Harry Potter 1”), Harry Potter and the Goblet of Fire (Rowling, 2000) (“Harry Potter 4”), 1984 (Orwell, 1949), The Hobbit (Tolkien, 1937), The Catcher in the Rye (Salinger, 1951), A Game of Thrones (Martin, 1996), Beloved (Morrison, 1987), The Da Vinci Code (Brown, 2003), The Hunger Games (Collins, 2008), Catch-22 (Heller, 1961), and The Duchess War (Milan, 2012). The public domain books are Frankenstein (Shelley, 1818) and The Great Gatsby (Fitzgerald, 1925). We obtained these books from the Books3 corpus, which was torrented and released in 2020.⁸⁸8We have a copy of this dataset for research purposes only, stored on a university research computing cluster.
我们仅保留了一份该数据集用于研究目的，存储在大学科研计算集群上。
书籍。我们尝试提取十三本书：其中十一本在美国受版权保护，两本已进入公共领域。我们主要选择了 Cooper 等人（2025 年）观察到 Llama 3.1 70B 高度记忆的书籍（附录 C.1）。美国受版权保护的书籍包括《哈利·波特与魔法石》（罗琳，1998 年）（我们在情节标签中有时将其缩写为“哈利·波特 1”）、《哈利·波特与火焰杯》（罗琳，2000 年）（“哈利·波特 4”）、《1984》（奥威尔，1949 年）、《霍比特人》（托尔金，1937 年）、《麦田里的守望者》（塞林格，1951 年）、《冰与火之歌》（马丁，1996 年）、《宠儿》（莫里森，1987 年）、《达·芬奇密码》（布朗，2003 年）、《饥饿游戏》（柯林斯，2008 年）、《第二十二条军规》（赫尔勒，1961 年）和《公爵夫人战争》（米兰，2012 年）。公共领域的书籍包括《弗兰肯斯坦》（雪莱，1818 年）和《了不起的盖茨比》（菲茨杰拉德，1925 年）。我们从 Books3 语料库获取了这些书籍，该语料库于 2020 年通过 BT 下载发布。 ⁸ Therefore, all of these books significantly pre-date the knowledge cutoffs of every LLM we test. Following Cooper et al. (2025), as a negative control we also test The Society of Unknowable Objects (Brown, 2025), published in digital formats on July 31, 2025. This date is long after the training cutoffs for all four LLMs, and therefore it is very unlikely that this original novel contains text that is in the training data.
因此，所有这些书都显著早于我们测试的每一个 LLM 的知识截止日期。根据 Cooper 等人（2025 年）的研究，我们作为负对照也测试了《不可知之物社》（Brown，2025 年），该书于 2025 年 7 月 31 日以数字格式出版。这个日期远在所有四个 LLM 的训练截止日期之后，因此该原创小说包含的训练数据中的文本的可能性非常小。

Configurations for the two-phase procedure and quantifying extraction success.  For Phase 1 (Section 3.1), we set a maximum BoN budget of $N=$10,000$$ for each experiment. In our initial experiments, we observed that we did not need to jailbreak Gemini 2.5 Pro or Grok 3 ($N=0$). For the initial prompt of the instruction and seed prefix, we generate up to $1000$ tokens as the response. We only attempt Phase 2 if Phase 1 succeeds, with the production LLM producing a response that is at least a loose approximation of the target suffix, i.e., $s\geq 0.6$ (Equation 2). We run the Phase 2 continuation loop (Section 3.2) for up to a maximum query budget, or until the production LLM responds with a refusal or stop phrase, e.g., “THE END”. The four production-LLMs APIs expose different, configurable generation parameters (e.g., frequency penalty). For all four LLMs, we set temperature to $0$, but other LLM-specific configurations vary (Appendix C.2). For instance, based on our exploratory initial experiments, we observed it was necessary to set the per-interaction maximum generation length differently for each LLM to evade output filters. For our extraction measurements (Algorithm 1), we use the same conservative configurations across all runs. For the first merge-and-filter, we set $\tau^{(1)}_{\mathrm{gap}}=2$, $\tau^{(1)}_{\mathrm{align}}=1$, and $l^{(1)}=20$; for the second, $\tau^{(2)}_{\mathrm{gap}}=10$, $\tau^{(2)}_{\mathrm{align}}=3$, and $l^{(2)}=100$ (Section 3.3.1 & Appendix B). We provide full details on experimental configurations in Appendix C.
两阶段流程的配置和量化提取成功率。对于第一阶段（第 3.1 节），我们为每个实验设置最大 BoN 预算为 $N=$10,000$$ 。在我们的初始实验中，我们观察到我们不需要破解 Gemini 2.5 Pro 或 Grok 3（ $N=0$ ）。对于指令和种子前缀的初始提示，我们生成最多 $1000$ 个 token 作为响应。我们仅在第一阶段成功时尝试第二阶段，即生产 LLM 生成至少是目标后缀的松散近似响应，即 $s\geq 0.6$ （公式 2）。我们运行第二阶段延续循环（第 3.2 节），直到达到最大查询预算或生产 LLM 响应拒绝或停止短语，例如“THE END”。四个生产 LLM API 暴露不同的、可配置的生成参数（例如，频率惩罚）。对于所有四个 LLM，我们设置温度为 $0$ ，但其他 LLM 特定配置有所不同（附录 C.2）。例如，根据我们的探索性初始实验，我们观察到需要为每个 LLM 设置不同的每交互最大生成长度以规避输出过滤器。 在我们的提取测量（算法 1）中，所有运行都使用相同的保守配置。对于第一次合并和过滤，我们设置 $\tau^{(1)}_{\mathrm{gap}}=2$ 、 $\tau^{(1)}_{\mathrm{align}}=1$ 和 $l^{(1)}=20$ ；对于第二次，设置 $\tau^{(2)}_{\mathrm{gap}}=10$ 、 $\tau^{(2)}_{\mathrm{align}}=3$ 和 $l^{(2)}=100$ （第 3.3.1 节 & 附录 B）。我们在附录 C 中提供了实验配置的详细信息。

Across all Phase 2 runs, we extract hundreds of thousands of words of text. We provide two concrete examples of extracted text from in-copyright books in Figure 6, but do not redistribute long-form generations of in-copyright material. We share lightly normalized diffs for Claude 3.7 Sonnet on Frankenstein and The Great Gatsby, which are books in the public domain. We do not include The Duchess War in plots; of the thirteen books we attempt to extract, this is the only book where Phase 1 failed for all four production LLMs. Similarly, we omit results for our negative control, The Society of Unknowable Objects; as expected, Phase 1 also failed for this book (Appendix D.1).
在所有第二阶段运行中，我们提取了数十万字的文本。我们在图 6 中提供了两个来自版权保护书籍的提取文本实例，但不会重新分发版权保护材料的长期生成内容。我们分享了 Claude 3.7 Sonnet on Frankenstein 和 The Great Gatsby 的轻微规范化差异，这两本书都在公共领域。我们不包含 The Duchess War 在图中；在我们试图提取的十三本书中，这是唯一一本所有四个生产 LLMs 在第一阶段都失败的书。类似地，我们省略了我们负控制组 The Society of Unknowable Objects 的结果；正如预期的那样，这本书在第一阶段也失败了（附录 D.1）。

Interpreting our bar plots.  In this section, each bar reflects results from a single, specifically configured run for a given production LLM and book; across bars, the underlying generation configurations vary. As a result, our results should be interpreted only as describing specific experimental outcomes: each bar in a plot conveys how much extraction we observed under the specified experimental settings; since these settings are not fixed across bars, our plots do not make evaluative claims about relative extraction risk across production LLMs. (See Chouldechova et al. (2025), and further discussion in Sections 1 and 5.1.)
解读我们的条形图。在本节中，每个条形图反映了一个特定配置的生产 LLM 和书籍的单一运行结果；不同条形图之间，底层的生成配置有所不同。因此，我们的结果应仅被理解为描述特定的实验结果：每个条形图传达了在指定实验设置下观察到的提取量；由于这些设置在不同条形图中并非固定不变，我们的条形图并未对生产 LLM 之间的相对提取风险做出评估性声明。（参见 Chouldechova 等人（2025 年）的研究，并在第 1 节和第 5.1 节中进一步讨论。）

Proportion of book extracted ($\mathsf{nv{\text{-}}recall}$).  Figure 5 plots $\mathsf{nv{\text{-}}recall}$ (Equation 7): the overall proportion of a book extracted in in-order, near-verbatim blocks (Section 3.3.1). For a given production LLM, we fix the same generation configuration across books; however, the generation configuration varies across LLMs. Overall, these results show that it is possible to extract text across books and frontier LLMs. Importantly, we did not jailbreak Gemini 2.5 Pro and Grok 3 in Phase 1 to obtain these results in Phase 2. For Claude 3.7 Sonnet and GPT-4.1, we use BoN with up to $N=$10,000$$ attempts in Phase 1. While in terms of dollar-cost BoN is cheap to run for this budget, we note that it almost always required significantly larger $N$—often $10{-}1000\times$—to jailbreak GPT-4.1 compared to Claude 3.7 Sonnet. In four cases, Claude 3.7 Sonnet’s generations recover over $94\%$ of the corresponding reference book. Two of these books—Harry Potter and the Sorcerer’s Stone and 1984—are in-copyright in the U.S., while the other two—The Great Gatsby and Frankenstein—are in the public domain. In three other cases for Claude 3.7 Sonnet, $\mathsf{nv{\text{-}}recall}\geq 32\%$. With respect to LLM-specific generation configurations, we extract significant amounts of Harry Potter and the Sorcerer’s Stone and other books from all four production LLMs.
提取书籍的比例（ $\mathsf{nv{\text{-}}recall}$ ）。图 5 绘制了 $\mathsf{nv{\text{-}}recall}$ （公式 7）：按顺序、近乎逐字提取的书籍整体比例（第 3.3.1 节）。对于给定的生产 LLM，我们在不同书籍中保持相同的生成配置；然而，生成配置在不同 LLM 之间有所不同。总体而言，这些结果表明可以在不同书籍和前沿 LLM 之间提取文本。重要的是，我们在第一阶段没有越狱 Gemini 2.5 Pro 和 Grok 3 以在第二阶段获得这些结果。对于 Claude 3.7 Sonnet 和 GPT-4.1，我们在第一阶段使用 BoN，最多尝试 $N=$10,000$$ 次。虽然从美元成本来看，BoN 在这个预算下运行很便宜，但我们注意到，与 Claude 3.7 Sonnet 相比，越狱 GPT-4.1 几乎总是需要显著更大的 $N$ ——通常是 $10{-}1000\times$ 。在四个案例中，Claude 3.7 Sonnet 的生成恢复了对应参考书籍的 $94\%$ 以上。其中这两本书——《哈利·波特与魔法石》和《1984》——在美国受版权保护，而另外两本——《了不起的盖茨比》和《弗兰肯斯坦》——则属于公共领域。在 Claude 3.7 Sonnet 的另外三个案例中， $\mathsf{nv{\text{-}}recall}\geq 32\%$ 。 关于特定于 LLM 的生成配置，我们从全部四个生产 LLM 中提取了大量《哈利·波特与魔法石》和其他书籍的内容。

We frequently query the production LLM to continue hundreds of times per Phase 2 run, without encountering guardrails. However, when we run Phase 2 for GPT-4.1, we hit a refusal fairly early on in the continuation loop. For instance, for Harry Potter and the Sorcerer’s Stone, this happens at the end of the first chapter. Therefore, while we report $\mathsf{nv{\text{-}}recall}$ with respect to the full book, near-verbatim extraction is limited to the first chapter for GPT-4.1. For the other three production LLMs, we almost always do not encounter refusals (Section 4.3), and so halt Phase 2 when either a maximum query budget is expended, the LLM returns a response containing a stop phrase (e.g., “THE END”), or the API returns an HTTP error. (Section 3.2).
在 Phase 2 的每个运行中，我们频繁地向生产 LLM 查询数百次，并未遇到任何限制措施。然而，在为 GPT-4.1 运行 Phase 2 时，我们在延续循环的早期就遇到了拒绝。例如，对于《哈利·波特与魔法石》，这发生在第一章的结尾。因此，虽然我们针对整本书报告 $\mathsf{nv{\text{-}}recall}$ ，但针对 GPT-4.1 的近乎逐字提取仅限于第一章。对于其他三个生产 LLM，我们几乎从不遇到拒绝（第 4.3 节），因此当达到最大查询预算、LLM 返回包含停止短语的响应（例如，“THE END”）或 API 返回 HTTP 错误时，我们会停止 Phase 2（第 3.2 节）。

The cost of the loop varies across runs, according to the provider’s billing policy, the number of queries, and the number of tokens returned per query. For instance, as shown in Table 1, it cost approximately $119.97 to extract Harry Potter and the Sorcerer’s Stone with $\mathsf{nv{\text{-}}recall}=95.8\%$ from jailbroken Claude 3.7 Sonnet and $1.37 for jailbroken GPT-4.1 ($\mathsf{nv{\text{-}}recall}=4.0\%$); it cost approximately $2.44 for not-jailbroken Gemini 2.5 Pro ($\mathsf{nv{\text{-}}recall}=76.8\%$) and $8.16 for not-jailbroken Grok 3 ($\mathsf{nv{\text{-}}recall}=70.3\%$).
循环的成本因运行而异，根据提供商的计费政策、查询次数以及每个查询返回的 token 数量而变化。例如，如表 1 所示，使用被破解的 Claude 3.7 Sonnet 从 $\mathsf{nv{\text{-}}recall}=95.8\%$ 中提取《哈利·波特与魔法石》的成本约为 119.97 美元，而使用被破解的 GPT-4.1 的成本为 1.37 美元（ $\mathsf{nv{\text{-}}recall}=4.0\%$ ）；使用未破解的 Gemini 2.5 Pro 的成本约为 2.44 美元（ $\mathsf{nv{\text{-}}recall}=76.8\%$ ），而使用未破解的 Grok 3 的成本为 8.16 美元（ $\mathsf{nv{\text{-}}recall}=70.3\%$ ）。

Absolute extraction.  For a sense of the scale of how much text we extracted, it is also useful to examine absolute word counts. In Figure 7, we show results for four books for the total number of words $m$ that we extracted in in-order, near-verbatim blocks (Equation 6). As points of comparison, the $\mathsf{missing}$ count estimates how much text from the reference book was not extracted, and $\mathsf{additional}$ estimates how much text in the generation is not contained in the reference book. These metrics reveal additional nuances. First, low percentages of $\mathsf{nv{\text{-}}recall}$ can of course reflect enormous amounts of extraction. For Harry Potter and the Sorcerer’s Stone, we extracted thousands of words near-verbatim from all production LLMs. Even for GPT-4.1, for which $\mathsf{nv{\text{-}}recall}=4.0\%$, we extracted approximately $m\approx 3200$ words from the book. For A Game of Thrones, which is a significantly longer book, $\mathsf{nv{\text{-}}recall}=1.3\%$ for Grok 3, which corresponds to $m\approx 3700$ words of near-verbatim extracted text. Further, separate from total near-verbatim extraction, the individual extracted blocks can also be quite long. In Table 1, we show the longest extracted block for each experiment in Figure 7. For Harry Potter and the Sorcerer’s Stone, the longest near-verbatim blocks are $6658$, $821$, $9070$, and $6337$ words for Claude 3.7 Sonnet, GPT-4.1, Gemini 2.5 Pro, and Grok 3, respectively. The longest verbatim string that Nasr et al. (2023) extracted from ChatGPT 3.5 was slightly over $4000$ characters.
绝对提取。为了了解我们提取文本的规模，检查绝对词数也很有用。在图 7 中，我们展示了四个书籍在按顺序、近乎逐字提取的块中提取的总词数 $m$ （公式 6）。作为比较基准， $\mathsf{missing}$ 计数估计了参考书中未被提取的文本量，而 $\mathsf{additional}$ 估计了生成文本中不在参考书中的文本量。这些指标揭示了更多的细节。首先，低百分比的 $\mathsf{nv{\text{-}}recall}$ 当然可以反映巨大的提取量。对于《哈利·波特与魔法石》，我们从所有生产 LLMs 中近乎逐字提取了数千个词。即使对于 $\mathsf{nv{\text{-}}recall}=4.0\%$ 的 GPT-4.1，我们也从书中提取了大约 $m\approx 3200$ 个词。对于篇幅显著更长的《冰与火之歌》，Grok 3 的 $\mathsf{nv{\text{-}}recall}=1.3\%$ （对应于 $m\approx 3700$ 个近乎逐字提取的词）。此外，除了总近乎逐字提取外，单独提取的块也可以相当长。在表 1 中，我们展示了图 7 中每个实验的最长提取块。 对于《哈利·波特与魔法石》，Claude 3.7 Sonnet、GPT-4.1、Gemini 2.5 Pro 和 Grok 3 的最长近乎逐字复制的块分别是 $6658$ 、 $821$ 、 $9070$ 和 $6337$ 个词。Nasr 等人（2023 年）从 ChatGPT 3.5 中提取的最长逐字字符串略超过 $4000$ 个字符。

Second, interpreting $\mathsf{additional}$ and $\mathsf{missing}$ in Figure 7 indicates some important caveats. Recall that both counts may contain some instances of valid extraction that our measurement procedure under-counts. Since our extraction metric $m$ counts contiguous near-verbatim blocks, potentially duplicated (still valid) extraction may contribute to $\mathsf{additional}$, and near-verbatim text that is generated out-of-order with respect to the reference book may be counted in both $\mathsf{additional}$ and $\mathsf{missing}$ (Section 3.3.1). For instance, we note that the diff for Claude 3.7 Sonnet’s generation and The Great Gatsby has extensive repeats of extracted text on pages 114–132, which contribute to $\mathsf{additional}$. Note that duplicates also have an effect on the quality of the overall reproduction of a book in extracted outputs. While for Claude 3.7 Sonnet we extract $\mathsf{nv{\text{-}}recall}=97.5\%$ of the reference book, we did not extract a pristine copy of the whole book. Qualitative inspection of diffs for Claude 3.7 Sonnet on Frankenstein, 1984, and Harry Potter and the Sorcerer’s Stone reveals that we extracted cleaner copies of the ground-truth text that lack repeated extraction.
其次，解读图 7 中的 $\mathsf{additional}$ 和 $\mathsf{missing}$ 表明存在一些重要的注意事项。回想一下，这两个计数都可能包含一些我们测量程序低估的有效提取实例。由于我们的提取指标 $m$ 计算连续的近乎逐字复制的块，潜在的重复（但仍然有效）的提取可能对 $\mathsf{additional}$ 有贡献，而与参考书顺序不一致的近乎逐字生成的文本可能同时被计入 $\mathsf{additional}$ 和 $\mathsf{missing}$ （第 3.3.1 节）。例如，我们注意到 Claude 3.7 Sonnet 生成 The Great Gatsby 的 diff 中，第 114 至 132 页有大量重复的提取文本，这增加了 $\mathsf{additional}$ 。请注意，重复也会影响提取输出中整本书复制的质量。虽然对于 Claude 3.7 Sonnet 我们提取了 $\mathsf{nv{\text{-}}recall}=97.5\%$ 的参考书，但我们并未提取整本书的原始副本。对 Claude 3.7 Sonnet 在 Frankenstein、1984 和 Harry Potter and the Sorcerer’s Stone 上的 diff 进行定性检查表明，我们提取了更干净的、缺乏重复提取的 ground-truth 文本副本。

Brief qualitative observations about $\mathsf{additional}$ generated text.  We perform limited qualitative analysis of the $\mathsf{additional}$ generated text. As noted above, a portion of this text may contain duplicated or out-of-order extraction. However, this is not always the case; often, the $\mathsf{additional}$ generated text is not extraction. Brief qualitative inspection of this text for all of our experiments reveals that, for all books and frontier LLMs, $\mathsf{additional}$ text frequently contains text that replicates plot elements, themes, and character names from the book from which the Phase 1 prefix is drawn. We provide two examples of such text in Figure 8; these examples are drawn from GPT-4.1-generated text following Phase 1 success with a seed prefix from A Game of Thrones. Note that $\mathsf{nv{\text{-}}recall}$ is exactly $0\%$ for GPT-4.1 for A Game of Thrones (Figure 5), as matched words $m=0$ (Figure 7(d)). We selected these two examples by randomly sampling an index in the generation, and then looking at the surrounding text. We then manually performed repeated searches for subsequences of the generated text in the reference book, to confirm that they do not reflect extraction. Since extraction is our focus, we do not make claims about this non-extracted text, and instead defer detailed analysis to future work.
关于 $\mathsf{additional}$ 生成的文本的初步定性观察。我们对 $\mathsf{additional}$ 生成的文本进行了有限的定性分析。如前所述，这部分文本可能包含重复或顺序错乱的提取。然而，这种情况并非总是发生；通常， $\mathsf{additional}$ 生成的文本并非提取。我们对所有实验中的这部分文本进行初步定性检查后发现，对于所有书籍和前沿 LLMs， $\mathsf{additional}$ 文本经常包含与 Phase 1 前缀所来源的书籍中重复的情节元素、主题和角色名称。我们在图 8 中提供了两个这样的文本示例；这些示例来自 GPT-4.1 生成的文本，该文本在以《权力的游戏》为种子前缀成功完成 Phase 1 后生成。请注意，对于 GPT-4.1 的《权力的游戏》， $\mathsf{nv{\text{-}}recall}$ 与 $0\%$ 完全相同（见图 5），匹配的词语 $m=0$ （见图 7(d)）。我们通过随机采样生成中的索引，然后查看周围的文本来选择这两个示例。随后，我们手动在参考书中反复搜索生成文本的子序列，以确认它们并非提取内容。 由于我们的重点是提取，因此我们不对这部分未提取的文本做任何声明，而是将详细分析留待未来的工作。

As noted in the prior section, our initial experiments revealed that different settings for the two-phase procedure had an impact on extraction for each production LLM. For instance, these initial experiments revealed that we did not need to jailbreak Gemini 2.5 Pro or Grok 3. They also revealed how different generation configurations for Phase 2 resulted in varying amounts of extraction. Here, we provide some more details about how varied settings impact extraction, according to production LLM. Full experimental configurations, additional results, and API cost information can be found in Appendices C.2 and  D.2.
如前一节所述，我们的初步实验表明，两阶段流程的不同设置对每个生产 LLM 的提取产生了影响。例如，这些初步实验表明我们无需越狱破解 Gemini 2.5 Pro 或 Grok 3。它们还揭示了第二阶段的不同生成配置如何导致提取量的变化。在此，我们根据生产 LLM，提供更多关于不同设置如何影响提取的细节。完整的实验配置、附加结果和 API 成本信息可在附录 C.2 和 D.2 中找到。

Gemini 2.5 Pro.  For all experiments, Gemini 2.5 Pro did not refuse to continue the seed prefix in Phase 1. In our initial exploratory experiments, after a number of turns in the Phase 2 continue loop, the Gemini 2.5 Pro API would stop returning text; it instead would provide an empty response with a metadata object, linking to documentation indicating that we had encountered guardrails meant to prevent the recitation of copyrighted material (Google AI for Developers, ). We found that we could mitigate this behavior by minimizing the “thinking budget,” and explicitly querying Gemini 2.5 Pro to “Continue without citation metadata.” In some runs, Gemini 2.5 Pro would occasionally return empty responses during Phase 2. When this occurred, we count this as a turn in the maximum query budget, and retry after a one-second delay (Appendix C.2.2).
Gemini 2.5 Pro。在所有实验中，Gemini 2.5 Pro 在第一阶段并未拒绝继续使用种子前缀。在我们的初步探索性实验中，在第二阶段的继续循环中经过若干回合后，Gemini 2.5 Pro API 会停止返回文本；相反，它会提供一个包含元数据对象且内容为空的响应，该元数据对象链接到文档，表明我们遇到了旨在防止引用受版权保护材料的防护措施（Google AI for Developers,）。我们发现，通过最小化“思考预算”，并明确查询 Gemini 2.5 Pro 要求“无引用元数据地继续”，可以缓解这种行为。在某些运行中，Gemini 2.5 Pro 在第二阶段偶尔会返回空响应。当这种情况发生时，我们将其计为最大查询预算中的一个回合，并在一秒延迟后重试（附录 C.2.2）。

We also found that Gemini 2.5 Pro’s responses would often repeat previously emitted text. We therefore experimented with different generation configurations for the maximum number of generated tokens, frequency penality, and presence penalty. Through a set of experiments on Harry Potter and the Sorcerer’s Stone, we found that a maximum of $2000$ tokens resulted in the highest $\mathsf{nv{\text{-}}recall}$. We fixed this parameter, and swept over different combinations of frequency and presence penalty. Setting frequency penalty to $2$ and presence penalty to $0.1$ resulted in the highest $\mathsf{nv{\text{-}}recall}$, so we fix these as the configurations for Gemini 2.5 Pro runs across books for the results shown in Section 4.2 (Figures 5 and 7). Nevertheless, as shown in Figure 9(a), variance in extraction can be significant depending on the choice of these settings. Given the cheap cost of running our experiments on Gemini 2.5 Pro, we provide results for all books testing each of these $9$ configurations in Appendix D.2.2. These results show that the single, fixed configuration for Gemini 2.5 Pro for the results in Section 4.2 do not always result in the highest $\mathsf{nv{\text{-}}recall}$ for every book.
我们还发现，Gemini 2.5 Pro 的响应经常会重复之前已生成的文本。因此，我们尝试了不同的生成配置，包括最大生成 token 数、频率惩罚和存在惩罚。通过在《哈利·波特与魔法石》上进行的一系列实验，我们发现最大 $2000$ token 数量产生了最高的 $\mathsf{nv{\text{-}}recall}$ 。我们固定了这个参数，并测试了不同的频率惩罚和存在惩罚组合。将频率惩罚设置为 $2$ ，存在惩罚设置为 $0.1$ ，产生了最高的 $\mathsf{nv{\text{-}}recall}$ ，因此我们将这些设置为 Gemini 2.5 Pro 在所有书籍中运行的结果配置，这些结果在 4.2 节（图 5 和图 7）中展示。然而，如图 9(a)所示，提取的方差可能因这些设置的选择而显著不同。鉴于在 Gemini 2.5 Pro 上运行实验的成本较低，我们在附录 D.2.2 中提供了针对所有书籍测试每个 $9$ 配置的结果。这些结果表明，为 4.2 节中的 Gemini 2.5 Pro 设置的单个固定配置并不总是为每本书产生最高的 $\mathsf{nv{\text{-}}recall}$ 。

Grok 3.  We encountered no guardrails for any experiments for Phase 1. Except for the run on 1984 (Figure 5), we did not encounter any guardrails for Phase 2. For 1984, Grok 3 produced verbatim text until the $24$th continue request, when it responded with a refusal and that it would instead “continue the narrative in a way that respects the source material.” During Phase 2, the Grok 3 API sometimes returned a generic HTTP 500 error code, indicating a provider-side issue with fulfilling API requests. In these cases, the continuation loop terminated before the max query budget was exhausted.
Grok 3. 我们在第一阶段实验中未遇到任何限制条件。除了在 1984 年（图 5）上的运行外，我们在第二阶段也未遇到任何限制条件。对于 1984 年，Grok 3 在收到 $24$ 继续请求之前一直逐字生成文本，当收到继续请求时，它拒绝并表示将“以尊重原始材料的方式继续叙述”。在第二阶段，Grok 3 API 有时会返回通用的 HTTP 500 错误代码，表明在满足 API 请求方面存在提供方问题。在这些情况下，继续循环在最大查询预算耗尽之前终止。

Claude 3.7 Sonnet.  Initial experiments to complete a seed prefix failed, which is why we experimented with BoN in Phase 1. Early runs with Harry Potter and the Sorcerer’s Stone revealed that, for BoN-jailbroken Claude 3.7 Sonnet, different response lengths in Phase 2 could trigger refusals. In iterative experiments, we reduced the maximum response length per continue query from $1000$ to $250$ tokens, which was sufficient to evade refusals in all future experiments. We also noticed that, when Claude 3.7 Sonnet reproduced an entire book near-verbatim, it often appended “THE END”. This is what inspired us to include a stop-phrase condition in Phase 2, in addition to checking for refusals or if a maximum query budget has been exhausted.
Claude 3.7 Sonnet。初步实验试图完成种子前缀失败，这就是我们在第一阶段尝试 BoN 的原因。早期使用《哈利·波特与魔法石》的运行表明，对于 BoN 破解的 Claude 3.7 Sonnet，第二阶段的不同响应长度可能会触发拒绝。在迭代实验中，我们将每个继续查询的最大响应长度从 $1000$ 减少到 $250$ 个 token，这足以在所有后续实验中避免拒绝。我们还注意到，当 Claude 3.7 Sonnet 近乎逐字地复现整本书时，它经常会附加“THE END”。这启发了我们在第二阶段加入一个停止短语条件，除了检查拒绝或查询预算是否已用尽。

GPT-4.1.  As discussed in Section 4.2, jailbreaking GPT-4.1 in Phase 1 generally took significantly more BoN attempts with our specific initial instruction than for Claude 3.7 Sonnet. Further, for jailbroken GPT-4.1, success of the continuation loop in Phase 2 was always curtailed by an eventual refusal. Except for Grok 3 on 1984, these constituted all refusals in our final experimental configurations. Therefore, for the experiments shown in Section 4.2, if we successfully extracted text from GPT-4.1 for a given book, that text was always from the first chapter, after which GPT-4.1 refused to continue. For instance, for Harry Potter and the Sorcerer’s Stone, the last response before refusal was “That is the end of Chapter One.”
GPT-4.1. 如第 4.2 节所述，在第一阶段中，使用我们特定的初始指令破解 GPT-4.1 通常需要显著更多的 BoN 尝试次数，这比破解 Claude 3.7 Sonnet 要困难得多。此外，对于破解后的 GPT-4.1，在第二阶段中，其延续循环的成功总是被最终的拒绝所限制。除了 1984 年的 Grok 3 之外，这些构成了我们最终实验配置中的所有拒绝。因此，对于第 4.2 节中展示的实验，如果我们成功从 GPT-4.1 中提取了某本书的文本，那么该文本总是来自第一章，之后 GPT-4.1 拒绝继续。例如，对于《哈利·波特与魔法石》，拒绝前的最后回应是“第一章到此结束。”

However, encountering a refusal at the end of the first book chapter does not necessarily mean that further text is not memorized by GPT-4.1. Rather, failure due to refusal simply indicates that we were unable to extract more text with our specific two-phase procedure. To explore this further, we ran an additional set of experiments to attempt to elicit additional memorization from GPT-4.1. For each book, we execute a chapter-by-chapter variant of the two-phase procedure: for each chapter, we use the first sentence as the seed prefix for BoN in Phase 1 to find a successful jailbreak prompt, and then run the Phase 2 continuation loop to attempt to extract the rest of the chapter. We also implemented a retry policy for if we encountered a refusal, as we noticed that refusals are not deterministic for GPT-4.1: it may refuse a request at one point in time, but after a time delay may fulfill the identical request and continue.⁹⁹9Non-determinism is another salient difference between our results and those in Cooper et al. (2025), which are deterministic.
非确定性是我们结果与 Cooper 等人（2025）结果之间的另一个显著差异，后者是确定性的。
然而，在第一本书章节的末尾遇到拒绝并不意味着 GPT-4.1 没有记住更多文本。相反，由于拒绝而失败仅仅表明我们无法用特定的两阶段程序提取更多文本。为了进一步探索这一点，我们运行了一系列额外的实验，试图从 GPT-4.1 中引出更多记忆。对于每一本书，我们执行了按章节进行的两阶段程序变体：对于每一章节，我们使用第一句话作为 BoN 在第一阶段中的种子前缀来找到一个成功的越狱提示，然后运行第二阶段的延续循环来尝试提取章节的其余部分。我们还实现了一个重试策略，因为我们注意到对于 GPT-4.1 来说，拒绝并不是确定的：它在某个时间点可能会拒绝一个请求，但在经过一段时间延迟后可能会满足相同的请求并继续。 ⁹ This more intensive approach—which also makes use of more ground-truth text from the reference book—is able to extract more training data.
这种更密集的方法——同时利用了参考书中更多的真实文本——能够提取更多训练数据。

In Figure 9(b), we compare $\mathsf{nv{\text{-}}recall}$ results for these per-chapter-with-retry experiments with the results of our main experiments involving a single two-phase run starting with a prefix from the first chapter (Figure 5). For the per-chapter-with-retry variant, we report the total proportion of the book extracted by taking the union over (non-overlapping/disjoint) near-verbatim blocks to compute $\mathsf{nv{\text{-}}recall}$ (Equation 7). Note: We ran these experiments to probe if our main extraction procedure under-counts possible extraction (and thus memorization). The underlying extraction procedures are not equivalent, in terms of effort expended to elicit extraction.
在图 9(b)中，我们将这些每章重试实验的 $\mathsf{nv{\text{-}}recall}$ 结果与我们的主要实验结果进行了比较，后者从第一章的提示词开始，进行一次两阶段运行（如图 5 所示）。对于每章重试的变体，我们通过取（非重叠/不相交）逐字复述块的并集来计算提取的书的总比例，以计算 $\mathsf{nv{\text{-}}recall}$ （公式 7）。注意：我们运行这些实验是为了探究我们的主要提取程序是否低估了可能的提取（从而导致记忆）。这些提取程序在付出的努力来引出提取方面并不等价。

Throughout this paper, we have highlighted limitations and caveats in italicized notes. Nevertheless, it is worth reiterating that the points we raise have an important impact on how our results should be interpreted.
在整个论文中，我们用斜体注释强调了局限性和注意事项。然而，值得重申的是，我们提出的观点对我们的结果解释具有重要影响。

A loose lower bound on memorization for specific books.  Separate from how our measurements for extraction are conservative (Section 3.3), it is well-known that extraction more generally under-counts the total amount of training data that LLMs memorize. While prior work has demonstrated this in other contexts (Nasr et al., 2023; Cooper et al., 2025), our results for GPT-4.1 show how changing the prompting strategy can significantly alter how much extraction we observe, and how much underlying memorization this reveals. Our main focus is attempting to extract specific books near-verbatim; so, in most experiments, we run the two-phase procedure only once, with Phase 1 using a seed prefix from the beginning of a given book. In most cases, qualitative inspection of diffs with reference books shows that this succeeds in extracting near-verbatim text from at least part of the first chapter, but then the generation often diverges from the true text. However, as is clear in Figure 9(b), seeding Phase 1 in different book locations (here, the start of each chapter) can reveal additional memorization that we did not capture with our main experiments.
针对特定书籍的记忆力下限的宽松估计。与我们提取测量的保守性（第 3.3 节）不同，众所周知，提取通常低估了 LLMs 记忆的总训练数据量。虽然先前工作在其他背景下已经证明了这一点（Nasr 等人，2023；Cooper 等人，2025），但我们的 GPT-4.1 结果展示了如何通过改变提示策略显著改变我们观察到的提取量，以及这种改变揭示了多少潜在的记忆力。我们的主要重点是尝试近乎逐字地提取特定书籍；因此，在大多数实验中，我们仅运行两阶段流程一次，第一阶段使用给定书籍开头的种子前缀。在大多数情况下，与参考书籍的 diff 定性检查表明，这成功地从第一章的部分内容中提取了近似的逐字文本，但随后生成的内容往往偏离真实文本。然而，如图 9(b)所示，在不同书籍位置（此处为每章的开头）对第一阶段进行种子化，可以揭示我们主要实验未能捕捉到的额外记忆力。

Relatively small scale of experiments and their cost.  It is challenging to study production settings, as APIs change over time. For the same reason, it is often also difficult to reproduce results on production LLMs. We limited our experiments to a specific time window, so that we could successfully complete testing on the same books for all four production LLMs. In all, we only ran experiments on fourteen specific books, so our results do not speak to memorization and extraction more generally. Cost also impacted the number of books we tested. While it was typically less than $1 to run the Phase 2 continuation loop for Gemini 2.5 Pro, it was more expensive for some production LLMs. Notably, for Claude 3.7 Sonnet, long-context generation is significantly more expensive; it often cost over $100 per run (Table 1 & Appendix D.2.1).
实验规模相对较小，且成本较高。研究生产环境具有挑战性，因为 API 会随时间变化。出于相同原因，在 production LLMs 上重现结果也常常很困难。我们将实验限制在特定的时间窗口内，以便能够成功地在所有四个 production LLMs 上对同一本书进行测试。总共，我们仅在十四本书上进行了实验，因此我们的结果不能推广到记忆和提取的更一般情况。成本也影响了我们测试的书籍数量。虽然运行 Gemini 2.5 Pro 的 Phase 2 延续循环通常不到 1 美元，但对某些 production LLMs 来说则更昂贵。值得注意的是，对于 Claude 3.7 Sonnet，长上下文生成成本显著更高；每次运行通常超过 100 美元（表 1 & 附录 D.2.1）。

LLM-specific configuration of the two-phase extraction procedure.  In our main experiments (Section 4.2), we test one relatively simple extraction procedure (Section 3), and we instantiate that procedure in different ways for different production LLMs. In Phase 1, we decided to make the jailbreak optional, and we only tested Best-of-$N$. For Gemini 2.5 Pro and Grok 3, it was remarkable that this procedure evaded safeguards—that we did not need to use a jailbreak to successfully extract training data. However, it is also possible that, if we had used BoN on these two LLMs, it may have changed how much extraction we observed. For Phase 2, we set temperature to $0$ for all generation configurations and use the same halting conditions, but we tuned LLM-specific parameters (e.g., frequency penalty for Gemini 2.5 Pro) to increase LLM-specific extraction success.
针对 LLM 的两阶段提取流程的特定配置。在我们的主要实验（第 4.2 节）中，我们测试了一种相对简单的提取流程（第 3 节），并根据不同的生产 LLM 以不同的方式实例化该流程。在第一阶段，我们决定将越狱设置为可选，并且只测试了 Best-of- $N$ 。对于 Gemini 2.5 Pro 和 Grok 3 来说，值得注意的是，该流程避开了安全防护——我们不需要使用越狱就能成功提取训练数据。然而，也有可能，如果我们在这两个 LLM 上使用 BoN，可能会改变我们观察到的提取量。对于第二阶段，我们将所有生成配置的温度设置为 $0$ ，并使用相同的停止条件，但我们调整了 LLM 特定参数（例如，Gemini 2.5 Pro 的频率惩罚）以提高 LLM 特定提取的成功率。

We do not make evaluative claims across LLMs.  Given the above, it bears repeating that every observation we make about our results is with respect to a specific production LLM, book, and instantiation and run of our specific two-phase procedure. In some cases, the specific conditions we test revealed an enormous amount of extraction; notably, we extracted two entire in-copyright books—Harry Potter and the Sorcerer’s Stone and 1984—from Claude 3.7 Sonnet near-verbatim. We only make descriptive statements about these results: we discuss outcomes concerning specific experimental choices, outputs, and determinations of extraction success (Chouldechova et al. (2025); Sections 1 and 4.2). This aligns with our goal: to see if it is possible to extract long-form books from production LLMs. However, we do not make broader evaluative claims across production LLMs. For instance, while our specific experiments extracted the most text from Claude 3.7 Sonnet (Section 4.2), we do not claim that these results indicate Claude 3.7 Sonnet in general memorizes more training data than the other three production LLMs. We do not claim that any production LLM is in general more robust to extraction than another. Our bar plots should not be interpreted as making such comparative claims. In order to make such evaluative, comparative claims, one would need to run a much larger scale study under more controlled conditions.
我们不在 LLMs 之间做出评价性声明。鉴于上述内容，需要再次强调的是，我们对结果的每一项观察都是针对特定的生产 LLM、书籍以及我们特定两阶段程序的特定实例和运行。在某些情况下，我们测试的具体条件揭示出大量的提取；值得注意的是，我们从 Claude 3.7 Sonnet 中几乎逐字提取了两本版权在册的完整书籍——《哈利·波特与魔法石》和《1984》。我们仅对这些结果做出描述性陈述：我们讨论与特定实验选择、输出以及提取成功判定相关的结果（Chouldechova 等人（2025）；第 1 节和第 4.2 节）。这符合我们的目标：看看是否有可能从生产 LLMs 中提取长篇书籍。然而，我们不在生产 LLMs 之间做出更广泛的评价性声明。例如，虽然我们的特定实验从 Claude 3.7 Sonnet 中提取了最多的文本（第 4.2 节），但我们并不声称这些结果表明 Claude 3.7 Sonnet 总体上比其他三个生产 LLMs 记忆了更多的训练数据。 我们不声称任何生产 LLM 在一般意义上比另一个更不容易被提取。我们的条形图不应被解释为做出此类比较性声明。为了做出此类评估性、比较性声明，需要在一个更受控的条件下进行更大规模的调查研究。

While we defer detailed copyright analysis to future work, we briefly address why our results may be of interest.
虽然我们将详细的版权分析留待未来的工作，但我们将简要说明为什么我们的结果可能引起兴趣。

Production LLMs memorize some of their training data, and extraction is sometimes feasible.  In copyright litigation concerning generative AI, extraction and memorization of training data are both central issues (Sections 1 & 2). Several lawsuits have addressed questions over whether production LLMs reproduce copyrighted training data in their outputs (i.e., have touched on extraction) (Kadrey Judgment, 2025; 2025). There has also been increased academic discussion (Cooper and Grimmelmann, 2024; Dornis, 2025) and litigation (GEMA v. OpenAI, ) over whether LLMs themselves are legally cognizable copies of the training data they have memorized. Regardless of how relevant these issues may be for potential findings of copyright infringement, our work reveals important technical facts: the four production LLMs we study memorized (at least some of) the books on which they were trained, and it is possible to extract (at least some of) those memorized books at generation time.
生产 LLMs 会记忆部分训练数据，有时可以进行提取。在涉及生成式 AI 的版权诉讼中，训练数据的提取和记忆都是核心问题（第 1 节和第 2 节）。几起诉讼已经探讨了生产 LLMs 的输出是否复制了受版权保护的训练数据（即涉及了提取）的问题（Kadrey 判决，2025 年；2025 年）。此外，学术界也出现了越来越多的讨论（Cooper 和 Grimmelmann，2024 年；Dornis，2025 年）和诉讼（GEMA 诉 OpenAI），讨论 LLMs 本身是否是它们记忆的训练数据的法律可认的副本。无论这些问题对于潜在的版权侵权认定有多么重要，我们的工作揭示了重要的技术事实：我们研究的四个生产 LLMs 记忆了（至少部分）它们所训练的书籍，并且有可能在生成时提取（至少部分）这些记忆的书籍。

Jailbreaks, adversarial use, and cost.  Some might qualify our experiments as atypical use, as we deliberately tried to surface memorized books. Adversarial use, like the use of jailbreaks, may matter for copyright infringement analysis (Lee et al., 2023b; Cooper and Grimmelmann, 2024). Further, for the cases in which we retrieved whole copies of near-verbatim books, it was often quite costly ($>\mathdollar 100$) to do so (Section 4.2, Table 1). As Cooper et al. (2025) note, even with respect to their significantly cheaper experiments using open-weight LLMs, “there are easier and more effective ways to pirate a book.” Nevertheless, it is important to emphasize that we did not use jailbreaks for two production LLMs during Phase 1. In Phase 2, we observed that all four production LLMs sometimes responded with large spans of in-copyright text. In all cases, successful extraction of training data would not have been possible if these LLMs had not memorized those data during training (Section 3.3.2).
越狱攻击、对抗性使用和成本。有些人可能将我们的实验视为非典型使用，因为我们故意试图揭示记忆中的书籍。对抗性使用，如越狱攻击的使用，可能对版权侵权分析有影响（Lee 等人，2023b；Cooper 和 Grimmelmann，2024）。此外，对于我们检索到近乎逐字复制的书籍完整副本的情况，这样做通常成本很高（ $>\mathdollar 100$ ）（第 4.2 节，表 1）。正如 Cooper 等人（2025 年）指出的那样，即使对于他们使用开放权重 LLMs 进行的明显更便宜的实验，“也有更容易和更有效的方法来盗版一本书。”然而，必须强调的是，在第一阶段，我们没有使用两个生产 LLMs 的越狱攻击。在第二阶段，我们观察到所有四个生产 LLMs 有时会以大段受版权保护文本的形式做出回应。在所有情况下，如果这些 LLMs 在训练期间没有记住这些数据，那么成功的提取训练数据将是不可能的（第 3.3.2 节）。

Best efforts and safeguards.  As others have noted, it may be infeasible to produce perfect safeguards; in such circumstances, preventing the generation of copyrighted or otherwise undesirable material may depend on “reasonable best efforts” (Cooper et al., 2024). As noted above, in our main experiments (Section 4.2), two production LLMs did not exhibit safeguards in Phase 1: Gemini 2.5 Pro and Grok 3 directly complied with our initial probes to complete prefixes from books. We used jailbreaks to get Claude 3.7 Sonnet and GPT-4.1 to comply in Phase 1. For GPT-4.1 and our chosen initial instruction, it frequently took a significant number of BoN attempts to achieve Phase 1 success. It often took far fewer than our maximum budget ($N=$10,000$$) to jailbreak Claude 3.7 Sonnet to complete a provided in-copyright book prefix. Jailbreaks aside, our experiments managed to evade system-level safeguards during Phase 2. We were able to run multiple—sometimes hundreds—of iterations of a simple continue loop for each production LLM, before (if ever) encountering filters intended to prevent generation of copyrighted material (Section 2).
最佳努力与安全措施。正如其他人所指出的，可能无法实现完美的安全措施；在这种情况下，防止生成版权保护或其他不希望的材料可能取决于“合理的最佳努力”（Cooper 等人，2024）。如上所述，在我们的主要实验（第 4.2 节）中，两个生产 LLM 在第一阶段没有表现出安全措施：Gemini 2.5 Pro 和 Grok 3 直接按照我们最初的探测要求完成书籍的前缀。我们使用越狱技术让 Claude 3.7 Sonnet 和 GPT-4.1 在第一阶段配合。对于 GPT-4.1 和我们选择的初始指令，经常需要大量 BoN 尝试才能实现第一阶段成功。越狱除外，我们的实验在第二阶段成功规避了系统级安全措施。我们能够在（如果遇到的话）旨在防止生成版权保护材料的过滤器之前，对每个生产 LLM 运行多个——有时是数百个——简单的继续循环迭代（第 2 节）。

Non-extracted, $\mathsf{additional}$ text.  In our experiments, we specifically investigate extraction of training data. However, when conducting our extraction analysis, we qualitatively observed that thousands of words of $\mathsf{additional}$ (Equation 8), non-extracted generated text from all four production LLMs replicate character names, plot elements, and themes (Figure 8, Section 4.2). Given that copyright law does not only apply to near-verbatim copying, such outputs may be interest. We stress that we do not perform rigorous, quantitative, at-scale analysis of this text, and instead defer this to future work.
未提取的 $\mathsf{additional}$ 文本。在我们的实验中，我们专门研究了训练数据的提取。然而，在进行提取分析时，我们定性地观察到，所有四个生产 LLMs 生成的 $\mathsf{additional}$ （公式 8）非提取文本都复制了角色名称、情节元素和主题（图 8，第 4.2 节）。鉴于版权法不仅适用于近乎逐字复制，这些输出可能引起兴趣。我们强调，我们没有对这个文本进行严格、定量、大规模的分析，而是将其留待未来工作。