本报告深入探讨了训练数据归因（TDA）技术，及其在降低人工智能极端风险中的潜在作用和可行性。TDA旨在识别对特定模型输出具有显著影响的训练数据点。报告评估了将现有TDA研究转化为可应用于前沿大型语言模型的有效工具所需的工作量，讨论了TDA工具对AI实验室的潜在研究益处，并分析了AI实验室公开训练数据意愿的限制因素。报告还探讨了TDA的社会效益，包括版权保护、事实核查和公众信任，并评估了TDA在缓解大规模AI风险方面的潜在影响。结论认为，TDA在提高AI技术安全性和可解释性方面具有巨大潜力。

📊 现代TDA技术主要分为三类：基于重训练、基于表示（或输入相似性）和基于梯度的方法。其中，基于梯度的方法（使用影响函数）被认为是实现实用TDA的最有希望的途径。

💰 目前使用影响函数进行TDA的最有效方法，其训练成本与预训练大型语言模型相当。尽管存储成本显著较高，但仍是可行的。然而，这些技术尚未在大型模型上进行测试，其准确性尚不明确。

💡 高效的TDA技术将对AI研究和大型语言模型开发产生重大积极影响，包括减少幻觉和错误信息、识别导致不良结果的训练数据、提高数据过滤和选择、缩小模型尺寸、提高可解释性和对齐性，以及改进模型定制和编辑。

🔒 AI实验室目前有很强的动机保持训练数据私有，因为公开这些数据可能会导致竞争优势减少、法律责任风险增加以及违反隐私或专有数据要求。但通过选择性允许TDA推断或返回来源而非精确数据，可以规避这些问题。

🛡️ TDA最有望的AI风险缓解益处是通过提高可解释性来增强大型语言模型的技术安全性。此外，TDA可能对技术AI安全和对齐工作产生重大影响。

Published on January 22, 2025 3:40 PM GMT

Note: This report was conducted in June 2024 and is based on research originally commissioned by the Future of Life Foundation (FLF). The views and opinions expressed in this document are those of the authors and do not represent the positions of FLF.

This report investigates Training Data Attribution (TDA) and its potential importance to and tractability for reducing extreme risks from AI. TDA techniques aim to identify training data points that are especially influential on the behavior of specific model outputs. They are motivated by the question: how would the model's behavior change if one or more data points were removed from or added to the training dataset? 

Report structure:

First, we discuss the plausibility and amount of effort it would take to bring existing TDA research efforts from their current state, to an efficient and accurate tool for TDA inference that can be run on frontier-scale LLMs. Next, we discuss the numerous research benefits AI labs will expect to see from using such TDA tooling.Then, we discuss a key outstanding bottleneck that would limit such TDA tooling from being accessible publicly: AI labs’ willingness to disclose their training data. We suggest ways AI labs may work around these limitations, and discuss the willingness of governments to mandate such access.Assuming that AI labs willingly provide access to TDA inference, we then discuss what high-level societal benefits you might see. We list and discuss a series of policies and systems that may be enabled by TDA. Finally, we present an evaluation of TDA’s potential impact on mitigating large-scale risks from AI systems.

Key takeaways from our report:

Modern TDA techniques can be categorized into three main groups: retraining-based, representation-based (or input-similarity-based), and gradient-based. Recent research has found that gradient-based methods (using influence functions) are the most likely path to practical TDA.The most efficient approach to conduct TDA using influence functions today has training costs on par with pre-training an LLM. It has significantly higher (but feasible) storage costs than an LLM model, and somewhat higher per-inference costs.Based on these estimates, TDA appears to be no longer infeasible to run on frontier LLMs with enterprise-levels of compute and storage. However, these techniques have not been tested on larger models, and the accuracy of these optimized TDA techniques on large models is unclear.Compressed-gradient TDA is already plausible to be used on fine-tuned models, which have orders of magnitude fewer training examples and parameters (on the order of millions or billions rather than hundreds of billions).Timing to achieve efficient and accurate TDA on frontier models is likely between 2-5 years, depending largely on specific incremental research results and amount of funding / researchers allocated to the space.Efficient TDA techniques will likely have a substantial positive impact on AI research and LLM development, including the following effects:
Mitigating the prevalence of hallucinations and false claimsIdentifying training data that produces poor results (bias, misinformation, toxicity), improved data filtering / selectionShrinking overall model size / improving efficiencyImproved interpretability & alignmentImproved model customization and editing
AI labs are likely already well-incentivized to invest in TDA research efforts because of the benefits to AI research.Public access to TDA tooling on frontier AI models is limited primarily by the unwillingness / inability of AI labs to publicly share their training data.
AI labs currently have strong incentives to keep their training data private, as publishing such data would have negative outcomes such as:
Reduced competitive advantages from data curationIncreased exposure to legal liabilities from data collectionViolating privacy or proprietary data requirements
AI labs may be able to avoid these outcomes by selectively permitting TDA inference on certain training examples, or returning sources rather than the exact training data.Governments are highly unlikely to mandate public access to training data.
If AI labs willingly provided public access to TDA, you could expect the following benefits, among others:
Preventing copyrighted data usage.Improved fact checking / content moderationImpacts on public trust and confidence in LLMsAccelerated research by external partiesIncreased accountability for AI labs
AI labs appear largely disincentivized to provide access to TDA inference, as many of the public benefits are disadvantageous for them.
Governments are highly unlikely to mandate public access to TDA.It seems plausible that certain AI labs may expose TDA as a feature, but that the majority would prefer to use it privately to improve their models.
Several systems that could be enabled by efficient TDA include:
Providing royalties to data providers / creatorsAutomated response improvement / fact-checkingTooling for improving external audits of training dataContent attribution tooling for LLMs, though it is unlikely to replace systems reliant on RAG
We believe that the most promising benefit of TDA for AI risk mitigation is its potential to improve the technical safety of LLMs via interpretability.
There are some societal / systematic benefits from TDA, and these benefits may be a small contributing factor to reducing some sources of risk. We don’t think these appear to move the needle significantly to reduce large-scale AI risks.TDA may meaningfully improve AI capabilities research, which might actually increase large-scale risk.TDA may eventually be highly impactful in technical AI safety and alignment efforts. We’d consider TDA’s potential impact on technical AI safety to be in a similar category to supporting mechanistic interpretability research. 

Discuss