Abstract: Currently, pretrained language models are widely used to learn general language representations from massive training corpora. The performance of downstream tasks in the field of natural language processing has been significantly improved after using the pretrained language model, but the overfitting phenomenon of the deep neural network makes the pretrained language model may have the risk of leaking the privacy of the training corpus. This paper selects T5, GPT, OPT and other widely used pretrained language models as research objects, and uses model inversion attacks to explore the factors that affect the data leakage of pretrained language models. During the experiment, the pretrained language model was used to generate a large number of samples, and the samples most likely to cause data leakage risk were selected for verification by indicators such as perplexity. It proved that different models such as T5 have different degrees of data leakage problems. For the same model,  the larger size of the model, the scale, the greater the possibility of data leakage; adding a specific prefix makes it easier to obtain leaked data. The future data leakage problem and its defense methods are prospected.

Key words:  , natural language processing, pretrained language models, private data leakage, model inversion attack, model architecture

摘要： 当前广泛使用的预训练语言模型是从海量训练语料中学习通用的语言表示.自然语言处理领域的下游任务在使用预训练语言模型后性能得到显著提升，但是深度神经网络过拟合现象使得预训练语言模型可能存在泄露训练语料隐私的风险.选用T5，GPT2，OPT等广泛使用的预训练语言模型作为研究对象，利用模型反演攻击探索影响预训练语言模型数据泄露的因素.实验过程中利用预训练语言模型生成大量样本，以困惑度等指标选取最有可能发生数据泄露风险的样本进行验证，证明了T5等不同模型均存在不同程度的数据泄露问题;同一种模型，模型规模越大数据泄露可能性越大;添加特定前缀更容易获取泄露数据等问题.对未来数据泄露问题及其防御方法进行了展望.

关键词: 自然语言处理, 预训练语言模型, 隐私数据泄露, 模型反演攻击, 模型架构