ChatGPT’s Security Threaten Research

Abstract

Abstract: With the rapid development of deep learning technology and natural language processing technology, the large language model represented by ChatGPT came into being. However, while showing surprising capabilities in many fields, ChatgPT also exposed many security threats, which aroused the concerns of academia and industry. This paper first introduces the development history, working mode, and training methods of ChatGPT and its series models, then summarizes and analyzes various current security problems that ChatGPT may encounter and divides it into two levels: user and model. Then, countermeasures and solutions are proposed according to the characteristics of ChatGPT at each stage. Finally, this paper looks forward to developing a safe and trusted ChatGPT and a large language model.

Key words: ChatGPT, large langue model, artificial intelligent security, data security, privacy preserve

摘要： 随着深度学习技术与自然语言处理技术的快速发展，以ChatGPT为代表的大型语言模型应运而生，然而其在诸多领域展现出令人惊讶的能力的同时，也暴露出诸多安全威胁，这引发了学术界与产业界的担忧.首先，介绍了ChatGPT及其系列模型的发展历程、工作模式与训练方式；然后，从用户和模型2个层面总结并分析了当前ChatGPT可能遇到的各类安全问题，并提出应对思路与方案；最后，对ChatGPT以及大型语言模型领域未来如何安全可信地发展进行了展望.

关键词: ChatGPT, 大型语言模型, 人工智能安全, 数据安全, 隐私保护

朱孟垚, 李兴华. ChatGPT安全威胁研究[J]. 信息安全研究, 2023, 9(6): 533-.

References

［1］Natalie. ChatGPT—Release notes［EBOL］. OpenAI, 2023［20230508］. https:help.openai.comenarticles6825453chatgptreleasenotes (Available 20230419)［2］Van Dis E A M,Bollen J, Zuidema W, et al. ChatGPT: Five priorities for research［J］. Nature, 2023, 614(7947): 224226［3］Du H, Teng S, Chen H, et al. Chat withChatGPT on intelligent vehicles: An IEEE TIV perspective［J］. IEEE Trans on Intelligent Vehicles, 2023, 8(3): 20202026［4］Wang F Y, Miao Q, Li X, et al. What doeschatGPT say: The DAO from algorithmic intelligence to linguistic intelligence［J］. IEEECAA Journal of Automatica Sinica, 2023, 10(3): 575579［5］Teubner T,Flath C M, Weinhardt C, et al. Welcome to the era of ChatGPT et al. the prospects of large language models［J］. Business & Information Systems Engineering, 2023, 65(2): 95101［6］Bubeck S, Chandrasekaran V, Eldan R, et al. Sparks of artificial general intelligence: Early experiments with GPT4［J］. arXiv preprint, arXiv:2303.12712, 2023［7］George A S, George A S H. A review of ChatGPT AI’s impact on several business sectors［J］. Partners Universal International Innovation Journal, 2023, 1(1): 923［8］Wang F Y, Li J, Qin R, et al. ChatGPT for computational social systems: From conversational applications to humanoriented operating systems［J］. IEEE Trans on Computational Social Systems, 2023, 10(2): 414425［9］Microsoft. The new Bing & EdgeUpdates to Chat［EBOL］. 2023 ［20230508］. https:blogs.bing.comsearchfebruary2023ThenewBingEdgeUpdatestoChat［10］Microsoft. ColetteStallbaumer. Introducing Microsoft 365 Copilot—A whole new way to work［EBOL］. 2023 ［20230508］. https:www.microsoft.comenusmicrosoft365blog20230316introducingmicrosoft365copilotawholenewwaytowork［11］Asch D A. An interview with ChatGPT about health care［J］. NEJM Catalyst Innovations in Care Delivery, 2023, 4(2): 18［12］Sallam M. ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns［COL］ Healthcare. 2023 ［20230426］. https:doi.org10.3390healthcare11060887［13］Liu S, Wright A P, Patterson B L, et al. Assessing the value of ChatGPT for clinical decision support optimization［JOL］. medRxiv. 2023 ［20230426］. https:doi.org10.11012023.02.21.23286254［14］Weissenbacher D, O’Connor K, Rawal S, et al. Automatic extraction of medication mentions from tweets—Overview of the BioCreative VII shared task 3 competition［JOL］. 2023 ［20230426］. https:doi.org10.1093databasebaac108［15］Yang X, PourNejatian N, Shin H C, et al. GatorTron: A large clinical language model to unlock patient information from unstructured electronic health records［J］. arXiv preprint, arXiv:2203.03540, 2022［16］Zhang Y, Tiňo P, Leonardis A, et al. A survey on neural network interpretability［J］. IEEE Trans on Emerging Topics in Computational Intelligence, 2021, 5(5): 726742［17］Wei J, Tay Y,Bommasani R, et al. Emergent abilities of large language models［J］. arXiv preprint, arXiv:2206.07682, 2022［18］Lu Q, Qiu B, Ding L, et al. Error analysis prompting enables humanlike translation evaluation in large language models: A case study on ChatGPT［J］. arXiv preprint, arXiv:2303.13809, 2023［19］Aljanabi M. ChatGPT: Future directions and open possibilities［J］. Mesopotamian Journal of Cybersecurity, 2023, 2023: 1617［20］Lund B,Agbaji D. Information literacy, data literacy, privacy literacy, and ChatGPT: Technology literacies align with perspectives on emerging technology adoption within communities［JOL］. SSRN Electronic Journal. 2023 ［20230426］. http:dx.doi.org10.2139ssrn.4324580［21］Future of Life Institute. Pause giant AI experiments: An open letter［EBOL］. Future of Life. ［20230426］. https:futureoflife.orgopenletterpausegiantaiexperiments［22］Satariano, A. ChatGPT is banned in Italy over privacy concerns［EBOL］. The New York Times. 2023 ［20230426］. https:www.nytimes.com20230331technologychatgptitalyban.html［23］Anthony Cuthbertson. Germany considers ChatGPT ban［EBOL］. Independent. 2023 ［20230426］. https:www.independent.co.uktechchatgptbangermanyaiprivacyb2314487.html［24］OpenAI. In the matter of OpenAI［EBOL］. Center for AI and Digital Policy. 2023 ［20230426］. https:www.caidp.orgcasesopenai［25］OpenAI. GPT4 technical report［J］. arXiv preprint, arXiv:2303.08774, 2023［26］Topal M O, Bas A, van Heerden I. Exploring transformers in natural language generation: Gpt, bert, and xlnet［J］. arXiv preprint, arXiv:2102.08036, 2021［27］Floridi L, Chiriatti M. GPT3: Its nature, scope, limits, and consequences［J］. Minds and Machines, 2020, 30(4): 681694［28］Nye M, Tessler M, Tenenbaum J, et al. Improving coherence and consistency in neural sequence models with dualsystem, neurosymbolic reasoning［J］. Advances in Neural Information Processing Systems, 2021, 34: 2519225204［29］Liu Y, Han T, Ma S, et al. Summary of ChatGPTGPT4 research and perspective towards the future of large language models［J］. arXiv preprint, arXiv:2304.01852, 2023［30］Peng B, Li C, He P, et al. Instruction tuning with GPT4［J］. arXiv preprint, arXiv:2304.03277, 2023［31］Liu Y, Iter D, Xu Y, et al. GPTEval: NLG evaluation using GPT4 with better human alignment［J］. arXiv preprint, arXiv:2303.16634, 2023［32］Liu Z, Yu X, Zhang L, et al.DeidGPT: Zeroshot medical text deidentification by GPT4［J］. arXiv preprint, arXiv:2303.11032, 2023［33］Mijwil M, Aljanabi M. Towards artificial intelligencebased cybersecurity: The practices and ChatGPT generated ways to combat cybercrime［J］. Iraqi Journal for Computer Science and Mathematics, 2023, 4(1): 6570［34］Wang F Y, Li J, Qin R, et al. ChatGPT for computational social systems: From conversational applications to humanoriented operating systems［J］. IEEE Trans on Computational Social Systems, 2023, 10(2): 414425［35］Hariri W. Unlocking the potential of ChatGPT: A comprehensive exploration of its applications, limitations, and future directions in natural language processing［J］. Technology, 2023, 15(2): 1639［36］Thomas K, Pullman J, Yeo K, et al. Protecting accounts from credential stuffing with password breach alerting［C］ Proc of USENIX Security Symp. Berkeley, CA: USENIX Association, 2019: 15561571［37］BaidooAnu D, Owusu Ansah L. Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning［JOL］. 2023 ［20230426］. http:dx.doi.org10.2139ssrn.4337484［38］Grbic D V, Dujlovic I. Social engineering with ChatGPT［C］ Proc of the 22nd Int Symp INFOTEHJAHORINA. Piscataway, NJ: IEEE, 2023: 15［39］Biswas S. Prospective role of chat GPT in the military: According to ChatGPT［JOL］. 2023 ［20230426］. https:doi.org10.323888WYYOD［40］Roose K. The brilliance and weirdness of ChatGPT［EBOL］. (20221205) ［20230508］. https:www.nytimes.com20221205technologychatgptaitwitter.html［41］Blake A. Someone just used ChatGPT to generate free Windows keys［EBOL］. Digitaltrends: Alex Blake. (20230404) ［20230419］ https:www.digitaltrends.comcomputingchatgptgeneratesfreewindowskeys［42］GozaloBrizuela R, GarridoMerchan E C. ChatGPT is not all you need: A state of the art review of large generative AI models［J］. arXiv preprint, arXiv:2301.04655, 2023［43］Deng J, Lin Y. The benefits and challenges of ChatGPT: An overview［J］. Frontiers in Computing and Intelligent Systems, 2022, 2(2): 8183［44］Sebastian G. Do ChatGPT and other AI chatbots pose a cybersecurity risk?: An exploratory study［J］. International Journal of Security and Privacy in Pervasive Computing, 2023, 15(1): 111［45］OpenAI. Privacy policy［EBOL］.OpenAI: OpenAI. (20230427) ［20230508］ https:openai.compoliciesprivacypolicy［46］OpenAI. Samsung workers made a major error by using ChatGPT［EBOL］. Techradar: Lewis Maddison. (20230404) ［20230419］. https:www.techradar.comnewssamsungworkersleakedcompanysecretsbyusingchatgpt［47］AlSibai N. Amazon begs employees not to leak corporate secrets to ChatGPT［EBOL］. Futurism: NOOR ALSIBAI. (20230126) ［20230419］. https:futurism.comthebyteamazonbegsemployeeschatgpt［48］González C. Account Takeover Vulnerabilities in ChatGPT are patched by OpenAI［EBOL］. ［20230419］. https:www.linkedin.compulseaccounttakeovervulnerabilitieschatgptpatchedguti%C3%A9rrezgonz%C3%A1lez［49］Gillioz A, Casas J, Mugellini E, et al. Overview of the transformerbased models for NLP tasks［C］ Proc of the 15th Conf on Computer Science and Information Systems (FedCSIS). Piscataway, NJ: IEEE, 2020: 179183［50］Arghire I. OpenAI patches account takeover vulnerabilities in ChatGPT［EBOL］. ［20230419］. https:www.security week.comopenaipatchesaccounttakeovervulnerabilitiesinchatgpt［51］Zhang J, Peng S, Gao Y, et al. APMSA: Adversarial perturbation against model stealing attacks［J］. IEEE Trans on Information Forensics and Security, 2023, 18: 16671679［52］Dash B, Sharma P. Are ChatGPT and deepfake algorithms endangering the cybersecurity industry? A review［J］. International Journal of Engineering and Applied Sciences, 2023, 10(1): 15［53］Salem A M G, Bhattacharyya A, Backes M, et al. Updatesleak: Data set inference and reconstruction attacks in online learning［C］ Proc of the 29th USENIX Security Symp. Berkeley, CA: USENIX Association, 2020: 12911308［54］Carlini N, Tramer F, Wallace E, et al. Extracting training data from large language models［C］ Proc of USENIX Security Symp. Berkeley, CA: USENIX Association, 2021: 26332650［55］金志刚, 周峻毅, 何晓勇. 面向自然语言处理领域的对抗攻击研究与展望［J］. 信息安全研究, 2022, 8(3): 202211［56］Shokri R,Stronati M, Song C, et al. Membership inference attacks against machine learning models［C］ Proc of 2017 IEEE Symp on Security and Privacy. Piscataway, NJ: IEEE, 2017: 318［57］Lacharité M S, Minaud B, Paterson K G. Improved reconstruction attacks on encrypted data using range query leakage［C］ Proc of 2018 IEEE Symp on Security and Privacy. Piscataway, NJ: IEEE, 2018: 297314［58］Hu H,Salcic Z, Sun L, et al. Membership inference attacks on machine learning: A survey［J］. ACM Computing Surveys , 2022, 54(11s): 137［59］Duan J, Kong F, Wang S, et al. Are diffusion models vulnerable to membership inference attacks?［J］. arXiv preprint, arXiv:2302.01316, 2023［60］Wang Z, Ma J, Wang X, et al. Threats to training: A survey of poisoning attacks and defenses on machine learning systems［J］. ACM Computing Surveys, 2022, 55(7): 136［61］Alfeld S, Zhu X, Barford P. Data poisoning attacks against autoregressive models［C］ Proc of the AAAI Conf on Artificial Intelligence. Menlo Park, CA: AAAI Press, 2016［62］Dang T K, Truong P T T, Tran P T. Data poisoning attack on deep neural network and some defense methods［C］ Proc of 2020 Int Conf on Advanced Computing and Applications (ACOMP). Piscataway, NJ: IEEE, 2020: 1522［63］White J, Fu Q, Hays S, et al. A prompt pattern catalog to enhance prompt engineering with ChatGPT［J］. arXiv preprint, arXiv:2302.11382, 2023［64］Perez F, Ribeiro I. Ignore previous prompt: Attack techniques for language models［J］. arXiv preprint, arXiv:2211.09527, 2022［65］Salem A, Backes M, Zhang Y. Get a model! Model hijacking attack against machine learning models［J］. arXiv preprint, arXiv:2111.04394, 2021［66］Shumailov I, Zhao Y, Bates D, et al. Sponge examples: Energylatency attacks on neural networks［C］ Proc of 2021 IEEE European Symp on Security and Privacy (EuroS&P). Piscataway, NJ: IEEE, 2021: 212231［67］Maddigan P, Susnjak T. Chat2vis: Generating data visualisations via natural language using ChatGPT, codex and GPT3 large language models［J］. arXiv preprint, arXiv:2302.02094, 2023［68］Zhao W X, Zhou K, Li J, et al. A survey of large language models［J］. arXiv preprint, arXiv:2303.18223, 2023［69］Shen Y,Heacock L, Elias J, et al. ChatGPT and other large language models are doubleedged swords［J］. Radiology, 2023, 307(2): 230163230163［70］Dignum V. Responsible Artificial Intelligence: Recommendations and Lessons Learned［M］. Berlin: Springer International Publishing, 2023: 195214［71］Tian E. GPTZero classrooms［EBOL］. (20230220) ［20230419］. https:gptzero.substack.compgptzeroclassrooms［72］谭毓安. 高度重视人工智能安全问题［J］. 信息安全研究, 2022, 8(3): 311313

[1]	. Consideration on Some Problems in the Development of GPT4 and Its Regulation Scheme [J]. Journal of Information Security Reserach, 2023, 9(6): 510-.
[2]	. Research on the Integration of Full Lifecycle Data Security Management and Artificial Intelligence Technology#br# [J]. Journal of Information Security Reserach, 2023, 9(6): 543-.
[3]	. Research on Security Risk Response for Internet of Body Applications [J]. Journal of Information Security Reserach, 2023, 9(5): 433-.
[4]	. Preliminary Study on Data Security Protection Scheme of Civil Aviation A-CDM System [J]. Journal of Information Security Reserach, 2023, 9(5): 482-.
[5]	. A Survey of Data Security Sharing Technology Development and Its Application in Power Domain [J]. Journal of Information Security Reserach, 2023, 9(3): 208-.
[6]	. Research on On-chain and Off-chain Data Supervision Scheme for Energy Big Data [J]. Journal of Information Security Reserach, 2023, 9(3): 235-.
[7]	. Research and Implementation of Crossplatform Encryption Technology for Mobile Storage Devices [J]. Journal of Information Security Reserach, 2023, 9(3): 271-.
[8]	. Research on Data Transfer Security Model of Digital Twin City Big Data Platform [J]. Journal of Information Security Reserach, 2023, 9(1): 48-.
[9]	. Road Analysis of Platform Data Security Governance Optimization —From the Perspective of “DiDi Cyber Security Review Case” [J]. Journal of Information Security Reserach, 2023, 9(1): 66-.
[10]	. Analysis on the Division of Data Security Management Responsibilities and Accountability Mechanism [J]. Journal of Information Security Reserach, 2023, 9(1): 73-.
[11]	. Methodological Exploration of Data Security Standard Construction in Health Industry [J]. Journal of Information Security Reserach, 2023, 9(1): 87-.
[12]	. Research on Data Security Endogenous Mechanism on Complex Adaptive Organizations [J]. Journal of Information Security Reserach, 2022, 8(9): 901-.
[13]	. Promotion and Regulation Path of Personal Financial Information Cross-border Flow in China [J]. Journal of Information Security Reserach, 2022, 8(9): 931-.
[14]	. [J]. Journal of Information Security Reserach, 2022, 8(6): 554-.
[15]	. [J]. Journal of Information Security Reserach, 2022, 8(6): 613-.