Abstract: With the widespread popularization of the Internet and smart devices, social media has become a major platform for news dissemination. However, it also provides conditions for the widespread of fake news. In the current social media environment, fake news exists in multiple modalities such as text and images, while existing multimodal fake news detection techniques usually fail to fully explore the intrinsic connection between different modalities, which limits the overall performance of the detection model. To address this issue, this paper proposes a hybrid model of crossmodal attention mechanism and weaksupervised contrastive learning(CMAWSCL) for fake news detection. The model utilizes pretrained BERT and ViT models to extract text and image features respectively, and effectively fuses multimodal features through the crossmodal attention mechanism. At the same time, the model introduces weaksupervised contrast learning, which utilizes the prediction results of effective modalities as supervisory signals to guide the contrast learning process. This approach can effectively capture and utilize the complementary information between text and image, thus enhancing the performance and robustness of the model in multimodal environments. Simulation experiments show that the CMAWSCL performs well on the publicly available Weibo17 and Weibo21 datasets, with an average improvement of 1.17 percentage points in accuracy and 1.66 percentage points in F1 score compared to the current stateoftheart methods, which verifies its effectiveness and feasibility in coping with the task of multimodal fake news detection.

Key words: fake news detection, multimodal fusion, crossmodal attention mechanism, contrastive learning, deep learning

摘要： 随着互联网和智能设备的广泛普及，社交媒体已成为新闻传播的主要平台.然而这也为虚假新闻的广泛传播提供了条件.在当前的社交媒体环境中，虚假新闻以文本、图片等多种模态存在，而现有的多模态虚假新闻检测技术通常未能充分挖掘不同模态之间的内在联系，限制了检测模型的整体性能.为了解决这一问题，提出了一种基于跨模态注意力机制和弱监督式对比学习的虚假新闻检测模型.该模型利用预训练的BERT和ViT模型分别提取文本和图像特征，通过跨模态注意力机制有效融合多模态特征.同时，该模型引入了弱监督式对比学习，利用有效模态的预测结果作为监督信号指导对比学习过程，能够有效捕捉和利用文本与图像间的互补信息，从而提升了模型在多模态环境下的性能和鲁棒性.仿真实验表明，提出的虚假新闻检测模型在公开的Weibo17和Weibo21数据集上表现出色，与目前最先进的方法相比，准确率平均提升了1.17个百分点，F1分数平均提升了1.66个百分点，验证了其在应对多模态虚假新闻检测任务中的有效性和可行性.

关键词: 虚假新闻检测, 多模态融合, 跨模态注意力机制, 对比学习, 深度学习