Malicious Traffic Detection Method Based on Bayesian Optimization and Spatiotemporal Attention

doi:10.12379/j.issn.2096-1057.2026.08.04

Abstract

Abstract: To address the limitations inherent in current malicious network traffic detection methods, specifically suboptimal detection accuracy and reliance on manual hyperparameter tuning, this study proposes a malicious traffic detection method based on Bayesian optimization and a spatiotemporal attention mechanism. It adopts bidirectional long short-term memory (BiLSTM) to extract temporal and spatial features of network traffic data, and leverages Bayesian optimization to search for optimal network hyperparameters.A bidirectional architecture extracts temporal and spatial features from network traffic data, while Bayesian optimization automatically searches for optimal network parameters. Innovatively, we introduce a Multi-scale Traffic Protocol Enhancement Module that significantly enhances spatiotemporal feature representation for complex attack chains through multi-level protocol parsing, dynamic feature fusion, and attention coordination. Concurrently, to establish deep correlations between spatiotemporal features, we propose a Multi-head Spatiotemporal Attention Mechanism. This employs four parallel attention heads to cooperatively model millisecond-level protocol transients, second-level interaction sequences, minute-scale attack chain evolution, and cross-protocol correlation characteristics, effectively improving detection accuracy for stealthy threats. Experimental validation on the CIC-IDS2017 dataset demonstrates that the proposed method achieves multiclass and binary classification accuracies of 99.46% and 99.59%, respectively, outperforming comparative approaches.

Key words: malicious traffic detection; Bayesian optimization; attention mechanism; recurrent neural network; data features

摘要： 针对当前恶意网络流量检测技术准确率不足、超参数依赖手动调整的问题，本文提出一种基于贝叶斯优化与时空注意力的双向循环网络恶意流量检测方法，通过双向长短时记忆网络提取网络流量数据的时间特征和空间特征，使用贝叶斯优化搜索最优的网络参数。本文创新性地提出多尺度流量协议增强模块，通过多级协议解析、动态特征融合与注意力协同机制，显著增强了对复杂攻击链的时空特征表征能力。同时，为建立时空特征间的深度关联，本文提出多头时空注意力机制，通过4组并行注意力头协同建模毫秒级协议瞬变、秒级交互时序、分钟级攻击链演进及跨协议关联特性，有效提升对隐蔽威胁的识别精度。采用CIC-IDS2017数据集进行实验，结果表明，本文方法在多分类和二分类的准确率分别达到99.46%和99.59%，性能均优于其他方法。

关键词: 恶意流量检测；贝叶斯优化；注意力机制；循环神经网络；数据特征

CLC Number:

TP393

References

[1] Fu C, Li Q, Shen M, et al. Frequency domain feature based robust malicious traffic detection[J]. IEEE/ACM Trans on Networking, 2023, 31(1): 452-467. [2] Hussain F, Abbas S G, Shah G A, et al. A framework for malicious traffic detection in IoT healthcare environment[J]. Sensors, 2021, 21(9): 3025. [3] Shafiq M, Tian Z, Bashir A K, et al. CorrAUC: A malicious bot-IoT traffic detection method in IoT network using machine-learning techniques[J]. IEEE Internet Things J, 2021, 8(5): 3242-3254. [4] Guo G, Wang H, Bell D, et al. KNN model-based approach in classification[C]//Proc of the OTM Confederated Int Conf on Cooperative Information Systems, Distributed Objects and Applications, and Ontologies Databases and Applications of Semantics. Berlin: Springer, 2003: 986-996. [5] Dongare A, Kharde R, Kachare A D, et al. Introduction to artificial neural network[J]. Int J Engineering and Innovative Technology, 2012, 2(1): 189-194. [6] Chandra M A, Bedi S S. Survey on SVM and their application in image classification[J]. Int J Information Technology, 2021, 13(5): 1-11. [7] Wu D, Chen X, Chen C, et al. On addressing the imbalance problem: A correlated KNN approach for network traffic classification[C]//Proc of the 8th Int Conf on Network and System Security. Cham: Springer, 2014: 138-151. [8] Jain M, Kaur G, Saxena V. A k-means clustering and SVM based hybrid concept drift detection technique for network anomaly detection[J]. Expert Systems with Applications, 2022, 193: 116510. [9] 陈财. 基于随机森林的恶意HTTP外连流量检测[D]. 武汉: 华中科技大学, 2020. [10] Dwivedi D, Bhushan A, Kumar Singh A, et al. Detection of malicious network traffic attacks using support vector machine[C]// Proc of the Int Conf on Advanced Network Technologies and Intelligent Computing. Cham: Springer Nature Switzerland, 2023: 54-68. [11] Kattenborn T, Leitloff J, Schiefer F, et al. Review on convolutional neural networks (CNN) in vegetation remote sensing[J]. ISPRS J Photogrammetry and Remote Sensing, 2021, 173: 24-49. [12] Yao Y, Wei Y, Gao F X, et al. Anomaly intrusion detection approach using hybrid MLP/CNN neural network[C]// Proc of the 6th Int Conf on Intelligent Systems Design and Applications. Jinan, China: IEEE, 2006: 1095-1102. [13] Darmawan M A, Aradea, Rahmatulloh A, et al. Deep Learning for Malware Traffic Analysis: A comprehensive review on model optimization and performance enhancement[C]// Proc of the 11th Int Conf on Wireless and Telematics. Lampung, Indonesia: IEEE, 2025: 1-6. [14] Nikitenko A, Bashkov Y. Construction of a network intrusion detection system based on a convolutional neural network and a bidirectional gated recurrent unit with attention mechanism[J]. Eastern-European Journal of Enterprise Technologies, 2024, 129(9): 6-15. [15] 李佳，云晓春，李书豪，等. 基于混合结构深度神经网络的HTTP恶意流量检测方法[J]. 通信学报，2019，40（1）：24-33. [16] 翟明芳，张兴明，赵博. 基于深度学习的加密恶意流量检测研究[J]. 网络与信息安全学报，2020，6（3）：66-77. [17] 陈子涵，程光，徐子恒，等. 互联网加密流量检测、分类与识别研究综述[J]. 计算机学报，2023，46（5）：1060-1085. [18] Hochreiter S, Bengio Y, Frasconi P, et al. Gradient flow in recurrent nets: The difficulty of learning long?term dependencies[M]. NJ: IEEE Press, 2001: 237–243. [19] Greff K, Srivastava R K, Koutník J, et al. LSTM: A search space odyssey[J]. IEEE Trans on Neural Networks and Learning Systems, 2017, 28（10）: 2222-2232. [20] Zhang J, Zhang X, Liu Z, et al. A network intrusion detection model based on BiLSTM with multi-head attention mechanism[J]. Electronics, 2023, 12(19): 4170. [21] Frazier P I. A tutorial on Bayesian optimization[EB/OL]. 2018[2026-03-18]. https://arxiv.org/abs/1807.02811.