关键词: 流量异常检测, 数据预处理, 数据约减, 层次聚类, 机器学习方法

Abstract: With the advent of the big data era, the attacks in network traffic are rising dramatically. Detecting malicious traffic through abnormal flow detection is vital. Nowadays, the equipment of abnormal flow detection used in industry mainly adopts statistical analysis method or simple machine learning method. However, the amount of flow data and redundant data is large. The precision rate is low and the false alarm rate is high. In order to solve these problems, this paper presents a new method to detect flow anomalies based on hierarchical clustering in data processing. This method first uses the hierarchical clustering algorithm to achieve the purpose of data reduction. Then based on seven different machine learning algorithms, an abnormal traffic model based on hierarchical clustering is constructed. The experimental results show that this method can detect the abnormal behavior on the DARPA dataset with a precision rate of 99% and a recall rate of 99%. At the same time, while maintaining the precision rate of 90%, the data reduction can be up to 47.58%, which greatly improves the detection efficiency.

Key words: flow anomaly detection, data preprocessing, data reduction, hierarchical clustering, machine learning methods