Abstract: At present, analysis on the homology of malware families mostly focuses on the closedset problem, that is, it is assumed that the samples to be tested must belong to a certain known class.However, there are many malware families in an open world, and the unknown classes usually account for the majority. The closedset recognition cannot accurately identify the malware families in an open world. Aiming at the above problems, this paper proposes a homology analysis method for malware families based on openset recognition. The malware executable files are converted into grayscale images through NGram sliding window and Doc2vec sentence embedding method, the features of the grayscale images are obtained based on the convolutional neural network model MobileNet, and the Open Longtailed Recognition model is used to realize openset recognition of malware families. Identifying 9 known classes and 9 unknown classes of malware families, the experimental results show that the proposed method can identify the malware family of the unknown classes while maintaining high accuracy on both known and unknown families.