Abstract: Trusted execution environment (TEE) is widely used, and its kernel security has become a significant area of focus. Fuzzing, a powerful technique for detecting vulnerabilities in operating system, has increasingly been applied to the security analysis of TEE. However, conventional fuzzing tools cannot be directly used for TEE kernels due to their isolation. Coverageguided fuzzers often discard test cases that trigger new states but cover the same code, which limits their effectiveness in discovering vulnerabilities. To address these challenges, a stateaware fuzzing method tailored for TEE kernels is proposed. Initially, a modeling and tracing approach is developed to represent the program state through statevariable values and retaining the test cases that trigger new states, overcoming the limitations of coverageguided fuzzers. Subsequently, we introduce an innovative communication scheme to tackle issues arising from TEE isolation. New seed retention and selection algorithms are proposed to better guide the fuzzer in exploring vulnerabilities. Finally, the NGram model is employed to enhance test case generation and optimize the framework’s performance. A prototype, named TrustyStatefuzz, has been implemented and evaluated on fuchsia, the selfdeveloped microkernel operating system Nebula, and OPTEE. The evaluation results show that TrustyStatefuzz is effective at detecting both new code and vulnerabilities. TrustyStatefuzz discovers 9 unknown vulnerabilities and 23 known vulnerabilities. Additionally, it achieves 13% higher code coverage and 27% higher state coverage than the stateoftheart fuzzer Syzkaller.

Key words: fuzzing, trusted execution environment, program state, kernel, NGram model

摘要： 可信执行环境(trusted execution environment, TEE)被广泛使用，其内核安全已成为一个重要的关注领域.模糊测试作为识别操作系统内核漏洞的有效方法，已广泛应用于TEE安全研究.然而，传统的模糊测试工具由于TEE的隔离性而不能直接用于TEE内核.覆盖引导的模糊器通常会丢弃触发新状态而覆盖相同代码的测试用例，限制了它们在发现漏洞方面的有效性.针对以上问题，提出了一种状态感知的TEE内核模糊测试方法.首先，设计了一种建模和跟踪方法，通过状态变量的值表示程序状态，保留触发新状态的测试用例，克服了覆盖引导的模糊器的局限性.其次，提出了新的通信方案以解决TEE的隔离性引发的问题.并提出了新的种子保存和选择算法，以更好地引导模糊器探索漏洞.最后，结合NGram模型指导测试用例生成过程，优化测试框架性能.目前已经实现了一个TrustyStatefuzz原型，并在fuchsia、自主开发的微内核操作系统nebula以及OPTEE上进行了模糊测试并评估.结果表明，TrustyStatefuzz在发现新代码和漏洞方面是有效的.它发现了9个未知漏洞和23个已知漏洞，比现有模糊测试工具Syzkaller提升13%的代码覆盖率和27%的状态覆盖率.

关键词: 模糊测试, 可信执行环境, 程序状态, 内核, NGram模型