Abstract
Buffer overflow is one of the most serious and common vulnerability that using memory beyond the allocated memory. It can cause many severe threats, including system crash, arbitrary code execution and data corruption. Buffer overflow vulnerability has two main characteristics, one is that buffer overflow vulnerability may not be discovered after its path has been executed. The other is that not every buffer overflow vulnerability could cause crash. Therefore, existing program analysis tools have more or less deficiencies in detecting buffer overflow vulnerability. This paper presents a BOF-Sanitizer method for buffer overflow vulnerability detection. We proposed a pre-process method that combine vulnerability metric and rank, which can effectively avoid the exploration of unnecessary paths and quickly locate potential bug positions. We present a custom memory model of concolic execution and a boundary detection engine based on a low fat pointer, which not only ensure that the executed path is safe, but also effectively reduces the system performance overhead through the real time interacting with the analysis engine and solver in the process of dynamic detection. We implement a prototype system of BOFSanitizer to detect buffer overflow in different application, the result show that BOFSanitizer accurately discovers 88% buffer overflow target points and the detection rate improve over 50% with compare to simple combination of existing program analysis tools.