Abstract
When testing applications that run on diverse client devices, operating systems, and browsers, it is necessary to check whether application screens display correctly in various environments and ensure that there is no presentation failure. Applications are typically composed of many screens, and there are many types and combinations of environments in which they may run. Accordingly, it is very labor intensive to visually confirm large numbers of such screens, and there is a strong possibility of missing presentation failures. Finding a way of efficiently confirming large numbers of application screen images is hence a challenge facing real-world testing and industry. Here, we propose a method to support discovery of presentation failures. It detects differences in screen elements, e.g., a disappearance of or change in the position of a button, by comparing the images of the correct screen and the target screen by utilizing computer vision techniques and displays those differences to the tester in an easy-to-understand way. As a result, the tester can quickly confirm the differences and find presentation failures efficiently. Our method uses only images of the application screen, therefore it does not depend on a specific implementation technology. Thus, it can be utilized for confirming the test results of various implementations (Web, android, iOS, etc.). In experiments with application screens in which presentation failures were intentionally embedded, we measured how much the proposed method increases failure discovery rate and determined whether it reduces the amount of labor.