Is using images to test web pages the solution to a Sisyphean task?

Abstract

This thesis describes a system, Sysifos, that automates capturing and segmentation of screen dumps of web pages. The system builds a model of the spatial structure of a page based on the segmentation. The model is used when comparing two screen dumps. The system uses image analysis techniques to segment the page. The model is then compared to a model generated earlier. The model comparison is packaged in a form so that it may be used as a test oracle in standard Java testing frameworks, for example Junit or TestnG. The motivation for the development of Sysifos is that there are currently no established ways to automate testing of browser rendering, although browser related bugs are impor- tant. In this thesis one operation running 56 high volume websites was investigated, and it was found that browser related bugs represented around 13% of all bugs needing developer attention. Sysifos was evaluated using a test set containing known errors. It found 100% of the errors it was expected to find, but reported one false positive. Test results are visualized using SVG. This thesis show that using a test oracle may be beneficial when testing browser rendering of web pages. Currently, the image capturing service of Sysifos is not satisfactory according to speed and reliability, but the results of the evaluation indicate that Sysifos has the potential to become a valuable tool if the image capturing service can be improved.