Environment model-based system testing of real-time embedded systems

Abstract

Real-time embedded systems (RTES) are part of a vast majority of computing devices available today. They are widely used in critical domains where high system dependability is required. These systems typically work in environments comprising of large numbers of interacting components. The interactions with the environment are often bound by time constraints. Missing such time deadlines, or missing them too often for soft real-time systems, can lead to serious failures resulting in threats to human life or the environment. There is usually a great number and variety of stimuli from the RTES environment with differing patterns of arrival times. Testing all possible sequences of stimuli is not feasible and only a fully automated testing approach can scale up to the testing requirements of industrial RTES. In this thesis, we take a black-box approach for system testing of RTES based on environment models. Our main motivation is to provide a practical approach to the model-based testing (MBT) of RTES. To do so, we enable system testers, who are often not familiar with the system design but are knowledgeable of the application domain, to model the environment using well-supported modeling standards, to enable test automation. Once the environment models are developed they can support the automation of three tasks: the code generation of an environment simulator to enable testing on the development platform, the selection of test cases, and the evaluation of their expected results (oracles).

Given the above objectives, a first contribution of this thesis is a detailed environment modeling methodology that fits the practical needs for industrial adoption of a RTES system testing approach. The methodology is based on modeling standards (1) that are at an adequate level of abstraction, (2) that software engineers are familiar with, and (3) that are well supported by commercial or open source tools. The methodology uses the Unified Modeling Language (UML), the profile for Modeling and Analysis of Real-time Embedded Systems (MARTE), and the Object Constraint Language (OCL). We also provide extensions to UML and introduce a profile for modeling concepts that are specific to our context. The models capture only the details in the environment that are visible and relevant to the SUT, including the nominal behavior and failure behavior of environment components. The environment behavioral models also capture what we call ‘error states’ that should never be reached if the SUT is implemented correctly. The ‘error states’ act as oracles for the test cases. The environment modeling methodology is applied on two industrial case studies. The results show that the modeling notations selected suffice to model the RTES environments for our test automation. The experiences learned by applying UML/MARTE in industrial contexts are also summarized in the form of a framework, which can help practitioners in bridging the gap between the modeling standards and industrial adoption.

A second contribution of this thesis is the definition of transformation rules for environment simulator generation. To convert environment models developed using UML state machines and class diagrams to their simulator code, we extend the well-known state pattern for our specific purpose and also resolve a number of UML semantic variation points. We evaluate the transformation rules by transforming models for five case studies, including two industrial case studies and use these models for testing. Our empirical evaluation based on the case studies shows that the developed rules are sufficient and that they are correct as far as fault detection is concerned. The automated simulator generation is expected to save a significant amount of effort during system testing.

The third contribution of this thesis is an efficient approach to solve constraints on the environment models written using OCL. For this purpose we define a set of heuristics for search algorithms and empirically evaluate their effectiveness on an industrial case study. These heuristics play an important role in test case generation from environment models. Results of the empirical study suggest that even for the most difficult constraints, with research prototypes and no parallel computations, we obtain test data within 2.96 seconds on average. This is a significant improvement compared to an existing OCL solver, which was not able to solve the same constraints even after several hours of execution.

The final contribution of the thesis is test case generation from environment models for black-box system testing of RTES. We conducted a number of experiments to investigate the effectiveness of testing algorithms, specifically, Genetic Algorithms (GA) and (1+1) Evolutionary Algorithm (EA), Adaptive Random Testing (ART), and Random Testing (RT) in our context. The goal of testing in our context is to reach an ‘error state’ of the environment with as few test case executions as possible. For search algorithms we provide and iteratively improve a fitness function for effective testing. The testing strategies are evaluated on an industrial case study and a number of artificial problems. On the industrial case study we were able to automatically find new, critical faults. Based on the results of our experiments, we propose a hybrid strategy, which combines the strengths of (1+1) EA and ART, to improve the overall performance of system testing that is obtained when using each single strategy in isolation. Results show that the hybrid strategy fares better and, unlike individual algorithms, its performance is not drastically affected by the characteristics of the environment models (i.e., low variance in results).