Testing Embedded Systems - 2 | Embedded Systems (Web) - Computer Science Engineering (CSE) PDF Download

Non-concurrent testing

This form of testing is either event-triggered (sporadic) or time-triggered (periodic), and is characterized by low space and time redundancy. Event-triggered testing is initiated by key events or state changes in the life of a system, such as start-up or shutdown, and its goal is to detect permanent faults. It is usually advisable to detect and repair permanent faults as soon as possible. Event-triggered tests resemble manufacturing tests. Time-triggered testing is activated at predetermined times in the operation of the system. It is often done periodically to detect permanent faults using the same types of tests applied by event triggered testing. This approach is especially useful in systems that run for extended periods, where no significant events occur that can trigger testing. Periodic testing is also essential for detecting intermittent faults. Periodic testing can identify latent design or manufacturing flaws that only appear under the right environmental conditions. 

Concurrent testing 

Non-concurrent testing [23] cannot detect transient or intermittent faults whose effects disappear quickly. Concurrent testing, on the other hand, continuously checks for errors due to such faults. However, concurrent testing is not by itself particularly useful for diagnosing the source of errors, so it is often combined with diagnostic software. It may also be combined with nonconcurrent testing to detect or diagnose complex faults of all types.

A common method of providing hardware support for concurrent testing, especially for detecting control errors, is a watchdog timer. This is a counter that must be reset by the system on a repetitive basis to indicate that the system is functioning properly. A watchdog timer is based on the assumption that the system is fault-free—or at least alive—if it is able to perform the simple task of resetting the timer at appropriate intervals, which implies that control flow is correctly traversing timer reset points. 

For critical or highly available systems, it is essential to have a comprehensive approach to on-line testing that covers all expected permanent, intermittent, and transient faults. In recent years, built-in-self-test (BIST) has emerged as an important method for testing manufacturing faults, and it is increasingly promoted for on-line testing as well.

Test Pattern Generation 

Test Plan 

Test plans are generated to verify the device specification, which comprise of the decision on test type, fault coverage, test time etc. For example, the test pattern generator and response analyzer may reside on an automatic test equipment (ATE) or on-chip, depending on the test environment. In the case of production testing in an industry, ATE may be the option, while on-site testing may require on-chip testers (BIST).

Test Programming 

The test program comprises modules for the generation of the test vectors and the corresponding expected responses from a circuit with normal behavior. CAD tools are used to automate the generation of optimized test vectors for the purpose [1,24]. Figure. 38.3 illustrates the basic steps in the development of a test program.  

Fig. 38.3 Test program generation 

Test Pattern Generation 

Test pattern generation is the process of generating a (minimal) set of input patterns to stimulate the inputs of a circuit, such that detectable faults can be sensitized and their effects can be propagated to the output.  The process can be done in two phases: (1) derivation of a test, and (2) application of a test. For (1), appropriate models for the circuit (gate or transistor level) and faults are to be decided. Construction of the test is to be accomplished in a manner such that the output signal from a faulty circuit is different from that of a good circuit. This can be computationally very expensive, but the task is to be performed offline and only once at the end of the design stage.

The generation of a test set can be obtained either by algorithmic methods (with or without heuristics), or by pseudo-random methods. On the other hand, for (2), a test is subsequently applied many times to each integrated circuit and thus must be efficient both in space (storage requirements for the patterns) and in time. The main considerations in evaluating a test set are: (i) the time to construct a minimal test set; (ii) the size of the test set; (iii) the time involved to carry out the test; and (iv) the equipment required (if external). Most algorithmic test pattern generators are based on the concept of sensitized paths. 

The Sensitized Path Method is a heuristic approach to generating tests for general combinational logic networks. The circuit is assumed to have only a single fault in it. The sensitized path method consists of two parts:  

1. The creation of a SENSITIZED PATH from the fault to the primary output. This involves assigning logic values to the gate inputs in the path from the fault site to a primary output, such that the fault effect is propagated to the output.
2. The JUSTIFICATION operation, where the assignments made to gate inputs on the sensitized path is traced back to the primary inputs. This may require several backtracks and iterations. In the case of sequential circuits the same logic is applied but before that the sequential elements are explicitly driven to a required state using scan based design-for-test (DFT) circuitry [1,24].

The best-known algorithms are the D-algorithm, PODEM and FAN [1,24]. Three steps can be identified in most automatic test pattern generation (ATPG) programs: (a) listing the signals on the inputs of a gate controlling the line on which a fault should be detected; (b) determining the primary input conditions necessary to obtain these signals (back propagation) and sensitizing the path to the primary outputs such that the signals and faults can be observed; (c) repeating this procedure until all detectable faults in a given fault set have been covered.

ATPG for Hardware-Software Covalidation

Several automatic test generation (ATG) approaches have been developed which vary in the class of search algorithm used, the fault model assumed, the search space technique used, and the design abstraction level used. In order to perform test generation for the entire system, both hardware and software component behaviors must be described in a uniform manner. Although many behavioral formats are possible, ATG approaches have focused on CDFG and FSM behavioral models. 

Two classes of search algorithms have been explored, fault directed and coverage directed. Fault directed techniques successively target a specific fault and construct a test sequence to detect that fault. Each new test sequence is merged with the current test sequence (typically through concatenation) and the resulting fault coverage is evaluated to determine if test generation is complete. Fault directed algorithms have the advantage that they are complete in the sense that a test sequence will be found for a fault if a test sequence exists, assuming that sufficient CPU time is allowed. For test generation, each CDFG path can be associated with a set of constraints which must be satisfied to traverse the path.

Because the operations found in a hardware-software description can be either boolean or arithmetic, the solution method chosen must be able to handle both types of operations. Constraint logic programming (CLP) techniques [27] are capable to handle a broad range of constraints including non-linear constraints on both boolean and arithmetic variables. State machine testing has been accomplished by defining a transition tour which is a path which traverses each state machine transition at least once 26ransition tours have been generated by iteratively improving an existing partial tour by concatenating on to it the shortest path to an uncovered transition [26 A significant limitation to state machine test generation techniques is the time complexity of the state enumeration process performed during test generation. 

Coverage directed algorithms seek to improve coverage without targeting any specific fault. These algorithms heuristically modify an existing test set to improve total coverage, and then evaluate the fault coverage produced by the modified test set. If the modified test set corresponds to an improvement in fault coverage then the modification is accepted. Otherwise the modification is either rejected or another heuristic is used to determine the acceptability of the modification. The modification method is typically either random or directed random. An example of such a technique is presented in [25] which uses a genetic algorithm to successively improve the population of test sequences. 

Embedded Software Testing 

Software Unit Testing

The unit module is either an isolated function or a class. This is done by the development team, typically the developer and is done usually in the peer review mode. Test data /test cases are developed based on the specification of the module. The test case consists of either: 

• Data-intensive testing: applying a large range of data variation for function parameter values, or
• Scenario-based testing: exercising different method invocation sequences to perform all possible use cases as found in the requirements. 

Points of Observation are returned value parameters, object property assessments, and source code coverage.  Since it is not easy to track down trivial errors in a complex embedded system, every effort should be made to locate and remove them at the unit-test level. 

Software Integration Testing  

All the unit modules are integrated together. Now the module to be tested is a set of functions or a cluster of classes. The essence of integration testing is the validation of the interface. The same type of Points of Control applies as for unit testing (data-intensive main function call or methodinvocation sequences), while Points of Observation focus on interactions between lower-level models using information flow diagrams. 

First, performance tests can be run that should provide a good indication about the validity of the architecture. As for functional testing, the earlier is the better. Each forthcoming step will then include performance testing. White-box testing is also the method used during that step. Therefore software integration testing is the responsibility of the developer. 

Software Validation Testing 

This can be considered one of the activities that occur toward the end of each software integration. Partial use-case instances, which also called partial scenarios, begin to drive the test implementation. The test implementation is less aware of and influenced by the implementation details of the module. Points of Observation include resource usage evaluation since the module is a significant part of the overall system. This is considered as white-box testing. Therefore, software validation testing is also the responsibility of the developer. 

System Unit Testing

Now the module to be tested is a full system that consists of user code as tested during software validation testing plus all real-time operating system (RTOS) and platform-related pieces such as tasking mechanisms, communications, interrupts, and so on. The Point of Control protocol is no longer a call to a function or a method invocation, but rather a message sent/received using the RTOS message queues, for example. Test scripts usually bring the module under test into the desired initial state; then generate ordered sequences of samples of messages; and validate messages received by comparing (1) message content against expected messages and (2) date of reception against timing constraints. The test script is distributed and deployed over the various virtual testers. System resources are monitored to assess the system's ability to sustain embedded system execution. For this aspect, grey-box testing is the preferred testing method. In most cases, only a knowledge of the interface to the module is required to implement and execute appropriate tests. Depending on the organization, system unit testing is either the responsibility of the developer or of a dedicated system integration team. 

System Integration Testing 

The module to be tested starts from a set of components within a single node and eventually encompasses all system nodes up to a set of distributed nodes. The Points of Control and Observations (PCOs) are a mix of RTOS and network-related communication protocols, such as RTOS events and network messages. In addition to a component, a Virtual Tester can also play the role of a node. As for software integration, the focus is on validating the various interfaces. Grey-box testing is the preferred testing method. System integration testing is typically the responsibility of the system integration team. 

System Validation Testing

The module to be tested is now a complete implementation subsystem or the complete embedded system. The objectives of this final aspect are several: 

• Meet external-actor functional requirements. Note that an external-actor might either be a device in a telecom network (say if our embedded system is an Internet Router), or a person (if the system is a consumer device), or both (an Internet Router that can be administered by an end user).
• Perform final non-functional testing such as load and robustness testing. Virtual testers can be duplicated to simulate load, and be programmed to generate failures in the system.
• Ensure interoperability with other connected equipment. Check conformance to applicable interconnection standards. Going into details for these objectives is not in the scope of this article. Black-box testing is the preferred method: The tester typically concentrates on both frequently used and potentially risky or dangerous use-case instances. 

Interaction Testing Technique between Hardware and Software in Embedded Systems 

In embedded system where hardware and software are combined, unexpected situation can occur owing to the interaction faults between hardware and software. As the functions of embedded system get more complicated, it gets more difficult to detect faults that cause such troubles. Hence, Faults Injection Technique is strongly recommended in a way it observes system behaviors by injecting faults into target system so as to detect interaction faults between hardware and software in embedded system. 

The test data selection technique discussed in [21] first simulates behaviors of embedded system to software program from requirement specification. Then hardware faults, after being converted to software faults, are injected into the simulated program. And finally, effective test data are selected to detect faults caused by the interactions between hardware and software. 

Conclusion 

Rapid advances in test development techniques are needed to reduce the test cost of million-gate SOC devices. In this chapter a number of state-of-the-art techniques are discussed for testing of embedded systems. Modular test techniques for digital, mixed-signal, and hierarchical SOCs must develop further to keep pace with design complexity and integration density. The test data bandwidth needs for analog cores are significantly different than that for digital cores, therefore unified top-level testing of mixed-signal SOCs remains major challenge. This chapter also described granular based embedded software testing technique.