Neural Network Dependability Kit

A toolbox to support safety engineering of artificial neural networks

In recent years, neural networks have been widely adapted in engineering automated driving systems with examples in perception, decision-making, or even end-to-end scenarios. As these systems are safety-critical in nature, problems during operation such as failed identification of pedestrians may contribute to risk behaviours. Importantly, the root cause of these undesired behaviours can be independent of hardware faults or software programming errors but can solely reside in the data-driven engineering process, e.g., due to unexpected results of function extrapolation between correctly classified training data.

The Neural Network Dependability Kit (NN-dependability-kit) is an open-source toolbox to support safety engineering of neural networks. It supports verification, test-case generation and metrics computation for neural networks. The key functionality includes:

• Formal reasoning engine for ensuring that the generalization does not lead to undesired behaviours, and

• Novel dependability metrics for indicating sufficient elimination of uncertainties in the product life cycle

• Runtime monitoring for reasoning whether a decision of a neural network in operation time is supported by prior similarities in the training data.

The NN-dependability-kit is available for download at GitHub, where further information can be found: https://github.com/dependable-ai/nn-dependability-kit

Examples of NN-dependability-kit use cases:

1. Formal Verification of a Highway Front Car Selection Network

Formally verifying properties of a neural network that selects the target vehicle for an adaptive cruise control (ACC) system to follow. The overall pipeline is illustrated in the figure, where two modules use images of a front facing camera of a vehicle to (i) detect other vehicles as bounding boxes and (ii) identify the ego-lane boundaries. Outputs of these two modules are fed into the third module called target vehicle selection, which is as a neural-network based classifier that reports either the index of the bounding box where the target vehicle is located, or a special class for “no target vehicle”.

The input features of the Target vehicle selection neural network are defined as follows: 1-8 (possibly up to 10) are bounding boxes of detected vehicles, E is an empty input slot, i.e., there are less than ten vehicles, and L stands for the ego-lane information.

A tutorial for this example is available at the GitHub repository: https://github.com/dependable-ai/nn-dependability-kit/blob/master/TargetVehicleProcessingNetwork_FormalVerification.ipynb

2. Perturbation Loss over German Traffic Sign Recognition Network

Analysing a neural network trained under the German Traffic Sign Recognition Benchmark with the goal of classifying various traffic signs. With NN-dependability-kit, one can apply the perturbation loss metric, in order to understand the robustness of the network subject to known perturbations.

A tutorial for this example is available at the GitHub repository: https://github.com/dependable-ai/nn-dependability-kit/blob/master/GTSRB_AdditionalMetrics.ipynb