How can designers evaluate and benchmark sensor fusion systems for autonomous machines?

It is always good to remember or be reminded that “If you cannot measure it, you cannot improve it,” as Lord Kelvin observed. This is certainly true for today’s newest design approaches, such as sensor fusion. For safety-critical and highly regulated situations such as advanced driver assistance systems (ADAS) and automated driving (AD), key performance indicators (KPIs) have become essential benchmarks for optimizing sensor capabilities and meeting regulatory standards. There are several aspects to design approaches that involve simulation, re-simulation, and synthetic data to provide accurate decision-making for these systems.

Key performance indicators for sensor fusion

KPIs help to determine the choice of appropriate sensors for specific ADAS features to ensure that each sensor aligns with the system’s design criteria. Determining the key performance indicators to measure is the first step toward design improvements. For sensor fusion, in addition to the usual KPIs for the sensors themselves, such as resolution, range, and reliability, systems-related KPIs include precision, recall, real-time processing, and more. In fact, other lower-level performance metrics—such as function and feature KPIs evaluate specific subsystems and individual features. In contrast, system KPIs measure how well the entire ADAS or AD system operates to meet design goals.

Tools and frameworks for evaluating fusion systems

Many machine learning environments, including sensor fusion evaluations, start with ground truth (accurate, verified data that serves as a benchmark for comparison) or simulated ground truth data.

A frequently used approach for calculating sensor KPIs records thousands of hours of road test data that is analyzed to draw bounding boxes around objects and estimate distances and speeds to create a ground truth for sensor performance measurements.

Figure 1. Boxing of images to establish ground truth. (Image: Autoware Universe)

One company has developed a platform that automatically computes a synthetic ground truth by fusing LiDAR, camera, and other sensor data, and then automatically calculates the KPIs for the perception system. Detecting true and false positives as well as true and false negatives, the KPIs are summarized in a comprehensive report. Failures are indicated in a timeline display for further detailed examination using an additional design tool.

Other tools for evaluating sensor fusion systems include CARLA (Car Learning to Act), an open-source simulator designed for autonomous driving research and development, and Gazebo.

Simulation environments to benchmark performance

Developed by the Computer Vision Center (CVC) at the Universitat Autònoma de Barcelona, CARLA provides users with realistic urban environments, sensor simulation capabilities, and ease of integration.

One group of researchers developed a benchmarking script to enable users to easily analyze the performance of CARLA in their environment. After configuring to run different/specific scenarios that combine various maps, sensors, and weather conditions, the script reports the average and standard deviation of frames per second (FPS) under the requested criteria.

Another approach is provided in Gazebo, a collection of open source software libraries for robot developers and designers by the Open Source Robotics Foundation. Gazebo-classic, specifically Gazebo 9 and Gazebo 11, publishes a message called /gazebo/performance_metrics / that allows designers to check the performance of each sensor in the system.

Handling edge cases and sensor failure modes in evaluations

For a thorough validation, edge cases or those situations that occur under rare, difficult-to-predict scenarios, such as sudden lane changes or extreme weather, must be part of the design process. One company has collected and consolidated edge cases for designers to test their autonomous vehicle systems. The data consists of fatal accident reports, non-fatal accident reports, and millions of examples of tight, tricky real-world interactions that occur between cars and other road users.

To address sensor faults in CARLA, researchers from one university developed a CARLA Robot Operating System (ROS) bridge (a middleware interface) to facilitate the integration of the CARLA simulation environment. The CARLA ROS bridge core functionalities include a modular and extensible sensor fault injection framework centered around a base class named FaultInjector. The class diagram is shown in Figure 2.

Figure 2. Fault injector class diagram. (Image: Polytechnic University of Coimbra)

Implemented in Python and integrated into the CARLA ROS bridge, the framework is built on ROS 2 (Humble version) and is compatible with the Autoware Universe repository of core functionalities of AD technologies. Each sensor type’s fault (such as LiDAR, IMU, and GNSS (Global Navigation Satellite System)) is handled by a dedicated subclass, such as LidarFaultInjector, IMUFaultInjector, and GNSSFaultInjector, and contributes to the FaultInjector base class.

Finally, to certify its capabilities of achieving SAE J3016 Level 3 autonomous vehicle status, Mercedes-Benz AG used the reliability analysis methods available in Ansys optiSLang. This analysis allowed them to determine the probability of failure for numerous traffic scenarios, including very rare events with a very low probability of failure (10^-9).