How to implement multi-sensor fusion algorithms for autonomous vehicles

To achieve autonomous vehicle (AV) operation, sensing techniques include radar, LiDAR, and cameras, as well as infrared (IR) and/or ultrasonic sensors, among others. No single sensing technique is adequate by itself, and each of these sensors has its strengths and weaknesses. As shown in Table 1, one or more sensing techniques overcome the weaknesses of another, and their combined capabilities also provide redundancy for safe autonomous vehicle operation. In addition to the sensors themselves, combining or fusing the data from multiple sensors to obtain a more accurate and reliable estimate requires coding (software algorithms), advanced filtering, and performance simulation.

Table 1. Strengths and weaknesses of LiDAR, radar, and camera sensors. (Image: Mindkosh)

Code examples/Tech stacks for integrating sensor data

Different coding/software approaches have been used to fuse or integrate the data from the key sensors. Similar to other widely pursued applications, tech stacks that combine programming languages, frameworks, databases, and tools are used to expedite the development of sensor fusion software for AV applications. In addition to C/C++, two popular approaches for developing sensor fusion software include Autoware and the Robot Operating System (ROS).

Since it released its first modular open-source software stacks for autonomous driving in 2015, the Autoware Foundation has periodically offered improved versions with the most recent architecture called Autoware Core/Universe.

ROS is not an operating system but a software development kit (SDK) – a set of software libraries and tools that help engineers build robot applications. This open-source robotics framework provides developers with the middleware required for communicating between hardware components and software algorithms.

In addition to these design approaches, Python, a multi-paradigm (object-oriented, procedural (imperative), functional, structured, and reflective) general-purpose programming language, is frequently used for sensor fusion of inertial measurement unit (IMU) outputs (accelerometer, gyroscope, and magnetic sensing signals).

Filtering techniques for combining sensor data

Effectively using the data from different sensors, especially when the output from one sensor conflicts with the output from another, requires specialized filtering of the data. Since three-dimensional readings must be evaluated, Kalman (Gaussian-based) filters provide a common starting point. For non-linear systems, the extended Kalman filter (EKF) uses linearization to model nonlinear functions, making it applicable for fusing data from LiDAR, radar, and camera sensors. A variation of non-linear Kalman filters includes the unscented Kalman filter (UKF) that provides an improvement over the EKF, using a deterministic sampling approach to address the EKF’s flaws in nonlinear estimations. Figure 1 shows how it works for LiDAR and radar data measurements.

Figure 1. Flow Chart with a predict-update cycle for UKF Implementation. (Image: Udacity SFND Course)

In contrast to Kalman filtering, particle filters combine predictions from a system dynamics model with new observations to update the estimated state of a system. The Bayesian filtering technique uses a set of weighted random samples, called “particles,” where the weight of each particle is updated based on how well it aligns with the readings from all sensors.

Simulating sensor fusion performance

Using simulation tools like MATLAB or Simulink, engineers can evaluate different approaches to design and evaluate their system.

Figure 2. Different options (including simultaneous localization and mapping (SLAM)) bring sensor data to perception algorithms. (Image: MathWorks)

To ease into the evaluation process, several simulation tools suggest starting with just two variables (see Figure 1). For example, the MathWorks multi-object tracker to fuse information from radar and video camera sensors uses Kalman filters to estimate the state of motion of a detected object. Another tracker generates an object-level track list from measurements of a radar and a LiDAR sensor and fuses them using a track-level fusion scheme.

Baidu, Inc., a Chinese multinational technology company, has an open platform called Apollo to help developers with simulations, an autonomous perception system, and high definition (HD) mapping data to minimize costs and improve the precision of their designs.

For all the design activities required for sensor fusion in AV applications, engineers have choices, including those mentioned above, as well as options from other providers.