What is multimodal sensing in physical AI?

Multimodal sensing in physical AI (PAI), sometimes called embodied AI, is the ability for AI to fuse diverse sensory inputs, like vision, audio, touch, lidar, text, and more, from its environment to build a richer and more complete situation awareness, enabling complex physical interaction, perception, and autonomous action in the real world.

A key application of multimodal sensing for PAI is spatial AI (SAI) that enables machines like autonomous robots to understand and navigate dynamic 3D environments in real time. That contrasts with conventional image recognition and classification, which are limited to 2D applications.

Multimodal AI enables robots or autonomous vehicles to naturally interact with their environment by simultaneously interpreting complex signals like visual information and spoken commands. The ability to adjust to changing conditions in real time improves flexibility, reliability, and safety.

The goal of SAI is to closely mimic human perception and understanding, paving the way for more intuitive and natural human-machine interactions. The multimodal AI architecture needed to support SAI typically consists of three functional blocks, including the input module, fusion module, and output module (Figure 1).

The use of multimodal sensing enables the system to implement complex tasks that are not possible using a single sensor type. It begins with the input module that typically includes a series of unimodal neural networks, one for each sensor. For example, LIDAR and cameras can provide complementary data.

AI techniques for LIDAR focus on processing unstructured 3D point clouds for precise spatial, geometric, and depth understanding. AI for camera inputs generally uses 2D convolutional neural networks (CNNs) to extract texture, color, and semantic information from dense, pixel-based images, focusing on pixel-wise classification.

Sensor fusion can be complex

Getting the unimodal sensor data is followed by fusing those inputs into a single model consisting of multiple modalities. It’s not as simple as merging all the various inputs; the key is fusing only the relevant inputs from the various modalities and combining them in an optimal manner.

That enables leveraging the strength of each modality and maximizing the results of the fusion process. It can involve simple tools like concatenation or advanced techniques like transformer AI models. Different fusion techniques are appropriate for specific PAI applications:

• Situation awareness often requires techniques like Bayesian networks to manage uncertainty, while deep learning filters handle object recognition from a combination of LIDAR, radar, and cameras.
• Navigation applications are more likely to use Kalman filters for fusing inertial measurement units (IMUs) and wheel encoders for precise localization and mapping, and support operation in complex environments.
• Robotic grippers can take advantage of radial basis function (RBF) neural networks that provide fast, accurate, and robust, non-linear function approximation to integrate data from multiple sensors like force/torque sensors (strain gauges), inductive/photoelectric sensors for item detection, and tactile sensors for surface texture and slip detection.

Once the various sensor inputs have been fused in a manner suited for the specific application, the output module produces the final prediction in a form that is suited for the task.  That can include controlling the speed and direction of movement, the amount of force applied, the brightness or frequency of laser pulses, and other physical parameters.

Diving deeper

The implementation of PAI systems employing sensor fusion can be challenging and involve sensor tradeoffs, multiple controllers, and power converters (Figure 2).

Several of the deeper considerations include:

• Fusion can occur at the data level or the decision level of the system. Data-level fusion combines raw data from the various sensors, resulting in detailed inputs, while decision-level fusion merges filtered data from the individual sensors and can be more robust, but less detailed.
• Balancing complementary and redundant sensing. The use of complementary sensors like radar, LIDAR, and cameras can produce richer outputs, while the use of redundant sensors supports higher system reliability.
• Quieting environmental noise is often an important consideration. Actual sensor data can be affected by environmental conditions, and separate algorithms are required to filter the noise, enabling the system to identify the actual signal. That can also require that the data of each sensor be weighed to account for uncertainties introduced by the environment.
• Precise alignment using temporal and spatial calibration can be needed to guarantee that data from different sensors, often with different latencies and at different locations on the PAI platform, accurately picture the same point in time and space.

Summary

Multimodal sensing can be used to support the human-like understanding needed for PAI operation in autonomous vehicles, robotics, and other real-world applications. It also supports SAI that provides situational awareness. The implementation of multimodal sensing in PAI systems involves tradeoffs in sensor performance and processing, and the use of multiple controllers and power converters.

References

Application of multi-sensor fusion localization algorithm based on recurrent neural networks, Nature scientific reports
Biomimetic multimodal tactile sensing enables human-like robotic perception, nature sensors
Multimodal AI Explained: Major Applications across 5 Different Industries, Daffodil
Multimodal AI in Robotics: Simplifying Automation Complexity, Akira
Multimodal Physical Interaction, Emergent Mind
Multimodal tactile sensing fused with vision for dexterous robotic housekeeping, nature communications
The Role of Multimodal AI in Autonomous Vehicles, Sapien
Top 10 Multimodal Use Cases, Encord
Transforming the Future of AI and Robotics with Multimodal LLMs, Arm
What is multimodal AI?, IBM
What is the role of multimodal AI in autonomous vehicles?, Milvus
Why Multimodal AI Is the Next Big Leap, Openstf

Related EE World content

What are the mathematics that enable sensor fusion?
How can AI/ML improve sensor fusion performance?
Sensor fusion levels and architectures
How does multi-sensor calibration work in autonomous vehicles?
How to implement multi-sensor fusion algorithms for autonomous vehicles