Sensor Fusion: Combining Evidence, Not Just Sensors

Sensor fusion turns separate measurements into one time-consistent estimate of the road scene. The hard part is not collecting more data; it is managing uncertainty, disagreement and failure without hiding them.

What fusion actually does

A camera records patterns of light. Radar measures radio reflections and radial velocity. Lidar samples optical range, while ultrasonic sensors cover the near field. Positioning sensors estimate where the vehicle itself is. Each modality observes a different projection of reality, with different blind spots.

Fusion software associates those observations with the same physical objects and estimates properties such as position, velocity, class and confidence. It may also combine sensor data with maps and vehicle-motion measurements. This is the bridge between the individual technologies described in cameras, radar, lidar, ultrasonic sensors and the perception stack used by driver-assistance and automated-driving systems.

Fusion happens at several levels

Raw or early fusion combines data close to the measurement level. It can retain detail but demands precise calibration, timing and large data flows.
Feature fusion combines extracted structures such as image edges, radar returns or lidar clusters.
Object or track fusion combines independently detected objects and their histories. It is easier to modularize, but information discarded by an earlier detector cannot be recovered.
Decision fusion combines outputs from substantially independent subsystems. This can support fault tolerance when the independence is real and the arbitration logic is defined.

Production systems can use several of these approaches at once. There is no universally superior fusion layer.

Calibration and time are part of perception

Measurements cannot be combined correctly unless the system knows where each sensor is, where it points and when each sample was taken. A small angular error can create a large position error at distance. Unsynchronized sensors can make a moving object appear in two places.

Fusion therefore depends on geometric calibration, clock synchronization, ego-motion compensation and monitoring of sensor alignment. Repairs, vibration, temperature changes, contamination and software changes can all affect the result. These dependencies are covered in calibration and sensor health.

Complementarity is not redundancy

Two sensors are complementary when one contributes information the other lacks. A camera may classify a traffic signal while radar supplies stable relative speed to a vehicle ahead. That can improve perception, but it does not automatically create a redundant safety channel.

Redundancy requires the system to continue meeting a defined requirement after a relevant failure. The channels must be sufficiently independent, their common-cause failures must be understood, and the system must detect the failure and enter a defined degraded or fallback state. Mobileye: True Redundancy describes one manufacturer's deliberate separation of a camera subsystem from a radar-and-lidar subsystem. Waymo: sixth-generation Waymo Driver describes a different architecture with overlapping camera, radar and lidar coverage.

Sensor count is therefore a poor proxy for redundancy. Ten cameras can share the same obscured windshield, power supply, clock or perception model. A camera and lidar can also share a dirty cover or a flawed object-tracking assumption.

Disagreement must remain visible

A weak fusion design can become overconfident because several correlated inputs appear to agree. It can also discard a correct minority observation because another sensor normally dominates. Safer designs preserve uncertainty and ask:

Are the observations statistically independent enough to combine?
Is one sensor outside its qualified operating conditions?
Does the map conflict with live perception?
Is the object association ambiguous?
Can the system still satisfy its function if one channel is removed?

The correct output is sometimes “unknown.” The driving policy must then increase margins, reduce speed, request the driver, or perform a minimum-risk maneuver according to the automation level and operational design domain.

Fusion does not repair missing evidence

Adding sensors cannot compensate for an undefined safety requirement or an unrepresentative test program. A fused stack still has to be validated for the complete chain from detection through prediction, planning and actuation. ISO 21448 addresses hazards caused by functional insufficiencies and foreseeable misuse even when no component has failed; it is directly relevant to complex perception systems. See ISO 21448:2022 — Safety of the intended functionality and the broader discussion in Safety and evidence for automated driving.

NIST research on evaluating fused colored point clouds illustrates why system output needs its own quality measures rather than inheriting the headline specifications of its component sensors. See NIST: quality evaluation of automotive sensor fusion.

The engineering question

The useful question is not “How many sensors does the car have?” It is:

What must the function perceive within its operational design domain, how is uncertainty represented, which failures can it detect, and what does it do next?

That question applies equally to camera-only and multimodal designs. sensor architecture choices examines those architecture choices without treating either label as a safety verdict.