Egocentric Data Modalities: RGB, Gaze, IMU, and Depth Explained

A single first-person video stream tells a model a lot. A synchronized set of sensor streams tells it far more. Exploring egocentric data modalities requires understanding how rich egocentric datasets combine several data channels at once, each capturing a different physical signal, and how training pipelines align them so a model learns how they relate to each other in time.

The Core Modalities

A comprehensive physical AI dataset includes 5 distinct sensor streams:

• Egocentric RGB Video: The baseline visual stream from a head-, chest-, or smart-glasses-mounted camera (see our introductory guide on what is egocentric data).
• Gaze & Eye-Tracking Data: Binocular eye fixations recording where the wearer looks before, during, and after an action — predicting intent for VLA models.
• 3D Hand & Body Pose Data: Keypoint tracking of joints, fingers, and wrist orientation as objects are manipulated (learn more about bimanual manipulation data).
• IMU & Depth Sensors: 9-axis accelerometer, gyroscope, and LiDAR/depth readings capturing head rotation velocity and spatial distance.
• Egocentric Audio: Spatial binaural sound capturing tool clicks, motor whirs, and environmental acoustics that visual-only feeds miss.

Why Synchronization Matters More Than Any Single Stream

A gaze-tracking stream is far more useful paired with the hand-motion data it precedes than viewed alone. Sensor-fusion alignment — the process of linking these streams accurately in time — is what turns a set of separate recordings into a genuinely rich training dataset. Poorly synchronized multimodal data is often worse than a single clean video stream, because it introduces noise a model can't reliably separate from signal.

How Capture Source Affects Modality Choice

• Human Wearable Rigs: Smart glasses and chest harnesses are ideal for large-scale, real-world multimodal capture across field settings.
• Robot Onboard Sensors: Record equivalent modalities directly from a robot's own camera and depth sensors during teleoperation.
• Simulated Environments: Generate data inside 3D engines, useful for scale but limited by real-world physical gaps.

Where Blue Projects Fits In

Blue Projects captures synchronized egocentric video, motion, and audio data across real-world Indian environments, with sensor-fusion alignment built into delivery so datasets are usable across modalities, not just as isolated video files (see our dedicated egocentric video data collection services).

Frequently Asked Questions on Egocentric Modalities