The ML Data Pipeline at Work

Stage 1 — Field Data Capture

The raw pipeline begins with field-based recording of native workers in their natural workspaces, focusing on the operator's egocentric point-of-view.

• Egocentric POV: Chest/head-mounted cameras (GoPro Hero 12, DJI Osmo) record the visual field and hand orientation of the worker.
• Overhead Workspace View: Synchronized static overhead cameras capture broad trajectories, spatial dynamics, and auxiliary equipment parameters.
• 3D Spatial Reconstruction: Multi-camera array placements align relative coordinates across complex bimanual workspace environments.

"The footage captures what a human sees and does — not a third-person observation."

Stage 2 — Video Preprocessing

Raw video streams undergo automated filtering and cleaning loops to protect participant identity and parse out unusable video frames.

• Frame Extraction: Raw videos are sliced into individual high-definition frame sequences (typically 1 to 30 frames per second).
• Temporal Synchronization: Dual and multi-camera perspectives are synchronized with sub-millisecond precision.
• Anonymization Filters: Automated face-blurring and license plate masking tools ensure strict PII compliance.
• Quality Audits: Frames containing extreme motion blur, lighting occlusion, or focus slips are flagged and auto-rejected.

Stage 3 — Data Annotation

Trained labelers annotate geometric, keypoint, and behavioral landmarks directly onto the parsed frame sequences.

Software employed: CVAT, Label Studio, Scale AI, Labelbox, Roboflow.

Stage 4 — Feature Extraction

Individual video frames are translated into highly structured, low-dimensional coordinate representations that deep learning models can ingest.

• Anatomical Extraction: Feeds frames to OpenPose/MediaPipe to map 21 wrist/finger joint coordinate matrices (X, Y, Z).
• Bounding Class Map: YOLO/Detectron2 networks resolve object locations and class probability arrays.
• Scene Embeddings: DINOv2 encoders extract global lighting, spatial context, and tool properties into feature vectors.

"The output is no longer a video — it's a numerical representation of what happened in that frame."

Stage 5 — Dataset Construction

Extracted features are curated, balanced, and prepared for neural net optimization cycles.

• Train / Val / Test Split: Standardized division (70% training, 15% validation, 15% testing splits).
• Demographic Balancing: Datasets balanced across participant hand sizes, lighting parameters, and factory shifts.
• Geometric Augmentation: Frames are rotated (+/- 15deg), flipped, and contrast-shifted to expand dataset scale.
• Export Ingestion: Formatted into COCO JSON, YOLO .txt, Pascal VOC, or PyTorch TFRecords.

Stage 6 — Model Fine-Tuning

General baseline AI frameworks are optimized with localized human data to resolve action trajectories and hand pose profiles.

• Behavior Cloning: Direct supervision where models learn mapping between camera frames and corresponding hand actions.
• Imitation Learning: Tracing spatial joint movements to copy trajectories: "when gripping a cylindrical object, apply X force at Y angle."
• Egocentric AI Vision: Equips wearable assistant models to recognize what is in a user's view and anticipate next-step tasks.

Stage 7 — Deployment & Feedback Loop

Models are deployed directly onto edge computers controlling physical robotics hardware or wearable software stacks.

• Edge Inference: Real-time processing (resolves visual feed ➔ joint coordinates ➔ robotic actuation).
• Out-of-Distribution Triggers: Active learning checks flags anomalies or failure cases (e.g. slips, mistargets).
• Recycling Loop: Failed cases are tagged and routed back to Stage 1 to launch fresh field capture campaigns.