RLDS, WebDataset, and HDF5: Dataset Formats Explained

A perfectly captured, well-labeled dataset is still useless if it arrives in a format your training pipeline can't ingest without weeks of reformatting work. Delivery format is a practical, often underdiscussed part of scoping a data collection engagement — and robotics datasets in particular tend to use a handful of specific formats worth understanding before a project starts.

RLDS (Reinforcement Learning Datasets)

A format built specifically for sequential decision-making data — episodes composed of observations, actions, and rewards in order. It's become a common standard for robotics manipulation datasets, particularly ones intended to be compatible with broader open robotics data initiatives, since it structures data around the episode-and-step format most imitation-learning and reinforcement-learning pipelines expect natively.

WebDataset

A format built around sharded tar archives, designed for efficient streaming access to large datasets during training — particularly useful when a dataset is too large to load into memory at once, or when training is distributed across multiple machines. It prioritizes fast sequential read performance over the more flexible random-access patterns other formats support.

HDF5

A hierarchical, self-describing binary format widely used across scientific computing and machine learning generally, well suited to storing large, structured multimodal data — video, sensor readings, and labels — in a single organized file with fast random access to specific records.

Choosing Between Them

• RLDS — best fit when a pipeline is built specifically around sequential robot demonstration data and expects that structure directly
• WebDataset — best fit for very large-scale training runs prioritizing fast streaming over random access
• HDF5 — a flexible general-purpose choice, particularly useful when a dataset combines many different data types that need to stay linked together

Why This Should Be Decided Before Capture, Not After

Reformatting a large dataset after the fact is time-consuming and occasionally lossy if metadata gets dropped or misaligned in the conversion. Specifying delivery format as part of the initial project brief avoids that rework entirely.

Frequently Asked Questions

Can a vendor deliver the same dataset in multiple formats?

Usually yes, though it's more efficient to specify the target format upfront rather than requesting a conversion after initial delivery.

Is one of these formats objectively better than the others?

No — the right choice depends entirely on your training pipeline's existing tooling and how it expects to read the data.

Where Blue Projects Fits In

Blue Projects delivers datasets in RLDS, WebDataset, HDF5, or a custom schema on request, matched to what a client's training pipeline actually expects rather than a one-size-fits-all default.

Frequently Asked Questions