Imagine teaching a robot to pick up a screwdriver, insert it into a tight gap, and turn it — all without letting the robot anywhere near the data you trained it on. Sounds impossible, right? That’s exactly what HiFi-UMI claims to do, and the numbers back it up.
Contents
The Robot Data Bottleneck
Robot learning has a dirty secret: it’s hungry for data, but good data is brutally expensive to produce. You can either use a real robot (teleoperation) — which gives you accurate trajectories but doesn’t scale because you need one robot arm per collector — or you can use robot-free data collection (UMI), where someone carries a handheld gripper with cameras and just… does the task. Cheap, scalable, but lower quality.
Current practice uses UMI data for pre-training (teaching the model general motor skills) and then a small amount of real-robot teleoperation data for post-training (fine-tuning for deployment). The real robot data is the “anchor” that keeps the policy grounded.
HiFi-UMI’s bold claim: raise the fidelity of UMI data high enough, and you can drop that anchor entirely. Zero real-robot data at post-training. Just robot-free data, start to finish.
What’s Wrong with Current UMI Data?
UMI (Universal Manipulation Interface) is brilliant in concept: a handheld gripper with a fisheye camera on the wrist, recording first-person video of a human doing a task. You collect thousands of hours, train a policy, and deploy it on a real robot. Simple.
But if you try to use that data for post-training (not just pre-training), four problems surface:
- Pose tracking drifts over time. UMI uses visual-inertial SLAM, which slowly accumulates error. After a few minutes, your “6 mm accurate” pose is nowhere close. It’s like recording a video with shaky hands — the first few seconds are fine, then things go blurry.
- Two-hand positioning is approximate. UMI reconstructs the relative pose between two grippers from two independent SLAM trajectories. It works, but it’s never quite right.
- Cameras aren’t synchronized. Each camera runs on its own clock. ~6 ms of jitter sounds small, but when you’re inserting a peg into a hole, 6 ms is the difference between “in” and “missed.”
- Narrow field of view. A single 155° fisheye means the gripper leaves the frame when you reach for something. It’s like driving with the windshield half-covered.
The result? Low usable-data rate. Most trajectories simply can’t be replayed on a real robot.
How HiFi-UMI Fixes It
HiFi-UMI rethinks the data collection hardware from the ground up. Every design choice targets a specific failure mode in the original UMI:
- Head-mounted stereo + offline SLAM — instead of wrist-mounted cameras, HiFi-UMI mounts stereo cameras on the collector’s head and runs offline (batch) SLAM after the session. Result: 3 mm workspace-local accuracy, no drift accumulation.
- Both hands in one camera view — one head-mounted camera sees both grippers natively. The relative pose between them is measured directly, not reconstructed. That’s a huge difference in accuracy.
- GPIO hardware trigger — all six cameras share a microsecond-level sync signal. The synchronization error drops from ~6 ms to <40 μs. That’s 150× better.
- Two ultra-wide cameras per hand — each gripper has two 200° fisheye lenses covering 6 × ~200° total field of view. The gripper is never out of frame, no matter how you reach.
- Full-palm glove — instead of a trigger grip that forces an awkward hand posture, HiFi-UMI uses a glove that wraps around your palm. The data reflects how humans actually grasp objects, not how a handle constrains you.
- Voice feedback for quality — an on-device monitor checks for dropped frames, sync loss, and pose quality. If something goes wrong, the system tells the collector right then, not hours later during processing.
| Feature | Original UMI | HiFi-UMI |
|---|---|---|
| Pose accuracy | ~6 mm (drifts) | 3 mm |
| Sensor sync | ~6 ms (software) | <40 μs (hardware) |
| Field of view | 1 × 155° | 6 × ~200° |
| Two-hand pose | Reconstructed | Native |
| Data usable rate | Low | ~96% |
The Data Engine: From Raw Capture to Training-Ready
Hardware is only half the story. HiFi-UMI also builds a six-stage data pipeline that turns raw captures into verified, simulation-validated trajectories:
- Collect — human performs the task with the HiFi-UMI rig.
- Offline SLAM — batch-optimize the full trajectory. No drift.
- Simulation replay — reconstruct the trajectory in a simulator and try to replay it. If the robot arm can follow the path without collision, it passes.
- AI annotate — automatically label task phases, objects, and keyframes.
- Human verify — a quick visual check confirms the annotation is correct.
- Export — the trajectory is ready for training.
The numbers are impressive. 98% of raw captures pass trajectory reconstruction, and 98% of those pass simulation replay. That gives you ~96% usable data overall — compared to the much lower rates typical of earlier UMI systems.
The team has collected over 20,000 hours of data across 480+ scenes, totaling 4.32 million episodes. The released subset, HiFi-UMI-2K, contains 2,000 hours of microsecond-synchronized, ultra-wide-FoV demonstrations. It’s released under a CC BY 4.0 license, with faces masked for privacy — so anyone can use it.
The Big Result: Zero-Robot Post-Training
This is where the rubber meets the road. HiFi-UMI tests its data on three different AI backbones — think of them as three different engines you could drop into a car:
- StarVLA-QwenPI — a Vision-Language-Action (VLA) model. These models take camera images and language commands and output robot joint positions.
- OpenPI-π₀.₅ — another VLA model, but publicly available (you can actually run it yourself).
- LingBot-VA — a World-Action-Model (WAM). Instead of predicting actions directly, it predicts future video frames and derives actions from them.
For each backbone, the team compares two scenarios: post-training on HiFi-UMI data only vs post-training on in-domain teleoperation data (the gold standard). The difference is measured in percentage points of task success rate:
| Backbone | Difference (HiFi-UMI vs Teleoperation) |
|---|---|
| StarVLA-QwenPI (VLA) | -2.5 pp |
| OpenPI-π₀.₅ (VLA) | +3.1 pp |
| LingBot-VA (WAM) | -0.6 pp |
Read that again. OpenPI-π₀.₅ actually performed better with zero-robot data than with teleoperation data. The other two backbones are within statistical noise of the teleoperation baseline. On the hardest evaluation task — precision insertion, where the robot must slide a peg into a tight hole — the best policy reaches 85% success rate. That’s with zero teleoperation data from the evaluation scene. Zero.
Across all experiments, the team ran 960 real-robot rollouts to get statistically meaningful results.
And pre-training? Using 4,000 hours of the same HiFi-UMI corpus for pre-training cut action error on ten unseen tasks by 41% and boosted StarVLA-QwenPI’s real-robot success by an additional 18.1 percentage points. That’s not just “comparable to teleoperation” — that’s meaningfully better.
Why This Matters
The bottleneck in robot learning has always been data that works at deployment time. You could collect UMI data cheaply, sure, but you still needed a real robot to post-train. That meant the number of robots you owned was the ceiling on how much deployable data you could generate.
HiFi-UMI breaks that ceiling. If robot-free data alone is enough for both pre-training and post-training, then anyone with the capture rig can contribute deployable demonstrations. No robot required. For a field that has been robot-bottlenecked since its inception, that’s a paradigm shift.
The dataset is already open-source (CC BY 4.0, 2,000 hours, faces masked). The hardware design is portable and costs a fraction of a robot arm. Future work will likely explore more diverse tasks, longer-horizon manipulation, and the question of whether even better hardware fidelity can close the remaining gap entirely.
Where to Learn More
- Paper: arXiv:2607.25895
- Dataset: HiFi-UMI-2K (2,000 hours, CC BY 4.0)
- Project page: cloud.simpleai.tech/simple-world-lab/hifi-umi/
If you use HiFi-UMI in your research, please cite:
@article{hifiumi2026,
title={HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone},
author={Wei, Yuteng and Ma, Jinming and Wang, Jiawei and Zhou, Weitao and Zuo, Yushen and Rui, Ke and Li, Minglei and Zhang, Jinhao and Pan, Zhikang and Wang, Xiang and Jia, Haoran and Du, Huan and Zeng, Zicheng and Ma, Jun and Qin, Guiyu and Zhang, Di and Li, Xiaofei},
journal={arXiv preprint arXiv:2607.25895},
year={2026}
}
