In the last piece I argued that no single data source wins for contact-rich manipulation. Human video scales but carries no action labels. Teleoperation is perfectly grounded but expensive per hour. Simulation is cheap but wrong in exactly the regime that matters, which is contact. My conclusion was that the layered pyramid architecture at the major labs reflects structural necessity rather than a temporary engineering compromise.
Since then one of those layers has been moving faster than the others.
Handheld grippers of the UMI type are showing up everywhere. FastUMI-100K, the ten thousand hour corpus behind RDT2, and now a twenty thousand hour corpus from a single system. AgiBot World assembled 2,976 hours of manipulation data from 100 dual-arm humanoids in a purpose-built 4,000 square meter facility. RoboMIND assembled 305 hours across four embodiments, each needing its own teleoperation hardware built to match that arm. A handheld gripper needs an operator and a room.
But the field has confined that data to a specific role. Robot-free data pretrains, teleoperation post-trains. The small real-robot fraction at the end is what grounds a policy for deployment, and nobody removes it. ActiveUMI and XRZero-G0 shrink it. RDT2 mixes it back in once deployment-grade performance on a specific arm is required. The anchor stays.
The interesting question is whether the anchor is there because post-training needs embodied data, or because robot-free data has not been good enough. Those are different claims and the field has mostly assumed the first without testing it.
Three things that are actually wrong with handheld data
I want to be precise about the limitations, because they are usually discussed as one undifferentiated problem and they are not.
The first is pose estimation of the device itself, and it is the one that gets the least attention despite sitting upstream of everything else. Every action label in a UMI corpus is a SLAM output. The camera doing that SLAM is mounted on the wrist, which means it is bolted to the fastest-moving link and pointed at the exact part of the scene that manipulation is changing. It gets occluded by the hand and by the object. It drifts over long horizons and fails under low texture or motion blur. AirExo-2 attributes handheld device shortcomings primarily to this and to limited field of view. For bimanual work it gets worse, because the relative pose between the two grippers is reconstructed after the fact from cross-camera co-visibility rather than measured, which puts the largest error precisely on the coordinated tasks where relative pose matters most.
What makes this failure mode challenging is that it is invisible downstream. A drifted trajectory is still kinematically valid. Nothing in the filtering stack catches it.
The second is kinematic and dynamic feasibility. The human arm has different reach, joint limits, and velocity envelope than the target robot, so some fraction of what gets demonstrated cannot be executed. UMI’s answer was a post-hoc filter against the robot’s known kinematics. That works, but the filter is not a random sample. It removes fast motions, motions near the workspace boundary, and high joint-rate segments, which is to say it removes the hardest and most informative parts of the distribution. FeasibleCap makes this point directly, that boundary-case motions are the ones that matter most for policy robustness and also the ones most likely to fail replay. I have not found any paper that quantifies the bias this introduces.
The third is the perception gap, and it splits into two channels that get conflated. One is the mismatch between the camera on the collection device and the camera on the robot, meaning the extrinsic transform from camera to fingertip and the intrinsic lens parameters. The other is the mismatch between the observation distribution of the robot data and the distribution the foundation model was pretrained on. I will come back to the second one, because I think it is the more consequential of the two and the one with the cleanest analogy to what does not happen in language models.
What co-design buys
HiFi-UMI (arXiv 2607.25895) attacks all three at the hardware level rather than in post-processing. The framing is what caught my attention. Rather than shrinking the real-robot fraction toward zero, they ask whether raising the fidelity of the robot-free data itself can remove the need for the anchor at all. That reframes the problem from a mixing-ratio question, which is what most of the field has been optimizing, to a data-quality question.
Instead of a camera on each wrist, a stereo rig with an IMU sits on the head and estimates the global head trajectory. Each hand carries a rigidly attached fiducial marker cube observed by those same head cameras. Composing the global head trajectory with the two hand-relative-to-head poses gives globally consistent trajectories for both hands. The head viewpoint is more stable than the wrist, and head motion is smaller than hand motion, so accumulated drift drops. Because both cubes appear in a single head-camera frame, the inter-gripper relative pose becomes a direct measurement with the same accuracy as the per-gripper pose rather than a derived quantity.
The sharpest decision in the paper is buried in that section. They run the optimization offline, which lets the optimizer use future observations as well as past ones, and they deliberately forgo global loop closure. The reasoning is that manipulation continually alters the scene, which violates the static-world assumption loop closure depends on. Instead they impose local consistency over a dynamic sliding window. Global drift stays at the centimeter level and local accuracy stays at the millimeter level.
That is a choice to be worse at the metric SLAM papers report and better at the metric manipulation needs. Absolute position in the room is irrelevant to a policy predicting relative end-effector increments. Local consistency inside the workspace is everything. The measured result is 3mm mean end-effector error over roughly 2m of accumulated head trajectory, against base-station ground truth used only for the evaluation.
On synchronization, every sensor is driven from a single GPIO hardware trigger, giving under 40 microseconds of cross-sensor offset. This one is easy to dismiss as engineering hygiene, so it is worth doing the arithmetic. If the image and the pose stream are misaligned by some interval and the hand is moving, the position error on the action label is velocity times that interval. At 0.5 m/s, a ten millisecond software-alignment residual is 5mm. Their entire claimed accuracy budget is 3mm. Software synchronization alone would blow it, and would do so as a velocity-dependent bias rather than as noise, meaning the corruption is largest during exactly the fast approach segments where precision matters and consistent enough that a policy can learn it.
On feasibility, they run a whole-body motion controller for the target embodiment and replay every reconstructed trajectory in simulation, discarding what is kinematically or dynamically infeasible. Reconstruction passes 98% and replay passes 98%, for roughly 96% cumulative yield from raw capture.
I would not read that 98% as evidence that feasibility is solved. It is partly a property of the target platform. Whole-body control on a bimanual system means the base and torso can reposition to bring a target into reach, which enormously expands the feasible set relative to a fixed-base arm. FeasibleCap reports a mean infeasible frame ratio of 83% on a fixed Franka doing pick-and-place. it is important to note that feasibility binds much less tightly when the embodiment has redundancy to spare.
They also catch problems at capture time rather than after it, with the device detecting underexposure, motion blur, excessive operator speed, and hands leaving the head cameras’ field of view, then issuing voice feedback so the operator corrects in situ. This is the ARCap lesson applied to a different failure class. ARCap’s user study found expert demonstrations without feedback trained policies to 25% success while expert demonstrations with feedback reached 70%, and novices with feedback beat experts without it. Filtering removes bad data. Feedback produces good data.
The first shift: what if this becomes the pretraining corpus
Pretraining on 4,000 hours of the same corpus lowers offline action error on ten unseen tasks by 41% and raises real-robot success by 18.1 percentage points on one backbone, matching the scratch-initialized baseline with a quarter of the task data. The scaling fit is a power law with exponent 0.268.
That raises a question the paper does not ask. If you can produce action-grounded data at this fidelity and this volume, what is web-scale pretraining still doing?
The reason the question is worth taking seriously is that the web corpus is misaligned with the robot observation distribution in a way that does not have a good analogue in language. In LLMs, fine-tuning data is out of distribution in content but in distribution in representation, because it is tokens over the same vocabulary. In manipulation the post-training data can be out of distribution geometrically. A 180 degree wrist fisheye image is not a rare sample from the web image distribution. It is a different projection of the world.
So in current web pretraining, you import a prior, then spend architecture budget stopping action gradients from destroying it, then spend data budget correcting it for your camera geometry.
I do not think that makes the case for dropping it, and I want to be clear that HiFi-UMI does not test this. All three of their backbones are initialized from pretrained VLMs. The robot pretraining sits on top of web pretraining, not instead of it.
Two reasons I think it stays. The corpus spans 480 scenes across 4.32 million episodes, which is roughly nine thousand episodes per scene. That density is exactly right for learning contact dynamics and exactly wrong for learning what objects are. And the language interface only works if the model can ground nouns it has never been shown, which is inherited capability rather than learned capability. The closest thing to a controlled test I found, “Rethinking VLM Representation for VLA Initialization” (arXiv 2605.25802), concludes that preserving the original VLM representation remains important even when the injected signal is robot trajectories.
The more useful framing is that these corpora differ in kind rather than in quantity. Robot data is dense in action-conditional physics and sparse in world coverage. Web data is the reverse.
The second shift: what if the anchor comes off entirely
This is the claim the paper is actually built around, and it is the one worth arguing about.
Across three backbones spanning reactive VLA and world-action-model families, post-training solely on handheld data differs from in-domain teleoperation by -2.5, +3.1, and -0.6 percentage points. The strongest policy reaches 85% on a precision insertion task.
The experimental design deserves credit on two counts. The asymmetry runs against them, because the teleoperation baseline was collected in the evaluation scene and no handheld trajectory was. And they froze the benchmark before evaluation, separated test-instance construction from policy execution, randomized policy order, and recorded termination reasons across 960 real-robot rollouts.
Now the caveats, which the authors mostly state themselves. Nine hundred sixty rollouts across six conditions is 160 per condition, and near a 70% success rate the standard error on a proportion at that sample size is around 3.6 points. A 3.1 point difference is inside noise. The correct reading is that the two are not distinguishable at this sample size, which is what parity means here rather than equality.
More significantly, they treat fidelity as a design principle rather than a variable isolated through controlled degradation, and explicitly leave to future work the ablation that would separate fidelity from sample count and scene coverage. Nothing in the paper attributes the result to offline SLAM versus hardware synchronization versus wide field of view versus curation versus volume. It is a system-level result and cannot be decomposed.
What I take from it
The finding I keep returning to is not the parity number. It is the claim that transfer depends more on whether pretraining covered a task’s kind of physical interaction than on whether its objects have been seen before.
If that holds, it inverts how most teams budget collection. The instinct is to broaden object and scene coverage. This says to broaden coverage of interaction dynamics, meaning contact modes, force regimes, and failure-and-recovery patterns. Notably they deliberately collect rare failure-and-recovery episodes for exactly this reason. That connects directly back to the argument in the last piece, because contact modes are precisely what video-only sources cannot supply.
One gap worth naming. The fidelity axes in their comparison table are pose, synchronization, views and field of view, relative pose, gripper morphology, and portability. There is no calibration axis. Like VISTA, they eliminate the device-to-robot perception gap by mounting matched hardware rather than by measuring it. I could not find any published sensitivity curve for how many millimeters of wrist-camera extrinsic offset costs how many points of success rate. That number does not exist as far as I can tell, and it should.
If high-fidelity robot-free data displaces teleoperation at post-training, and web pretraining stays because it supplies something orthogonal, then teleoperation is squeezed from both sides. It is not the scale layer and it may no longer be the grounding layer. What it is still for is a question worth answering before anyone builds another 4,000 square meter facility.