Why Robots Are Still Just Big Remote-Controlled Toys

The generalization problem in robotics

In August 2026, Dyna Robotics published a comparison between its lab and its customers’ sites. On in-house evaluations, Dyna-1 and Dyna-2 both passed at close to 100%. At customer sites the models had never seen in training, graded against the customers’ own requirements for quality, speed, and reliability, the pass rates were 46% and 87%. The two generations used the same task post-training data and the same number of training steps.^[1]

The result shows progress, and it also shows how hard generalization still is for robots. After learning a set of demonstrated motions in the training scene, a robot still struggles to adapt to a new site, new objects, and new operating conditions.

In everyday household chores, picking up a cup means choosing a grasp point, judging whether the cup will slip, and adjusting force after contact. Any of these can run into a situation the training data never covered. A model can “recognize the cup” and “understand” the instruction to pick it up and still fail while executing.

Language models and video generation models already generalize well across many tasks. Why is a model that predicts actions still so limited? That is the question this article works through. If you have run into the same puzzle in your own experiments, read on.

The next two sections cover background on the models and on scaling laws. If you already know VLA, WAM, and scaling laws, skip ahead to Sources of error in action prediction.

VLA and WAM

A vision-language-action model (VLA) takes camera frames, a language instruction, and the robot’s own state as input and generates actions. Many VLAs inherit a large pretrained vision-language model and add an action output module on top. The action can be a change in end-effector pose, a set of joint targets, or a continuous trajectory segment, which a controller turns into execution commands.^[2],[3],[4]

A world-action model (WAM) also learns to predict future observations. In Dyna-2, video and actions use separate representations and network modules that interact through attention, and the model used for the scaling experiments is trained on video prediction and action prediction at the same time. The video objective shapes the shared representation, and at control time the model can generate actions directly.^[1]

These models mostly rely on imitation learning, and the common method within it is behavior cloning: record the observations and actions while a demonstrator performs the task, then train the model to predict the action from the frames, the task description, and the robot’s own state. The data can come from a person teleoperating the robot, or from handheld grippers, wearable devices, or video of humans doing the task. The latter sources reduce the dependence on the robot body itself, and they bring their own problems with action labeling and cross-embodiment transfer.

Reinforcement learning improves a policy from the rewards received after acting, and it can use interaction data from simulation or from real robots. It can also be combined with behavior cloning: get an initial policy from demonstrations, then keep training on the policy’s own execution records, corrections, and rewards.^[5]

Pretraining, imitation learning, and reinforcement learning each supply a different kind of experience. A vision-language model has seen a huge number of objects and concepts, demonstrations record how a person performs the task, and interaction data contains the feedback a robot gets after executing an action. To apply this experience to new objects and tasks, the model still has to learn the relationship between the current observation and the action.

Scaling laws for robots

The scaling laws of language models are an attractive reference point. In 2020, Kaplan and colleagues found that, within the range they studied, a language model’s test loss follows a fairly stable power law in model size, data volume, and compute.^[8] The later Chinchilla work went on to study how to balance parameter count against training data under a fixed compute budget.^[9]

Written in shorthand, the relationship between data volume and loss is:

L(D) ≈ k · D^−α + L_∞

Here D is the amount of data, α describes how fast the improvable part of the loss falls as data grows, k sets its magnitude, and L_∞ is the asymptotic loss term in the fit. Every doubling of the data multiplies L(D) − L_∞ by 2^−α. The part that is a straight line on log-log axes is this improvable loss; once a nonzero asymptotic term is included, the curve gradually flattens.

Robotics research is testing for similar relationships, but “data volume” here lumps together very different material. RoboTurk released 111 hours of teleoperation data, DROID about 350 hours, and AgiBot World about 2,976 hours. Qwen-RobotManip’s tally of nine major open-source robot datasets comes to roughly 11,000 hours.^{[10],[11],[12],[13]} Dyna-2’s pretraining corpus exceeds one million hours, mostly first-person video of humans doing manipulation.^[1]

Data volume by collection method. Orange: recorded robot actions. Blue: data collected from humans.^{[10],[11],[12],[14],[15],[16],[17],[18],[19]} The sources, sensors, and labeling methods differ. The trend lines describe only these published samples, not an industry-wide growth rate. A million-hour commercial corpus and the total of the open-source datasets are also not a like-for-like comparison.

An hour of human video and an hour of robot teleoperation provide different action supervision. The first usually requires extracting actions from wrist trajectories, finger distances, or device readings. The second records the commands and states of a specific robot performing the task. Hour counts reflect collection scale; judging training value also requires looking at label quality, task coverage, and the difference in bodies.

Model parameter counts have similar limits. GR00T N1 has 2.2B parameters, π0 about 3B, OpenVLA 7B, and RT-2 once used a 55B backbone.^{[20],[4],[3],[2]} OpenVLA outperformed the larger RT-2-X in its paper’s evaluation, but the two differ in data and training recipe, so the comparison alone says nothing about the effect of parameter count. Robots are also constrained by control frequency and inference latency, so scaling up a model has to account for execution speed.^[7]

Dyna-2’s ablations look at the training objective more directly. The researchers extracted pseudo-actions from human video with hand-pose annotations and, at 5,000, 50,000, and 100,000 hours, compared three recipes: predicting actions only, jointly predicting video and actions, and adding video without action labels on top of that. In zero-shot offline evaluation on 39 robot tasks, action-only training showed severe and unstable overfitting. Only after adding extra video-prediction data did cross-embodiment transfer improve steadily with data volume.^[1]

This result shows that, under this data, architecture, and evaluation setup, the training objective and the data mix affect the return on scale. To explain why, we need to look at the action labels themselves.

Sources of error in action prediction

The training data records the actions the demonstrator took. It does not necessarily record why. A person who reaches for the right side of a cup handle might be avoiding an obstacle on the left or might simply be in the habit. Both cases can leave similar action labels, and the model has to tell them apart from the information in its input.

Below, x_t stands for the observation history, task instruction, and robot state the model receives, a_t for the demonstrated action, and o_t+1 for the next observation. The camera frame records only part of the physical state. The demonstrator’s choice of detour, their habits, and the forces they felt on contact are not necessarily in these inputs either.

Hidden state

Two people go around an obstacle from the same position. One goes left, the other right. Before the fork, their position histories can be identical. If the model receives only the instruction “go to target O”, nothing in these inputs determines which side the demonstrator will pick.

Add the phrase “go around on the left” and the choice becomes definite. If the demonstrator picks a side at random each time and the recording contains no clue, collecting more demonstrations can help the model learn the left-right ratio, but it still cannot predict any one random pick. This separates two things: learning the distribution of possible actions, and guessing the specific action in one recording. When both paths complete the task, the robot only needs to commit to one of them and execute it.

When is video prediction easier? Suppose the model already knows the action about to be executed. Then it can predict how the scene changes from that action. Given a “move left” command, for example, the detour direction is settled, and the model can learn how position changes with motion. The two tasks condition on different things:

Action-conditioned future prediction: p(o_t+1 | x_t, a_t)
Action prediction: p(a_t | x_t)

The action condition supplies the choice already made; action prediction has to learn how to make the choice from the observation. Genie predicts the next frame from the video history and a latent action, where the latent action is learned from changes in the video and can be specified by the user at generation time.^[26] Designs like this remove part of the uncertainty in future prediction. Conditions such as friction and contact location may still be missing, so a given action does not guarantee a fully determined prediction.

The specific training setup of a WAM needs to be checked case by case, though. The Dyna-2 variant used for the scaling experiments shares a backbone and trains separate flow-matching objectives for video and actions. Both tasks condition on the same history, and the video branch does not directly receive the true future action.^[1] So this set of experiments cannot be attributed simply to the video branch getting an extra a_t. General video generation also often has to model several possible futures at once.^[25]

When the input omits the conditions behind an action choice, the model can only learn the action distribution from the information that was recorded. With limited samples, it may tie an incidental detail of a scene to one action choice. Training loss keeps falling, and predictions get worse on a new batch of demonstrations. This is one path that can lead to overfitting. Confirming it requires checking missing information, sample coverage, and training behavior separately. Two loss curves, one for actions and one for video, are not enough to identify the cause.

We built a simple 2D obstacle-avoidance simulation to see how a hidden choice affects prediction. Each demonstration picks a detour side at random, with a style difference in how early it turns and jitter on the command. Both models receive the last two positions. The observation head predicts the next position; it corresponds to the vision head in the figures. The action head predicts the raw action command. The observation head here is just a position regression model, unlike a VLA’s vision encoder or a WAM’s video branch.^[21]

The simulation also has inertia: execution velocity depends on the previous velocity and the current command together, so brief command jitter gets smoothed. Predicting the raw command and predicting the position after motion are therefore disturbed in different ways, and the errors of the two targets have to be interpreted separately.

Left: detour side and turning style. Right: raw command versus executed motion. Neither prediction head receives the current action, the detour side, or the style parameter. This experiment does not test the effect of adding an action condition to video prediction.

# Hidden per episode: side (detour side), turn_dist (turning distance).
if x < OBS_X and (OBS_X - x) < turn_dist and abs(y) < 1.0:
    target = (x + 0.5, side * 1.2)
else:
    target = GOAL
d = unit_vector(target - pos)
cmd = SPEED * d + gauss(0, 0.05)
exec_v = 0.6 * exec_v + 0.4 * cmd
pos = pos + exec_v

# Both predictions use the last two positions:
# observation head: pos_next
# action head: cmd
# Nearest neighbor: 200 training episodes, 200 validation episodes.
# Small MLP: 4 -> 240 tanh -> 2; also compared at 8 and 200 episodes.

Missing state during contact

The obstacle example above assumed both sides were passable. If one side has a blockage the camera cannot see, the detour choice also affects whether the task succeeds. This kind of missing information shows up in many tasks, and contact manipulation is one example.

Two cups that look alike can differ in weight and surface friction, so the grip force they need may differ too. A demonstrator feels the cup start to slip. A model that receives only the image from the same viewpoint has no basis for that extra squeeze. FACT’s analysis of contact-rich tasks separates force-control failures from other problems with vision and positioning precision.^[22]

Tactile and force feedback can supplement these observations, but the amount collected is still small. T-Rex reports about 100 hours of tactile manipulation data, a long way from million-hour video corpora.^[23] How much a task depends on touch varies. Deformation visible in the image, motion history, and active probing can also supply some of the physical information.

Contact also amplifies small unobserved differences. In our stick-slip simulation, with the friction parameter varied by ±1%, the spread of displacement outcomes near the static-friction threshold was about 83 times that of a smoothly damped control; away from the threshold the difference shrank sharply.^[21] The factor of 83 is a result of this simulation and cannot be converted into a uniform amplification rate for real robot error. Hardware studies of planar pushing have also measured a distribution of outcomes from repeated, nearly identical pushes.^[24]

Friction and contact location affect the frame after the action too. Without this information, video prediction also runs into trouble. When comparing how actions and video generalize, first check which state each input contains and which of the conditions that determine the outcome are still missing.

Accidental correlations and overfitting

How does a model use a demonstration with incomplete information? Suppose that in the training set, by chance, every demonstration on a blue table goes left and every one on a white table goes right, while the task allows either side. Table color then helps the model lower its training error. Change the table or change the demonstrator, and that correlation can break. With richer inputs, the network can also identify a particular training trajectory from tiny position differences and remember its actions and jitter.

This is what “memorizing noise” means here. One instance of jitter becomes part of the training label, and the model uses incidental features of the sample to fit it. Those features have no stable relationship to the next instance of jitter. If the model keeps strengthening this kind of mapping, training loss can fall while validation loss rises. Random-label experiments show that deep networks are capable of memorizing mappings with no real regularity, and studies of the memorization process have observed the difference between learning patterns and memorizing samples.^[27],[28] They support the possibility of this mechanism; whether a specific robot model does this still needs ablation.

Diffusion Policy, π0, and action-token models can all represent several possible actions.^[29],[4],[2] For a task where either side is passable, the model can learn both options and sample one to execute. For a task where grip force has to adjust to weight, no matter how rich the action distribution, the model cannot conjure up the weight of this particular cup.

A distributional model can also concentrate too much probability on the actions in the training samples. Whether a model can represent a multimodal distribution and whether it can learn that distribution accurately under new conditions are two different questions. The squared error in flow matching is used to learn the denoising velocity field, and its final output cannot be read as “averaging the left and right actions”.^[4]

Incomplete information does not necessarily make validation loss rise, either. As long as the model learns the stable pattern or action distribution the input can support, validation performance may level off. Overfitting also depends on data coverage, model capacity, and the training process. The small regression experiment below shows only one of these paths.

Our small MLP was trained on only eight demonstrations for 4,000 epochs. Observation training and validation NMSE reached 0.0014 and 0.0019. Action validation error fell to about 0.53 at epoch 45 and then climbed back to about 0.60, while training error fell to about 0.31.^[21]

Action validation error first falls and then rises, showing the overfitting in this training run. The “floor” in the figure is a lower bound on regression error estimated from local conditional variance, and it applies only to this input, this target, and the NMSE metric. It does not constrain the task success rate of modern generative action models.

The coverage of demonstration data

If training contains only a few scenes and operating habits, the model has a hard time telling which cues remain valid in a new task. Adding demonstrations helps, and how much depends on how many different conditions the new data brings.

For example, a two-second translation can be recorded as dozens of frames. Adjacent frames and actions are very close together and mostly repeat the conditions of the same motion. Raising the sampling rate adds frames and helps capture fast changes, but it does not automatically add variety in detour choices, object weights, or contact modes. Both video frames and action labels have this kind of repetition in time.

In the obstacle simulation, with the training frame count fixed at about 170, spreading the data from eight episodes to sixteen lowers the action head’s validation NMSE from 0.85 to 0.76, and going to thirty-two gives about 0.75. Keeping eight episodes and only halving the frames changes action validation NMSE from 0.85 to 0.88.^[21] In this comparison, adding different episodes was more effective than keeping more adjacent frames within the same episodes, because the different episodes contain new combinations of detour side and turning.

Real-robot research has gone further and compared coverage of environments and objects. Lin, Hu, and colleagues collected more than 40,000 demonstrations and ran more than 15,000 real-robot execution tests. In their tasks, adding environment and object variety helped more than repeating the same set of conditions, and in some experiments the returns fell off clearly after about fifty demonstrations per environment-object combination.^[30]

That study examines generalization across environments and objects. Adjacent frames within each new demonstration are still similar; what grows is the set of objects, backgrounds, and operating conditions the training set covers. Only by completing the same cup-grasping task at different positions, with different cups, on different tables, does the model get a chance to distinguish which action cues can be reused across scenes.

AgiBot’s research also found value in task diversity and discussed the interference that different operators’ habits introduce into training; debiasing for speed differences helped.^[32] MimicLabs analyzed the effect of variation in camera pose and spatial layout.^[33] All of these check one concrete thing: which conditions the new data changes, and whether those changes improve the corresponding generalization test.

Hour counts are therefore not enough on their own to measure the training value of action data. Extra demonstrations in the same scene can add fine corrections or rare contact events; changing the scene tests whether previously learned relationships still hold. The more thoroughly the existing conditions are already covered, the more clearly the next batch of data needs a stated gap to fill.

Limits on transferring pretrained knowledge

Vision-language pretraining provides far broader knowledge than robot demonstrations do. A model already knows cups and cup handles. After action training, why can it still fail when the cup or the viewpoint changes? Answering this requires checking whether the knowledge is preserved during training and whether the learned representation can be used for a new embodiment and new operating conditions.

Joint training and knowledge retention

When the action output goes wrong, the failure may sit earlier, in visual processing. One viewpoint-transfer study localized a specific failure to the visual tokens and improved performance with a small adapter while keeping the policy frozen.^[6] This offers a control: examine the input representation and the action module separately, so that an action failure is not automatically blamed on the action head.

Dyna-2’s results support video prediction helping cross-embodiment transfer.^[1] A study from Toyota Research Institute compared 89 policies across 58,000 simulated and 2,835 real-robot execution tests. Effective combinations of vision-language data, trajectory-language annotations, and cross-embodiment robot data improved performance under distribution shift, on unseen tasks, and on instruction following, while training on robot data alone eroded the backbone’s vision-language ability.^[31]

Physical Intelligence’s Knowledge Insulation work looks at interference between the action expert and the pretrained backbone. Attaching an untrained continuous action module directly can hurt training speed and knowledge transfer; isolating the relevant gradients and keeping suitable supervision on the backbone mitigates the problem.^[7]

These experiments locate the difficulty of transfer in another place besides the input: once pretrained knowledge enters action learning, it is also subject to the training objective and the gradient updates. Even when the raw input contains usable cues, the representation after training may not have kept them or used them well.

Gains from the action representation likewise depend on training conditions. The discrete action-token schemes compared by Toyota Research Institute brought no statistically significant gain, and some reduced generalization.^[31] That limits what those schemes achieved in that experiment; it does not rule out action chunking, distribution modeling, or other representations in other tasks.

Cross-embodiment transfer from human video

Dyna-2 was trained on subsets of 1,000, 10,000, 100,000, and 1,000,000 hours of human video, keeping the source mix, validation set, and training setup fixed. Offline action metrics on both the human held-out set and robot data excluded from pretraining improved as the pretraining data grew.^[1]

Left: Dyna-2’s same-embodiment evaluation against its cross-embodiment evaluation. Right: fitted exponents collected from different studies.^[1],[8],[34] The studies use different losses, units, tasks, and fitting ranges, so the curves show only each one’s trend; the slopes cannot be used to rank models by generalization ability.

Dyna-2 reports a persistent gap between human-action and robot-action error, roughly a factor of two to three across data scales. Same-embodiment and cross-embodiment prediction still differ, then. The specific gap has to be read against the tasks and action representations of the two datasets and cannot be converted directly into task success rates.

Transfer requires a concrete action mapping. Dyna extracts wrist pose from video as end-effector trajectory supervision and constructs a continuous grasp signal from the distance between thumb and index finger.^[1] These labels provide a representation close to robot actions, but they still leave out the robot’s own dynamics, contact forces, and execution error. Mapping hand motion onto a two-finger gripper and onto a dexterous hand also requires different handling.

We refit the published scale points with an asymptotic term, and the extrapolation range came out very wide.^[35] The data covers only four scales, and the fit is sensitive to the functional form and the asymptotic term. The existing curves support continuing to test the transfer benefit of human data. They are not enough to predict how many more hours would erase the embodiment difference, let alone to estimate a date for reaching human level.

Adaptation information for a new task

Transfer from pretraining still has to be judged together with the extra information a model gets on the new task. A few demonstrations can fill in object placement, action ordering, or the handling of a specific embodiment, so direct transfer, in-context demonstration, and parameter fine-tuning need to be kept apart.

In Dyna-2’s real-robot scaling experiment, models with different pretraining scales were post-trained on the same fourteen tasks, with at most ten hours per task. Holding that data fixed, more human-video pretraining still improved real-robot results: on the lockbox key-turning task, 0/10 at 100,000 hours of pretraining and 9/10 at a million.^[1]

In this comparison the amount of robot demonstration stays fixed, and broader human pretraining improves the later learning. The measured gain comes from the “human pretraining plus task post-training” setup. From this result alone, we do not know how the model would do with robot demonstrations removed entirely.

The amount of information adaptation needs is already changing. Generalist’s GEN-1.5 reports that adding a 3-to-12-second perception-and-action demonstration to the context, with no gradient updates, gives an average success rate of 59% across ten tasks with a reported standard deviation of 10%. After ten gradient steps on five minutes of data per task, the average is 83% with a standard deviation of 9%. These ”±” figures are standard deviations and should not be read as 95% confidence intervals.^[36]

The demonstration can be collected with a handheld gripper or taken from the robot’s own execution record, and it includes sensor data and action trajectories. The researchers restrict these experiments to simple, short-horizon tasks and note that skills learned in context remain more brittle than fine-tuned models.^[36] So the result supports using a short demonstration for part of task adaptation. It does not generalize to “any video can teach a robot a skill”.

Task experience can also keep accumulating during deployment. Physical Intelligence’s RECAP improves the model with policy execution results, corrections, and rewards.^[5] This lets data collection cover the states the policy itself encounters. Whether it reduces later human demonstration has to be judged against intervention counts, failure recovery, and collection cost.

Execution drift and state distribution

Another kind of generalization difficulty arises during execution. The robot’s first grasp is slightly off, the cup slides to a new position, and the next step has to approach again from there. If training contains only demonstrations where the cup was picked up cleanly, the model may never have learned this kind of correction. Even with the same objects and the same site, the robot’s own errors carry it into states the training data barely covers.

Open-loop evaluation places the model on a recorded trajectory and compares the predicted action with the demonstrated one step by step. At each step it receives the state the demonstrator left behind; if the model just predicted wrong, the next step still returns to the recorded input. Closed-loop execution carries forward the consequences of the model’s own actions, and the model has to handle the states after drifting away. Checking open-loop loss alone misses this part of the generalization requirement.

Open-loop error is computed on the states the demonstrator visited; closed-loop results depend on the states the learned policy actually visits. Whether an action deviation can be corrected is something only subsequent execution reveals.

In symbols, open-loop loss is usually averaged over the expert’s state distribution d_expert, while deployment performance depends on the state distribution d_π̂ produced by the policy π̂. A policy can predict accurately near the expert trajectories and still be poor at recovering from states it drifted into.

History inputs can also supply prediction cues that have nothing to do with the task. Actions are correlated in time, so copying the previous action can score reasonably well on an offline recording. The copycat studies analyze models that rely on past-action cues and downweight the current observation; the causal-confusion work discusses the mismatch in imitation learning between correlations and the actual causes of control.^[37],[38]

LIBERO-PRO perturbs objects, initial states, instructions, and environments beyond the standard benchmark and finds that models with high scores there fall to 0% in some of the generalization settings. After the target object was swapped or the instruction corrupted, models could still perform similar actions.^[39] Another study analyzed shortcut learning in multi-source robot data, where a model may use background and embodiment cues to identify the data source.^[40] These results support checking whether a model follows the task conditions; locating the specific failure still requires ablation.

Our obstacle simulation also compared open loop against closed loop. Two networks were trained on a hundred demonstrations, one of which also received the previous raw command as input; the command noise was set to low-frequency drift with temporal correlation. The extra history input lowered open-loop NMSE from 0.414 to 0.287, an improvement of about 31%. In two hundred closed-loop tests each, the success rates were 70% and 73%.^[21]

At this sample size, a three-point difference is not enough to confirm a stable gain. It also does not prove the gain is exactly zero, or that all of the open-loop improvement came from noise prediction. What can be said is that a 31% drop in open-loop error did not correspond to a rise in task success of the same size.

Autonomous driving research has reported metric divergence too. An imitation-learning study on 30,000 hours of data observed open-loop performance improving with data as a power law, but the same relationship did not appear in closed-loop simulation.^[34] A separate Waymo study on 500,000 hours of data reports that closed-loop metrics also improve with scale.^[41] The data, models, and evaluation protocols differ, so the two cannot be merged into a single conclusion. Together they ask researchers to verify the relationship between proxy metrics and actual control performance.

Error accumulation and failure recovery

The classic analysis of imitation learning considers error accumulation. If a policy’s single-step error rate on expert states is δ, then for a task of length T, the worst-case gap in accumulated cost can have the following upper bound:

C(π̂) − C(π_expert) ≤ O(δ · T²)

This assumes bounded per-step cost and uses a corresponding classification-style definition of error. An early mistake can put the policy in a state that is rare in training and affect many of the remaining steps; the longer the task, the more chances this accumulation has.^[42]

The bound describes a worst case under specific assumptions. Work on the sample complexity of imitation learning goes further into the roles of whether the state transitions are known, the policy class, and conditions on the expert.^[43] The analysis by Foster and colleagues shows that when realizability, the range of cumulative return, and the complexity of policy learning are controlled, behavior cloning with log loss can get a better dependence on horizon.^[44] Continuous control tasks cannot simply take the quadratic growth as an empirical law.

Recovery data from deployment therefore has its own value. Recording only successful demonstrations can leave out how to continue after a gripper slips, a grasp lands off-center, or the position drifts. Only by adding these states, corrective actions, and task outcomes to training can we test whether the policy learns to recover.^[5]

@article{yexling2026enactionbottleneck,
  title = {Why Robots Are Still Just Big Remote-Controlled Toys},
  author = {{Yex Ling}},
  journal = {yexling.com},
  year = {2026},
  month = {August},
  url = {https://yexling.com/blog/action-bottleneck/}
}