Physical AI · evidence and evaluation

Fine-tuning a foundation model for control: what web pretraining does and does not transfer

Coupling a pretrained vision-language model to robot actions transfers semantic knowledge the robot data never contained. It does not transfer physical competence, and the evidence for each is different.

How it works

Two reported systems mark the approach. In RT-2, Brohan and colleagues express robot actions as text tokens and co-fine-tune a pretrained vision-language model on robot trajectories together with internet-scale tasks such as visual question answering; they report improved generalisation to novel objects, the ability to follow commands absent from the robot training data, and rudimentary reasoning such as picking the smallest object or identifying a suitable implement, over some 6,000 evaluation trials. OpenVLA follows the same pattern as an open 7B model trained on 970k real demonstrations, which its authors report outperforming a 55B closed model by 16.5 percentage points of absolute task success across 29 tasks. What transfers is recognition and language grounding, because that is what the web corpus contains. What does not transfer is contact, force and timing, because no amount of image-text data observes them. The practical reading is that these systems generalise best along the axis the pretraining covered — naming and identifying things — and remain bounded by the robot data on the axis of actually moving them.

A concrete case

A model correctly identifies which of several objects is a suitable hammer, having never seen one in its robot data, and then fails to grasp it because its fingers meet a surface geometry the demonstrations never contained. Both outcomes are what the training mix predicts.

What this establishes

That co-fine-tuning on web and robot data is reported to add semantic generalisation beyond the robot dataset, and that the authors of each system report their own gains on their own evaluations.

What it does not

Both sets of figures are the authors’ own, on their own robots and evaluation suites, with no independent replication cited here. RT-2’s abstract reports capabilities rather than a headline success rate, and neither result establishes behaviour on a different embodiment, a different task set, or an unrehearsed setting.

Questions worth asking

  • Ask which capability is claimed to come from pretraining and which from the robot data, and what evidence separates them.
  • Ask whether the reported evaluation is the authors’ own, and whether anyone outside the group has reproduced it.
  • Ask what fine-tuning data would be needed for your embodiment, since the reported models were trained on specific ones.
  • Be precise that a semantic success — naming the right object — is not a manipulation success.

Sources

  • Brohan et al. — RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control ↗

    The authors express robot actions as text tokens and co-fine-tune a pretrained vision-language model on robot trajectory data together with internet-scale vision-language tasks such as visual question answering. They report improved generalisation to novel objects, the ability to follow commands absent from the robot training data, and rudimentary reasoning such as selecting the smallest or largest object or identifying a suitable implement, evaluated over some 6,000 trials.

    Boundary: The authors’ own reported results on their own robots and evaluation set. The abstract gives capabilities rather than a headline success-rate figure, no independent replication is cited, and emergent behaviour on chosen probes is not a guarantee of behaviour elsewhere.

    Locator, anchor and reuse basis

    Read at: Abstract; §1 Introduction. Inspected 2026-09-20. Paraphrase and link to the open preprint; no figures or tables reproduced.

    Verify by searching the source for: 6k evaluation trials. If that phrase is not there, or does not carry the meaning stated above, this citation is wrong and we want to know.

  • Kim et al. — OpenVLA: An Open-Source Vision-Language-Action Model ↗

    A vision-language-action model couples a pretrained vision-language backbone to robot action outputs; OpenVLA is a 7B-parameter open model trained on 970k real robot demonstrations, reported by its authors to exceed a 55B closed model by 16.5 percentage points of absolute task success across 29 tasks and several embodiments.

    Boundary: Those are the authors’ own reported results on their evaluation suites and embodiments. They are not an independent replication, and success rates on 29 tasks do not describe behaviour in an unseen setting.

    Locator, anchor and reuse basis

    Read at: Abstract; §1 Introduction. Inspected 2026-09-19. Original paraphrase and link to the open preprint; reported figures attributed to the authors.

    Verify by searching the source for: outperforming closed models such as RT-2-X (55B) by 16.5%. If that phrase is not there, or does not carry the meaning stated above, this citation is wrong and we want to know.

Continue

Maha Strategies publishes explanation and evaluation method. We build no robots, run no physical experiments, and report no benchmark results of our own. Hardware, safety and evidence-intake questions live in the robotics section.