Physical AI · evidence and evaluation

Vision-language-action models: what the success rate means

A vision-language-action model maps images and an instruction directly to robot actions, using a pretrained vision-language backbone. Reported success rates describe the tasks, objects and embodiments that were evaluated, and generalise no further on their own.

How it works

OpenVLA is a 7B-parameter open model trained on 970k real robot demonstrations, which its authors report outperforming a much larger closed model by 16.5 percentage points of absolute success across 29 tasks and multiple embodiments. Two things follow. First, the recipe is real: internet-scale vision-language pretraining plus robot demonstrations produces policies that follow language instructions across objects better than task-specific training. Second, every number of that kind is an average over a named task set, scored by a rule the authors chose, on hardware they controlled. Reading “generalist” as “works in my setting” is the error that a procurement decision cannot afford.

A concrete case

A model reported at 60% success on a 29-task suite may be near 100% on the pick-and-place subsets and near zero on anything requiring force control. A per-task table answers that; a headline average does not.

What this establishes

That the architecture and the authors’ reported comparisons exist as published. It does not establish any figure independently.

What it does not

Self-reported results from the authors’ own suite, not an independent replication. Success rate says nothing about failure severity, recovery behaviour or safety.

Questions worth asking

  • Ask for per-task results and the definition of success used for each.
  • Ask which embodiment and which cameras produced the numbers, and how they differ from yours.
  • Ask what happens on failure: does the policy stop, retry, or continue into contact?

Sources

  • Kim et al. — OpenVLA: An Open-Source Vision-Language-Action Model ↗

    A vision-language-action model couples a pretrained vision-language backbone to robot action outputs; OpenVLA is a 7B-parameter open model trained on 970k real robot demonstrations, reported by its authors to exceed a 55B closed model by 16.5 percentage points of absolute task success across 29 tasks and several embodiments.

    Boundary: Those are the authors’ own reported results on their evaluation suites and embodiments. They are not an independent replication, and success rates on 29 tasks do not describe behaviour in an unseen setting.

    Locator, anchor and reuse basis

    Read at: Abstract; §1 Introduction. Inspected 2026-09-19. Original paraphrase and link to the open preprint; reported figures attributed to the authors.

    Verify by searching the source for: outperforming closed models such as RT-2-X (55B) by 16.5%. If that phrase is not there, or does not carry the meaning stated above, this citation is wrong and we want to know.

Continue

Elsewhere on this site

Maha Strategies publishes explanation and evaluation method. We build no robots, run no physical experiments, and report no benchmark results of our own. Hardware, safety and evidence-intake questions live in the robotics section.