Physical AI · evidence and evaluation

What a benchmark score is evidence of

A benchmark score is evidence about that benchmark. Its value depends on whether the tasks, objects and scoring resemble the decision you are making.

How it works

Meta-World supplies 50 distinct simulated manipulation tasks, and its authors report that while individual tasks and their variations could be learned, the algorithms they evaluated struggled to learn several tasks at once — even with as few as ten training tasks. That result is more useful than any leaderboard position, because it describes a limit that transfers: multi-task competence is not the sum of single-task competences. When reading any robotics or physical-AI benchmark, the questions are the same: how were tasks sampled, what counts as success, how many seeds, and does the reported number come from the same conditions the baseline used.

A concrete case

Two papers report on the same suite with different reset policies — one re-randomises object pose each episode, the other does not. The scores are not comparable, and nothing in the headline number says so.

What this establishes

That a published multi-task manipulation benchmark exists with 50 tasks, and that its authors reported multi-task learning difficulty as a finding.

What it does not

Simulated tasks with the benchmark’s own models and rewards. No benchmark score is a statement about a physical deployment.

Questions worth asking

  • Ask how many seeds, and whether variance across seeds is reported.
  • Ask whether the baseline was run by the same authors under the same conditions.
  • Ask which aspects of your decision the task distribution does not cover at all.

Sources

  • Yu et al. — Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning ↗

    The benchmark provides 50 distinct simulated manipulation tasks; the authors report that while individual tasks and their variations can be learned, the algorithms they evaluated struggled to learn several tasks at once, even with as few as ten training tasks.

    Boundary: Simulated manipulation with the benchmark’s own object models and reward structure. A score here is not a statement about a physical robot.

    Locator, anchor and reuse basis

    Read at: Abstract; §1 Introduction. Inspected 2026-09-19. Original paraphrase and link to the open preprint.

    Verify by searching the source for: even with as few as ten distinct training tasks. If that phrase is not there, or does not carry the meaning stated above, this citation is wrong and we want to know.

  • Kim et al. — OpenVLA: An Open-Source Vision-Language-Action Model ↗

    A vision-language-action model couples a pretrained vision-language backbone to robot action outputs; OpenVLA is a 7B-parameter open model trained on 970k real robot demonstrations, reported by its authors to exceed a 55B closed model by 16.5 percentage points of absolute task success across 29 tasks and several embodiments.

    Boundary: Those are the authors’ own reported results on their evaluation suites and embodiments. They are not an independent replication, and success rates on 29 tasks do not describe behaviour in an unseen setting.

    Locator, anchor and reuse basis

    Read at: Abstract; §1 Introduction. Inspected 2026-09-19. Original paraphrase and link to the open preprint; reported figures attributed to the authors.

    Verify by searching the source for: outperforming closed models such as RT-2-X (55B) by 16.5%. If that phrase is not there, or does not carry the meaning stated above, this citation is wrong and we want to know.

Continue

Elsewhere on this site

Maha Strategies publishes explanation and evaluation method. We build no robots, run no physical experiments, and report no benchmark results of our own. Hardware, safety and evidence-intake questions live in the robotics section.