Compare controller revisions without moving the goalposts

Evidence and interpretation

REP 2004 includes version policies and performance-regression requirements for package-quality declarations. Maha’s proposed evaluation applies the general discipline to a small task comparison, without claiming REP compliance. Bind the controller identifier to its implementation artifact rather than trusting a mutable label. If the environment, dataset or task definition also changes, call it a multi-factor comparison. Retain the previous package so a reviewer can see which condition changed and which results are no longer directly comparable.

Proposed evidence workflow

1. Pair old and new controllers against the identical frozen case list.
2. Compare terminal outcomes, assistance and step counts without hiding adverse cases.
3. Preserve both packages, the comparison code version and its limits.

Worked illustration

Direct-path v1 moves toward the goal without routing around obstacles. Grid-search v2 considers unblocked neighboring cells. Both still fail the injected grasp-failure and tight-budget cases. Any improvement is attributable to these code paths in this toy world, not evidence that v2 is generally more intelligent.

Limits

The example is not a statistical benchmark, a quality-level declaration or release authorization. External observation and domain-specific acceptance criteria remain separate.

Sources and review

Continue reading

• Count all planned robot evaluation attempts
• Write a robot task contract before collecting results