Separate perception evidence from task completion

Evidence and interpretation

For Maha’s proposed intake, separate the sensor observation, the estimated object pose, the downstream action and the outcome judgement. This lets a reviewer ask whether a successful grasp depended on accurate localization, a forgiving fixture or intervention. Calibration and reference error belong in the evidence, not in a footnote asserting perfect ground truth. A missing stream should produce an explicit observation gap. It should not be replaced by an inference from the final image.

Proposed evidence workflow

1. Identify sensor configuration, frame, timestamp and calibration revision.
2. Distinguish measured reference values from labels inferred by the same model being tested.
3. Retain missed detections and unusable observations with their attempt IDs.

Worked illustration

A supplied target coordinate lets our toy controller navigate without detecting an object. That is privileged simulator state. The example therefore cannot be scored as a perception benchmark, even when its transfer succeeds.

Limits

NIST identifies perception as an evaluation component. These intake checks are Maha’s proposed method; they do not prove sensor accuracy or remove the need for independent measurement.

Sources and review

Continue reading

• Robot evidence needs explicit frames and units
• Keep simulation and hardware evidence separate