Robotics evidence guide · automated editorial preparation, not expert review
Count all planned robot evaluation attempts
Publish the denominator with the numerator. Register expected attempts before execution, preserve failures and assistance, and distinguish missing records from unsuccessful recorded trials.
Evidence and interpretation
Maha’s proposed comparison is a paired evaluation: each controller runs the same registered cases. Report completed, assisted and failed attempts separately. A dropped log is not automatically a physical failure, but it prevents a complete evidence claim. An excluded case needs a reason and a preserved record; otherwise selective reporting can look like an improvement. Success rate alone also hides whether gains are concentrated in one easier condition. Break down results by case and failure reason before creating an aggregate.
Proposed evidence workflow
- Freeze expected IDs and conditions before generating results.
- Require one terminal outcome per controller/case pair; reject duplicates or omissions.
- Calculate autonomous successes divided by all planned attempts, and report assisted completion separately.
Worked illustration
A controller completing two of six synthetic cases autonomously is reported as 2/6, not 2/2 after dropping failures. The second controller receives the same six cases. These are deliberately selected fixtures, not a random sample; no population confidence interval or generalization claim follows.
Limits
This denominator policy is Maha-authored. NIST motivates task-oriented metrics but does not prescribe this six-case benchmark. A complete ledger can still contain untruthful external observations.
Sources and review
Locator: Summary; Description: Framework, Capability Models and Test Methods
The completed project decomposes assembly performance into perception, mobility, dexterity and safety-related components.
Boundary: A research-program description, not evidence of Maha performance or an operational safety approval.
Inspection and reuse basis
Selected sections inspected 2026-09-14. Link and original paraphrase only; no third-party images or publication copied. This is not independent expert review of this article.