Required inputs
All eligible tasks
Prediction or abstention per task
Locked scoring rule
Coverage threshold
Ordered execution
Work the protocol
- 1
Require one submission state for every task.
- 2
Count abstention in coverage.
- 3
Score accuracy both overall and when answered.
- 4
Apply the preregistered abstention penalty.
- 5
Report confidence intervals and missingness.
Outputs
- Coverage
- Overall score
- Conditional score
- Abstention count
Done only when
- Tasks equal answered plus abstained.
- Coverage accompanies accuracy.
- Policy matches preregistration.
Fail closed
Refusal conditions
- Abstained tasks vanish.
- Coverage threshold changes after scoring.
- Only correct answered tasks are reported.