Prospective evaluation and falsifiability protocol

Temporal train–validation–test split

Prevent future outcomes from influencing fitting, model selection, or feature construction.

Required inputs

Chronological outcome corpus

Outcome availability time

Validation boundary

Test boundary

Feature version

Ordered execution

Work the protocol

  1. 1

    Sort by when each outcome became available.

  2. 2

    Fit only before the validation boundary.

  3. 3

    Select allowed hyperparameters on validation only.

  4. 4

    Lock the model before test begins.

  5. 5

    Score test rows once without refitting.

Outputs

  • Immutable split manifest
  • Locked model digest
  • Held-out score set

Done only when

  • Every row belongs to exactly one split.
  • Training precedes validation and test.
  • Test remains untouched until lock.

Fail closed

Refusal conditions

  • Random splitting leaks future information.
  • Test performance selects a model.
  • Feature definitions change during test.