Physical AI · evidence and evaluation

Uncertainty estimates and the inputs a model has never seen

A useful uncertainty estimate rises when the input is unlike training data. Reported methods do this on average; none of them is a guaranteed detector, and an average is not an operating point.

How it works

Deep ensembles — several independently trained networks whose predictions are combined — were reported by Lakshminarayanan and colleagues to give uncertainty estimates competitive with approximate Bayesian approaches, and to express higher uncertainty on out-of-distribution examples. For a physical system the question is what that buys operationally. An uncertainty signal is only useful if something acts on it: slow down, request assistance, refuse the action. That requires a threshold, and a threshold requires knowing the false-alarm and missed-detection rates at the chosen operating point on data like the deployment, which a benchmark average does not give you.

A concrete case

An ensemble flags a novel object with high variance in 80% of cases. If the remaining 20% includes the case that causes contact, the monitor does not remove the need for the fallback behaviour; it only reduces how often the fallback is needed.

What this establishes

That ensembles are a published, scalable approach reported to express higher uncertainty out of distribution on classification and regression benchmarks.

What it does not

Benchmarks, not robots. No detection guarantee, no operating point, and no claim that an uncertain prediction is a safe one.

Questions worth asking

  • Ask what action the uncertainty signal triggers, and at what threshold.
  • Ask for the false-alarm and missed-detection rates at that threshold on representative data.
  • Ask what happens when the monitor itself is out of distribution.

Sources

  • Lakshminarayanan, Pritzel and Blundell — Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles ↗

    The authors report that an ensemble of independently trained networks produces uncertainty estimates competitive with approximate Bayesian methods, and expresses higher uncertainty on out-of-distribution test examples.

    Boundary: Classification and regression benchmarks, not robot control. Higher uncertainty on out-of-distribution inputs is a reported tendency, not a detector with a guaranteed operating point.

    Locator, anchor and reuse basis

    Read at: Abstract; §1 Introduction. Inspected 2026-09-19. Original paraphrase and link to the open preprint.

    Verify by searching the source for: express higher uncertainty on out-of-distribution examples. If that phrase is not there, or does not carry the meaning stated above, this citation is wrong and we want to know.

Continue

Maha Strategies publishes explanation and evaluation method. We build no robots, run no physical experiments, and report no benchmark results of our own. Hardware, safety and evidence-intake questions live in the robotics section.