Confidence Is a Budget: A Small Study of Selective Prediction

Ezequiel Juarez Garcia, ejuarezg.com

This short, deliberately synthetic study asks a practical question: when should a classifier decline to answer? We score 120 toy predictions, retain only examples above a confidence threshold, and compare coverage with accuracy. The exercise is not an empirical claim. It is a compact demonstration of Nota's academic-paper theme, document-aware definitions, cross-references, mathematics, figures, and bibliography.

1 Motivation

Probability estimates are useful only when their meaning survives contact with a decision. A model that reports $0.8$ confidence should be correct about eight times in ten on comparable examples. Calibration measures this agreement [ $Brier 1950$ ][ $Guo et al. 2017$ ].

For a confidence score $s (x) \in [0, 1]$ and threshold $τ$ , define a

A selective predictor returns the model's answer only when

s (x) \geq τ

; otherwise it abstains. Its coverage is the fraction of examples receiving an answer.

Click the reference to selective predictor whenever the term appears. Nota keeps its definition available in context instead of forcing the reader to search backward through the paper.

For predictions partitioned into 6 confidence bins, expected calibration error is

ECE = b = 1 \sum 6 \frac{∣ B _{b} ∣}{n} ∣ acc (B_{b}) - conf (B_{b}) ∣ .

The formula is static and readable without JavaScript.

2 The Coverage--Accuracy Tradeoff

Raising $τ$ removes low-confidence cases. The resulting relationship is structural rather than empirical:

Theorem 1: Monotone coverage

Let

A_{τ} = {x ∣ s (x) \geq τ}

be the answered set. If

τ_{1} \leq τ_{2}

, then

A_{τ_{2}} \subseteq A_{τ_{1}}

and therefore coverage cannot increase as the threshold rises.

The small table in Figure 1 illustrates Theorem 1. Its accuracy values are invented for this demonstration, while the decline in coverage follows directly from the theorem.

Threshold $τ$	Coverage	Accuracy on answered cases
0.50	100%	78%
0.65	79%	86%
0.80	42%	93%

Figure 1:

A synthetic selective-prediction result. Higher thresholds answer fewer questions but can reserve answers for easier cases.

The operating point is therefore a policy choice, not merely a model metric. In a low-cost recommendation system, broad coverage may matter most. In a high-consequence workflow, abstention can create room for review, a second model, or a request for more information.

3 Conclusion

A confidence score becomes useful when it controls an explicit action. Even this tiny example benefits from a paper's structure: the abstract states the claim, selective predictor fixes the terminology, Theorem 1 separates a guarantee from the toy observations, and the bibliography preserves provenance. Those relationships are what Nota adds beyond styling alone.

References

Glenn W. Brier

. 1950. Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review (1).

Chuan Guo

Geoff Pleiss

Yu Sun

, and

Kilian Q. Weinberger

. 2017. On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning.