Model calibration

A model is calibrated when the confidence it states matches its actual hit rate: of all answers given with 0.9 confidence, about 90% should be correct.

The straight line is the calibrated model; the lower one is an overconfident model, which says 0.9 and is right only 60% of the time. LLMs tend to fall into this second group.

Why it matters: a system that is right 95% of the time can only be automated if it can point out which 5% of cases are doubtful. With calibration, a simple threshold does the job:

if answer.confidence >= 0.85:
    act(answer.choice)
else:
    escalate(case)  # a person or a bigger model

How to check it: take a set of examples with known answers, group the predictions by confidence band and compare each band with its hit rate. Common metrics are ECE (Expected Calibration Error, the average gap between confidence and accuracy) and the Brier score.

Seen in JEV, which is trained precisely to have calibrated probabilities (a claim that still needs independent verification).

• Reading: On Calibration of Modern Neural Networks (Guo et al., 2017)