One conformal band across cell types hides a 23× spread in the band each type needs

Conformal prediction gives a surrogate a calibrated interval with a coverage guarantee, and the standard recipe fits one quantile over a held-out set. We did that, shipped it, and then asked whether one number was the right shape.

The measurement

Fitting the conformal quantile separately per SFQ cell type:

A 23× spread. The global fit lands somewhere in between and is wrong at both ends — far too wide for the easy cell, far too narrow for the hard one.

Too wide is a nuisance. Too narrow is the dangerous one, because it means serving a prediction as trustworthy when it is not. Splitting the band cut the cells we wrongly served as functional from 11 to 3.

Why the single band was never going to work

The coverage guarantee is marginal: it holds on average across whatever population you calibrated on. If that population mixes objects whose difficulty differs by an order of magnitude, average coverage is met while conditional coverage fails on both halves — and the half where it fails unsafely is the one you care about.

This is not a flaw in conformal prediction. It is what marginal coverage means, and the fix is the standard one: Mondrian, or per-group, conformal. The point is that the grouping is a modelling decision nobody makes explicitly, and the default of "one band" is a decision too.

What to check in your own setup

Fit the quantile separately on whatever groups you suspect differ, and look at the spread before deciding one band is enough. If the ratio is small, the global fit is fine and you have cheap evidence for it. If it is 23×, you have been shipping a number that is wrong in both directions at once.

A related discovery from the same audit is worth stating: for a long time our conformal block was never fitted by any committed code at all — the only writer was a test fixture. The band was real, and it was not reproducible. That is recorded separately in the same audit.