August 16, 2026
Random splits overstate surrogate accuracy on qubit design data by 31×
The finding
A gradient-boosted model scores 0.12% under random K-fold and 3.81% under a grouped split that holds out whole geometries — the same model, the same data, a 31× difference. The field read this as needing more data. It is a split-methodology problem.
Superconducting qubit design has a small amount of public data and a lot of people trying to learn from it. The usual diagnosis for why those attempts underperform is that the databases are too small. We think the evidence points somewhere else.
The measurement
Two ways of splitting the same SQuADDS capacitance data, same model, same targets:
| Split | Gradient-boosted trees |
|---|---|
| Random K-fold | 0.12% |
Grouped (whole cross_length values held out) | 3.81% |
A factor of 31.
Random K-fold on a dense 3-D grid asks only for interpolation between immediate neighbours. Nearly every point has a near-twin on the other side of the split, so the model is graded on a question it cannot get wrong. The grouped split holds out whole geometries, so the model has to predict a device it has never seen at any claw length — which is what a designer actually needs.
Why it matters more than it looks
That gap is the overfitting the SQuADDS authors reported, quantified. And the fix was not more coverage. Capacitance is a smooth, near-power-law function of geometry, so a degree-4 polynomial in log–log space — a physics-motivated interpolation, fitted globally rather than locally — is 12.7× better where it matters.
More data would not have closed a gap that came from how the data was split.
We found the same disease in our own work, which is the only reason we trust the result. An internal benchmark reported 0.29% under a random fold where the grouped number is 3.85% — a 13.4× overstatement, in a card we had shipped. Both were corrected.
What we are claiming, and what we are not
This is not a claim that our model is better than someone else's. We ran that comparison too, pre-registered, and the honest answer is unheadline: roughly 1.2× and 1.8× on two dimensions, with confidence intervals excluding parity — and our p95 on one of them is worse, so we carry the heavier tail.
The methodology result is the one that stands on its own. If you are benchmarking a surrogate on device geometry, a random split will flatter it, and the factor is large enough to reverse conclusions. Hold out whole geometries. Report both numbers if you like, but quote the grouped one.