How much better could a model still get?

One loss curve, two different questions

When a model makes a mistake, is it missing a pattern—or is the observation itself ambiguous? Those situations call for different responses. More training might help with the first; better measurements might be needed for the second.

OracleFlow studies this distinction in a controlled generative setting. The associated paper, Beyond the Loss Curve, investigates scaling, posterior targets, distribution shift, and active learning.

A distribution with an answer you can inspect

Class-conditional normalizing flows provide a specified generative distribution whose class probabilities can be evaluated. This makes it possible to compare a learner with the posterior of that distribution, rather than only with a single observed label.

“Exact” here is relative to the specified oracle distribution. It does not mean we have recovered the true probabilities of the real world. Sampling and evaluation must describe the same distribution for that interpretation to hold.

Make the gap measurable

For a known posterior p and a prediction q, expected log loss is the entropy of p plus the forward KL divergence from p to q. Entropy measures the ambiguity that remains even with the right answer distribution; KL measures the additional cost of the model’s prediction.

The example below lets you vary those two distributions independently. It is a binary thought experiment, not an OracleFlow benchmark result. It illustrates why evaluating uncertainty requires knowing what distribution the numbers refer to.

Papers & resources

Beyond the Loss Curve

Scaling Laws, Active Learning, and the Limits of Learning from Exact Posteriors · arXiv, 2026