PriorBPB: Reducing the Answer Length Confound of Bits-per-Byte While Improving Accuracy Prediction

Pre-training evaluations and the bits-per-byte metric

Before a large language model can chat or follow instructions, it is first pre-trained by learning to predict the next piece of text across trillions of words. When building models from scratch, pre-training can make up most of a model development’s compute budget. For example, pre-training our recent Kolibri model took about 392,000 GPU hours on 768 NVIDIA B200 GPUs, roughly an order of magnitude more than the compute of its final reinforcement learning run. Teams like ours therefore run many smaller experiments first to decide things like which data to train on or what architecture to pick. Each of those experiments needs a score that says whether a change helped.

Pre-trained (base) models usually can’t do what users care about, such as holding a conversation or carrying out multi-step tasks, because those abilities mostly come from post-training. Scoring base models on accuracy for such tasks can give flat or noisy signals (Schaeffer et al., 2023) and depend heavily on the prompt or answer formatting (Gu et al., 2025). So model builders want to find measurements which are more fine-grained and change more smoothly as models improve. If that score is biased, a team can pick the wrong recipe. In this post we show that bits-per-byte (BPB), a widely used score of this kind (Gao et al., 2020; Magnusson et al., 2024; Olmo Team, 2026), carries a bias from something which should be generally unrelated to model quality: the length of the correct answer. Our open-source evaluation package, eval-framework, includes many pre-training evaluations of this kind, and also includes a proposed fix, PriorBPB.

To compute BPB, give the model a prompt and score it on a pre-defined gold answer string one token at a time. For each token, take its model-assigned conditional probability and sum $-\log_2$ of those probabilities. The total, $L$, is the model’s surprisal on the answer in bits, i.e., the number of bits needed to describe the gold answer under the model’s own predictive distribution. Dividing $L$ by the number of tokens would leave scores incomparable across models with different tokenisers, so BPB divides instead by the answer’s UTF-8 byte length $n$, giving the per-item score $\mathrm{BPB}_i = L_i / n_i$. A lower BPB is interpreted as the model predicting the gold answer string more fluently, given the prompt string. Over a task’s $N$ items, evaluation frameworks typically report the mean of per-item ratios BPB, $(1/N) \sum_i L_i / n_i$, or the corpus BPB as $\sum_i L_i / \sum_i n_i$, where $L_i$ is the total surprisal (in bits) of the answer string of item $i$ and $n_i$ is the string length in bytes.

Shorter responses are evaluated more harshly than longer ones

The following table shows BPB scores for Qwen3-4B and OLMo-3-7B on answer variants to the question What is the capital of France?. Despite each answer variant having the same semantics, the more verbose answers receive increasingly better scores (lower is better).

Within one task (MMLU full text) scored with Qwen3-4B, we sorted the items by gold answer length and split them into five length groups. The group with the shortest answers (6 bytes on average) scores about 11× worse (11× higher mean-of-ratios BPB) than the group with the longest answers (96 bytes on average).

Summary

The problem. BPB divides the model’s surprisal on a gold answer by that answer’s UTF-8 byte length. Bytes avoid the tokeniser dependence of bits-per-token, but answer length remains a property of the task and of each of its samples. Any cost the model pays once per item is divided by whatever length the benchmark author(s) chose. Tasks with short answers then score worse than tasks with long answers at the same per-byte difficulty, and short items score worse than long items inside every task.

The finding. Fit each item’s bits as a fixed overhead $\alpha$ plus a rate $\beta$ per byte, by least squares. Corpus BPB then equals $\alpha / \mathrm{mean}(n) + \beta$ exactly, with $\mathrm{mean}(n)$ the task’s mean gold length. This equality is algebraic and holds whatever the residuals might be. This makes any fixed cost the model pays per item included in the score, divided by the task’s mean answer length. Empirically, that fixed cost accounts for a large fraction of BPB scores.

In the table above, the longest answer costs Qwen3-4B only about 15% more bits than “It is Paris.” (27.7 vs 24 bits), but it is almost 3× as long (36 vs 13 bytes). Bits appear to grow much more slowly than length, so dividing by length rewards longer answers in general. We see the same pattern across benchmarks: a large part of the cost doesn’t grow with the answer’s length, and BPB spreads it over however many bytes the answer has. Across our 19-task suite, the standard BPB metrics (mean-of-ratios and corpus BPB) order tasks largely by their mean gold length, and how much of the cost actually scales with answer length varies from task to task, meaning no simple normalisation method fits all of them. The mean-of-ratios BPB aggregation adds a further, task-dependent inflation. On Qwen3 models, the combined effect is large enough to produce an apparent, but misleading, inverse-scaling trend between the 0.6B, 1.7B, and 4B models.

The fix. We introduce PriorBPB. For each model and task, we estimate a marginal per-byte cost curve from the per-token surprisal, then average that curve under one length prior shared across tasks. Every byte of every gold answer contributes, and with each task’s own length distribution as the prior, PriorBPB equals corpus BPB.

The result. Across more than 60 checkpoints from five model families, PriorBPB is more length-neutral than standard BPB metrics. The dependence on gold answer length which dominates standard BPB scores largely disappears, both in how tasks are ordered against each other and in how short and long answers compare within a task. PriorBPB also correlates with accuracy more closely than standard BPB metrics, somewhat on base checkpoints and substantially on instruct checkpoints, making it also a more useful proxy metric for evaluating pre-training recipes.

Answer length correlates with BPB scores

Fit total surprisal against gold answer length, per model-task with $L_i \approx \alpha + \beta n_i$. This affine form is an observation of Oostermeijer (2026), who documents it across tokenisers and benchmarks and builds Bayesian accuracy on it, a multiple-choice metric that subtracts a fitted length trend rather than dividing by length. The intercept $\alpha$ is a length-independent cost, and it is largely contributed to by the first answer token surprisal. The slope $\beta$ is the per-byte continuation cost rate for the remaining tokens in the answer string.

The least-squares residuals sum to zero, which means corpus BPB, total bits over total bytes, can be written as

The second term, $\beta$, is a per-byte rate. The other term is a fixed cost divided by the task’s mean answer length, so it decays as answers get longer. Two tasks with identical $\beta$ but different $\mathrm{mean}(n)$ therefore receive different scores due to the first term, meaning a task with shorter answers can score higher and therefore worse via only the shortness of the answer lengths and not because they are more ‘difficult’ from a model knowledge perspective.

Across the 19 tasks tested on multiple models, the fixed-cost term $\alpha / \mathrm{mean}(n)$ correlates strongly with mean gold length, while the per-byte rate $\beta$ barely does.

Are long answers just easier?

Another possible explanation for this phenomenon is that tasks with long answers are easier. We test this by holding the model and task fixed, binning a single task’s own items by gold length, and comparing the shortest fifth with the longest fifth of answer strings. The short items score worse on every task we tested, typically about twice as badly (median top/bottom quintile ratio ≈ 0.43). So when Qwen3-0.6B scores an order of magnitude better on MMLU full text than on SQuAD2, that might say more about the gold-length distributions of those tasks than about their relative difficulty.

The first token is the most expensive

Per-token traces show the first gold token often costing an order of magnitude more than the tokens that follow it. On SQuAD2 the later tokens cost very little, probably because the answer can be copied from the passage. If we delete the answer-bearing sentence from the prompt, the total bits rise sharply, so the relatively flat $\beta$ we observed is quite likely some form of ‘copyability’ effect.

How much of the cost scales with length therefore varies widely by task (even when tested on the same model), from almost nothing (e.g., SQuAD2) to the majority of the apparent signal (e.g., OLMo-3-7B reaches 96% on GSM8K). As a consequence, any single, constant length normalisation method like dividing by the number of tokens or bytes cannot fit all cases.

Mean-of-ratios BPB adds a second error

Some model evaluation reports use the mean of per-item ratios, $(1/N) \sum_i L_i/n_i$, which is a mean of rates and not itself a rate. Since $E[1/n] \ge 1/\mathrm{mean}(n)$, it includes a larger fixed-cost term than corpus BPB and overweights short items. In our suite, this bias inflates the reported score, for some model-task combinations by >100%. Mean-of-ratios BPB scores are therefore more length-dependent than corpus BPB and are inflated by a task-dependent amount. Switching to corpus BPB helps combat the aggregation part of this inflation.

An apparent inverse-scaling trend

Under corpus BPB, the smallest model in the Qwen3-instruct series of models, 0.6B, gets a better score on 15 of 19 tasks we test, compared to its 1.7B and 4B counterparts.

Decomposing the cost with the least-squares length fit, the fixed cost $\alpha$ is larger at 4B than at 0.6B on every task, though by very different amounts (e.g., 1.2× on SQuAD2 but 7× on GSM8K), while the per-byte rate $\beta$ shows no such trend. On a token-per-token basis, we find that the first token gets more expensive on average than later ones.

Corpus BPB mixes the fixed cost into that score through $\alpha / \mathrm{mean}(n)$, so a growing intercept appears as an inverse-scaling result even though it may not reflect an ‘actual’ capability difference.

We suspect this growth in $\alpha$ might appear due to prompt format rather than model quality. These completion-style gold answers are wrapped in an instruct chat template, and a larger, better instruction-tuned model may pay more to emit the bare continuation which such pre-training benchmarks expect. Re-scoring the Qwen3 base checkpoints without a chat template attenuates but does not remove the apparent fixed-cost growth, though base and instruct checkpoints differ in more than the template alone. We also ran a template-only control (with enable_thinking=False), which did not remove the anomaly, so the format-mismatch explanation remains a possible hypothesis. The metric-level point holds in any case, however, since corpus BPB mixes $\alpha$ and $\beta$ through $\alpha / \mathrm{mean}(n)$, and whatever inflates the fixed cost appears among these models as an inverse-scaling trend.

Introducing PriorBPB

Our goal is to measure the bits the model needs per gold byte, whatever the answer’s length. Although $\beta$ could appear as a natural proxy, the affine fit models only the fixed cost and the length term. We therefore choose to estimate the cost curve non-parametrically.

From per-token bits and per-token byte lengths we build the byte-indexed cumulative cost $C_i(k)$, the bits required to cover the first $k$ content bytes of item $i$. The marginal per-byte cost curve for each model and task, $R(k)$, is then the mean of $C_i(k) - C_i(k-1)$ over the items with $n_i \ge k$. $R(k)$ is therefore a non-negative-weighted mean of non-negative marginal bits, and so it is monotone in those costs1.

To get a scalar, we average the curve against a fixed length prior $s(k) = P(N \ge k)$ that is the same for every task, to arrive at

We use a geometric prior, $s(k) = (1 - 1/\mu)^{k-1}$, with median length $\mu = 111$ bytes, the pooled median answer length across the evaluations suite we tested. Because $\mu$ is set from the length distribution alone, before any score is computed, it cannot be tuned to the results. Every position has positive weight, so the late bytes of a long gold answer are weighted by the shared prior rather than by how many items in that task happen to be that long.

If the weights are instead set to a task’s own length-survival function, $s(k) = P_{\mathrm{data}}(N \ge k)$, PriorBPB equals corpus BPB exactly. The two are the same weighted average of the same cost curve, differing only in whether the weights come from the data or are fixed in advance. Because PriorBPB keeps the byte denominator, it inherits the tokeniser-independence of BPB.

Results

On 19 length-varying tasks, we compared PriorBPB with corpus and mean-of-ratios BPB on Qwen3-instruct (0.6B/1.7B/4B) and OLMo-3-7B-Instruct, with further checks on Qwen3-Base, SmolLM2, Qwen2.5, OLMo-2, and Pythia models.

Within-task. Holding task and model fixed, we split a task’s items into gold-length quintiles. Under corpus and mean-of-ratios BPB, the shortest-gold quintile scores far worse than the longest (top/bottom quintile ratio ~0.43). PriorBPB’s per-byte rate curve $R(k)$ — the quantity it integrates — is by contrast essentially length-invariant: comparing the long- and short-gold quintiles at matched content-byte positions under the one shared prior, the top/bottom ratio is median 0.98 (95% credible interval [0.94, 1.01]). That is, at a fixed content position the per-byte cost barely depends on the gold string’s eventual length.

Cross-task. Corpus and mean-of-ratios BPB correlate strongly with mean gold length. PriorBPB’s dependence is much weaker (mean |ρ| 0.68 → 0.18 for 62 base checkpoints), though not zero (e.g., −0.34 on Qwen3-0.6B). A per-task prior (using a geometric prior with $\mu$ set to each task’s own median length) behaves much like corpus BPB, leaving the bias mostly intact, which suggests the shared prior is what brings about this benefit.

Correlation with accuracy. In a paired, clustered Bayesian comparison against mean-of-ratios BPB, PriorBPB follows accuracy more closely along model-capability ladders. The gain is small on base checkpoints and larger on instruct checkpoints.

Primary target: raw accuracy, statistic −Spearman. Higher means the metric correlates with accuracy more closely.

Limitations

$R(k)$ at position $k$ averages only over items that reach byte $k$, so its late-position values describe a length-selected sub-population. This survivorship bias may contribute to some of the residual length correlation we see in PriorBPB. Dropping short items outright does not remove it.

The residual also grows in magnitude with the prior mean (from about $-0.01$ at $\mu = 40$ to $-0.41$ at $\mu = 160$), while the cross-task score ordering agrees with that at the frozen $\mu = 111$ (Spearman $\ge 0.84$ for $\mu \ge 60$). So while PriorBPB is more length-neutral, some residual length dependence remains, depending on the choice of $\mu$, and we therefore report $\mu$.

Recommendations

1. Do not naïvely average mean-of-ratios or corpus BPB across tasks, as it weights tasks by length.
2. Report corpus BPB only within a fixed task when you are happy to assume (and have, ideally, verified) a stable fixed cost $\alpha$ across models, since the length mix will also be shared across the models and $\alpha / \mathrm{mean}(n)$ will be a common offset.
3. When a single cross-task scalar is needed or unavoidable, use PriorBPB with a frozen, shared length prior, stating the prior.
4. Do not report mean-of-ratios or corpus BPB without something like the $(\alpha, \beta)$ decomposition alongside it to differentiate fixed costs from per-byte costs (except within a fixed task as above), especially since mean-of-ratios BPB is not a rate and can inflate scores by ∼2×.

Conclusion

Moving the denominator from tokens to bytes removes the tokeniser dependence of the denominator but leaves a length dependence. Because part of each item’s cost does not scale with answer length, cross-task BPB partly reflects the length composition of the benchmarks, and in one case we find this is enough to produce an apparent inverse-scaling trend. PriorBPB states one shared length prior and integrates a per-byte cost curve against it, recovering corpus BPB as the biased special case in which the prior is each task’s own length distribution. It substantially reduces, though does not entirely eliminate, the length dependence of BPB evaluations while improving their accuracy prediction.

Try out PriorBPB yourself in our Apache 2.0 evaluations package at https://github.com/Aleph-Alpha-Research/eval-framework/.

If you found this interesting and would like to read more, check out the full paper on OpenReview:

@inproceedings{burns2026priorbpb,
  title     = {Prior{BPB}: Reducing the Answer Length Confound of Bits-per-Byte While Improving Accuracy Prediction},
  author    = {Thomas F Burns and Prabhu Teja S and Frank Schneider and Martin Reinhardt},
  booktitle = {NeurIPS 2026 Trust-AI-Eval Workshop},
  year      = {2026},
  url       = {https://openreview.net/forum?id=fiMZJJzZ7g}
}

Acknowledgements

This work arose out of a team effort with many members of Aleph Alpha. In particular, we wish to thank all contributors to the eval-framework repository, as well as evaluations, infrastructure, and model factory teams for helping us run evaluations smoothly and reproducibly. We are grateful to Koen Oostermeijer and Letitia Parcalabescu for research discussions, Fabien Benureau, Yasser Jadidi, Peter Rodenkirch Nfissi, Paige Reddington, Helena Treeck, and Samuel Weinbach for careful reviews and valuable feedback on this blog post, Svenja Fahlisch for coordination, Noé Beckerle for design assistance, and Alexander Wortmeier for publishing it on this website.

This project has received funding from the European Union’s Digital Europe Programme under grant agreement No. 101195233 (OpenEuroLLM).

The contents of this publication are the sole responsibility of the OpenEuroLLM consortium and do not necessarily reflect the opinion of the European Union.