Beyond token fragmentation: byteification put to the test in Nature

Conventional large language models process text by breaking it into word fragments called subword tokens. The approach is computationally efficient, but the token structure hides individual characters, which gets in the way of working with source code, biological sequences and spelling tasks. To get past this limit without paying for retraining from scratch, a research team that includes the Allen Institute for AI, the University of Cambridge and LMU Munich has described a method called byteification in Nature. It adapts existing architectures to process raw bytes directly, using, according to the authors, less than one percent of a normal pre-training budget.

The conversion takes 49.1 billion tokens in total, split into two stages. In the first stage, 9.8 billion tokens, the global model stays frozen and only the new byte-level components are trained. The second stage, 39.3 billion tokens, uses end-to-end training. The architecture pairs the original transformer with local mLSTM encoders and decoders and a predictor for byte-patch boundaries. Across the four converted systems — Bolmo 1B, Bolmo 7B, Blama 8B and Bwen 8B — the change in parameter count relative to the base versions stays minimal. Nature credits Bolmo 7B with an absolute gain of 16.5 percentage points on STEM tasks over BLT 7B, a byte-level model trained from scratch. As Valentin Hofmann, junior professor at LMU and the study's last author, put it, conventional models struggle with tasks that require character-level skills, while the converted systems show a clear improvement on this front.

Although the training code and source data are publicly available, the technical documentation gives specific reasons for caution. Testing stopped at models of at most 8 billion parameters, so the method's effectiveness on the largest frontier architectures is unverified. In the controlled comparison, the architecture also scores worse on Code pass@1 than simple continued training. Some secondary summaries attribute the +16.5 to Bwen 8B, while Nature assigns it to Bolmo 7B. Others cite the University of Edinburgh, Edoardo M. Ponti's affiliation in the preprint, whereas Nature lists Imperial College London. Ai2's page does not state the licence for the weights, and Nature does not report the one for the code.

Reusing existing structures to cut training costs is a pragmatic and rigorous approach. What remains to be seen is whether moving away from tokens holds up at larger compute scales without hurting efficiency in code generation. — Olya