TriGLU v0.3.0 · results
A 40-layer decoder keeps its quality, its benchmark capability and its long-context retrieval with 57.5% of its attention layers replaced by token-local blocks — and gives back the same share of key/value cache and roughly double the prefill throughput. The attention that cannot be removed is at the front of the network.
−57.5%
key/value cache
34 KB/token vs 80
1.99×
prefill throughput
measured at 4K
+49%
cached decode
measured at 4K
57.5%
of attention removed
at better quality than baseline
41 training runs across 25 layer plans, each a 152-million-parameter, 40-layer decoder trained on two billion FineWeb-Edu tokens at 4K context. Every plan has an identical parameter count, so only the attention map differs. Headline comparisons are repeated at three seeds and reported against the resulting spread.
An attention layer costs memory and compute twice over at long context: it holds a key/value cache that grows linearly with the context, and its share of prefill grows with the square of it. A token-local block holds no cache and grows linearly. Removing 57.5% of the attention layers therefore removes 57.5% of the cache exactly — not approximately, and not less as context grows.
Each point is a separately trained 40-layer model, identical in parameter count and token budget, differing only in which layers keep attention. The headline configurations were each trained at three seeds so the differences can be read against run-to-run variation rather than assumed.
Validation loss alone can hide a capability that has quietly gone missing, so the same checkpoints were scored on the standard LM Evaluation Harness — LAMBADA, which requires resolving a dependency across a whole passage, and WikiText perplexity.
The reason to remove attention is long-context serving cost, so the configurations were also asked to retrieve a fact buried at varying depths in a haystack — the capability most obviously at risk when attention layers are taken away.
Two findings, one useful and one cautionary, came out of holding the attention budget fixed and moving it around.
Taking the same small budget of attention layers and moving it from the front of the network to the end produced the worst model in the study, by a wide margin. The early layers build a representation the rest of the network refines; adding attention back near the output recovers next-word prediction but never general quality.
Within the region that can be replaced, several spacing schemes we expected to matter did not separate from even spacing once each was run at three seeds. Keeping a solid block of early attention matters; the schedule of what follows it, at this scale, does not.
These are controlled results at 152 million parameters and a 4K trained context. They do not establish equivalence at substantially larger scale, and the 128,000-token cache figures are exact arithmetic on a quantity that scales linearly — not a measurement of quality at that length.
Run-to-run variation at this scale turned out to be larger than most of the differences we were originally chasing. Two effects that looked real at a single seed did not survive replication, and are reported as the null results they are. Where a comparison sits near the significance threshold we say so rather than rounding it into a claim.
The token-local block at the center of the study belongs to a known family of gated units. The contribution here is the controlled map — how much attention can go, from where, and what it is worth — not the algebra.
Every number on this page is backed by a raw record. The release ships the implementation, the generated configuration for all 25 layer plans, and an evidence archive covering all 41 runs — resolved configs, metric streams, environment metadata, data hashes, retrieval results and the benchmark records behind the cost figures — along with the analysis tooling that produced the reported verdicts. Apache-2.0 licensed.
Code & study on GitHub → Release v0.3.0 → Archived record (DOI) → Back to news →