Why did attention residuals get so much attention

I've thought about it for a while now and I don't really understand why the attention residuals paper got as much attention as it did online. Online being the imperative word to be fair. I likely don't know what I don't know here. First, an overview of the paper.

When training a model, on the depth axis, there are an increasing number of layers leading:

• information in earlier layers to be increasingly hidden
• the relative influence of any individual layer diluting
• the residual stream accumulating every layer with unit weight (+1)

Attention residuals are essentially a fix to this phenomenon that the paper calls PreNorm Dilution by applying softmax attention over the outputs of previous layers. In this video from Kimi founder, he speaks to how this is the attention paper rotated 90 degrees ie softmax attention moved from sequence/token to depth/layer. Block AttnRes makes this less computationally expensive by summarizing layers within blocks and creating block level summaries. With so much of recent improvement in AI attributed entirely to scaling laws + more compute, it is a novel contribution on the relatively under explored axis of depth.

Here are some reasons why it's unclear to me whether the findings here are particularly meaningful:

1. In the paper, they run scaling-law experiments on 5 smaller Kimi-Linear-style MoE models that range from 194M to 528M active params and a larger experiment on Kimi Linear itself at 3B active params. Kimi Linear is ~11× smaller than Kimi K2 and ~12× smaller than DeepSeek-V3 on active params. So it’s unclear if the gains will hold at the scale of the most intelligent open-weight models, let alone frontier models.

2. Evals:

   1. Most of the evals in the paper are old and largely saturated (MMLU, GPQA-Diamonds, MMLU-Pro), so gains on them carry little signal versus others.
   2. Code generation is the category of knowledge work where LLMs have proven most useful so far. Across evals for code, HumanEval and MBPP improvements are minimal versus the reasoning benchmark that they led with.

3. Prenorm dilution increases with depth and layer count, not parameters. Among the largest OSS MoEs, (DeepSeek, Kimi, Llama 4) the models appear to scale through expert count and width while holding depth roughly flat. Depth does not reliably increase with scale, and in some cases drops.

   I'd expect this means that Attn Res is not that impactful for the largest frontier models.

   ---