What exactly is different about Attention Residuals vs standard Transformer residuals?

Standard residuals use fixed 1:1 additive weights where each layer only sees the previous layer. Attention Residuals replace this with learnable depth-wise attention weights—each layer dynamically selects information from ALL prior layers with learned importance weights. Block AttnRes partitions layers into blocks (e.g., 8 layers each) to make this computationally practical, reducing overhead from O(L²) to O(L·k).

What does '1.25x compute advantage' mean in practice?

It means an AttnRes model trained with 100B tokens achieves the same performance as a standard Transformer trained with 125B tokens (or equivalently: same performance with ~20% less compute). At frontier training scale ($1B+ budgets), this translates to hundreds of millions in potential savings—or proportionally stronger models for the same budget.

Has the Attention Residuals result been independently verified?

Not yet as of March 2026. Results are based on Kimi's internal testing on their 48B parameter MoE architecture (Kimi Linear). The paper and code are published on GitHub for community reproduction. The AI research community is actively attempting to verify whether the 1.25x advantage holds on other architectures (dense models, different parameter scales). Independent validation is the critical next step.

— AI DAILY

Sources

MarkTechPost
Reddit r/LocalLLaMA
NerdSchalk