What is Attention Residuals and how is it different from normal Transformers?

Traditional Transformers use fixed additive residuals (each layer's output + previous layer, fixed 1:1 weighting). Attention Residuals replaces this with a depth-wise attention mechanism that lets each layer selectively weight contributions from ALL prior layers. Block AttnRes makes this practical by partitioning layers into blocks. The result: ~1.25x compute efficiency over standard PreNorm baselines.

How significant is the 1.25x compute advantage?

Very significant at scale. In absolute terms: training a $100M model with AttnRes delivers the same performance as a $125M model trained conventionally—a $25M saving. At frontier scale ($1B+ training budgets), this translates to $200M+ in savings per run. The advantage also amplifies at larger scales due to lower scaling loss.

Has Attention Residuals been independently verified?

Not yet as of March 2026. The results are based on Kimi's own testing on their MoE architecture. The paper and code have been published on GitHub for community reproduction. Independent verification across different architectures and scales is needed before the community can fully endorse the claims.

— AI DAILY

Sources

Kimi/Moonshot on X
MarkTechPost