CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry

Abstract

Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value. We present two efficiency methods that exploit this asymmetry. CascadeFormer tapers width with depth to match the uneven information flow across layers, achieving comparable perplexity to a uniform baseline at the same training budget while reducing latency by 8.6% and increasing throughput by 9.4%. CascadeFlow Pruning removes layers using accumulated training gradients and outperforms standard heuristics on perplexity and rank-stability (while remaining competitive on downstream accuracy), without expensive post hoc analysis. To motivate these methods, we propose Gradient Fan-in Asymmetry (GFA) as a structural account of why deeper layers contribute less. In Pre-LayerNorm residual stacks, the gradient at a layer is the sum of an identity path and all downstream functional paths, producing a gradient fan-in that decays linearly with depth (and quadratically under deep supervision), yielding rich signals early and sparse for later layers. We provide correlational and interventional evidence for GFA on models trained from scratch up to 1.2B parameters: across Transformers and ResNets, accumulated training gradients follow the theoretical fan-in and are associated with post hoc layer importance; and two interventional experiments are consistent with structure (not magnitude) as the bottleneck, since equalizing per-layer gradient norms does not restore late-layer value, whereas increasing downstream path counts via parameter-shared repetition restores and elevates their impact. Whether gradient magnitude proxies fan-in beyond high-rank regimes, and how these dynamics behave at the 100B+ scale, remain open questions.

BibTeX

@misc{ahmad2025cascadeformer,
  title        = {CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry},
  author       = {Ahmad, Huzama and Nam, Cao Viet Hai and Yun, Se-Young},
  howpublished = {arXiv},
  year         = {2025},
  url          = {https://arxiv.org/abs/2606.26538}
}