DeepSeek, the Chinese AI lab whose R1 reasoning model rattled the industry in early 2025, has published a paper that questions one of the oldest design decisions in modern deep learning. The paper, “mHC: Manifold-Constrained Hyper-Connections,” was posted to arXiv on December 31, 2025. It proposes a way to widen the flow of information between the layers of a neural network while keeping training stable, a combination earlier attempts had failed to achieve.
If the approach holds up at larger scale, it would offer model builders a lever other than simply adding more layers, data and hardware.
The decade-old design it targets
Large language models are built from stacks of layers. An input passes through the first layer, is transformed, and moves to the next, until an answer emerges. During training, an error signal called a gradient flows backward through the same stack. Researchers found a decade ago that this signal tends to fade or blow up as it passes through many layers, which made very deep networks nearly impossible to train.
The fix was the residual connection, introduced in 2015 and now present in essentially every large model. A residual connection gives each layer a shortcut that passes its input straight through alongside its output, so information and gradients can travel cleanly across dozens or hundreds of layers. The design works and scales, and over time it hardened from a choice into infrastructure. Almost everything else in model architecture has changed; this part has not.
The trade-off, as the DeepSeek authors describe it, is that all information between layers travels through a single, fixed stream. That was not a visible problem while gains came from bigger models and more data. It becomes a constraint when models are asked to do deeper multi-step reasoning.
Hyper-connections and why they stalled
One proposed answer, called hyper-connections, replaced the single residual stream with several parallel streams that can exchange information with one another through learned mixing matrices. On paper this widens the network’s internal bandwidth. In practice, the authors note, the unconstrained mixing let signals grow or shrink without bound as training went on. Runs could proceed normally for a long time and then collapse, a risk that is unacceptable when a single training run costs millions of dollars. The idea never reached mainstream use.
What mHC changes
DeepSeek’s method keeps the multiple streams but constrains how they mix. The mixing matrices are projected onto a specific mathematical space, a manifold, in which the total signal strength is preserved: information can be redistributed across streams but not amplified or attenuated overall. The paper uses the Sinkhorn-Knopp algorithm, an iterative procedure for normalizing matrices, to keep the mixing matrices on that manifold throughout training.
The result, according to the authors, is stability that is guaranteed by the structure of the method rather than by careful tuning of hyperparameters. The network retains the robustness of residual connections while gaining several times the internal capacity for passing information between layers.
Results and engineering
DeepSeek tested the approach by training models with 3 billion, 9 billion and 27 billion parameters. Across a set of standard benchmarks, including GSM8K for grade-school math, BBH for reasoning and MMLU for broad knowledge, the paper reports that mHC models outperformed both conventional baselines and models using unconstrained hyper-connections, with the largest gains on reasoning and math tasks.
Widening the residual stream normally carries a heavy memory and compute cost. The paper describes a reworked training stack, with custom GPU kernels, selective recomputation of intermediate values and overlapping of communication with computation, that allowed a fourfold expansion of the internal data flow at a cost of about 6.7 percent additional training time. Memory bandwidth is one of the tightest constraints in training frontier models, so that overhead figure is central to whether the technique is practical.
Why it matters
The paper continues a pattern DeepSeek established with R1: pursuing efficiency and architectural changes rather than relying only on larger hardware budgets, and publishing the results openly. The findings have not yet been independently reproduced at frontier scale, and a 27-billion-parameter model is small by current standards. But the broader question the work raises is a pointed one for the field. If widening information flow between layers proves more effective than stacking more of them, other parts of the transformer design that are treated as settled may deserve a second look.
