HC outperforms mHC across model depth

Oct 3, 2026

mHC solves training instabilities in HC, but collapses effective rank to 1 at deeper layers.

HC & mHC

Overview

Residual connections are the pathways for transformers in which updates from transformer layers get added to. This serves as a good optimization technique and allows clean differentiation of gradients during backprop. However, it causes deeper layers' updates to dilute and become less effective.

For better depth utilization, DeepSeek introduced Hyper Connections (HC), which is a redesign of the residual stream. HC widens the residual stream by having many parallel residual streams, aggregating them into a single input into attention/ffn/norm layers, and then expanding to parallel streams again before applying an $n \times n$ map to the block update.

However, these $n \times n$ maps can cause training instability as they are unconstrained matrices. mHC solves the training instability by constraining the $n \times n$ map to a manifold.

These methods work well for regular training so far. But recent research (DepthBench) has shown that mHC’s benefits don’t translate for deeper models.

Hyper Connections:

Hyper connections were introduced to boost residual capacity by adding more parallel residual streams. To allow the streams to share information with each other, they introduce an $n \times n$ matrix $H_l^{\mathrm{res}}$ which exchanges / mixes information before the layer update.

• N parallel residual streams → learned mix → attn / ffn / norm layer → learned split → N parallel residual streams

Before the attention or feedforward or normalization layers, the residual streams are aggregated into a single input since these layers cannot handle wide residual shapes.

To better understand this let’s backtrack a bit to standard residual connections and see what it's doing:

$$x_{l+1} = x_l + F_l(x_l)$$

It adds layer updates into a single residual stream.

But now with hyper connections, we have $n$ parallel residual streams

$$X_l = \begin{bmatrix}x_l^{(1)} \\ x_l^{(2)} \\ x_l^{(3)} \\ x_l^{(4)}\end{bmatrix}$$

so the update becomes:

$$X_{l+1} = H_l^{\mathrm{res}} X_l + H_l^{\mathrm{post}} F_l(H_l^{\mathrm{pre}} X_l)$$

what res mapping, pre and post mapping mean

$H_l^{\mathrm{res}}$ → mixes information between the parallel residual streams

$H_l^{\mathrm{pre}}$ → reads all $n$ residual streams into a single matrix input before the layer. Also uses a sigmoid function to maintain non-negative values to keep the values from cancelling each other out

$H_l^{\mathrm{post}}$ → broadcasts back the layer output into its $n$ parallel residual streams. This also maintains non-negative values.

the $H_l^{\mathrm{res}}$ matrix is a square matrix. the singular values of this matrix are unconstrained and can therefore blow up or vanish across layers, making training very unstable. mHC fixes this.

Manifold-constrained Hyper Connections (mHC)

To prevent the $H_l^{\mathrm{res}}$ from exploding or vanishing singular values, mHC enforces a geometric constraint to the matrix. It runs the Sinkhorn-Knopp normalization on $H_l^{\mathrm{res}}$ to make it doubly stochastic.

A doubly stochastic matrix is when both the rows and the columns sum to 1.

But how does a doubly stochastic matrix ensure stable training?

It preserves the norm of the matrix. In a doubly stochastic matrix, the matrix is essentially a convex combination of the inputs. (convex combination is just a weighted average where the weights are non-negative and add to 1).

Another thing is that the product of doubly stochastic matrices is also doubly stochastic. Which means that its stability persists across depth.

This is how mHC constrains outputs across layers and ensures it does not blow up.

What this means for computational depth

For narrow-deeper models, the DepthBench paper conducts a set of robust experiments to prove that HC performs better than mHC across depth.

why?

mHC over-constrains the $H_l^{\mathrm{res}}$ matrix across layers and this causes all unique residual paths to collapse into one identical path. mHC’s products across layers end up having much lower effective rank compared to HC, reducing deeper layers' effectiveness.

we should rethink HC variants that constrain the $H_l^{\mathrm{res}}$ matrix but also preserve its effective rank and the residual path directions.