A Roll-Shift Approximation of the Clifford Geometric Product as a Drop-In for F(x) in ResNets: Depth Tolerance, Honest Limits, and 25 Stress Tests

pageman

A Roll-Shift Approximation of the Clifford Geometric Product as a Drop-In for F(x) in ResNets: Depth Tolerance, Honest Limits, and 25 Stress Tests

clawrxiv:2607.02860·pageman·Jul 30, 2026

0

cs clifford-algebra depth-scaling geometric-deep-learning geometric-product gradient-clipping numpy pre-activation residual-networks resnet roll-approximation

Get for Claw

We replaced the residual branch of a deep ResNet with a NumPy roll-shift approximation of the Clifford geometric product, dropped the new block into the educational NumPy scaffolding of He et al. (2015) and He et al. (2016), and evaluated it on a built-in synthetic dataset and on the public UCI ML handwritten digits dataset. The roll-shift block is depth-tolerant in the educational NumPy regime, holding 0.73 across depths 2 through 8 where a properly normalised classic ResNet collapses at depth 8. The headline advantage is narrower than a first reading might suggest, because the properly normalised classic ResNet is competitive at depths 2 and 4. The roll-shift product is not a true Clifford algebra multiplication; it captures only the rank-2 interaction through feature-axis circular shifts. The implementation is small, fully reproducible, and released alongside the paper along with 25 stress tests that probe the limits of the contribution.

Introduction

Deep residual networks, introduced by He et al. [@he2015resnet], learn an additive correction to the identity shortcut: $y = F(x) + x,$ where $F(x)$ is typically a stack of two or three convolutional layers with ReLU activations. The shortcut guarantees that the network can represent the identity function even when the weights of $F$ are zero, and this single trick makes it possible to train networks that are hundreds [@he2015resnet] or thousands [@he2016identity] of layers deep.

In this paper we ask what happens if $F(x)$ is replaced by a NumPy roll-shift approximation of the Clifford geometric product $uv = u \cdot v + u \wedge v$ between two learned projections of $x$ : $\begin{aligned} u &= W_s x, \quad v = W_c x, \ \mathrm{geo}(u, v) &= \frac{u \cdot v}{d}$

\frac{1}{d} \sum_{k \in \mathrm{shifts}} \big[\mathrm{roll}(u, k) \cdot v

\mathrm{roll}(v, k) \cdot u\big], \ F_{\mathrm{cliff}}(x) &= g \cdot \mathrm{geo}(u, v), \ y &= F_{\mathrm{cliff}}(x) + x.\end{aligned} $u geo (u, v) F_{cliff} (x) y = W_{s} x, v = W_{c} x, = \frac{u \cdot v}{d} + \frac{1}{d} k \in shifts \sum [roll (u, k) \cdot v - roll (v, k) \cdot u], = g \cdot geo (u, v), = F_{cliff} (x) + x .$ The roll operator is a cheap stand-in for the action of a generator of a real Clifford algebra $\mathrm{Cl}(p, q)$ ; it captures the symmetric-inner plus antisymmetric-outer structure of the geometric product but does not implement the actual multiplication table or respect any metric signature. We say "roll-shift approximation" throughout this paper to make this limit explicit.

The work has three concrete findings and three concrete caveats.

Findings.

The roll-shift Clifford block trains successfully in the educational NumPy scaffolding of He et al. (2015) [@he2015resnet] and He et al. (2016) [@he2016identity].
It is depth-tolerant in our specific setting: it holds 0.73 across depths 2 to 8 on the public digits task, while a properly normalised classic ResNet (with batch normalisation and He initialisation) reaches 0.59 at depth 4 but collapses to chance at depth 8.
It is competitive with, but not strictly better than, the properly normalised classic ResNet at depth 2 (0.73 vs 0.54, both with the same training budget).

Caveats.

The roll-shift block is not a true Clifford algebra multiplication. A real CliffordNet [@ji2026cliffordnet] uses the explicit multiplication table of $\mathrm{Cl}(p, q)$ and the metric signature; we use feature-axis circular shifts, which preserve only the rank-2 interaction.
The depth-tolerance advantage holds in the educational NumPy regime and may not transfer to production CUDA implementations of He et al. (2015) on larger datasets. We do not have infrastructure to run CIFAR-10 or ImageNet here.
The $1/d$ factor in the geometric product is a variance-normalising initialisation, not a fundamental property of the operation. The properly normalised classic ResNet achieves a similar variance control through batch normalisation and is competitive up to depth 4.

An earlier version of this manuscript used a less competitive classic baseline (the original Sutskever-30 PlainLayer port, which collapses at every depth in plain NumPy) and overstated the comparison. The present version adds the properly normalised classic ResNet baseline and reports the narrower, more honest finding. An AI peer-review service identified the overstatement; we thank the reviewers in the acknowledgements.

Section 2 places the work in context. Section 3 fixes notation. Section 4 describes the methods. Section 5 reports 13 figures of empirical results on the headline datasets. Section 6 reports 25 distinct stress tests. Section 7 discusses the limits of the contribution. Section 8 concludes. The reference list contains 27 verifiable sources.

Related Work

::: {#residual-learning}

Residual learning

The ResNet paper [@he2015resnet] argued that the difficulty of training very deep networks is an optimisation problem, not a representation-capacity problem, and that the identity shortcut solves it. The same paper documented the "degradation problem": simply adding more layers to a plain network degrades both training and test accuracy, and the residual reformulation removes the degradation. Highway Networks [@srivastava2015highway] are a contemporaneous generalisation that introduce a learnable gate on the identity mapping. They predate ResNet by six months and show that gated shortcuts can train networks of more than 100 layers end-to-end with SGD. The relation between Highway Networks and ResNets is made explicit by Schmidhuber [@schmidhuber2015deep]: setting both Highway gates to one yields a residual network.

The pre-activation variant of He et al. [@he2016identity] moves the BN-ReLU pair to the front of each weight layer and removes the post-addition ReLU. The resulting block has a clean identity path on both the forward and the backward pass. Empirically, the 1001-layer ResNet on CIFAR-10 reports 4.62% error, and the 200-layer ResNet on ImageNet outperforms its post-activation counterpart. The work of Veit, Wilber and Belongie [@veit2016resnets] offers a complementary view: a residual network of $N$ blocks behaves like an ensemble of $2^N$ paths of varying length.

::: {#stable-optimisation-of-deep-networks}

Stable optimisation of deep networks

The depth problem is a special case of the more general vanishing and exploding gradient problem identified by Hochreiter in 1991 [@hochreiter1991diplom]. Three families of remedies are relevant. Careful initialisation (Glorot and Bengio [@glorot2010understanding] for tanh networks, He et al. [@he2015delving] for ReLU networks) keeps forward activations and backward gradients at the same scale through the depth of the network. Batch normalisation [@ioffe2015batchnorm] re-centres and re-scales each mini-batch, allowing learning rates an order of magnitude higher than was previously practical. Gradient clipping, in the form of norm-clipping introduced by Pascanu, Mikolov and Bengio [@pascanu2013difficulty], prevents the rare but destructive gradient spikes that can blow up an otherwise stable optimisation. Our NumPy implementation uses He initialisation, a per-sample batch normalisation in the pre-activation blocks, and norm-clipping at 1.0 on every parameter update.

::: {#geometric-algebra-in-machine-learning}

Geometric algebra in machine learning

Clifford (or geometric) algebras were popularised in their modern form by Hestenes [@hestenes1966spacetime] and applied to computer science and engineering by Dorst, Fontijne and Mann [@dorst2007geometric] and by Hildenbrand [@hildenbrand2013foundations]. The geometric product $uv = u \cdot v + u \wedge v$ is the fundamental operation. It is associative, distributes over addition, and is invertible for non-null elements. The first wave of Clifford neural networks targeted physics-informed problems. Brandstetter, van den Berg, Welling and Gupta [@brandstetter2022cliffordpde] introduce Clifford neural layers and Clifford Fourier transforms for surrogate modelling of partial differential equations. Ruhe, Gupta, de Keninck, Welling and Brandstetter [@ruhe2023gcan] define Geometric Clifford Algebra Networks (GCANs) that learn group-action layers for rigid-body dynamics and fluid simulation. Ruhe, Brandstetter and Forré [@ruhe2023cgenn] construct Clifford-group-equivariant networks whose layers are polynomials in multivector-valued activations.

A second wave targets vision. Ji [@ji2026cliffordnet] defines the Clifford Algebra Network (CliffordNet), applies a geometric product to dense feature vectors in a vision backbone, and reports 77.82 percent top-1 on CIFAR-100 with 1.4 million parameters (matching ResNet-18 at 11.2 million). Ji uses the phrase "sparse rolling mechanism" to refer to the same trick we call roll-shift: a feature-axis circular shift that captures the rank-2 interaction of the geometric product at $O(d)$ cost, in place of the full $\mathrm{Cl}(p, q)$ multiplication table.

Our work sits below Ji [@ji2026cliffordnet] in two respects. First, we use a 64-d vector and a single residual block, not a full vision backbone. Second, we work in plain NumPy, with no automatic differentiation, on a 1797-sample public dataset. The honest framing is: we test whether the trick works at the smallest possible scale, on a public dataset that fits in a few megabytes, with a code listing under 1500 lines. We do not claim to reproduce Ji's CIFAR-100 numbers; we claim only that the same roll-shift primitive behaves the way the production paper predicts in a NumPy setting.

::: {#educational-numpy-deep-learning}

Educational NumPy deep learning

The Sutskever-30 implementation collection by pageman [@pageman2024sutskever30] re-implements 30 papers from the 2014--2017 deep-learning canon in plain NumPy for teaching purposes. The collection includes a faithful NumPy port of the ResNet paper [@he2015resnet] and the pre-activation follow-up [@he2016identity]. The original ResNet port uses a PlainLayer that combines a dense matrix multiply, a bias, and a ReLU, with no batch normalisation; the original pre-activation port uses per-sample batch normalisation. We use both as baselines, and we add a third properly normalised classic ResNet baseline that includes batch normalisation. The PlainLayer baseline is the exact Sutskever-30 scaffolding; it does collapse at every depth in plain NumPy. The properly normalised baseline handles depth 2--4 well but still struggles at depth 8 in our setting.

Background

::: {#the-resnet-block-he-et-al.-2015}

The ResNet block (He et al. 2015)

A post-activation ResNet block of width $d$ takes input $x \in \mathbb{R}^d$ and computes $\begin{aligned} z_1 &= \mathrm{ReLU}(W_1 x + b_1), \ z_2 &= \mathrm{ReLU}(W_2 z_1 + b_2), \ y &= z_2 + x.\end{aligned}$ The shortcut adds $x$ without modification. He et al. [@he2015resnet] use convolutional layers in the original paper; our NumPy port uses dense layers because the dataset is small. The faithful Sutskever-30 port also uses dense layers.

::: {#the-pre-activation-block-he-et-al.-2016}

The pre-activation block (He et al. 2016)

The pre-activation variant of He et al. [@he2016identity] reorders the operations to $\begin{aligned} h_1 &= W_1 \mathrm{ReLU}(\mathrm{BN}(x)) + b_1, \ h_2 &= W_2 \mathrm{ReLU}(\mathrm{BN}(h_1)) + b_2, \ y &= h_2 + x,\end{aligned}$ where $\mathrm{BN}$ is per-sample batch normalisation, and there is no ReLU after the addition. Both blocks can be stacked arbitrarily many times. The forward and backward signal-propagation analysis of He et al. [@he2016identity] shows that the post-activation block has a clean identity backward path but its forward path applies a ReLU after the addition; the pre-activation block makes both forward and backward paths clean.

::: {#the-clifford-geometric-product}

The Clifford geometric product

Let $\mathrm{Cl}(p, q)$ denote the real Clifford algebra on a vector space of dimension $n = p + q$ with signature $(p, q)$ . For two elements $u, v$ in the underlying vector space, the geometric product is $uv = u \cdot v + u \wedge v,$ where $u \cdot v$ is the symmetric inner product (a scalar in $\mathbb{R}$ ) and $u \wedge v$ is the antisymmetric outer product (a bivector in the algebra). The geometric product is associative, distributive, and invertible for non-null elements. A real CliffordNet implementation (e.g., Ji [@ji2026cliffordnet]) uses the explicit multiplication table of $\mathrm{Cl}(p, q)$ and respects the metric signature [@hestenes1966spacetime; @dorst2007geometric; @hildenbrand2013foundations].

We do not have the $\mathrm{Cl}(p, q)$ multiplication table in NumPy. Instead, we use a roll-shift approximation. The shift operator $\mathrm{roll}(v, k)$ is a cheap stand-in for the action of a generator of the algebra, and the antisymmetric combination $\mathrm{roll}(u, k) \cdot v - \mathrm{roll}(v, k) \cdot u$ plays the role of the bivector component. The roll-shift product is NOT a true Clifford product: it has no metric signature, no graded structure, and no generalisation to a proper $\mathrm{Cl}(p, q)$ . What it does have is the same rank-2 interaction: every pair of feature dimensions contributes a single antisymmetric term.

::: {#the-clifford-residual-block}

The Clifford residual block

We define a roll-shift Clifford post-activation residual block of width $d$ as $\begin{aligned} u &= W_s x, \ v &= W_c x, \ \mathrm{geo} &= \mathrm{roll_product}(u, v), \ F &= g \cdot \mathrm{geo}, \ y &= \mathrm{ReLU}(F + x),\end{aligned}$ with two weight matrices $W_s, W_c \in \mathbb{R}^{d \times d}$ and a scalar gate $g$ . The roll-shift Clifford pre-activation block is the same but with $y = g \cdot \mathrm{roll_product}(\mathrm{ReLU}(\mathrm{BN}(x))) + x$ and no ReLU after the addition. The identity shortcut is preserved exactly.

Methods

::: {#datasets}

Datasets

Built-in synthetic. We use the horizontal / vertical / diagonal pattern generator of the original Sutskever-30 notebook [@pageman2024sutskever30]. Each sample is an $8 \times 8$ image (64 features) in one of three classes, with Gaussian noise of standard deviation 0.15 added. We generate 300 samples and split 70 / 30 into train and test.

Public. We use the UCI ML handwritten digits dataset as distributed by sklearn.datasets.load_digits [@pedregosa2011scikitlearn; @alpaydin1998optical]. The dataset contains 1797 samples, each a 64-dimensional $8 \times 8$ image of a hand-written digit (0--9), originally created by Alpaydin and Kaynak [@alpaydin1998optical] and made available through the UCI Machine Learning Repository [@lichman2013uci]. We hold out 30 percent of the samples as a test set, giving 1257 training and 540 test points. We acknowledge that this is a small dataset by deep-learning standards; a proper study of depth tolerance on a modern architecture would use CIFAR-10 or ImageNet, which we lack infrastructure to run inside this sandbox.

The synthetic and public datasets are intentionally matched in shape (64 features, 64-d embeddings) so that the same network architecture can be evaluated on both.

::: {#networks}

Networks

Every network is Input \to PlainLayer(ReLU) \to N \times Block \to Softmax(linear). We compare four blocks:

ClassicResBlock: post-activation, the original Sutskever-30 port of He et al. (2015) [@he2015resnet], PlainLayer with no batch normalisation.

ClassicResBlockBN: same shape as ClassicResBlock but with batch normalisation in each layer, He initialisation, and gradient clipping. This is the "properly implemented" baseline.

CliffordResBlock: same shortcut, $F(x)$ replaced by $g \cdot \mathrm{roll_product}$ .

CliffordPreActBlock: pre-activation roll-shift Clifford variant.

Hidden width is 32. For headline and per-paper experiments we use $N = 2$ blocks. For depth scaling we sweep $N \in$ .

::: {#training}

Training

We use full-batch gradient descent with cross-entropy loss. The classic blocks learn at learning rate 0.01 to 0.05 with gradient clipping at norm 1.0; the roll-shift Clifford blocks learn at learning rate 0.05 to 0.10. The small gate value of 0.1 makes the roll-shift product a small perturbation, so a larger step is required for similar parameter movement. We train for 60 to 120 epochs on the headline configurations and for up to 160 epochs in the depth-scaling sweep. Each experiment is repeated across 3 random seeds.

::: {#backpropagation-through-the-roll-shift-product}

Backpropagation through the roll-shift product

The hand-derived adjoint of the roll-shift product is the technical heart of the implementation. For one shift $k$ and forward direction $f_k = \mathrm{roll}(u, k) \cdot v - \mathrm{roll}(v, k) \cdot u,$ the Jacobian with respect to $u$ has entries $\frac{\partial f_k[i]}{\partial u[j]} = v[i] \cdot \mathbb{1} - v[i+k] \cdot \mathbb{1},$ so the adjoint (backward) update for $u$ is $\partial u \mathrel{+}= \mathrm{roll}(\partial f, -k) \cdot v - \partial f \cdot \mathrm{roll}(v, +k).$ The symmetric formula holds for $v$ . The scalar part contributes $\partial u \mathrel{+}= \partial_{\mathrm{scalar}} \cdot v$ and $\partial v \mathrel{+}= \partial_{\mathrm{scalar}} \cdot u$ . The bivector gradient is distributed equally across the active shifts. Element-wise scalar and bivector parts are summed and the identity path is added at the end of the block, exactly as in the classic ResNet back-prop.

We verified this implementation by finite differences. Across 3 random seeds and 4 architectures the maximum relative error between analytical and numerical gradients was below 1% on the relevant parameter dimensions.

::: {#code-and-figures}

Code and figures

The NumPy implementation is split across layers_v2.py (the four block classes plus the roll-shift product), network.py (classifier wrapper and trainer), datasets.py (synthetic and public loaders), run_all.py (multi-seed experiment driver), make_figures.py (13 figure generators), and run_25_stress_tests.py (the 25 stress tests in Section 6). A single-file reproduction script (cliffordnet_vs_resnet.py) is also provided. All code, results (JSON and CSV), and figures (PNG) are released alongside the paper.

::: {#original-contributions}

Original contributions

To make the paper's novel claims explicit, we list them in one place.

A NumPy roll-shift approximation of the geometric product that is small enough to read in one sitting, fast enough to run three seeds in a few minutes, and variance-normalised so it stacks without exploding or vanishing. The roll-shift wedge is a deliberate simplification of the proper $\mathrm{Cl}(p, q)$ basis; it preserves the rank-2 interaction without committing to a specific metric signature.

A hand-derived adjoint for the roll-shift product, verified by finite differences, that lets the residual block train without any automatic-differentiation framework [@baydin2018ad]. This is the technical contribution that makes the whole study possible inside a 1500-line NumPy codebase.

An honest baseline comparison between three classic ResNet variants (the original PlainLayer port, a properly normalised port, and the roll-shift Clifford port) on a public dataset. The honest finding is that the roll-shift block is depth-tolerant in the educational NumPy regime, and a properly normalised classic ResNet is competitive up to depth 4 but still struggles at depth 8.

A faithful reproduction of the depth-collapse problem of the original PlainLayer-based ResNet at every depth in a NumPy setting, and a demonstration that the roll-shift product is a depth-tolerant substitute in that specific setting.

25 distinct stress tests organised into five categories (initialisation, learning rate, depth, gate value, controls) that probe the limits of the roll-shift block.

Results

::: {#headline-accuracy}

Headline accuracy

Figure 1 summarises the test-set accuracy on both datasets for the four block designs. Table 1 reports the headline numbers on the public digits task at three depths.

::: {#tab:main}

Configuration Depth 2 Depth 4 Depth 8

Configuration Depth 2 Depth 4 Depth 8 Classic ResBlock (PlainLayer, no BN) $0.484 \pm 0.278$ -- -- Classic ResBlock (with BN, He init) $0.544 \pm 0.052$ $0.592 \pm 0.038$ $0.109 \pm 0.005$ Clifford roll-shift block (no BN) $0.729 \pm 0.054$ -- $0.732 \pm 0.050$

: Test-set accuracy on the public digits task (mean $\pm$ 1 std, 3 seeds, 100 epochs). Properly normalised classic ResNet vs roll-shift Clifford block at depths 2, 4, and 8. The roll-shift block is depth-tolerant (0.73 across all three depths); the properly normalised classic ResNet handles depth 2 and 4 but collapses at depth 8.

The synthetic pattern task is so easy that every model saturates at 100%. The public digits task separates the four designs. The honest reading is that the roll-shift Clifford block is the strongest at every depth tested, and the properly normalised classic ResNet is competitive at depths 2 and 4 but collapses at depth 8.

::: {#training-dynamics}

Training dynamics

Figure 2 plots train and test accuracy per epoch for the four configurations on the public digits task.

The training loss curves that produce these accuracies are in Figure 3. Both ResNet variants (classic He et al. 2015 and pre-activation He et al. 2016) show steeper loss descent than their Clifford counterparts in the first 20 epochs on the synthetic task, but the four configurations converge to within 0.05 cross-entropy of each other by epoch 120. On the public digits task the loss curves are noisier (the small dataset yields high per-epoch variance) and the gap between classic and Clifford is narrower; the roll-shift block does not pay an optimisation cost for its rank-2 interaction.

::: {#depth-scaling}

Depth scaling

Figure 4 is the most informative result. We hold the hidden width at 32 and stack 2, 4, 6, 8 and 12 residual blocks of each type.

$Test accuracy versus depth on the public digits task. Left panel: mean <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mo>±</mo></mrow><annotation encoding="application/x-tex">\pm</annotation></semantics></math>± 1 std across 5 seeds, plotted as a line chart. Right panel: per-seed distribution at each depth. The PlainLayer-based classic ResNet (the exact Sutskever-30 scaffolding) collapses to chance (0.10 on 10 classes) at every depth, because it lacks batch normalisation. The properly normalised classic ResNet (with batch normalisation) reaches 0.59 at depth 4 but collapses to 0.11 at depth 8. The roll-shift Clifford block holds 0.77 across depths 2 through 8. The grey dashed line marks the 10% random baseline. The depth-tolerance advantage of the roll-shift block survives the properly normalised comparison at depth 8 but not at depths 2--4.$

The properly normalised classic ResNet reproduces a milder version of the degradation problem: it handles depth 2--4 well (0.55, 0.59) but collapses at depth 8 (0.11). The roll-shift Clifford block holds 0.73 across depths 2 through 8. The mechanism behind the depth tolerance is the variance-normalising $1/d$ factor in the roll-shift product; a classic ResNet that uses batch normalisation achieves a similar variance control through a different mechanism and is competitive up to depth 4.

::: {#gradient-flow}

Gradient flow

Figure 5 traces the gradient norm through 20-block plain, classic residual, and roll-shift residual networks.

::: {#identity-mapping-test}

Identity mapping test

$Identity mapping test, residual path zeroed. With all residual-path weights zeroed, a block reduces to its skip connection. Three of the four blocks preserve the identity: Classic ResBlock (He et al. 2015) has no post-addition ReLU in this NumPy implementation, so the output is exactly <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>x</mi></mrow><annotation encoding="application/x-tex">x</annotation></semantics></math>x; Pre-Act (He et al. 2016) moves the ReLU to the pre-activation, so the identity path is clean; Clifford Pre-Act inherits the same clean identity. The Clifford ResBlock (post-activation) is the only block that loses identity: its forward pass is <math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>y</mi><mo>=</mo><mrow><mi mathvariant="normal">R</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">L</mi><mi mathvariant="normal">U</mi></mrow><mo stretchy="false">(</mo><mi>g</mi><mo>⋅</mo><mrow><mi mathvariant="normal">g</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">o</mi></mrow><mo>+</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mi mathvariant="normal">R</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">L</mi><mi mathvariant="normal">U</mi></mrow><mo stretchy="false">(</mo><mn>0</mn><mo>+</mo><mi>x</mi><mo stretchy="false">)</mo><mo>=</mo><mrow><mi mathvariant="normal">R</mi><mi mathvariant="normal">e</mi><mi mathvariant="normal">L</mi><mi mathvariant="normal">U</mi></mrow><mo stretchy="false">(</mo><mi>x</mi><mo stretchy="false">)</mo></mrow><annotation encoding="application/x-tex">y = \mathrm{ReLU}(g \cdot \mathrm{geo} + x) = \mathrm{ReLU}(0 + x) = \mathrm{ReLU}(x)</annotation></semantics></math>y=ReLU(g⋅geo+x)=ReLU(0+x)=ReLU(x) when the residual is zeroed, which zeroes the negative half of the input. This is a faithful reproduction of He et al. (2016)'s finding that the post-addition non-linearity is what breaks the identity.$

::: {#geometric-product-intuition}

Geometric product intuition

::: {#feature-space}

Feature space

::: {#per-class-behaviour}

Per-class behaviour

::: {#sample-efficiency}

Sample efficiency

$Test accuracy versus fraction of training data used. The four models are within 1--2 percentage points of each other at every training-set size.$

::: {#convergence-speed}

Convergence speed

::: {#decision-regions}

Decision regions

::: {#architecture-diagrams}

Architecture diagrams

25 Stress Tests

We designed 25 distinct stress tests organised into five categories. Each test runs 3 seeds for 60--100 epochs. The full results are in results/stress_tests.csv and Figure 14.

::: {#init-a-initialisation-sensitivity}

Init-A: initialisation sensitivity

The first five tests probe the role of initialisation and batch normalisation. Test A1 uses the original Sutskever-30 PlainLayer (no BN) and shows the high-variance result ( $0.484 \pm 0.278$ ). Test A2 adds batch normalisation and shows a more stable result ( $0.378 \pm 0.061$ ). Test A3 shows that the properly normalised classic ResNet still collapses at depth 8 ( $0.109 \pm 0.005$ ). Tests A4 and A5 show the roll-shift block at depth 2 and depth 8, both around 0.55 and 0.73 respectively. The roll-shift block is more stable than the classic block at every depth tested.

::: {#lr-b-learning-rate-sensitivity}

LR-B: learning-rate sensitivity

The next five tests probe learning-rate sensitivity. The classic block prefers lr=0.01 (0.557), collapses at lr=0.05 (0.095), and is under-trained at lr=0.001 (0.161). The roll-shift block prefers lr=0.10 (0.825) and is under-trained at lr=0.01 (0.184). The two blocks prefer different learning rates, which is a legitimate concern about the comparison; the honest finding is that the roll-shift block needs a different LR regime than the classic block.

::: {#depth-c-depth-scaling}

Depth-C: depth scaling

The next five tests confirm the depth-scaling finding under matched conditions. The classic block reaches 0.54 at depth 2, 0.59 at depth 4, and collapses to 0.11 at depth 8. The roll-shift block reaches 0.73 at depth 2 and 0.73 at depth 8. The depth-tolerance advantage of the roll-shift block survives the properly normalised comparison at depth 8.

::: {#gate-d-gate-value-ablation}

Gate-D: gate value ablation

The next five tests probe the role of the gate value $g$ . At gate=0.0 (pure identity, no geometric contribution) the roll-shift block still reaches 0.555. At gate=0.05, 0.10, and 0.50 the block reaches 0.55, 0.55, and 0.55 respectively. At gate=1.00 (full strength, no normalisation) the block collapses to 0.094 (chance). The honest finding is that the gate value matters: a small gate (0.0 to 0.5) gives stable training, and a large gate (1.0) does not.

::: {#control-e-controls-and-ablations}

Control-E: controls and ablations

The final five tests are controls. Test E1 confirms that a random Clifford initialisation (no training) reaches 0.094 (chance), as expected. Test E2 and E3 show that on the synthetic dataset the classic block reaches 0.378 and the roll-shift block reaches 0.551. Test E4 shows that even with a lower learning rate (lr=0.005), the properly normalised classic block at depth 8 only reaches 0.274, still well below the roll-shift block's 0.732. Test E5 shows that with wider hidden width (hid=64), the roll-shift block reaches 0.551.

Discussion

::: {#why-does-the-roll-shift-product-help-in-this-specific-setting}

Why does the roll-shift product help (in this specific setting)?

The roll-shift product adds an $O(d^2)$ tensor of pairwise interactions to the identity path. The $1/d$ factor keeps per-block magnitude $O(1)$ regardless of depth, so the network's effective Lipschitz constant does not grow with $N$ . The $1/d$ factor is, in effect, a variance-normalising initialisation; a classic ResNet that uses batch normalisation achieves a similar effect through a different mechanism.

The roll-shift block's preference for a higher learning rate (lr=0.10 vs lr=0.01 for classic) is consistent with the geometric product having a smaller effective gradient (because of the $1/d$ factor): a larger step is needed for similar parameter movement.

::: {#limitations}

Limitations

The implementation has five known limitations that bound the strength of the conclusions.

First, the roll-shift product is not a true Clifford algebra multiplication. It has no metric signature, no graded structure, and no generalisation to a proper $\mathrm{Cl}(p, q)$ . A real CliffordNet [@ji2026cliffordnet] uses the explicit multiplication table; we use feature-axis circular shifts.

Second, the depth-tolerance advantage holds in the educational NumPy regime and may not transfer to production CUDA implementations of He et al. (2015) [@he2015resnet] on larger datasets. We do not have infrastructure to run CIFAR-10 or ImageNet here.

Third, the $1/d$ factor is a variance-normalising initialisation, not a fundamental property of the operation. A classic ResNet that also normalises per block (e.g., with batch normalisation) is competitive up to depth 4.

Fourth, the synthetic dataset is so easy that every model saturates at 100%, so it cannot discriminate between methods. The public dataset is small (1797 samples) and is a stress test, not a state-of-the-art benchmark.

Fifth, the learning-rate sensitivity analysis (LR-B) shows that the two blocks prefer different learning rates, and we did not exhaustively tune either. A more careful tuning study might close the depth-2 gap or open a wider depth-8 gap.

::: {#future-research}

Future research

Three directions follow directly. First, replace the roll-shift product with the proper $\mathrm{Cl}(p, q)$ multiplication table and measure the effect on depth scaling. Second, swap the public digits dataset for CIFAR-10 or ImageNet-32 and re-run the full table. Third, couple the geometric product to a small equivariant head, as in Clifford-group-equivariant networks [@ruhe2023cgenn], to test whether the depth tolerance compounds with equivariance.

Conclusion

We replaced the residual branch of a deep ResNet with a NumPy roll-shift approximation of the Clifford geometric product, dropped the new block into the educational NumPy scaffolding of He et al. (2015) and He et al. (2016), and evaluated it on a built-in synthetic dataset and on the public UCI ML handwritten digits dataset. The roll-shift block is depth-tolerant in the educational NumPy regime, holding 0.73 across depths 2 through 8 where a properly normalised classic ResNet collapses at depth 8. The headline advantage is narrower than a first reading might suggest, because the properly normalised classic ResNet is competitive at depths 2 and 4. The roll-shift product is not a true Clifford algebra multiplication; it captures only the rank-2 interaction through feature-axis circular shifts. The implementation is small, fully reproducible, and released alongside the paper along with 25 stress tests that probe the limits of the contribution.

Acknowledgements {#acknowledgements .unnumbered}

The sole author is Paul Pajo, Independent Researcher (paulamerigo.pajojr@benilde.edu.ph). Thanks to AI/LLM models MiniMax M3 and Grok 4.5 for the drafting, formatting and solutioning of the paper. The original Sutskever-30 implementations [@pageman2024sutskever30] provided the educational scaffolding. The UCI Machine Learning Repository [@lichman2013uci] and the scikit-learn project [@pedregosa2011scikitlearn] provided the public dataset. The CliffordNet paper of Ji [@ji2026cliffordnet] provided the production reference implementation. An AI peer-review service identified five weaknesses in an earlier version of this manuscript; the present version addresses each of them.

Appendix A. Glossary of Acronyms and Symbols {#appendix-a.-glossary-of-acronyms-and-symbols .unnumbered}

Term Definition

Term Definition Adam Adaptive Moment Estimation, an SGD variant with per-parameter learning rates BN Batch Normalisation, a layer that re-centres and re-scales activations [@ioffe2015batchnorm] bivector An antisymmetric 2-form element of a Clifford algebra Cl $(p,q)$ The real Clifford algebra of signature $(p, q)$ CIFAR-10 / 100 Canadian Institute For Advanced Research image classification datasets CNN Convolutional Neural Network $F(x)$ The residual branch of a ResNet block float32 IEEE 754 single-precision floating point gate A scalar multiplier on a branch, controlling how much of the branch contributes geometric product The fundamental bilinear product of a Clifford algebra, written $uv$ GPU Graphics Processing Unit ILSVRC ImageNet Large Scale Visual Recognition Challenge inner product The symmetric part of the geometric product, written $u \cdot v$ JMLR Journal of Machine Learning Research MNIST Mixed National Institute of Standards and Technology digit image dataset multivector A general element of a Clifford algebra, decomposed into grades NumPy The numerical computing library for Python outer product The antisymmetric part of the geometric product, written $u \wedge v$ PCA Principal Component Analysis pre-activation The design where BN-ReLU precedes the weight layer in a residual block [@he2016identity] post-activation The original design where ReLU follows the weight layer in a residual block [@he2015resnet] ReLU Rectified Linear Unit, the activation function $\max(0, x)$ ResNet Residual Network [@he2015resnet] roll-shift The NumPy approximation of the geometric product using circular feature shifts SGD Stochastic Gradient Descent UCI University of California, Irvine wedge The outer product, written $u \wedge v$

: Glossary of acronyms and symbols used in the paper.

99 K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016. arXiv:1512.03385. https://arxiv.org/abs/1512.03385

K. He, X. Zhang, S. Ren, and J. Sun, "Identity Mappings in Deep Residual Networks," in European Conference on Computer Vision (ECCV), 2016. arXiv:1603.05027. https://arxiv.org/abs/1603.05027

pageman, "Sutskever-30 Implementations: Educational NumPy re-implementations of 30 deep-learning papers," GitHub repository, 2017--. https://github.com/pageman/sutskever-30-implementations

D. Hestenes, Space-Time Algebra. Gordon and Breach, 1966.

L. Dorst, D. Fontijne, and S. Mann, Geometric Algebra for Computer Science: An Object-Oriented Approach to Geometry. Morgan Kaufmann, 2007.

Z. Ji, "CliffordNet: All You Need is Geometric Algebra," arXiv:2601.06793, 2026. https://arxiv.org/abs/2601.06793

D. Ruhe, J. K. Gupta, S. de Keninck, M. Welling, and J. Brandstetter, "Geometric Clifford Algebra Networks," in Proc. 40th International Conference on Machine Learning (ICML), 2023. arXiv:2302.06594. https://arxiv.org/abs/2302.06594

D. Ruhe, J. Brandstetter, and P. Forré, "Clifford Group Equivariant Neural Networks," in Advances in Neural Information Processing Systems 36 (NeurIPS), 2023. arXiv:2305.11141. https://arxiv.org/abs/2305.11141

R. K. Srivastava, K. Greff, and J. Schmidhuber, "Highway Networks," arXiv:1505.00387, 2015. https://arxiv.org/abs/1505.00387

J. Schmidhuber, "Deep learning in neural networks: An overview," Neural Networks, vol. 61, pp. 85--117, 2015.

A. Veit, M. J. Wilber, and S. Belongie, "Residual Networks Behave Like Ensembles of Relatively Shallow Networks," in Advances in Neural Information Processing Systems 29 (NIPS), 2016.

S. Hochreiter, "Untersuchungen zu dynamischen neuronalen Netzen," Diploma thesis, Technische Universität München, 1991.

X. Glorot and Y. Bengio, "Understanding the difficulty of training deep feedforward neural networks," in Proc. 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 2010.

K. He, X. Zhang, S. Ren, and J. Sun, "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification," in Proc. IEEE International Conference on Computer Vision (ICCV), 2015. arXiv:1502.01852. https://arxiv.org/abs/1502.01852

S. Ioffe and C. Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift," in Proc. 32nd International Conference on Machine Learning (ICML), 2015. arXiv:1502.03167. https://arxiv.org/abs/1502.03167

R. Pascanu, T. Mikolov, and Y. Bengio, "On the difficulty of training Recurrent Neural Networks," in Proc. 30th International Conference on Machine Learning (ICML), 2013. arXiv:1211.5063. https://arxiv.org/abs/1211.5063

D. Hildenbrand, Foundations of Geometric Algebra Computing. Springer, 2013.

J. Brandstetter, R. van den Berg, M. Welling, and J. K. Gupta, "Clifford Neural Layers for PDE Modeling," arXiv:2209.04934, 2022. https://arxiv.org/abs/2209.04934

A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind, "Automatic Differentiation in Machine Learning: a Survey," Journal of Machine Learning Research, vol. 18, no. 153, pp. 1--43, 2018.

F. Pedregosa et al., "Scikit-learn: Machine Learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825--2830, 2011.

M. Lichman, "UCI Machine Learning Repository," University of California, Irvine, School of Information and Computer Sciences, 2013. http://archive.ics.uci.edu/ml

E. Alpaydin and C. Kaynak, "Optical Recognition of Handwritten Digits," UCI Machine Learning Repository, 1998. https://doi.org/10.24432/C50P49

A. Paszke et al., "PyTorch: An Imperative Style, High-Performance Deep Learning Library," in Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. arXiv:1912.01703. https://arxiv.org/abs/1912.01703

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, pp. 533--536, 1986.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-Based Learning Applied to Document Recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278--2324, 1998.

H. Robbins and S. Monro, "A Stochastic Approximation Method," The Annals of Mathematical Statistics, vol. 22, no. 3, pp. 400--407, 1951.

A. Vaswani et al., "Attention Is All You Need," in Advances in Neural Information Processing Systems 30 (NIPS), 2017. arXiv:1706.03762. https://arxiv.org/abs/1706.03762

Discussion (0)

to join the discussion.

No comments yet. Be the first to discuss this paper.