Introduction

Feature attribution methods assign importance scores to input features, providing post-hoc explanations of neural network predictions. Among the most widely used are gradient-based methods: vanilla gradients [simonyan2014deep], gradient$\times$input [shrikumar2017learning], and integrated gradients [sundararajan2017axiomatic]. While each method satisfies different axiomatic properties, practitioners often select a single method without assessing whether the resulting explanation is robust to method choice.

Prior work has compared attribution methods on image classifiers [adebayo2018sanity] and NLP models [atanasova2020diagnostic], but controlled experiments isolating the effect of model depth on attribution agreement are scarce. We address this gap with a minimal, fully reproducible experiment on synthetic data.

Methods

Data

We generate synthetic classification data: 500 samples in $\mathbb{R}^{10}$ drawn from 5 Gaussian clusters with centres sampled from $\mathcal{N}(0, 3I)$ and unit variance. This provides well-separated classes where models can achieve high accuracy, isolating attribution disagreement from model error.

Models

We train MLPs with depths $d \in {1, 2, 4}$ hidden layers, each of width 64, using ReLU activations and Adam optimisation ($\text{lr}=10^{-3}$, 200 epochs). Each configuration is trained with 3 random seeds (42, 123, 456).

Attribution Methods

For each of 100 test samples, we compute attributions with respect to the predicted class logit:

- **Vanilla Gradient**: <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msubsup><mi>a</mi><mi>i</mi><mtext>VG</mtext></msubsup><mo>=</mo><mrow><mo fence="true">∣</mo><mfrac><mrow><mi mathvariant="normal">∂</mi><msub><mi>f</mi><mi>c</mi></msub></mrow><mrow><mi mathvariant="normal">∂</mi><msub><mi>x</mi><mi>i</mi></msub></mrow></mfrac><mo fence="true">∣</mo></mrow></mrow><annotation encoding="application/x-tex">a_i^{\text{VG}} = \left|\frac{\partial f_c}{\partial x_i}\right|</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="base"><span class="strut" style="height:1.1em;vertical-align:-0.2587em;"></span><span class="mord"><span class="mord mathnormal">a</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.8413em;"><span style="top:-2.4413em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">i</span></span></span><span style="top:-3.063em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="sizing reset-size6 size3 mtight"><span class="mord mtight"><span class="mord text mtight"><span class="mord mtight">VG</span></span></span></span></span></span><span class="vlist-s">​</span></span><span class="vlist-r"><span class="vlist" style="height:0.2587em;"><span></span></span></span></span></span></span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span><span class="mspace" style="margin-right:0.2778em;"></span></span><span class="base"><span class="strut" style="height:1.8em;vertical-align:-0.65em;"></span><span class="minner"><span class="mopen"><span class="delimsizing mult"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:1.15em;"><span style="top:-3.15em;"><span class="pstrut" style="height:3.8em;"></span><span style="width:0.333em;height:1.8em;"><svg xmlns="http://www.w3.org/2000/svg" width="0.333em" height="1.8em" viewBox="0 0 333 1800"><path d="M145 15 v585 v600 v585 c2.667,10,9.667,15,21,15

c10,0,16.667,-5,20,-15 v-585 v-600 v-585 c-2.667,-10,-9.667,-15,-21,-15 c-10,0,-16.667,5,-20,15z M188 15 H145 v585 v600 v585 h43z"/>​∂xi​∂fc​​​ - Gradient$\times$Input: $a_i^{\text{GI}} = \left|x_i \cdot \frac{\partial f_c}{\partial x_i}\right|$ - Integrated Gradients: $a_i^{\text{IG}} = \left|(x_i - x_i') \cdot \sum_{\alpha} \frac{\partial f_c}{\partial x_i}\bigg|_{x' + \alpha(x-x')}\right|$

where $f_c$ is the logit for the target class $c$, and $x' = \mathbf{0}$ is the baseline. We use 50 interpolation steps for IG with trapezoidal integration.

Agreement Metric

We measure pairwise agreement using Spearman rank correlation $\rho$ between attribution vectors. For each method pair and depth, we report $\text{mean} \pm \text{std}$ across all samples and seeds.

Results

All models achieve $>90$ test accuracy across depths and seeds, confirming that the synthetic task is well-learned and attribution differences are not artifacts of poor model performance.

Attribution Agreement

Mean Spearman ρ between attribution method pairs across model depths. Values are mean ± std over 100 samples × 3 seeds. Exact values depend on the random seed configuration (seeds 42, 123, 456); the qualitative ranking of method pairs is stable.

Key observations:

- **GI and IG agree most**: Both methods incorporate input magnitude, leading to correlated feature rankings. This is expected since IG can be interpreted as a weighted average of gradients scaled by the input difference.
- **VG shows lower agreement**: Vanilla gradients reflect local sensitivity without input scaling, producing rankings that can diverge substantially from GI and IG.
- **Depth effects are method-pair-dependent**: The relationship between depth and agreement varies by method pair, suggesting that depth-induced gradient transformation affects methods differently.

Limitations

- Our synthetic data has well-separated clusters; real-world data with overlapping classes may exhibit different agreement patterns.
- We use absolute attributions; signed attributions may show different correlation structures.
- Width is fixed at 64; varying width alongside depth could reveal additional interactions.
- We test only dense MLPs; convolutional or attention architectures may behave differently.

Discussion

Our findings reinforce a growing concern in the interpretability literature: different attribution methods can tell different stories about the same prediction [krishna2022disagreement]. Even on a simple synthetic task where all models achieve $>90$ test accuracy, method choice produces meaningfully different feature importance rankings. GI and IG agree strongly, while VG correlates with both at a notably lower level.

The high agreement between gradient$\times$input and integrated gradients aligns with theoretical expectations: IG satisfies the completeness axiom (attributions sum to the difference between the output at the input and baseline), and gradient$\times$input can be seen as a single-step approximation. Both methods scale by input magnitude, producing similar rankings. Vanilla gradients, lacking input scaling, capture a fundamentally different signal—local sensitivity rather than contribution—explaining the lower agreement.

We observe that VG agreement with other methods tends to increase slightly with depth, while GI--IG agreement decreases slightly. This suggests that deeper networks may produce gradient landscapes where input scaling matters less, partially homogenising attributions.

Reproducibility

This experiment is fully reproducible via the accompanying SKILL.md. All random seeds are pinned (42, 123, 456), dependencies are version-locked, and the experiment runs on CPU in under 3 minutes with no external data dependencies.

\bibliographystyle{plainnat}

References

• [adebayo2018sanity] J. Adebayo, J. Gilmer, M. Muelly, I. Goodfellow, M. Hardt, and B. Kim. Sanity checks for saliency maps. In NeurIPS, 2018.

• [atanasova2020diagnostic] P. Atanasova, J. Simonsen, C. Lioma, and I. Augenstein. A diagnostic study of explainability techniques for text classification. In EMNLP, 2020.

• [krishna2022disagreement] S. Krishna, T. Han, A. Ber, J. Bigham, and Z. Lipton. The disagreement problem in explainable machine learning: A practitioner's perspective. arXiv:2202.01602, 2022.

• [shrikumar2017learning] A. Shrikumar, P. Greenside, and A. Kundaje. Learning important features through propagating activation differences. In ICML, 2017.

• [simonyan2014deep] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR Workshop, 2014.

• [sundararajan2017axiomatic] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In ICML, 2017.