Abstract

Chain-of-thought (CoT) prompting enables complex reasoning in LLMs, but its robustness to adversarial perturbations is poorly understood. We study five perturbation types across four benchmarks and three model families, finding that CoT accuracy degrades by 18.4% under perturbations that barely affect direct prompting (2.1% drop). We introduce the Reasoning Fragility Index (RFI = 8.7), showing CoT reasoning is nearly an order of magnitude more fragile than direct prediction.

1. Introduction

Chain-of-thought prompting [1] has been celebrated as a breakthrough in eliciting reasoning capabilities from large language models. By prompting models to generate intermediate reasoning steps, CoT dramatically improves performance on arithmetic, symbolic, and commonsense reasoning benchmarks [2, 3].

However, genuine reasoning should be robust: a valid logical argument remains valid regardless of how its premises are phrased. If CoT reasoning were truly compositional, minor perturbations to the input—synonym substitutions, premise reordering, numerical formatting changes—should not substantially affect the model's ability to arrive at correct conclusions.

We test this hypothesis systematically and find it decisively refuted.

2. Perturbation Taxonomy

We define five perturbation classes:

Critically, $\mathcal{P}_1$, $\mathcal{P}_3$, $\mathcal{P}_4$, and $\mathcal{P}_5$ are meaning-preserving: a human reader would extract the same information from the perturbed input.

3. Experimental Setup

3.1 Models and Benchmarks

3.2 Prompting Conditions

For each (model, benchmark, perturbation) triple, we evaluate under two conditions:

• Direct: "Answer the following question: [perturbed input]"
• CoT: "Think step by step and answer: [perturbed input]"

We use greedy decoding (temperature 0) and 8-shot exemplars for GSM8K/MATH, 0-shot for ARC/StrategyQA.

4. Results

4.1 Overall Fragility

Averaged across all models and benchmarks:

The RFI for premise reordering ($\mathcal{P}_5$) is 59.0—meaning CoT accuracy is 59 times more sensitive to reordering than direct prompting.

4.2 Benchmark-Specific Results

Mathematical reasoning (GSM8K, MATH) is most fragile, consistent with the hypothesis that CoT exploits sequential numerical patterns rather than mathematical understanding.

4.3 Model Comparison

All models show extreme fragility to premise reordering. The MoE architecture (Mistral) is slightly more fragile overall, possibly due to routing instability under input perturbations.

4.4 Error Analysis

We manually categorize 200 CoT errors induced by $\mathcal{P}_5$ (premise reordering) on GSM8K:

4.5 Does Increasing Exemplars Help?

We test whether more few-shot exemplars improve robustness on GSM8K:

More exemplars provide marginal robustness improvement but do not fundamentally resolve the fragility. Even at 16-shot, the RFI remains above 35.

5. Discussion

5.1 What Does CoT Actually Learn?

Our results suggest that CoT prompting elicits a form of "reasoning" that is heavily dependent on the sequential presentation order of information. This is inconsistent with genuine logical reasoning (which is order-invariant for independent premises) but consistent with an autoregressive generation process that extends surface-level patterns.

The fact that direct prompting is robust to reordering while CoT is not implies that the reasoning trace is more fragile than the reasoning conclusion. Models can often arrive at correct answers through pattern matching without explicit reasoning, and this implicit computation is more robust than the explicit chain generated by CoT.

5.2 Implications for Evaluation

Current CoT evaluation protocols implicitly assume that the model's reasoning is robust to presentation variations. Our RFI metric provides a simple diagnostic: if $\text{RFI} \gg 1$ for a given benchmark, the benchmark is measuring pattern sensitivity rather than reasoning capability.

We recommend that benchmark designers report perturbation-robust accuracy alongside standard accuracy.

5.3 Limitations

1. Perturbation design: Our perturbations are automated and may not represent naturally occurring variation. Adversarial optimization could find even more effective perturbations.

2. Model scale: We evaluate 46-72B parameter models. The relationship between scale and RFI is an open question—it is possible that larger models (400B+) show different fragility patterns.

3. Greedy decoding: Self-consistency decoding [4] across multiple samples may improve robustness, though at significant computational cost.

4. No fine-tuning: We evaluate only instruction-tuned models without perturbation-aware training. Data augmentation with perturbed inputs during fine-tuning may reduce fragility.

6. Conclusion

We demonstrated that chain-of-thought reasoning in LLMs is systematically fragile under meaning-preserving perturbations, with an average Reasoning Fragility Index of 8.7. Premise reordering causes the most severe degradation (RFI = 59), revealing that CoT relies on sequential pattern matching rather than order-invariant logical inference. These findings urge the community to adopt perturbation-robust evaluation and reconsider claims about the depth of LLM reasoning.

References

[1] J. Wei et al., "Chain-of-thought prompting elicits reasoning in large language models," NeurIPS, 2022.

[2] T. Kojima et al., "Large language models are zero-shot reasoners," NeurIPS, 2022.

[3] K. Cobbe et al., "Training verifiers to solve math word problems," arXiv:2110.14168, 2021.

[4] X. Wang et al., "Self-consistency improves chain of thought reasoning in language models," ICLR, 2023.

[5] S. Yao et al., "Tree of thoughts: Deliberate problem solving with large language models," NeurIPS, 2023.

[6] A. Shi et al., "Detecting pretraining data from large language models," ICLR, 2024.

[7] Y. Oren et al., "Proving test set contamination in black box language models," ICLR, 2024.

[8] P. Nakkiran et al., "Deep double descent: Where bigger models and more data can hurt," JSTAT, 2021.