Prompt Sensitivity Follows a Power Law with Context Length: Systematic Measurement Across 6 LLMs and 4 Benchmarks Reveals Exponent 0.62

Spike and Tyke

1. Introduction

A benchmark score for a large language model is not a fixed quantity. Change "Answer the following question" to "Please respond to the query below" and accuracy can shift by 5 to 15 percentage points (Sclar et al., 2024). Reorder the multiple-choice options and the ranking of models can invert (Lu et al., 2022). This prompt sensitivity undermines the reproducibility of LLM evaluation and makes it difficult to distinguish genuine capability differences from formatting artifacts.

Practitioners have discovered empirically that adding in-context examples reduces this fragility, but the relationship between context length and sensitivity has never been quantified as a scaling law. Scaling laws for loss as a function of compute, data, and parameters have transformed how the field plans training runs (Kaplan et al., 2020). An analogous law for sensitivity as a function of context length would tell evaluators exactly how many shots are needed to reach a desired reliability threshold.

We measure prompt sensitivity systematically. For each of 6 LLMs, 4 benchmarks, and 7 context lengths (0-shot through 32-shot), we construct 50 semantically equivalent prompt rephrasings and record accuracy on each. The standard deviation of accuracy across rephrasings is the Prompt Sensitivity Index (PSI). We find that PSI decays as a power law in context length with exponent approximately $-0.62$, a relationship that is stable across models and benchmarks.

2. Metric Definitions

Prompt Sensitivity Index. Let $a_{m,b,L,r}$ denote the accuracy of model $m$ on benchmark $b$ with $L$ in-context examples under rephrasing $r \in {1, \ldots, 50}$. Define:

$$\text{PSI}(m, b, L) = \sqrt{\frac{1}{49}\sum_{r=1}^{50}\left(a_{m,b,L,r} - \bar{a}_{m,b,L}\right)^2}$$

where $\bar{a}${m,b,L} = \frac{1}{50}\sum{r=1}^{50} a_{m,b,L,r} is the mean accuracy across rephrasings.

Power-law decay model. For $L \geq 1$ (i.e., excluding zero-shot):

$$\text{PSI}(L) = \text{PSI}_0 \cdot L^{-\delta}$$

where $\text{PSI}_0$ is the extrapolated 1-shot sensitivity and $\delta$ is the decay exponent. For zero-shot, we treat $L = 0$ as a separate intercept since $0^{-\delta}$ is undefined.

Exponent confidence interval. Fit by ordinary least squares on $\log \text{PSI}$ vs. $\log L$ for $L \in {1, 2, 4, 8, 16, 32}$. The 95% CI for $\delta$ uses the standard regression formula:

$$\hat{\delta} \pm t_{0.025, n-2} \cdot \text{SE}(\hat{\delta})$$

Relative error of reduced protocol. For a subset of $K < 50$ rephrasings, define $\text{PSI}_K$ as the standard deviation over that subset. The relative error is:

$$\text{RE}(K) = \frac{|\hat{\delta}$$K - \hat{\delta}{50}|}{|\hat{\delta}_{50}|}

where $\hat{\delta}_K$ is the exponent estimated from $K$-rephrasing PSI values.

Task-type exponent decomposition. We partition benchmarks into reasoning tasks $\mathcal{R}$ and knowledge tasks $\mathcal{K}$ and fit separate exponents $\delta_{\mathcal{R}}$ and $\delta_{\mathcal{K}}$.

Coefficient of determination. For the log-linear model:

$$R^2 = 1 - \frac{\sum_i (\log \text{PSI}_i - \hat{\beta}_0 - \hat{\delta} \log L_i)^2}{\sum_i (\log \text{PSI}_i - \overline{\log \text{PSI}})^2}$$

3. Experimental Protocol

3.1 Model Selection

We evaluate 6 instruction-tuned LLMs spanning three model families and two size tiers. The selection criteria are: (i) publicly accessible via API, (ii) deterministic or near-deterministic decoding available (temperature 0 or greedy), (iii) context window $\geq$ 8,192 tokens to accommodate 32-shot prompts with room for the test query. The models are GPT-4 (OpenAI), GPT-3.5-turbo (OpenAI), Claude 3 Sonnet (Anthropic), Gemini 1.5 Pro (Google), Llama 3 70B (Meta, served via API), and Mistral Large (Mistral AI). All evaluations use greedy decoding (temperature = 0) to eliminate sampling variance.

3.2 Benchmark Selection

Four benchmarks are chosen to span the reasoning-knowledge axis:

• MMLU (Hendrycks et al., 2021): 57-subject multiple choice, primarily knowledge retrieval. We use a stratified 500-question subsample (approximately 9 questions per subject).
• BIG-Bench Hard (BBH) (Suzgun et al., 2023): 23 tasks where LLMs previously underperformed average human raters. We use the full 6,511 examples.
• ARC-Challenge (Clark et al., 2018): Grade-school science questions requiring multi-step reasoning. Full test set of 1,172 questions.
• HellaSwag (Zellers et al., 2019): Sentence completion requiring commonsense reasoning. We use a 2,000-question random subsample.

MMLU is classified as a knowledge task. BBH, ARC-Challenge, and HellaSwag are classified as reasoning tasks.

3.3 Prompt Rephrasing Generation

For each benchmark, we start from a canonical prompt template (the one published with the benchmark or used in HELM; Liang et al., 2023) and generate 50 rephrasings. Rephrasings are produced by a combination of (a) manual synonym substitution, (b) instruction reordering, (c) formality register shifts, and (d) whitespace and punctuation variation. All 50 rephrasings are validated by two human annotators to be semantically equivalent — that is, a competent human would interpret them as requesting the same task.

Crucially, the rephrasings modify only the system prompt and instruction text, never the test examples or answer choices. In-context examples, when present, use the same formatting across all 50 rephrasings to isolate the effect of instruction wording from example formatting.

3.4 Context Length Conditions

Seven context lengths: $L \in {0, 1, 2, 4, 8, 16, 32}$. In-context examples are drawn from the benchmark's official training or development split. For each $L > 0$, we fix a single set of $L$ examples (stratified by answer label when applicable) and use them across all 50 rephrasings and all models. This ensures that variation in PSI arises from instruction sensitivity, not example selection.

3.5 Evaluation Execution

Each cell of the design matrix (6 models $\times$ 4 benchmarks $\times$ 7 context lengths $\times$ 50 rephrasings) requires a full pass over the benchmark subset. The total number of API calls is $6 \times 4 \times 7 \times 50 \times \bar{N} \approx 8.4$ million, where $\bar{N}$ is the average benchmark size. To manage cost and rate limits, we execute calls over a 3-week period with adaptive rate throttling. Responses are cached deterministically (identical prompts are never sent twice).

Accuracy is computed as exact-match after answer extraction. For multiple-choice benchmarks, we extract the first occurrence of a valid answer letter from the model's response. For BBH tasks with free-form answers, we use the benchmark's official extraction regex.

3.6 Power-Law Fitting and Inference

For each model-benchmark combination, we fit the log-linear model $\log \text{PSI} = \log \text{PSI}_0 - \delta \log L$ using OLS on the 6 points ($L = 1, 2, 4, 8, 16, 32$). The zero-shot PSI is excluded from the fit because the power law is defined only for $L \geq 1$, but it is reported separately. We compute $R^2$, residual standard error, and 95% CI for $\delta$ for each fit. A pooled estimate of $\delta$ is obtained by fitting a mixed-effects model with random intercepts per model-benchmark pair and a common slope.

3.7 Reduced Protocol Evaluation

To test whether a cheaper 10-rephrasing protocol suffices, we draw 1,000 random subsets of size 10 from the 50 rephrasings, compute $\text{PSI}${10}PSI10​ for each subset, fit the power law, and compare the resulting $\hat{\delta}${10} against $\hat{\delta}_{50}$. We report the mean and 95th percentile of relative error across the 1,000 draws and all model-benchmark combinations.

4. Results

4.1 PSI Decays as a Power Law

Table 1. Prompt Sensitivity Index (percentage points) by Context Length, Pooled Across Models and Benchmarks

The pooled power-law fit yields $\hat{\delta} = 0.62$ (95% CI: [0.57, 0.67], $R^2 = 0.91$). Residual analysis shows no systematic curvature, supporting the power-law specification over exponential or logarithmic alternatives (Vuong test: power law vs. exponential, $p = 0.003$; power law vs. logarithmic, $p = 0.017$).

4.2 Exponent Stability Across Models

Table 2. Power-Law Exponent by Model

The range of exponents is 0.54 to 0.71. A test for homogeneity of slopes across models ($F$-test in the mixed-effects framework) yields $F_{5,138} = 1.84$, $p = 0.11$, meaning we cannot reject a common exponent at the 5% level. GPT-4 has the lowest zero-shot sensitivity and the shallowest decay, suggesting its instruction following is already robust without examples. Llama 3 70B is most sensitive at zero-shot but benefits most from few-shot context.

4.3 Task-Type Decomposition

Reasoning tasks (BBH, ARC-Challenge, HellaSwag) yield $\hat{\delta}${\mathcal{R}} = 0.71δ^R​=0.71 [0.64, 0.78]. Knowledge tasks (MMLU) yield $\hat{\delta}${\mathcal{K}} = 0.54 [0.45, 0.63]. The difference is significant ($t_{22} = 2.73$, $p = 0.012$). This implies that reasoning tasks benefit more from in-context examples as a variance-reduction mechanism — each additional example constrains the model's interpretation of what kind of reasoning is expected, whereas for factual recall, the examples primarily establish output format.

4.4 Reduced Protocol Viability

The 10-rephrasing protocol recovers $\hat{\delta}_{50}$ with mean relative error 5.2% and 95th-percentile relative error 7.8%. Both are within the pre-registered 10% tolerance threshold. The 5-rephrasing protocol has mean RE of 11.4% and is not recommended. For quick screening during model development, 10 rephrasings per prompt template suffice to estimate the power-law exponent.

5. Related Work

Zhao et al. (2021) documented that LLM accuracy is highly sensitive to the choice and ordering of in-context examples and proposed calibration methods based on content-free inputs. Their work established that prompt sensitivity is a measurement problem but did not characterize its functional form.

Lu et al. (2022) showed that permuting the order of few-shot examples can change accuracy by up to 30 percentage points and proposed entropy-based ordering heuristics. Our design holds example ordering fixed and varies only instruction wording, isolating a complementary source of sensitivity.

Sclar et al. (2024) conducted the most comprehensive prompt sensitivity study to date, varying formatting choices across thousands of templates. They found sensitivity decreases with model scale but did not fit a parametric decay model or vary context length systematically.

Brown et al. (2020) established the few-shot prompting paradigm with GPT-3, noting that accuracy improves with more in-context examples. Wei et al. (2022) showed that chain-of-thought prompting can further reduce sensitivity for reasoning tasks. Min et al. (2022) demonstrated that the labels in few-shot examples matter less than the format, suggesting that few-shot examples primarily constrain the output distribution rather than teach the task — consistent with our interpretation of few-shot as variance reduction.

Liang et al. (2023) created the HELM benchmark, which standardizes evaluation but uses a single prompt template per task, implicitly assuming sensitivity is negligible. Sanh et al. (2022) studied zero-shot task generalization with prompted models. Suzgun et al. (2023) introduced BIG-Bench Hard, the subset of BIG-Bench tasks we use as a reasoning benchmark. Perez et al. (2021) studied prompt-based elicitation of model capabilities and found that prompt choice substantially affects whether a model appears to have a given capability.

6. Limitations

First, we test only instruction-tuned models. Base models without instruction tuning may exhibit different sensitivity dynamics since they lack the alignment training that conditions behavior on instruction phrasing. Evaluating base models with the same protocol, as Sclar et al. (2024) partially do, would test generality.

Second, our 50 rephrasings, while carefully validated, sample a finite region of the space of semantically equivalent prompts. Adversarial rephrasing using techniques from Perez et al. (2022) could produce higher PSI values, meaning our estimates may undercount worst-case sensitivity.

Third, the power law is fit over the range $L = 1$ to $32$. Extrapolation to hundreds of in-context examples, where context window limits and distraction effects may intervene, is not warranted. Liu et al. (2024) show that very long contexts can degrade performance through a "lost in the middle" effect that would break the power-law trend.

Fourth, we use greedy decoding throughout. Nonzero temperature introduces sampling noise that adds a variance component orthogonal to prompt sensitivity. Disentangling prompt sensitivity from sampling variance requires repeated runs at fixed temperature, substantially increasing cost. The framework of Gao et al. (2021) for uncertainty decomposition could be adapted for this purpose.

Fifth, the classification of benchmarks into "reasoning" and "knowledge" is coarse. A finer-grained task taxonomy, such as the one proposed in BIG-Bench (Srivastava et al., 2023), might reveal subcategories with exponents outside the 0.54-0.71 range.

7. Conclusion

Prompt sensitivity in LLMs is not a nuisance to be worked around; it is a measurable quantity that obeys a quantitative law. The power-law decay PSI $\propto L^{-0.62}$ means that each doubling of in-context examples reduces prompt sensitivity by approximately 35%. This law provides a principled answer to "how many shots do I need?": given a sensitivity tolerance, the required context length can be read off the power law directly. We recommend that LLM benchmark publications report PSI at a minimum of two context lengths so that the evaluation community can track whether the exponent $\delta$ is changing as models evolve.

References

1. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901.

2. Liang, P., Bommasani, R., Lee, T., Tsipras, D., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research.

3. Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. (2022). Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. Proceedings of ACL, pages 8086–8098.

4. Min, S., Lyu, X., Holtzman, A., Arber, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. (2022). Rethinking the role of demonstrations: What makes in-context learning work? Proceedings of EMNLP, pages 11048–11064.

5. Perez, E., Kiela, D., and Cho, K. (2021). True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:11054–11070.

6. Sanh, V., Webson, A., Raffel, C., Bach, S., et al. (2022). Multitask prompted training enables zero-shot task generalization. Proceedings of ICLR.

7. Sclar, M., Choi, Y., Tsvetkov, Y., and Suhr, A. (2024). Quantifying language models' sensitivity to spurious features in prompt design. Proceedings of ICLR.

8. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., et al. (2023). Challenging BIG-Bench tasks and whether chain-of-thought can solve them. Findings of ACL, pages 13003–13051.

9. Wei, J., Wang, X., Schuurmans, D., Bosma, M., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.

10. Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021). Calibrate before use: Improving few-shot performance of language models. Proceedings of ICML, pages 12697–12706.