1. Introduction

Retrieval-augmented generation (RAG) has become the dominant paradigm for grounding large language models in external knowledge. At the core of every RAG pipeline lies a semantic similarity computation: given a query and a corpus of documents, an embedding model maps both to a shared vector space, and cosine similarity determines which documents are retrieved. The quality of this retrieval step directly affects downstream generation quality.

Modern embedding models, particularly instruction-tuned variants, accept an optional text prefix—a "prompt template" or "instruction prefix"—that is prepended to the input text before encoding. Model cards typically recommend a specific prefix (e.g., "Represent this sentence for retrieval: " for BGE models, or "search_query: " and "search_document: " for asymmetric retrieval). Practitioners generally adopt these recommendations without further testing, treating the prefix as a fixed configuration parameter rather than a variable that could meaningfully affect retrieval outcomes.

This paper demonstrates that prompt template choice is, in fact, a hidden variable with substantial impact on similarity scores. We show that:

1. Template choice shifts similarity by up to 0.49 points on a [-1, 1] cosine similarity scale, with mean shifts of 0.20 (MiniLM) and 0.15 (BGE-large) across 100 diverse sentence pairs.
2. 15 sentence pairs in BGE-large cross standard classification thresholds solely due to template choice—the same pair of sentences is judged "similar" or "dissimilar" depending on which prefix is used.
3. Even a nonsense prefix increases similarity scores compared to no prefix, confirming that the effect is architectural rather than semantic.
4. Template sensitivity is correlated across models (Spearman ρ = 0.86), indicating that certain sentence pairs are inherently more template-sensitive regardless of the model.
5. The non-instruction-tuned model (MiniLM) is more sensitive to templates than the instruction-tuned model (BGE-large), suggesting that instruction tuning partially stabilizes representations against prefix perturbation.

These findings have immediate practical implications. Template choice can matter as much as threshold choice for retrieval classification. RAG practitioners should treat the prompt template as a hyperparameter that requires systematic evaluation, not a default to copy from documentation.

2. Background

2.1 Sentence Embeddings and Semantic Similarity

Modern sentence embedding models build on transformer architectures (Devlin et al., 2019) to produce fixed-dimensional vector representations of variable-length text. Sentence-BERT (Reimers & Gurevych, 2019) demonstrated that fine-tuning BERT with a siamese network structure produces embeddings where cosine similarity correlates with semantic similarity. Subsequent models have refined this approach, producing increasingly powerful embedding spaces for retrieval, clustering, and classification tasks.

In practice, semantic similarity is computed as the cosine of the angle between two embedding vectors. A threshold is applied to classify pairs as "similar" (above threshold) or "dissimilar" (below threshold). Common thresholds in production systems range from 0.65 to 0.85, depending on the application and model.

2.2 Instruction-Tuned Embeddings

Recent embedding models have adopted instruction tuning, where a natural-language instruction prefix is prepended to the input text before encoding. BGE (Xiao et al., 2023) popularized this approach in the embedding domain, using prefixes like "Represent this sentence for retrieval: " to condition the model's behavior for different downstream tasks.

The mechanical process is straightforward: the prefix string is concatenated with the input text, the combined string is tokenized, and the full token sequence is passed through the transformer. The prefix tokens participate in self-attention with the input tokens, and all tokens (including prefix tokens) contribute to the final pooled representation (typically via mean pooling over the last hidden states).

This means that the prefix does not merely "instruct" the model in a symbolic sense—it materially alters the computation by introducing additional tokens that shift attention patterns and the pooling centroid. This mechanical effect is the focus of our investigation.

2.3 The Gap: Quantifying Template Sensitivity

Model cards for instruction-tuned embedding models typically specify a recommended prefix and sometimes note that using no prefix will degrade performance. However, to our knowledge, no systematic study has quantified the magnitude of the similarity shift induced by different templates, the conditions under which template choice changes retrieval outcomes, or whether template sensitivity is a property of the model, the data, or both. We address these questions directly.

3. Experimental Setup

3.1 Models

We evaluate two models representing distinct points in the embedding model design space:

all-MiniLM-L6-v2 (MiniLM): A 22M-parameter model based on a 6-layer MiniLM architecture, distilled from a larger model and fine-tuned for semantic similarity using the Sentence-Transformers framework. This model was not trained with instruction prefixes—its training data consists of raw sentence pairs without any template. It serves as a baseline: any template sensitivity observed in this model is purely an artifact of the architecture (tokenization, attention, and pooling), not learned instruction-following behavior.

BAAI/bge-large-en-v1.5 (BGE-large): A 335M-parameter model based on a 24-layer BERT-large architecture, trained with contrastive learning on large-scale text pairs with instruction prefixes (Xiao et al., 2023). This model was explicitly trained to use the prefix "Represent this sentence: " and its variants for different tasks. It represents the instruction-tuned paradigm.

Both models use mean pooling over the last hidden layer to produce 384-dimensional (MiniLM) and 1024-dimensional (BGE-large) embeddings. Experiments were conducted using sentence-transformers 3.0.1 with PyTorch 2.4.0.

3.2 Templates

We evaluate 10 prompt templates spanning four categories:

No prefix (baseline):

• none: Empty string (raw input, no prefix)

Standard retrieval prefixes:

• query: "query: "
• search_query: "search_query: "
• search_document: "search_document: "
• passage: "passage: "

Instruction-style prefixes:

• represent: "Represent this sentence: "
• represent_retrieval: "Represent this sentence for retrieval: "

Task-specific prefixes:

• clustering: "clustering: "
• classification: "classification: "

Noise control:

• noise: "xyzzy: " (a nonsense prefix to test whether any prefix shifts similarity, regardless of semantic content)

The same template is applied symmetrically to both sentences in each pair. This isolates the effect of the template itself, independent of asymmetric query-document formatting.

3.3 Sentence Pairs

We construct 100 sentence pairs across 7 semantic categories designed to probe different aspects of similarity:

The adversarial categories (negation, entity_swap, temporal, quantifier, hedging) represent known failure modes of embedding models, where surface-level lexical similarity is high but semantic meaning differs substantially. These categories let us test whether template sensitivity interacts with model failure modes.

3.4 Metrics

For each sentence pair under each template, we compute cosine similarity. Per pair, we derive:

• Standard deviation (SD) across the 10 templates: measures overall template sensitivity
• Maximum shift: the difference between the highest and lowest similarity across templates
• Template flip: whether the pair crosses a similarity threshold under different templates (i.e., classified as "similar" under one template and "dissimilar" under another)

We also compute Spearman rank correlations between templates to assess whether template choice changes the relative ordering of pairs, not just their absolute scores.

4. Results

4.1 Template Sensitivity Overview

Both models exhibit substantial template sensitivity, but with notable differences in magnitude:

The most striking finding is the magnitude of the shifts. A mean max shift of 0.20 points (MiniLM) means that, on average, the most favorable and least favorable templates for any given pair differ by one-fifth of the full similarity scale. For 38% of pairs in MiniLM and 33% in BGE-large, this shift exceeds 0.20 points—a range that spans typical threshold boundaries in production systems.

The non-instruction-tuned model (MiniLM) is consistently more sensitive than the instruction-tuned model (BGE-large). MiniLM's mean SD is 50% higher (0.061 vs 0.040), and its mean max shift is 35% higher (0.199 vs 0.147). This is counterintuitive: one might expect a model that was not trained with prefixes to simply ignore them. Instead, the untrained model's representations are more easily perturbed by prefix tokens. We discuss the mechanism in Section 5.

4.2 Per-Template Ranking

The ranking of templates by mean similarity differs between models:

MiniLM (highest to lowest mean similarity):

1. represent_retrieval: 0.708
2. clustering: 0.697
3. search_query: 0.695
4. noise: 0.678
5. represent: 0.657
6. search_document: 0.656
7. passage: 0.613
8. classification: 0.600
9. query: 0.587
10. none: 0.538

BGE-large (highest to lowest mean similarity):

1. passage: 0.862
2. clustering: 0.838
3. represent: 0.812
4. search_query: 0.804
5. represent_retrieval: 0.802
6. query: 0.794
7. search_document: 0.790
8. classification: 0.775
9. noise: 0.765
10. none: 0.722

Several patterns emerge:

"None" is always lowest. In both models, using no prefix produces the lowest mean similarity. The gap between "none" and the highest template is 0.170 for MiniLM and 0.140 for BGE-large. This means that any prefix—even a nonsense one—pushes similarity scores upward.

The models disagree on the best template. MiniLM peaks with "represent_retrieval" while BGE-large peaks with "passage." The recommended BGE prefix ("Represent this sentence...") ranks 3rd in BGE-large but 5th in MiniLM.

"Clustering" ranks high in both models (2nd for both), despite not being a retrieval-oriented prefix.

The range across templates is large. MiniLM spans 0.170 points between the best and worst template; BGE-large spans 0.140 points. For context, the difference between a "good" and "bad" embedding model on standard benchmarks is often smaller than this.

4.3 Template Flips: When Template Choice Changes Classification

A template flip occurs when a sentence pair crosses a similarity threshold under different templates—meaning that template choice alone determines whether the pair is retrieved or not. This is the most practically concerning finding.

At a standard threshold of 0.70, BGE-large exhibits flips on 16 sentence pairs (16% of the dataset). These flips are concentrated in two categories:

Negative control pairs (dissimilar sentences): Several pairs that are genuinely unrelated (e.g., "The weather is very cold today" vs "The company filed for bankruptcy") are pushed above the threshold by certain templates. The "passage" and "clustering" templates are the most aggressive inflators, with "passage" pushing three otherwise-dissimilar negative control pairs above 0.70. This means that in a RAG system using the "passage" prefix, these unrelated documents would be incorrectly retrieved.

Quantifier pairs: Pairs like "All patients responded to treatment" vs "No patients responded to treatment" sit near the threshold boundary (cosine similarity around 0.73–0.91 depending on template). With the "none" template, BGE-large correctly assigns lower similarity (0.731) to this semantically opposed pair; with "clustering," it assigns 0.906—well above any reasonable threshold. Template choice determines whether the system recognizes the semantic opposition or treats the sentences as equivalent.

These flips have direct consequences for production RAG systems:

• A document retrieval system using one template might return irrelevant results that a system using a different template would correctly filter out.
• Evaluation benchmarks computed with one template may not reflect production performance with another template.
• A/B testing template changes without controlling for this effect could produce misleading results.

4.4 Category Analysis: What Makes a Pair Template-Sensitive?

Template sensitivity varies dramatically across semantic categories. The following table shows mean max shift per category:

The pattern is clear: dissimilar sentence pairs are far more template-sensitive than similar pairs. Negative control pairs (unrelated sentences) have mean max shifts 2.8x (MiniLM) to 4.8x (BGE-large) higher than positive control pairs (paraphrases).

This makes intuitive sense from a geometric perspective. When two sentences are genuinely similar, their embeddings are close in the vector space, and a template-induced centroid shift moves both embeddings in roughly the same direction, preserving their relative proximity. When two sentences are unrelated, their embeddings point in different directions, and the template-induced shift has different projections onto their respective directions, changing their cosine similarity more dramatically.

Entity swap pairs are virtually template-insensitive (max shift < 0.02 in both models). This is because entity swaps preserve almost all tokens—only their order changes—so the template-induced shift is nearly identical for both sentences.

Temporal pairs are also low-sensitivity (max shift ~0.035), likely because the temporal markers ("before" vs "after") are single tokens that minimally interact with the template tokens.

Quantifier pairs show an interesting divergence: BGE-large is actually more template-sensitive on quantifier pairs than MiniLM (0.149 vs 0.135). This may reflect the instruction-tuned model's greater sensitivity to semantic nuance in the quantifier dimension, which interacts differently with different prefixes.

4.5 Cross-Model Comparison

A key question is whether template sensitivity is a property of the model or the data. If certain sentence pairs are inherently template-sensitive regardless of the model, this would suggest that the phenomenon is driven by the geometry of the sentence pair (how the prefix interacts with the tokens of each sentence), not by model-specific learned behavior.

We find strong evidence for this:

• Spearman correlation of per-pair SD across models: ρ = 0.86 (p < 1e-30)
• Spearman correlation of per-pair max_shift across models: ρ = 0.86 (p < 1e-30)
• 12 of the top 20 most sensitive pairs overlap between models (60% overlap vs 4% expected by chance)

This high correlation indicates that template sensitivity is primarily a property of the sentence pair itself, not the model. Pairs with high lexical overlap but different meanings (like quantifier pairs) or pairs with low lexical overlap (like negative controls) are consistently more template-sensitive across both architectures.

4.6 Rank Correlation Across Templates

While absolute similarity scores shift substantially with template choice, the relative ordering of pairs is more stable. The mean Spearman rank correlation between templates is 0.964 for both models, with minimum correlations of 0.929 (MiniLM: none vs search_query) and 0.937 (BGE-large: represent_retrieval vs passage).

This means that if pair A is more similar than pair B under one template, it is very likely to remain more similar under a different template. The template effect is primarily a shift in absolute scores, not a reordering of pairs. However, the rank correlations are not perfect, and the minimum values (0.93) indicate that approximately 7% of the pairwise ordering variance is attributable to template choice. For pairs near a decision boundary, this reordering is sufficient to change retrieval outcomes—which is exactly what we observe in the template flip analysis.

5. Mechanistic Analysis

5.1 Why Templates Shift Similarity

The mechanism by which prompt templates affect similarity is architectural, not semantic. When a prefix is prepended to the input text, three things happen:

1. Additional tokens enter the sequence. The prefix "Represent this sentence for retrieval: " adds approximately 7 tokens to the input. These tokens participate in self-attention with the input tokens and contribute to the mean-pooled representation.

2. The embedding centroid shifts. Mean pooling averages over all token representations, including prefix tokens. The prefix tokens add a constant (template-dependent) bias to the embedding centroid. This bias is roughly the same for all inputs using the same template, but its effect on pairwise cosine similarity depends on the geometric relationship between the two sentence embeddings and the bias vector.

3. Attention patterns are modified. Prefix tokens attend to input tokens and vice versa through the transformer's self-attention mechanism. This means that the prefix does not merely add a constant offset—it also modulates the representations of the input tokens themselves, in a content-dependent way.

5.2 The Noise Prefix Effect

The most revealing result is the behavior of the noise prefix ("xyzzy: "). This is a nonsense string with no semantic content related to any task. Yet it significantly increases similarity scores compared to no prefix in both models:

In MiniLM, 98 out of 100 pairs have higher similarity with the "xyzzy: " prefix than with no prefix. The mean increase is 0.140 points—higher than many "real" templates. This confirms that the similarity-increasing effect is primarily mechanical (the extra tokens shift the centroid) rather than semantic (the prefix does not "instruct" the model to find more similarity).

The effect is smaller in BGE-large (0.044 vs 0.140), which we attribute to instruction tuning: BGE-large has learned to partially compensate for prefix tokens that differ from its training distribution. But even in BGE-large, the effect is highly significant (p < 1e-16) and positive in 91% of pairs.

5.3 Why the Non-Instruction-Tuned Model is More Sensitive

A counterintuitive finding is that MiniLM (no instruction training) is more template-sensitive than BGE-large (instruction-trained). We propose three contributing factors:

First, instruction tuning acts as a regularizer. BGE-large was trained with various prefixes during contrastive learning. This training implicitly teaches the model to produce consistent similarity judgments despite variation in prefixes. The model has learned to down-weight the contribution of prefix tokens to the final representation, or to project away the prefix-dependent component during pooling.

Second, model capacity matters. BGE-large has 15x more parameters than MiniLM (335M vs 22M) and uses 1024-dimensional embeddings vs 384-dimensional. The higher-dimensional space provides more room for the prefix-induced shift to be orthogonal to the semantically meaningful dimensions, reducing its impact on cosine similarity.

Third, MiniLM's smaller model amplifies perturbations. With only 6 layers and 384 dimensions, MiniLM has less capacity to separate the prefix signal from the input signal. The prefix tokens' influence propagates through fewer layers and is pooled into a lower-dimensional space, where it occupies a proportionally larger fraction of the representation.

6. Practical Implications

6.1 Always Test Multiple Templates

Our results demonstrate that template choice is not a "set and forget" configuration parameter. We recommend the following protocol:

1. Evaluate at least 3-5 templates on a representative sample of your data before deploying an embedding model.
2. Include the "no prefix" baseline to understand the template-free behavior of the model.
3. Include a noise prefix (e.g., "xyzzy: ") to establish a lower bound on the mechanical (non-semantic) effect of adding any prefix.
4. Measure both absolute similarity and rank ordering across templates. If rank ordering is stable but absolute scores shift, you may need to adjust thresholds but can use any template. If rank ordering changes, template choice directly affects retrieval quality.

6.2 Template Choice Can Matter as Much as Threshold Choice

In production RAG systems, the similarity threshold is a carefully tuned hyperparameter that balances precision and recall. Our results show that template choice induces shifts comparable to typical threshold adjustments:

• The mean template-induced shift (0.20 for MiniLM, 0.15 for BGE-large) is the same order of magnitude as the typical range of threshold values practitioners consider (0.65–0.85).
• The template effect (Cohen's d = 1.17 for MiniLM, 1.09 for BGE-large, comparing no-prefix to best prefix) is a "large" effect by conventional standards.

This means that changing from one template to another without adjusting the threshold could have the same impact on retrieval behavior as moving the threshold by 0.15–0.20 points.

6.3 Document-Side and Query-Side Templates Should Be Tested Independently

Many retrieval systems use asymmetric templates: one prefix for queries and a different prefix for documents (e.g., "search_query: " for queries and "search_document: " for documents). Our experiment uses symmetric templates (same prefix for both sentences), which represents a simplified case. In asymmetric setups, the interaction between query-side and document-side templates introduces an additional dimension of variability.

We recommend testing query-side and document-side templates independently and in combination. The total number of template combinations is manageable (e.g., 5 query templates × 5 document templates = 25 combinations), and the payoff is substantial.

6.4 Consistency Matters More Than Optimality

Given that template sensitivity is a property of the sentence pair (not just the model), it is impossible to find a single template that is optimal for all pairs. Instead, we recommend prioritizing consistency: choose a template that minimizes variance across your data distribution, even if it does not maximize mean similarity.

A template that produces high similarity on some pairs but low similarity on others is harder to threshold than a template that produces moderately high similarity consistently. The goal is not to maximize retrieval scores but to maximize the separability between relevant and irrelevant pairs.

7. Limitations

Number of models. We evaluate only two embedding models. However, these models represent two distinct architectural paradigms (instruction-tuned vs not), and the finding that both exhibit substantial template sensitivity—with correlated sensitivity patterns—strengthens the generality of our conclusions. Future work should extend this analysis to additional models, including multilingual models, decoder-based embedding models, and models with different pooling strategies (e.g., CLS token pooling).

Number of sentence pairs. Our dataset of 100 pairs, while diverse across 7 semantic categories, is small compared to standard benchmarks. However, the 100 pairs × 10 templates × 2 models = 2000 measurements yield highly significant statistical results (all key comparisons at p < 1e-16), suggesting that the sample is sufficient to establish the phenomenon. Larger-scale validation on benchmark datasets such as STS, MTEB, or domain-specific corpora would be valuable.

Symmetric template application. We apply the same template to both sentences in each pair. Production RAG systems often use asymmetric templates. The symmetric case is a conservative estimate of template sensitivity; asymmetric templates may introduce additional variability.

Threshold sensitivity. Our template flip analysis uses a fixed threshold. In practice, thresholds vary by application, and the number of flips depends on the threshold chosen. Our analysis at the 0.70 threshold is illustrative; practitioners should evaluate flips at their own operational thresholds.

Single language. All experiments use English-language text. Template sensitivity may differ for other languages, particularly for multilingual models where the prefix language may not match the input language.

8. Conclusion

We have demonstrated that prompt template choice is a hidden variable in semantic search that shifts embedding similarity by up to 0.49 points, changes retrieval classification for up to 16% of sentence pairs, and exhibits large effect sizes (Cohen's d > 1.0). Even a nonsense prefix significantly alters similarity scores, confirming that the effect is architectural rather than semantic.

Our key findings challenge common assumptions in the RAG ecosystem:

• Assumption: "Use the recommended prefix and move on." Reality: The recommended prefix is not always the best choice for your data, and the difference between templates can be substantial.
• Assumption: "Instruction-tuned models need instructions; non-instruction models ignore them." Reality: Non-instruction models are more sensitive to templates, not less.
• Assumption: "Template choice is a minor implementation detail." Reality: Template choice can shift similarity by as much as changing the model or the threshold.

We recommend that RAG practitioners treat the prompt template as a first-class hyperparameter: test multiple templates, measure their impact on your specific data distribution, and document the chosen template as part of the system configuration. The hidden variable deserves to be made explicit.

References

• Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT.
• Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of EMNLP-IJCNLP.
• Xiao, S., Liu, Z., Zhang, P., & Muennighoff, N. (2023). C-Pack: Packaged Resources to Advance General Chinese Embedding. arXiv preprint arXiv:2309.07597.

Appendix A: Template Prefix Definitions

Appendix B: Per-Category Sensitivity Statistics

MiniLM

BGE-large

Appendix C: Experimental Environment

• Hardware: Standard compute instance
• Framework: sentence-transformers 3.0.1, PyTorch 2.4.0
• Models: sentence-transformers/all-MiniLM-L6-v2, BAAI/bge-large-en-v1.5
• Similarity metric: Cosine similarity via util.cos_sim()
• Encoding: Default settings (normalize_embeddings=True where applicable)