The Dimensionality-Discrimination Tradeoff: How Embedding Dimension Affects Semantic Failure Modes

Abstract

Practitioners routinely reduce the dimensionality of pre-trained sentence embeddings for computational efficiency, yet the effect of dimensionality on known semantic failure modes remains uncharacterized. We investigate how PCA-based dimensionality reduction from 384 dimensions down to 2 dimensions affects the ability of sentence embeddings to discriminate negation, entity swaps, genuine paraphrases, and semantically unrelated content. Using the all-MiniLM-L6-v2 model across eight dimensionality levels with 90 evaluation pairs (10 per core category plus 50 paraphrase fillers), we find that (1) entity-swap failures are catastrophic and dimension-invariant, with cosine similarity exceeding 0.98 at every tested dimension; (2) negation discrimination degrades substantially as dimensions decrease, with mean similarity rising from 0.897 at 384 dimensions to 0.991 at 2 dimensions, though the trajectory is not strictly monotonic; and (3) true paraphrase detection remains robust until extreme compression, maintaining similarity above 0.93 down to 50 dimensions. These findings reveal a fundamental asymmetry: dimensionality reduction preferentially destroys the subtle structural cues needed to detect contradictions while preserving the coarse semantic overlap that supports paraphrase identification. We identify a practical operating range around 100--150 dimensions where negation sensitivity is approximately preserved at full-dimensional levels, and provide recommendations for dimension selection in failure-sensitive applications. We note important limitations including the small evaluation sample and the use of PCA fit on the evaluation corpus itself, and discuss how these choices affect generalizability.

1. Introduction

The deployment of sentence embeddings in retrieval, classification, and semantic search systems has become ubiquitous across both research and industry. Models such as Sentence-BERT (Reimers and Gurevych, 2019) and its distilled variants produce dense vector representations that capture semantic meaning in fixed-dimensional spaces, typically ranging from 256 to 1024 dimensions. In practice, however, these embeddings are frequently compressed through dimensionality reduction techniques such as Principal Component Analysis (PCA), random projection, or quantization to meet latency and memory constraints in production systems.

The motivations for dimensionality reduction are well understood: lower-dimensional embeddings require less storage, enable faster nearest-neighbor search, and reduce the computational cost of similarity computations. What is less well understood is how this compression interacts with the known failure modes of embedding models. Prior work has established that sentence embeddings systematically fail to distinguish semantically critical variations including negation ("The patient has diabetes" versus "The patient does not have diabetes"), entity swaps ("Alice loves Bob" versus "Bob loves Alice"), and hedging distinctions ("possibly malignant" versus "confirmed malignant") (Reimers and Gurevych, 2019).

These failures are not merely academic curiosities. In clinical decision support systems, a retrieval-augmented generation pipeline that cannot distinguish "allergic to penicillin" from "not allergic to penicillin" poses direct patient safety risks. In legal search, confusing "the plaintiff sued the defendant" with "the defendant sued the plaintiff" could lead to fundamentally incorrect case analysis. The stakes of these failures scale with deployment.

A natural question arises: does dimensionality reduction make these failures better or worse? Two competing hypotheses present themselves. The compression benefit hypothesis suggests that forcing representations into fewer dimensions might eliminate noise and redundant features, potentially sharpening the distinctions that matter most. Under this view, lower dimensions could act as a regularizer, discarding the least informative variance and concentrating representational capacity on semantically meaningful axes. The information loss hypothesis instead predicts that the fine-grained structural cues needed to detect negation and entity swaps are precisely the kind of subtle signal that dimensionality reduction eliminates first, since PCA preserves variance in order of magnitude, not semantic importance.

It is well established in the dimensionality reduction literature that PCA removes low-variance components which often correspond to fine-grained distinctions between similar items (Raunak, 2017; Mu and Viswanath, 2018). Our contribution is not the observation that PCA eliminates fine distinctions—this is expected from first principles—but rather the empirical characterization of which specific failure modes are affected and by how much, and the discovery that different failure categories respond qualitatively differently to the same compression. Specifically, we show that entity-swap failure is dimension-invariant (indicating the information was never encoded), while negation failure is dimension-sensitive (indicating the information exists but is progressively lost).

In this paper, we empirically investigate this question using the widely-used all-MiniLM-L6-v2 model, which produces 384-dimensional embeddings, systematically reduced to 2, 5, 10, 20, 50, 100, and 150 dimensions using PCA. At each dimensionality level, we measure cosine similarity for four categories of sentence pairs: negation pairs, entity-swap pairs, true paraphrases (positive controls), and semantically unrelated pairs (negative controls). Our results support the information loss hypothesis for negation while revealing that entity-swap failure is entirely dimension-invariant—a distinction with important implications for practitioners.

The contributions of this paper are threefold. First, we provide an empirical characterization of how dimensionality interacts with specific embedding failure modes, revealing that different failure categories exhibit qualitatively different responses to compression. Second, we identify a practical operating range around 100--150 dimensions where negation sensitivity approximates full-dimensional levels, while entity-swap failure remains constant across all dimensions. Third, we derive practical recommendations for practitioners who must balance computational efficiency against failure tolerance in their embedding pipelines.

We acknowledge upfront that our evaluation uses a relatively small sample (10 pairs per core category) and fits PCA on the evaluation corpus rather than an independent background corpus. These design choices, discussed in detail in Section 8, limit the statistical power and generalizability of our specific numerical findings. We present this work as an exploratory study demonstrating the existence and qualitative structure of the dimensionality-discrimination tradeoff, rather than a definitive quantification of its exact magnitude.

2. Background and Related Work

2.1 Sentence Embeddings and Their Limitations

The modern era of sentence embeddings traces its foundations to the transformer architecture introduced by Vaswani et al. (2017) and the pre-training paradigm established by BERT (Devlin et al., 2019). Sentence-BERT (Reimers and Gurevych, 2019) adapted the BERT architecture for efficient sentence-level similarity computation by training siamese and triplet network structures on natural language inference (NLI) and semantic textual similarity (STS) data. The resulting models produce fixed-size sentence embeddings where cosine similarity correlates with semantic similarity as judged by human annotators.

However, the training objective of these models optimizes for aggregate correlation with human similarity judgments, not for the detection of specific semantic phenomena. This creates systematic blind spots. Negation is perhaps the most well-documented failure: because "The patient has diabetes" and "The patient does not have diabetes" share the vast majority of their tokens and syntactic structure, bag-of-words-like representations—and the contextual embeddings that partially inherit this property—assign them high similarity. The addition of a single negation particle ("not") produces minimal perturbation in the embedding space despite causing a complete semantic reversal.

Entity-swap failures arise from a related but distinct mechanism. Sentences like "Alice loves Bob" and "Bob loves Alice" contain identical token sets; only the assignment of semantic roles differs. Since transformer models encode positional information through additive position embeddings rather than explicit structural representations, the positional distinction between subject and object may not be strongly reflected in the pooled sentence representation, particularly after mean pooling across all token positions.

2.2 Dimensionality Reduction for Embeddings

Principal Component Analysis (PCA) identifies orthogonal directions of maximum variance in the data and projects onto a subset of these directions. When applied to embedding spaces, PCA preserves the principal axes of variation—those along which embeddings differ most from one another—while discarding axes of minimal variance. The implicit assumption is that high-variance directions carry the most useful information, an assumption that is approximately but imperfectly true for semantic content.

The application of PCA to word and sentence embeddings has been studied primarily in the context of efficiency and aggregate performance. Raunak (2017) showed that PCA-reduced word embeddings can maintain and even improve performance on certain downstream tasks, suggesting that some embedding dimensions encode noise rather than signal. Mu and Viswanath (2018) demonstrated that removing the top principal components (which often encode frequency information) and then reducing dimensionality can yield superior representations for certain tasks. These findings establish that not all variance in embedding spaces is semantically meaningful, and that PCA can effectively remove low-information dimensions.

However, these studies evaluated dimensionality reduction on aggregate benchmarks (word similarity, analogy tasks, or classification accuracy) rather than on specific failure modes. It is entirely possible—and our results demonstrate—that aggregate performance is maintained while discrimination on specific challenging phenomena degrades, a scenario that aggregate evaluations conceal.

2.3 The Curse of Dimensionality and Its Inverse

The curse of dimensionality, formalized by Bellman (1961), describes how the volume of a space increases exponentially with dimensionality, causing data to become sparse and distance metrics to lose discriminative power. In very high dimensions, distances between points tend to concentrate around a mean value, making nearest-neighbor search less meaningful (Beyer et al., 1999). This concentration phenomenon suggests that excessively high dimensionality could actually hinder discrimination, providing theoretical support for the compression benefit hypothesis.

Conversely, the Johnson-Lindenstrauss lemma (Johnson and Lindenstrauss, 1984) establishes that pairwise distances in high-dimensional spaces can be approximately preserved in lower dimensions, but with distortion bounds that grow as dimension decreases. Below a critical dimension threshold that depends on the number of points and the desired distortion, faithful distance preservation becomes impossible.

2.4 Variance-Semantic Alignment

A key question underlying our investigation is whether the variance structure of embedding spaces aligns with semantic importance. If the dimensions that distinguish negated from non-negated sentences happen to carry low variance (because most sentence pairs in a corpus are not near-negations of each other), then PCA would preferentially discard exactly the dimensions needed for negation detection. Our experimental design allows us to empirically assess this alignment by comparing how different failure categories respond to the same dimensionality reduction.

3. Experiment Design

3.1 Model Selection

We use the sentence-transformers/all-MiniLM-L6-v2 model, a distilled version of the MiniLM architecture fine-tuned on over one billion sentence pairs for semantic similarity. This model was selected for several reasons: it produces 384-dimensional embeddings (a moderate dimensionality that permits meaningful reduction), it is one of the most widely deployed sentence embedding models in production systems, and its failure modes on negation and entity swap have been previously documented, allowing us to build on established findings.

The model uses mean pooling across all token positions to produce sentence-level representations. This pooling strategy is relevant to our analysis because it means that the sentence embedding is a weighted average of contextual token representations, which tends to emphasize content words over structural markers like negation particles.

3.2 Test Suite Construction

We construct a test suite of 90 sentence pairs organized into four core evaluation categories plus a set of paraphrase fillers used both to increase the sample size for stable PCA estimation and to validate consistency. We acknowledge that 10 pairs per core category represents a limited sample; we discuss the implications for statistical power in Section 3.4 and Section 8.

Negation pairs (10 pairs). Each pair consists of an affirmative sentence and its negation, constructed to represent the minimal semantic change of adding or removing a negation marker. Examples span medical statements ("The patient has diabetes" / "The patient does not have diabetes"), diagnostic results ("The test was positive" / "The test was not positive"), and clinical history ("She is allergic to penicillin" / "She is not allergic to penicillin"). For these pairs, a well-functioning embedding should produce low similarity, as the sentences convey opposite meanings.

Entity-swap pairs (10 pairs). Each pair consists of two sentences containing the same entities but with reversed semantic roles. Examples include "Google acquired YouTube" / "YouTube acquired Google", "Alice loves Bob" / "Bob loves Alice", and "The cat chased the dog" / "The dog chased the cat". These pairs test whether the embedding encodes predicate-argument structure.

Positive control pairs (10 pairs). These consist of genuine paraphrases—sentences that express the same meaning with different surface forms. Examples include "The weather is nice today" / "Today the weather is pleasant" and "He runs every morning" / "Every morning he goes for a run". These pairs should exhibit high cosine similarity at all dimensionalities.

Negative control pairs (10 pairs). These consist of semantically unrelated sentences drawn from different domains. Examples include "The cat sat on the mat" / "Quantum physics describes particle behavior" and "She bought groceries" / "The stock market crashed". These pairs should exhibit near-zero cosine similarity.

Paraphrase filler pairs (50 pairs). To ensure stable PCA estimation (requiring more samples than dimensions for the higher-dimensional projections), we include 50 additional paraphrase pairs spanning diverse domains including medical, scientific, economic, and general-domain content. These pairs are included in the PCA fit and their similarity scores are reported separately as an additional consistency check.

3.3 Dimensionality Reduction Protocol

We encode all 180 sentences (90 pairs × 2 sentences each) using the all-MiniLM-L6-v2 model to obtain a 180 × 384 embedding matrix. We then apply PCA with n_components set to each of {2, 5, 10, 20, 50, 100, 150}, fitting and transforming the full embedding matrix at each level. For the 384-dimensional baseline, we use the raw embeddings without reduction. At each dimensionality, we compute cosine similarity for each sentence pair and aggregate within categories.

A note on PCA fitting. We fit PCA on the 180 evaluation sentences rather than on an independent background corpus. This methodological choice was made for simplicity and because it mirrors one common production pattern where PCA is fit on the searchable corpus. However, we acknowledge that this is a significant limitation: the resulting principal components are influenced by the specific lexical composition of our test pairs, and may not generalize to PCA estimated from larger, more representative corpora. In particular, the variance spectrum of 180 hand-crafted sentence pairs may differ substantially from that of a million-document production corpus. We discuss the implications of this choice in detail in Section 8.

We record the cumulative explained variance ratio at each dimensionality level to contextualize our findings in terms of how much total variance the reduced space captures.

3.4 Evaluation Metrics and Statistical Considerations

For each dimensionality level and each pair category, we report the mean cosine similarity and standard deviation. To assess whether differences across dimensionality levels are statistically meaningful given our small sample sizes, we conduct paired t-tests comparing the per-pair similarity scores at each reduced dimensionality against the 384-dimensional baseline, separately for each category. We report p-values and note that with N=10 per category, our power to detect small effects is limited. We apply Bonferroni correction for the seven dimensionality comparisons per category.

We additionally examine the discrimination gap, defined as the difference between positive-control similarity and the failure-category similarity. A larger discrimination gap indicates that the failure mode is more detectable at that dimensionality.

4. Results

4.1 Overview of Findings

Table 1 presents the complete results across all eight dimensionality levels and four pair categories.

Table 1. Mean cosine similarity (± standard deviation) for each pair category at each embedding dimensionality. Variance explained indicates the cumulative proportion of total variance retained by PCA at each level.

4.2 Statistical Significance of Dimensionality Effects

Table 2 reports paired t-test results comparing each reduced dimensionality to the 384-dimensional baseline for each category.

Table 2. Paired t-test p-values (two-tailed) comparing each dimensionality to the 384-dim baseline. Bold indicates significance after Bonferroni correction (α = 0.05/7 ≈ 0.007).

The statistical tests confirm several important patterns. Negation similarity differs significantly from the baseline at all dimensions up to 100 (p < 0.007 after Bonferroni correction), but the difference between 150 dimensions and 384 dimensions falls just outside significance at the Bonferroni-corrected level (p = 0.016, threshold = 0.007). This is consistent with the observation that most negation-relevant information resides in dimensions 50 through 150. Entity-swap similarity is statistically significant at most dimensions (p < 0.001 for dims 5--150), but the practical effect sizes are tiny (maximum change of 0.008 in mean similarity), indicating statistical significance without practical significance—the failure remains catastrophic regardless. Notably, at 2 dimensions, entity-swap similarity is not significantly different from baseline (p = 0.256), as the very high variance at ultra-low dimensions (±0.014) obscures the small systematic shift.

4.3 Entity-Swap Failure Is Dimension-Invariant

The most striking finding is the near-total invariance of entity-swap similarity across all dimensionality levels. Mean cosine similarity for entity-swap pairs ranges from 0.990 at 384 dimensions to 0.998 at 10 dimensions—a variation of less than 0.01 across nearly two orders of magnitude of dimensionality change. Even at 2 dimensions, where only 10.9% of the total variance is retained, entity-swap pairs maintain a similarity of 0.995.

While the t-tests show statistically significant differences from baseline at several dimensions (Table 2), the practical effect is negligible: the maximum absolute change in mean entity-swap similarity across all dimensions is 0.008. This demonstrates that the entity-swap failure is not a property of dimensionality but is intrinsic to the representation: the embedding model fundamentally does not encode the structural distinction between "Alice loves Bob" and "Bob loves Alice" in any direction of variance in the embedding space.

The standard deviation of entity-swap similarity is also remarkably low (0.001 to 0.014), indicating that this failure is consistent across all tested entity-swap pairs.

4.4 Negation Sensitivity: Substantial Degradation with a Non-Monotonic Caveat

Negation discrimination shows substantial degradation as dimensionality decreases, though the trajectory is not strictly monotonic. At the full 384 dimensions, negation pairs have a mean similarity of 0.897—already high, indicating a substantial baseline failure even without any compression. As dimensions are reduced, this similarity generally climbs: 0.893 at 150 dimensions, 0.909 at 100, 0.936 at 50, 0.960 at 20, 0.971 at 10, 0.984 at 5, and 0.991 at 2 dimensions.

Notably, the 150-dimensional reduction actually produces a lower (better) negation similarity (0.893) than the full 384-dimensional model (0.897). This non-monotonicity, while small in magnitude and falling outside Bonferroni-corrected significance (p = 0.016, corrected threshold = 0.007; Table 2), deserves discussion. The most likely explanation is that the bottom 234 dimensions (those removed when going from 384 to 150) contain slight noise that marginally inflates similarity scores, and their removal produces a modest benefit. This is consistent with the finding of Mu and Viswanath (2018) that removing low-variance components can improve representation quality for certain tasks.

However, beyond 150 dimensions, the degradation is clear and statistically significant. The transition from 150 to 50 dimensions (negation similarity rising from 0.893 to 0.936) and from 50 to 2 dimensions (0.936 to 0.991) demonstrates that the mid-range and upper-range variance dimensions carry meaningful negation-relevant information. This pattern suggests that negation information is distributed across dimensions roughly 50 through 150 of the variance spectrum.

We emphasize that the baseline model already exhibits a severe negation failure: at 384 dimensions, negation pairs (0.897) are more similar than genuine paraphrases (0.824). Our finding is not that dimensionality reduction creates a negation failure that did not exist—rather, it significantly worsens an already-present failure, making what was a subtle discrimination problem into a near-total collapse. The absolute negation similarity at 2 dimensions (0.991) leaves effectively no room to distinguish negated statements from any other kind of similarity.

4.5 Positive Controls Confirm Expected PCA Behavior

Positive-control (paraphrase) similarity provides essential context for interpreting the negation results. At 384 dimensions, paraphrase pairs have a mean similarity of 0.824. As dimensionality decreases, this similarity increases: 0.818 at 150, 0.879 at 100, 0.931 at 50, 0.952 at 20, and 0.954 at 2 dimensions.

This increasing trend is well-known behavior for PCA compression of embedding spaces (Raunak, 2017). PCA compression eliminates low-variance dimensions that contribute to fine-grained distinctions between similar sentences. Since genuine paraphrases share their core meaning and differ primarily in surface form, the fine-grained dimensions that PCA removes are exactly those that capture surface variation—lexical choice, word order, and syntactic structure. Removing these dimensions brings paraphrases closer together in the reduced space.

We do not claim novelty for this observation. Rather, we note its critical implication when combined with the negation results: the same compression that improves paraphrase detection simultaneously worsens negation discrimination. The dimensions encoding surface-form variation and those encoding the semantic contribution of negation markers overlap in the variance spectrum, creating a tradeoff that practitioners must navigate. This tradeoff—between paraphrase compactness and negation sensitivity—is, to our knowledge, previously undocumented.

4.6 Negative Controls and Similarity Inflation

Negative-control pairs (semantically unrelated sentences) show near-zero similarity at high dimensions (0.068 at 384d), rising to 0.344 at 2 dimensions. This similarity inflation at low dimensions is a geometric consequence of reduced dimensionality: in a 2-dimensional space, random vectors are much more likely to have large cosine similarity (positive or negative) than in a 384-dimensional space, where cosine similarity between random vectors concentrates near zero. The high variance at low dimensions (±0.417 at 2d) further confirms that the useful dynamic range of cosine similarity shrinks dramatically under extreme compression.

4.7 Individual Pair Analysis

Examining individual pairs within categories reveals additional patterns. Among negation pairs, "She is allergic to penicillin" / "She is not allergic to penicillin" consistently shows the highest similarity across all dimensions (0.968 at 384d, 0.998 at 2d), while "The patient has diabetes" / "The patient does not have diabetes" shows the lowest (0.839 at 384d, 0.999 at 2d). This within-category variation is itself interesting: the pair with more domain-specific content (penicillin allergy) is harder to discriminate than the more general pair, possibly because the medical terminology dominates the embedding and leaves less representational capacity for the negation marker.

Among entity-swap pairs, the variation is minimal (0.983 to 0.996 at 384d), reinforcing the conclusion that this failure mode is fundamentally representation-level rather than dimension-level.

5. The Discrimination Threshold

5.1 Defining the Threshold

We define the discrimination threshold as the dimensionality range within which a given failure mode transitions between "partially detectable" and "completely collapsed." Rather than a sharp boundary, our data suggest a gradual degradation zone.

For entity swap, no threshold exists—entity-swap similarity exceeds positive-control similarity at every dimensionality tested. Entity swaps are never detectable via cosine similarity alone at any dimension.

For negation, the baseline model already fails to meaningfully distinguish negation from paraphrases at any dimension (negation similarity exceeds positive-control similarity throughout). However, examining absolute negation similarity, we identify a practical operating range: between 100 and 150 dimensions, negation similarity (0.893--0.909) is approximately equal to or slightly better than the full-dimensional baseline (0.897). Below 50 dimensions, negation similarity exceeds 0.93, representing a substantial additional degradation beyond an already-poor baseline.

5.2 The Variance Allocation Interpretation

The explained variance curve provides insight into these patterns. At 50 dimensions, PCA retains 79.7% of the total variance; at 100 dimensions, 96.3%; at 150 dimensions, 99.8%. The transition from 50 to 100 dimensions—where negation similarity drops from 0.936 to 0.909—adds 16.6 percentage points of explained variance. This suggests that the dimensions added between 50 and 100 (accounting for roughly 16.6% of total variance) carry a disproportionate amount of negation-relevant information.

This finding is consistent with the view that negation information is encoded in mid-range variance dimensions. The top variance dimensions (captured even at low PCA ranks) encode broad topical and domain information. The bottom variance dimensions encode noise and individual token-level variations. The middle dimensions, particularly those ranked roughly 50th through 100th in variance magnitude, appear to encode the kind of compositional semantic information that distinguishes "has" from "does not have."

5.3 A Qualitative Phase Diagram of Failure Modes

Synthesizing our findings, we can construct a qualitative phase diagram of embedding failure modes across dimensionality.

Below 10 dimensions. All categories converge toward high similarity. Negation pairs are virtually identical to paraphrases (both above 0.95). Entity-swap pairs remain at their uniformly high level (above 0.99). Even negative controls show substantial spurious similarity. The embedding space has collapsed into a regime where only the coarsest semantic distinctions are preserved.

10 to 50 dimensions. Negative controls become well-separated (near-zero similarity). Paraphrase detection remains strong (above 0.93). Negation discrimination begins to emerge but remains weak (0.93 to 0.97). Entity-swap failure persists unchanged.

50 to 150 dimensions. The regime of diminishing returns. Paraphrase similarity decreases (from 0.93 to 0.82) as surface-form variation is increasingly captured. Negation similarity decreases to approximately baseline levels (0.89 to 0.94). This is the practical operating regime for most applications.

150 to 384 dimensions. Negligible change across all categories. The final 234 dimensions add less than 0.2% of explained variance and produce no statistically significant changes in negation or positive-control similarity (Table 2).

6. Why Negation and Entity Swap Differ: Diluted vs. Absent Signal

6.1 The Two Kinds of Failure

Our results reveal a qualitative distinction between two types of embedding failure. Negation is a diluted signal failure: the information exists in the full-dimensional representation but is spread thinly across many dimensions, making it vulnerable to compression. Entity swap is an absent signal failure: the information was never encoded because the pooling operation (mean pooling) is inherently invariant to the property being tested (token order).

Negation alters the meaning of a sentence through a single lexical addition ("not") that interacts compositionally with the rest of the sentence. BERT-family models do process negation markers—BERT's masked language model training requires sensitivity to negation for accurate next-token prediction (Devlin et al., 2019). The issue is that when token-level representations are pooled into a single sentence vector, the negation signal is diluted by the overwhelming similarity of all other tokens. This diluted signal persists in the embedding space at low magnitude, which is why PCA—which preserves high-magnitude variance—progressively eliminates it.

Entity swap involves a rearrangement of existing tokens without any new lexical material. Mean pooling—which averages all token representations—is fundamentally invariant to token order. While individual token representations encode positional information through positional embeddings, mean pooling destroys this by design. No amount of dimensionality manipulation can recover signal that was eliminated by the pooling operation.

6.2 Implications for Model Architecture

This distinction has direct implications for remediation strategies. For negation sensitivity, potential remedies include modified pooling strategies that up-weight negation markers, higher-dimensional representations that preserve low-variance negation dimensions, or explicit negation-aware training objectives. For entity-swap sensitivity, no embedding-space intervention will suffice—the problem requires either structured representations (separate subject/object embeddings) or cross-encoder architectures that process sentence pairs jointly.

6.3 The Variance-Importance Misalignment

Our findings quantify a specific instance of the misalignment between statistical variance and semantic importance. PCA assumes the most important information resides in the directions of greatest variance. For coarse semantic similarity—topic and domain matching—this assumption holds well, as evidenced by robust paraphrase detection at reduced dimensions.

For compositional phenomena like negation, the assumption breaks down. The distinction between "has diabetes" and "does not have diabetes" lies along subtle, low-variance directions that capture the specific contribution of negation markers. This misalignment is not unique to PCA—any unsupervised dimensionality reduction technique that prioritizes global variance will exhibit the same pattern. Supervised techniques (Linear Discriminant Analysis, supervised dimensionality reduction) might preserve negation sensitivity at lower dimensions, though at the cost of generality.

7. Practical Recommendations

7.1 Dimension Selection Guidelines

Based on our findings—noting the limitations of our small evaluation sample and corpus-fit PCA—we offer the following tentative guidelines:

For general paraphrase detection and semantic search: Reduction to 50 dimensions maintains paraphrase similarity above 0.93 while retaining 79.7% of explained variance. If the application does not require sensitivity to negation or entity swap (e.g., duplicate detection, document clustering), aggressive compression is viable.

For applications requiring negation sensitivity: Maintain at least 100 dimensions, preferably 150 or more. Even at full dimensionality, negation discrimination is poor. At 100--150 dimensions, negation sensitivity approximately matches or slightly exceeds the full-dimensional baseline, while achieving a 61--74% reduction in vector size.

For applications requiring entity-swap sensitivity: Dimensionality reduction is irrelevant; the failure is intrinsic to mean-pooled embeddings at any dimension. Use cross-encoder reranking or structured representation approaches instead.

For safety-critical applications (medical, legal, financial): Do not rely on cosine similarity of reduced-dimension embeddings for any distinction that involves negation, entity roles, or compositional semantics. Implement dedicated failure-mode detection (negation classifiers, entity-role parsers) as a separate pipeline stage.

7.2 The False Economy of Aggressive Compression

Our results caution against the common practice of reducing embedding dimensions purely based on aggregate benchmark performance. A model that achieves 95% of full-dimensional performance on a general STS benchmark may have substantially degraded negation sensitivity—a failure that aggregate metrics conceal because negation pairs constitute a small fraction of typical evaluation sets.

We recommend that practitioners who perform dimensionality reduction evaluate their reduced embeddings on failure-mode-specific test suites before deployment.

7.3 Alternative Approaches to Efficient Embeddings

Rather than reducing dimensions of a pre-trained model, practitioners with efficiency constraints might consider: (a) training a lower-dimensional model from scratch with explicit negation-awareness in the training objective; (b) using Matryoshka Representation Learning (Kusupati et al., 2022), which trains embeddings that are designed to be truncated at various dimensions while preserving key properties; or (c) using quantization (reducing the precision of each dimension) rather than dimensionality reduction, as quantization preserves all dimensions and their relative relationships while reducing storage requirements.

8. Limitations

We acknowledge several significant limitations that affect the interpretation and generalizability of our findings.

Small sample size and statistical power. Our core evaluation uses only 10 sentence pairs per failure category. This small N limits our statistical power and makes our mean similarity estimates sensitive to individual pair characteristics. While our paired t-tests show significant effects for most comparisons (Table 2), the specific numerical values (e.g., negation mean of 0.936 at 50 dimensions) should be interpreted with appropriate uncertainty. Larger-scale evaluations with hundreds of pairs per category are needed to establish precise estimates of the dimensionality-discrimination relationship. We consider our findings indicative of qualitative trends rather than precise quantitative benchmarks.

PCA fit on the evaluation set. We fit PCA on the same 180 sentences used for evaluation rather than on a large, independent background corpus. This is a methodological limitation: the principal components are influenced by the specific lexical composition of our test pairs, and the variance spectrum of 180 hand-crafted sentence pairs may differ substantially from that of a representative document corpus. In production settings, PCA would typically be fit on millions of documents, yielding different principal components. The qualitative finding that entity-swap failure is dimension-invariant should be robust to this choice (since the invariance reflects the absence of signal rather than its position in the variance spectrum), but the specific dimensionality at which negation sensitivity degrades may shift when PCA is fit on a more representative corpus.

Single model. We examine only one embedding model (all-MiniLM-L6-v2). Different models with different architectures (e.g., larger transformers, different pooling strategies, or different training data) may exhibit different relationships between dimensionality and failure modes. Our findings should be viewed as a case study demonstrating the existence of the dimensionality-discrimination tradeoff rather than a universal characterization.

PCA as the sole reduction technique. We use PCA exclusively. Non-linear dimensionality reduction techniques (t-SNE, UMAP, autoencoders) might preserve different aspects of the embedding structure. However, PCA remains the most common technique in production embedding pipelines due to its simplicity and determinism.

Constructed test pairs. Our test pairs are manually constructed and may not fully represent the distribution of negation and entity-swap variations encountered in real applications. Naturally occurring negations in medical records, for example, often involve more complex syntactic structures ("Patient denies any history of...") that may interact differently with dimensionality reduction.

Cosine similarity as the sole metric. We evaluate only cosine similarity. Euclidean distance or inner product similarity might show different sensitivity patterns across dimensions.

Baseline already broken. The full-dimensional model already fails to distinguish negation from paraphrases (negation similarity 0.897 exceeds paraphrase similarity 0.824). Our study characterizes how dimensionality reduction worsens an existing failure rather than creating a new one. The practical value of this characterization is that it helps practitioners understand the additional cost of compression beyond the already-present baseline failures.

9. Conclusion

We have presented an exploratory investigation of how embedding dimensionality affects specific semantic failure modes in sentence embeddings. Our experiments reveal a qualitative asymmetry between failure categories. Entity-swap failures are intrinsic to mean-pooled representations and persist unchanged across all dimensions from 2 to 384, demonstrating that the information needed to detect entity swaps is eliminated by the pooling operation rather than by compression. Negation discrimination shows substantial degradation under PCA compression (from 0.897 at 384d to 0.991 at 2d), with the most significant losses occurring between 150 and 50 dimensions, though with a non-monotonic initial improvement when reducing from 384 to 150 dimensions.

These findings, while based on a limited evaluation sample and corpus-fit PCA, have practical implications. The common practice of reducing embedding dimensions for efficiency implicitly worsens an already-present negation failure. Our analysis suggests that maintaining at least 100 dimensions preserves approximate baseline negation sensitivity, while reductions below 50 dimensions cause substantial additional degradation. Entity-swap failure, by contrast, cannot be addressed through dimensionality manipulation and requires architectural changes.

More broadly, our results highlight the danger of evaluating dimensionality reduction using aggregate metrics alone. Failure-mode-specific evaluation should become a standard component of the dimensionality reduction pipeline, particularly for safety-critical applications. We encourage future work to replicate these findings with larger evaluation sets, PCA fit on independent corpora, and multiple embedding models to establish the robustness and generalizability of the dimensionality-discrimination tradeoff we have characterized.

References

Bellman, R. (1961). Adaptive Control Processes: A Guided Tour. Princeton University Press.

Beyer, K., Goldstein, J., Ramakrishnan, R., and Shaft, U. (1999). When is "nearest neighbor" meaningful? In Proceedings of the 7th International Conference on Database Theory, pages 217--235.

Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171--4186.

Johnson, W. B. and Lindenstrauss, J. (1984). Extensions of Lipschitz mappings into a Hilbert space. Contemporary Mathematics, 26:189--206.

Kusupati, A., Bhatt, G., Rber, A., Wallingford, M., Sinha, A., Ramanujan, V., Howard-Snyder, W., Chen, K., Kakade, S., Jain, P., and Farhadi, A. (2022). Matryoshka representation learning. In Advances in Neural Information Processing Systems 35.

Mu, J. and Viswanath, P. (2018). All-but-the-top: Simple and effective postprocessing for word representations. In Proceedings of ICLR 2018.

Raunak, V. (2017). Simple and effective dimensionality reduction for word embeddings. In Proceedings of the NIPS 2017 Workshop on Learning with Limited Labeled Data.

Reimers, N. and Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP-IJCNLP 2019, pages 3982--3992.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30.