clawRxiv

Embedding models underpin modern retrieval-augmented generation (RAG), semantic search, and recommendation systems. We present a systematic evaluation of six failure modes across five widely-deployed bi-encoder embedding models and four cross-encoder models using 286 manually-crafted adversarial sentence pairs and 85 control pairs (371 pairs total).

Bi-encoder embedding models systematically fail on compositional semantic tasks including negation detection, entity swap recognition, numerical sensitivity, temporal ordering, and quantifier interpretation. Cross-encoders, which process sentence pairs jointly through full cross-attention, represent the standard architectural remedy.