Embedding Models Misjudge Physical Units in 24-Model Test

What the paper reports

On 17 September 2026 Juri Opitz and Andrianos Michail, both at the Department of Computational Linguistics of the University of Zurich, posted "Embedding Models Measure in Peculiar Ways" (arXiv 2609.20821, cs.CL). The paper asks a narrow and testable question: do text embeddings reflect physical measurements of mass, distance, time and volume? These quantities have one objective notion of equivalence and one objective notion of distance. "1 meter" and "100 centimeters" denote the same quantity. "105 centimeters" is physically closer to "1 meter" than "15 kilometers" is. If embedding similarity really measures meaning, it should respect this.

The authors' answer is no. Across all 24 embedding models they tested, physical measurement is only weakly modeled, and the models show what the authors call "quite peculiar" measurement patterns. They also report that similarity between measurement strings is strongly associated with superficial string similarity, and that recalibrating the similarity function does not fix the problem. Code is public at github.com/flipz357/embed-and-measure.

How the test works

The setup is simple. An embedding model maps a text to a vector. The authors build a string from a number and a unit, for example "10 kilometers" and "1 meter", embed both, and compute the standard cosine similarity. They repeat this for every pair of values in a range and plot the result as a heatmap. In an ideal heatmap, similarity falls smoothly as the physical distance between the two values grows.

The quantities are length (meters), mass (kilograms), volume (liters) and time (seconds). The ranges are: local (0 to 10), medium (0 to 1,000), log (up to 100,000), sign (-100 to 100, so negative numbers are included) and scientific (exponential notation). Each range is cut into roughly 20 steps. The local range is also written in words ("five meters") for each integer from zero to ten, plus one-hundred, so the authors can compare digits with words.

The 24 models run from older BERT-style encoders to recent LLM-based decoders. Embedding sizes range from 384 (MiniLM) to 4096 (Qwen3-Embedding-8B). For close reading the authors pick all-mpnet-base-v2 (an encoder) and Qwen3-Embedding-0.6B (a decoder), because both are among the most downloaded embedding models. Results for the other models are in the appendix.

Key findings from the source

Heatmaps look erratic. Besides the diagonal, where identical strings match trivially, the patterns are mostly irregular. The authors point to non-monotonic artifacts, horizontal and vertical bands, and local neighborhoods that do not follow physical distance. Both example models can roughly separate negative from positive values. On the local range there is a "checker pattern": integer-integer pairs align, integer-float pairs do not. For both models, 2.5 of a unit is more similar to 7.5 of that unit than to 3 of that unit. This is the reverse of the physical order. Digits versus words. Embeddings barely capture that "one" and "1" are equal. Qwen gives high similarity between one and 1, but also between one and 0, most strongly for liters and kilograms. mpnet gives one and 1 a markedly higher similarity than one and any other number. Qwen rates "nine liters" as dissimilar to all liter quantities written in digits, and its self-comparison for nine/9 and ten/10 is comparatively weak. Except for one-hundred/100, the older mpnet aligns words and digits better than Qwen. Unit conversion is fuzzy. The authors compare 1 meter with 100 centimeters, 1000 millimeters and 0.001 kilometers, and similar sets for the other quantities. The ideal is a flat line near 1.0. No line is flat and near 1.0. The Qwen lines run slightly higher, and mpnet drops off sharply as quantities grow. The best case is seconds against milliseconds in Qwen, the only line that exceeds 0.9 and stays high. A small benchmark, PhysScore. The authors score each model with Kendall's tau between embedding similarity and physical distance, and with pairwise accuracy (PAC = 50 + tau/2). The random baseline has tau 0 and PAC 50. All models beat it, but none exceeds a tau of 54. The best is multilingual-e5-large-instruct with tau 53.23 (PAC 76.61), followed by LaBSE at 51.31 and e5-large-v2 at 47.35. The weakest are nomic-embed-text-v1.5 at 18.57 and embeddinggemma-300m at 24.40. The average is 37.59. Scale does not help: Qwen3-Embedding-0.6B scores 40.47, the 4B model 34.22 and the 8B model 32.27. The authors note that this is still optimistic, because the random baseline suffers from the trivial diagonal that every model gets right.

Which measurements are hardest. Mass is hardest, with an average tau of 34.54 across models, against 37.03 for volume, 38.48 for length and 40.30 for time. Some model and range combinations fall below zero, which means the model orders values in the opposite direction. The medium range is hardest on average. Recalibration does not rescue it. One objection is that plain cosine gives every dimension equal weight and may blur useful information. The authors test this with a linear probe, a linear regression on the absolute difference of the two embedding vectors, using a train-test split. The average tau moves from 33.4 to 36.4, a gain of 3.0. The largest gain is +12.3 for e5-large-v2, and it still ends below 54. Some models get worse, for example granite-embedding-107m-multilingual (-8.9) and embeddinggemma-300m (-7.3). The larger Qwen models gain more than the small one, but overall alignment stays low. String similarity is the likely driver. The authors generate random integers and floats between -1000 and 1000, pair them, and correlate embedding similarity with three references: character-level Levenshtein distance, token-level Levenshtein distance, and true numerical distance. In Table 4 embedding similarity follows the text form more than the value. For Qwen3-Embedding-0.6B the correlation over all pairs is 47.6 with characters and 11.7 with numeric proximity. For embeddinggemma-300m it is 51.6 against 3.3. The paper calls the embeddinggemma result one of the strongest links to string similarity and one of the weakest to numerical distance.

Background and why it matters

Contrastive training pulls similar texts together and pushes dissimilar texts apart. The resulting vectors are used for classification, retrieval and clustering.

Benchmarks such as MTEB score models on many tasks at once, while this paper takes a narrow "micro lens" on one property. That matters because the same vectors are presented as semantic representations. If "2 kilometers" sits no closer to "2000 meters" than to "2 meters", then quantity-sensitive uses have a hidden weak spot.

Our reading

The claim is precise and the evidence is consistent across three different probes: heatmaps, a ranking benchmark and a string-overlap analysis. The result that bigger or newer models do not clearly do better fits the authors' own explanation. If the training objective never asks for numeric order, more capacity may just learn surface form more thoroughly. The authors say so themselves: contrastive training may provide too little pressure for physical structure, and such alignment would have to be an emergent capability or an explicit benchmark goal. They also argue against a dimensionality limit, because they see no trend with embedding size.

The authors are balanced about the other side. For many retrieval and clustering jobs, quantities matter little, and lexical similarity is a useful bias for identifiers, version numbers, dates and product codes. Their point is that a model that promises semantic similarity should still rank "2 kilometers" closer to "2000 meters" than to "2 meters".

Limits and open questions

The authors state that their set of models is necessarily limited. The correlations in Table 4 show association, not proof of cause, and the paper words its explanation as evidence and a plausible reason. We read only the paper text, not the code or the appendix tables.

In the main text, Table 3 lists cosine scores for the split experiment that differ from Table 1 (for example e5-large-v2), and the text quotes the best probe result as 47 tau while the table shows 48.1. The text does not explain these differences, so treat the small numbers with care. The test strings in the paper are short, clean, English-style phrases. It stays open how models behave with messy real text, or whether a training objective with explicit quantity supervision would close the gap.

Practical takeaways

Developers who use embeddings for search over specs, logs, recipes or lab data should not assume that numeric closeness or unit equivalence survives embedding. A simple check is cheap: embed "2 kilometers", "2000 meters" and "2 meters" with your model and compare.

If the ordering is wrong, normalize units and numbers before embedding, or combine embeddings with structured filters. Teams that build agentic or scientific systems should treat this as a known blind spot, not a rare edge case. Model builders can borrow the paper's idea and use physical measurement as a small extra quality metric.