Test Before You Trust

My workstation is an AMD Strix Halo. It has an NPU, and the NPU can run an embedding model – embed-gemma-300m-FLM, served by Lemonade alongside the same model on the integrated GPU. Same weights, two runtimes. Moving embeddings to the NPU is more an experiment in using available hardware than anything else. I’ve experimented with Whisper transcription on the NPU successfully with (dicta)[]; it was acceptable, slower than the iGPU, slightly worse, but functional enough that I use it and the GPU stays free for more demanding tasks. This seemed like a natural follow up when I saw it in (Lemonade)[]’s list of NPU models.

That is the kind of idea that sounds finished before you have tested anything. Free capability, one config change.

So I tested it. The results were not what I expected, and neither was the first set of results I got, which were wrong.

The obvious measurement

Three things matter for an embedder: how fast is it, does it work, and how well. Speed first, because it is easy:

Twenty-six times slower. Same machine, same model, both returning 768 dimensions. That is not a tradeoff to weigh, that is an answer.

But “slower” is not “broken”, and I wanted to know which. An embedder’s job is to put related text near related text. So: embed a few sentences, check that paraphrases score higher than unrelated ones.

I ran that, and the NPU looked catastrophic. Paraphrases scored 0.8507. Completely unrelated sentences scored 0.8345. A gap of 0.017 – everything similar to everything, which is the one thing an embedder must not do. The iGPU path on the same sentences gave 0.8109 against 0.3550.

Collapsed embeddings. I had my answer, and it was a good story: the NPU returns vectors that look fine and mean nothing.

The measurement was wrong

The test had two problems. It rested on exactly two pairs, one related and one unrelated, which is not a usefully-large sample. And every sentence in it began with the same twenty-character prefix, because I had prepended the model’s document prompt by hand. If a runtime’s pooling favours early tokens, an identical prefix on every input inflates every similarity by itself.

So the test had a plausible mechanism for producing precisely the result I got, independent of whether the model was any good.

I ran it more properly: four topics, two paraphrases each, no shared prefix, all twenty-eight pairs compared.

The NPU model is not collapsed. It separates related from unrelated text, about 3.7 times more weakly than the alternative. My conclusion had been right and my reason for it had been wrong.

The number that matters is not the average

Look at the last column rather than the separation.

Separation is the difference between two means, and means hide their spread. Worst margin is the weakest paraphrase pair minus the strongest unrelated pair. When it is positive, every paraphrase in the sample outranks every unrelated pair – the ordering holds. When it is negative, the two distributions overlap: somewhere in there, two unrelated sentences score higher than two paraphrases.

That is the difference between an embedder that is blunt and an embedder whose ranking is unreliable. Blunt is survivable. Unreliable is not, because retrieval is nothing but ranking.

nomic-embed-text-v2-moe makes the point from the other side. Its separation, 0.2810, is close to embeddinggemma’s 0.3011 – but its worst margin is 0.0491 against 0.1199. On the average it looks nearly equivalent. On the margin there is considerably less room before the orderings start to cross.

The second thing testing found

While checking that the NPU model got the right prompt prefixes, I checked whether the working one did.

EmbeddingGemma is trained with task prefixes: Google’s model card specifies two, each including its trailing space:

queries:   task: search result | query: 
documents: title: none | text: 

My go-embedding library applies both, correctly, for a model named embeddinggemma.

It applied neither for google/embeddinggemma-300m. Or unsloth/embeddinggemma-300m-GGUF:embeddinggemma-300M-Q8_0.gguf. Or onnx-community/embeddinggemma-300m-ONNX. Or nomic-ai/nomic-embed-text-v1-GGUF:Q4_K_S.

Those are not exotic strings. They are what the model cards say, what serving stacks report, and what anyone would paste into a config. An organization prefix never denotes different weights – google/, unsloth/ and onnx-community/ all publish the same model under their own account – but the lookup was exact-match, so every namespaced spelling resolved to nothing and text reached the model unwrapped.

Nothing errors when that happens. The embedder works. The vectors are the right shape. Every result is quietly a bit worse, and the natural diagnosis is that the model is mediocre.

The fix was small once seen: strip the namespace segment, register the version spellings by name. The bug had been sitting there because the only way to notice it is to look, and people tend to look at their own config rather than hypothetical alternatives. Mine had worked on nomic, running on A380 hardware because why not use my media servers for embedding, until I got a corpus large enough to make the switch to faster hardware useful for re-embedding passes.

So I wrote the check

Both failures are invisible from the outside, and both are cheap to detect. So the probe became a command, embedcheck:

DISCRIMINATION      (four topics, two paraphrases each, within this model's own space)
  paraphrase pairs  +0.7041 mean cosine
  unrelated pairs   +0.3532 mean cosine
  separation        +0.3509
  worst margin      +0.1180  every paraphrase outranks every unrelated pair

It reports whether the name resolves and whether prefixes are being applied, the similarity statistics, timings, and a verdict: healthy, weak, or broken. Exit status is 0, 1 and 3, so it can gate a deploy. It takes a few seconds and needs no labelled dataset – the question “does a paraphrase outrank an unrelated sentence” is answerable without one.

It also prints measured numbers for known embedders next to yours, because a number you cannot compare to anything is not much of a measurement. The last row of that table is the NPU path: a real backend, returning perfectly well-formed vectors, that you should not use (at least in the version I was testing, which was the first one I had seen, so likely early).

The thresholds come from those four measurements rather than from intuition, and a test asserts the published reference numbers still imply the verdicts printed beside them. If I fat-finger the table, the suite fails rather than every reader being quietly misled – which is the same failure mode as the rest of this post, one level up.

One more caveat. This is a tiny test corpus, and a proper test would be to run it over a larger, representative sample of your own corpus. My willingness to do that ended at 26x slower, since my real corpus is non-trivial.

The point of the tool is to sanity check your proposed embedding model config cheaply.

Test before you trust

Three things nearly went by on plausibility here.

The NPU model looked usable because it returned well-formed vectors of the right size. The library looked correct because it applied the right prefixes to the one model name I had happened to configure. And my own conclusion looked sound, while resting on a measurement that could not have supported it.

Only the third one bothers me. The first two are ordinary bugs, and bugs are what testing is for. The third is testing that produced an answer, and produced it confidently, and would have been cited later as evidence.

The correction was not a better tool or more data. It was one question – what is this actually comparing? – asked before the result got written down.

Test before you trust. Including your test.