Fast Is Slow, Slow Is Smooth, Smooth Is Fast

My D&D transcription pipeline got about a third faster. I did it by taking away twenty-four of the thirty-two threads it was using, and by not using the fastest GPU in the house.

Neither of those was the plan. The plan was to throw a 5090 at it.

Fast is slow

I’ve written before about how the session reports get made: record the Discord audio, transcribe it with speaker labels, hand the transcript to an LLM, edit the result into narrative prose. That post described a pair of bash scripts wrapping WhisperX. Since then the scripts have been replaced by a single Go binary that does the same job without PyTorch, without a Hugging Face token, and without a Python environment to rot.

The stages are the same either way. Extract the audio stream from the video. Run voice activity detection over it to find the speech and cut it into chunks. Send each chunk to a Whisper server over HTTP. Separately, run speaker diarization over the whole recording to work out who was talking when. Merge the two.

A four-hour game night is a lot of audio, and the numbers show it. A recent session ran 3h02m31s, and the pipeline took 21m48s to process it end to end. That is not painful – it runs while I do something else – but it is slow enough to notice, and the shape of it bothered me. Whisper transcription took 14m51s of that, against an 8 GB Radeon RX 7600 in a Thelio workstation. There is a machine two rooms away with an RTX 5090 in it.

So: point the transcription at the 5090 and take the win.

Same model, same software, same 120-second test clip. The RX 7600 did it in 2.4 seconds. The 5090 took 43.

That is not a disappointing result. It is an absurd one. A card with roughly 1.8 TB/s of memory bandwidth lost by a factor of eighteen to a midrange part with 288 GB/s, running identical code. When a benchmark comes back eighteen times wrong in the wrong direction, the benchmark is not measuring what you think it is.

Fast is slow. The fastest hardware I own was the slowest thing in the fleet, and it took most of a night to find out why.

Slow is smooth

The first useful thing was to stop guessing and measure everything. Four machines, same clip, same Whisper-Large-v3-Turbo model, one request each:

Both NVIDIA cards were the slow ones, and the 3080 – the weaker card, in the machine with the weaker CPU – was slowest of all. That pattern says the GPUs were not involved at all, and that what I was actually measuring was two CPUs.

They weren’t. Watching nvidia-smi during a transcription showed 0% utilization and 15 W, flat, for the entire run, while the inference server’s health endpoint cheerfully reported the device as gpu.

The cause was three failures stacked on top of each other, each of which individually looks like success.

The server config had the Whisper backend set to cuda. That is not a valid value – the error, once I asked for it directly, is 'whispercpp.backend' must be one of: auto, vulkan, cpu. There is no CUDA Whisper backend in that stack at all; only the LLM and image backends have one. Every transcription request had been failing at model load.

Setting it to auto fixes the error and resolves to Vulkan. Vulkan then fails to reach the card, because inside the container the NVIDIA driver’s Vulkan ICD does not load: libGLX_nvidia.so.0 exports no vk_icdGetInstanceProcAddr, and libXext.so.6 is missing from the image entirely. The Vulkan loader falls back to llvmpipe, which is a software rasterizer. Software rasterizer means CPU.

And the health endpoint reports gpu regardless, because it answers from a sysfs probe of what hardware exists, not from what the running process actually opened. Three green lights, one of them lying, and a transcription running on a 9950X3D while a 5090 idled at 15 W beside it.

That explains slow. It does not explain whether the card could be fast, so I built whisper.cpp from source against it. The stock CUDA container image ships no compiled kernels for Blackwell and JIT-compiles PTX at runtime instead, which works and is dreadful: 1386 ms per encode window. Rebuilt with -DCMAKE_CUDA_ARCHITECTURES=120, the same encode window takes 26.6 ms. Fifty-two times faster, same card, same model, same audio – the entire difference is whether the build knew what chip it was targeting.

For scale, that same encode step on the CPU takes 7606 ms. The “working” CUDA build was only five times better than no GPU at all, which is exactly the sort of number that passes a smell test and shouldn’t.

So the 5090 can transcribe a 120-second clip in 1.5 seconds. Through the real pipeline, chunked the way the tool actually chunks, it did 300 seconds of audio in 3.4 seconds against the RX 7600’s 32.

Then I put it in the pipeline and the wall clock didn’t move.

Transcription and diarization run concurrently, and diarization was the longer of the two – 16m38s against Whisper’s 14m51s. Making Whisper nine times faster took the shorter of two overlapping stages and made it shorter. The critical path never changed. I had spent the evening optimizing something that was already free.

Which meant the real question was one I hadn’t asked: why does diarization take sixteen minutes?

Both diarization and voice activity detection run through ONNX models, and both take a thread count. That count defaulted to runtime.NumCPU(), which on this workstation is 32. So I swept it:

The default was the worst setting on the table for both stages.

Voice activity detection is the stark case: one thread is twenty-four times faster than thirty-two. Not marginally better. Twenty-four times.

The reason is granularity. Silero VAD is a 629 KB model, and it is fed 512 samples at a time – at 16 kHz, that is 32 milliseconds of audio per step. A three-hour recording is about 342,000 of those steps, and each one is a separate call into the model. Divide the sweep above by its step count and one thread comes out at 53 microseconds per step, while thirty-two threads take 1.29 milliseconds.

Nothing got harder. The arithmetic in a 512-sample window is trivial and identical either way. What changed is that thirty-two threads have to be woken, handed a slice of a problem, and synchronized back together, 342,000 times. The coordination is not overhead on the work. The coordination is the work, and the actual computation is a rounding error inside it.

Diarization behaves differently because the models are real – pyannote segmentation at 5.7 MB and TitaNet-large at 96.7 MB, chewing on 2700 speaker turns. That work genuinely parallelizes, and it improves steadily out to about eight threads. Past eight, the same synchronization cost catches up and starts winning. Thirty-two threads landed at almost exactly twice the time of eight.

There was also a plain bug underneath, and it’s the reason nobody caught this earlier. A single --diarize-threads flag fed both stages. Anyone tuning that flag was tuning it for diarization, watching diarization improve as they raised it, and pushing VAD further into the ditch on every increment – with the damage hidden behind a flag named after the other stage.

Slow is smooth. Doing less at once, on purpose, is what let the thing run cleanly.

Smooth is fast

The fix is a defaults change: VAD gets one thread, diarization gets min(NumCPU, 8), and the two flags are separated so neither can drag the other around. Small machines still get every core they have.

The same 3h02m31s session, same machine, same backend, before and after:

Voice activity detection went from four and a half minutes to eighteen seconds. Diarization more than halved. Whisper landed within noise of where it started, because I never touched it – it is still running on the same 8 GB Radeon it always was, and it is now the longest stage and the critical path.

The output is identical. Same 3303 segments, same 1786 speech regions, same 1445 chunks, and the transcripts match at every thread count I tested. There was no speed-versus-quality trade available here to make. The old configuration was simply paying for coordination it had no use for.

Thirty percent of the wall clock, recovered by using fewer resources.

That is the general shape, and it is why this is worth writing down beyond my own weekly batch job. Parallelism is not free and it is not linear. Every parallel unit of work carries a cost – dispatch, synchronization, cache lines bouncing between cores that were each holding them privately. That cost does not care how small the work is, and it’s not fixed; it scales with the number of cores in use, so it costs more as you use more cores. When the work per unit shrinks below the cost of coordinating it, adding workers makes the program slower, and keeps making it slower the more workers you add.

NumCPU is the default that feels responsible. It looks like using the machine you paid for. What it actually encodes is an assumption that the work is big enough to be worth dividing into as many pieces as you can run in parallel, and nobody checks that assumption because the code that made it runs fine and you expect 3 hours of audio to be slow, it’s only a question of how slow. The penalty scales with how good your hardware is, which is a genuinely nasty property: this bug got worse every time I upgraded, and a 32-thread workstation is exactly where it hurts most.

It’s worth also pointing out the elephant in the room before someone else does. The 9950X3D has 16 cores, and 32 threads. That’s 16 real parallel processing units and an extra set of registers to optimize task switching between two processes on each one. That’s where 32 comes from: 16 cores, x2 register sets, 32 threads. There are workloads where that doubles what you can get done. This isn’t one of them.

I made the same mistake twice. Pointing the job at the 5090 meant reaching for the biggest number on hand without checking whether the work could use it; the path to that card turned out to be broken in three places, and nothing said so. Handing Silero thirty-two threads was the same reach. Thirty-two milliseconds of audio does not divide thirty-two ways cleanly.

Measure the thing you’re actually waiting on. Then give it the smallest amount of machinery that does the job.

I never did get it working on the 5090, because I had already spent an evening’s work to gain 6 minutes. That’s another kind of optimization. I’ll go back to it at some point.

The tool is at github.com/matthewjhunter/transcribe .