Running a local LLM server is the easy part. Getting three separate pieces of infrastructure to agree that a model is downloaded, reachable, and worth waiting for is where the afternoon goes. Over the past two weeks I shipped three fixes across two open-source projects to get AMD’s Lemonade serving models behind the Olla proxy on my Strix Halo box. None of them was hard in the algorithmic sense – the diffs are a struct field, a config key, and a prepended path. They all came out of the same goal: point Olla at Lemonade on a Radeon and get a chat completion back.

Two of the three shipped in Olla v0.0.28 ; the third shipped today in Lemonade v10.8.1 . This is the write-up of all three, because the lemonade was not pouring at three different layers and each one is a small lesson about what “load on demand” does to infrastructure that wasn’t expecting it.

The stack

Lemonade is AMD’s local inference server – an OpenAI-compatible endpoint that runs llama.cpp, whisper.cpp, and friends on ROCm, loading models into VRAM on demand. Olla is a high-performance proxy and load balancer that sits in front of one or more inference backends, health-checks them, and routes each request to an endpoint that actually has the requested model. Put Olla in front of Lemonade and you get a single OpenAI-compatible front door to a pool of local backends.

My box for this is a Strix Halo machine – Radeon 8060S, gfx1151. It is exactly the kind of hardware where the AMD path is newer than the NVIDIA one, which is a polite way of saying I was going to be the person who found the bugs.

Bug 1: the glass is full and the bartender says it’s empty

The first symptom was the dumbest one. Olla health-checked the Lemonade endpoint fine – green, up, reachable – and then refused to route a single model on it. A chat request for a model I could see sitting on disk came back 404 "No openai endpoints available".

Lemonade reports each model in /api/v1/models with a downloaded flag. Olla’s discovery parser dropped it. LemonadeModel declared neither downloaded nor size, so both fell on the floor at unmarshal, and MapModelState (in internal/adapter/unifier/model_builder.go) had nothing to key on – every Lemonade model resolved to state=unknown. The router treats unknown-state models as living on an unhealthy endpoint and excludes them. So the endpoint was up, the model was on disk, and the request had nowhere to go.

The lemonade was made and in the fridge, and the host kept telling guests there was nothing to drink.

The fix (#161 ) is one field. Capture downloaded on LemonadeModel, and in the parser map downloaded: true to available. This mirrors what Olla already does for a pulled, load-on-demand Ollama model: present on disk and serveable counts as available for routing, even though nothing is resident in VRAM yet. Lemonade loads on demand too; downloaded: true is the same readiness signal. After the change the same request returns X-Olla-Routing-Decision: routed and X-Olla-Routing-Reason: model_found.

I deliberately left size on the floor – Lemonade reports it as a GiB float and Olla’s internal size is bytes, so capturing it cleanly needs a unit conversion the routing fix doesn’t depend on. One bug, one field.

Bug 2: don’t walk out before the first pour

With routing fixed, the second pour of the day worked and the first one didn’t. A cold request – the one that actually triggers Lemonade to load the model into VRAM – aborted with 502 "request timeout after 30.0s". Once the model was resident, everything was instant. The bug only bit on the cold start.

Olla’s transport hardcoded ResponseHeaderTimeout to DefaultResponseHeaderTimeout, which is 30 seconds, with no override. An on-demand loader sends no response header until the model is resident, so any cold load that takes longer than 30 seconds trips the timeout before the first byte. The natural thing to reach for, response_timeout, doesn’t help – that governs a later deadline and never applies to the header wait. There was simply no knob for the thing that was timing out.

This is the ice-cold-lemonade bug: the machine is still warming up, the glass isn’t full yet, and Olla was walking out at the 30-second mark every time.

The fix (#164 ) adds a proxy.response_header_timeout key, defaulting to the same 30 seconds so existing behaviour is unchanged, threaded through to the transport via GetResponseHeaderTimeout(). It’s zero-value-means-default and OLLA_-overridable like the rest of Olla’s tunables, and it’s now honoured by both the Olla and Sherpa proxy engines. Set it to 180s and a cold load of a large model that used to 502 at thirty seconds now completes. The two Olla fixes are independent – one makes the model routable, the other keeps the cold start from aborting – but you need both before a fresh Lemonade backend answers its first real request through the proxy.

Bug 3: the hard lemonade

The first two were YAML and Go in the proxy. The third meant dropping out of config entirely and into the C++ that launches the inference server – the hard lemonade.

I switched Lemonade’s whisper backend to ROCm and the whisper-server aborted at startup. Exit 134, failing to dlopen libamd_comgr.so.3. The llama.cpp and stable-diffusion servers on the same box came up fine on ROCm; only whisper died.

The reason was a missing line of wiring. On Linux, the ROCm backends need TheRock’s ROCm library directory prepended to LD_LIBRARY_PATH so the bundled server can find libamd_comgr.so.3 and its neighbours. llamacpp_server.cpp and sd_server.cpp both did this; whisper_server.cpp was the one launcher that didn’t. So the newest ROCm path was missing the lib wiring its older siblings already had.

The fix (#2293 ) mirrors the existing logic, gated on the rocm backend:

// src/cpp/server/backends/whisper_server.cpp, in WhisperServer::load
if (whispercpp_backend == "rocm") {
    std::string rocm_arch = SystemInfo::get_rocm_arch();
    if (!rocm_arch.empty()) {
        std::string therock_lib = BackendUtils::get_therock_lib_path(rocm_arch);
        if (!therock_lib.empty()) {
            lib_path = therock_lib + ":" + lib_path;
        }
    }
}

One wrinkle worth flagging for anyone reading the diff: the gate keys on the raw whispercpp_backend == "rocm" option value, not the resolved_backend == "rocm-stable" form that llama.cpp and sd_server use. That’s intentional – whisper has no resolve step and uses the literal "rocm" string throughout. Matching the other two here would have been the idiomatic-looking choice and the wrong one.

Tested on the gfx1151 box: with the exe-dir-only path (pre-patch) the bundled rocm whisper-server aborts on dlopen; with TheRock’s dir prepended it reports found 1 ROCm devices ... Radeon 8060S Graphics, gfx1151 and transcribes. This one landed in Lemonade itself and shipped today in v10.8.1, credited in the “Radeon GPUs are back!” section of the release alongside the AMD folks who fixed the install gates and GPU detection.

Three bugs, two repos, one stack. The unifying thread isn’t a clever abstraction – it’s that on-demand model loading quietly breaks anything that assumes a model is either present and working or absent and broken. For Lemonade, “downloaded but not resident” is the normal resting state of every model on the box, and both Olla bugs were the proxy failing to model that third state: once by calling it unknown and refusing to route, once by calling it too slow and hanging up. The whisper crash was a different animal – the newest ROCm backend missing wiring the others had – but it rhymes, because being early on AMD means being the one who finds the path nobody walked yet.

The diffs are small because the bugs were small. The work was reading enough of two unfamiliar codebases to be sure each fix was the narrow correct one and not a plausible-looking patch over the symptom – and writing the test that proves it. Being the person who hits the wall first, files the fix, and leaves the next person a working stack is most of what open source actually is.

Three glasses poured. The Radeon path is a foot wider than it was two weeks ago.