The expensive hypothesis was wrong; the cheap check was right

Local LLM agent

2026

OllamaopencodeCUDA

Python

What it is

A local LLM agent in opencode took about a minute per reply and timed out. The value of the case is the diagnosis, not the config: the symptom pointed at the model, the real cause was one layer down.

Generation~1 tok/s~16 tok/s

Prompt eval~1.7 tok/s~38 tok/s

GPUVulkan/NVKCUDA 12.4

Proof

llm

$ nvidia-smi

command not found

# that was the whole root: 1 tok/s → 16 tok/s after the driver

The path to the fix

01The symptom misled

Qwen3.5 emits hundreds of reasoning tokens before answering, so replies timed out. Trying different quantizations and modes changed nothing.

02A proxy, not a fork

opencode uses the /v1 API and won't forward custom params. A small Python reverse-proxy injects reasoning_effort:none — only for the fast profile.

03The real root cause

Even after the fixes: ~1 tok/s at '100% GPU'. There was no NVIDIA driver — the card ran through the open Vulkan stack. Driver + CUDA + libcuda1 (installed separately) gave the 16× speedup.

The lesson: run the cheap environment check before the expensive hypotheses. I was tuning the bodywork while the engine wasn't even running.

Back to all work