Benchmarks and real-world performance of language models running on ordinary consumer hardware.
I was in the middle of a measurement when two remote servers went unreachable at the same time. The coding llama-server on port 8080, and the occupier server I’d started on port 8081 for that measurement. 8080 was the same server that had been answering normally a few minutes earlier. curl waited more than eight…
A month ago I published a post on this site about the same card. An image generation app was doing nothing at all and holding 2.65GB of VRAM, and my local coding model generated at 28.2 tok/s instead of 41.0. Not one error was thrown. That post had two points on it. So I never…
A title went up on r/LocalLLaMA: “Qwen3.8 27B at 50 tok/s with 100k Context on a 16GB GPU!” 100k of context on 16GB, with a 27B model, at 47–50 tokens a second. Those three figures are the OP’s report, not my measurement. The method is KVarN, a feature of the llama.cpp fork beellama.cpp that tiers…
I started writing this blog in English, which meant showing up on Reddit and Hacker News. Reading was fine. Writing was the bottleneck. Every comment went through the same loop. Think it in Korean, convert it to English I’d be willing to post, check it again. For a two-paragraph reply that loop cost me minutes,…
I run a local coding model and an image generation app on the same card. The image app was doing nothing at the time. Its queue was empty and nothing was being drawn. It was holding 2,651MB of VRAM anyway. I stopped it and sent the same prompt again. Generation went from 28.2 t/s to…
My GPU worked for 28 minutes and 37 seconds without a single complaint. Every step ran, and sampling finished. Then the last node in the graph printed this. RuntimeError: expected input to have 48 channels, but got 16 Decoding never started. Everything those 28 minutes and 37 seconds produced went straight in the bin. The…
I built a thing that answers when I talk to it. It takes the microphone, turns what I said into text, hands that to a model, and speaks the reply back out loud. I never touched a cloud speech API. The brain in the middle is a different story, and I deal with it head…
Two local embedding models ranked the same 24 bookmarks identically — 8 of 8 correct each. Their similarity scores sat in almost non-overlapping ranges, and one hardcoded 0.75 cutoff silently erased every result from one of them.
Fitting a 30B MoE model into 16GB of VRAM comes down to one flag. Full n-cpu-moe sweeps on a 16GB and an 11GB card — including the silent VRAM overflow that costs 2.5x with no error.
Same model, same build, same test — one on an RTX 2080 Ti (11GB, CUDA), one on an RX 9070 XT (16GB, Vulkan). The result split: 1.9x prompt processing one way, 1.3x generation the other.