Bring your own GPU on /sys/admin/blog

Room for two

Sat, 10 Oct 2026 00:00:00 +0000

With vLLM running and nobody connected, nvidia-smi shows this:

GPU 0: 22,112 MiB / 24,576 MiB used, 0% utilisation
GPU 1: 22,134 MiB / 24,576 MiB used, 0% utilisation

My first thought was a memory leak, but vLLM takes that memory on purpose. My config sets --gpu-memory-utilization 0.90, so when vLLM starts it claims 90% of each GPU and manages that space itself. “Memory used” tells me nothing about load.

Two cards, too many variables

Mon, 31 Aug 2026 00:00:00 +0000

In April I rented a machine with two RTX 3090s on Vast.ai, to see what a second card would buy me before paying for one. On the first benchmark, two cards were 21 times slower per token than one. My write-up that day ended with “Don’t buy a 2nd 3090 for the local rig right now.”

Every token reads the whole model

Tue, 02 Jun 2026 00:00:00 +0000

For about a year I used local models the easy way, asking codellama:13b in Ollama a question now and then. Then I started using them for real work: coding agents like Claude Code and Pi, and Hermes. An agent sends big prompts with tool definitions and files in them, runs in a loop, and carries the whole conversation in every request. The models worked, but they were slow, and they got slower the longer a session went on.