Gpu

Bring your own GPU · Part 1

Every token reads the whole model

nvidia-smi showed my 3090 at 100% while my agents got slower. Timing prefill and decode separately showed a card that was mostly waiting on memory.

• ai
• inference
• llama.cpp
• gpu
• sysadmin