Bring your own GPU ยท Part 1
Every token reads the whole model
nvidia-smi showed my 3090 at 100% while my agents got slower. Timing prefill and decode separately showed a card that was mostly waiting on memory.
Topic
Llama.cpp
Index
1 entriesTopic
Index
1 entriesBring your own GPU ยท Part 1
nvidia-smi showed my 3090 at 100% while my agents got slower. Timing prefill and decode separately showed a card that was mostly waiting on memory.