Quantization

Bring your own GPU · Part 2

Two cards, too many variables

My first benchmark said two RTX 3090s were 21 times slower per token than one. A rerun looked fine, and I had changed too many things to know why. That changed how I test.

• ai
• inference
• vllm
• benchmarking
• quantization