Why Is My Setup Slow?

Quick Diagnostic

Cause #1: Memory Offloading

The most common cause of slow inference. When a model doesn't fully fit in VRAM, layers get offloaded to system RAM or even disk.

How to Detect

• VRAM usage at or near 100%
• System RAM usage increasing
• Speed drops dramatically with longer context
• Inference starts okay, slows down during generation

Why It's So Slow

Solutions

1. Use more aggressive quantization: Q4 instead of Q6 can make a model fit
2. Use a smaller model: A 13B model that fits beats a 70B that doesn't
3. Reduce context length: Shorter context = smaller KV cache
4. Get more VRAM: Sometimes the only real fix

Cause #2: Thermal Throttling

GPUs reduce clock speeds when they get too hot to prevent damage.

How to Detect

• GPU temperature >83°C (check with nvidia-smi or GPU-Z)
• Performance degrades over time during a session
• Fans running at maximum speed
• Clock speeds dropping (visible in monitoring tools)

Solutions

1. Improve airflow: Case fans, remove obstructions
2. Repaste the GPU: Thermal paste degrades over time
3. Undervolt: Reduce voltage for less heat with minimal performance loss
4. Add cooling: Aftermarket coolers, deshroud and add case fans
5. AC or ambient temp: Room temperature matters

Cause #3: Wrong Backend or Settings

Your inference might be running on CPU when it should be on GPU, or missing optimizations.

How to Detect

• GPU utilization near 0% while generating
• CPU at 100% instead
• Performance way below what others report for same hardware

Common Fixes

llama.cpp / Ollama

• Ensure GPU layers flag is set: -ngl 99 or --n-gpu-layers 99
• Check that CUDA/Metal/ROCm build is being used
• Verify with nvidia-smi that GPU memory is being used

PyTorch / Transformers

• Ensure model is on GPU: model.to("cuda")
• Check torch.cuda.is_available() returns True
• Verify correct CUDA version installed

General

• Flash Attention enabled? Significant speedup if supported
• Using optimized kernels? (cuBLAS, Metal, etc.)
• Batch size appropriate? (Usually 1 for interactive use)

Cause #4: KV Cache Overflow

Even if weights fit, the KV cache grows with context length and can overflow VRAM.

How to Detect

• Speed is fine at start of conversation, degrades over time
• Long prompts cause significant slowdown
• VRAM usage increases during generation

KV Cache Size Examples (FP16)

Solutions

1. Use shorter context: Don't use 32K if you only need 4K
2. Enable KV cache quantization: INT8 KV cache uses half the memory
3. Start new conversations: Clear accumulated context

Cause #5: Multi-GPU Inefficiency

Adding a second GPU doesn't double performance — interconnect bandwidth limits scaling.

How to Detect

• Multi-GPU setup but performance barely better than single GPU
• One GPU at 100%, others lower
• PCIe bandwidth showing as bottleneck in profiling

Why It Happens

GPUs need to communicate during inference. PCIe is slow compared to NVLink:

• PCIe 4.0 x16: ~32 GB/s
• NVLink: ~900 GB/s

Solutions

1. Use NVLink if available: Much better scaling
2. Pipeline parallelism: Can reduce communication needs
3. Accept the limitation: Multi-GPU over PCIe helps with capacity, less with speed

Cause #6: It's Actually Normal

Sometimes the speed is correct for your hardware — you just expected more.

Expected Performance (Approximate)

If you're in this ballpark, your setup is probably working correctly. The theoretical maximum is limited by memory bandwidth.

Diagnostic Commands

NVIDIA

# Watch GPU stats in real-time
nvidia-smi -l 1

# Check CUDA version
nvcc --version

# Monitor detailed GPU metrics
nvidia-smi dmon

Apple Silicon

# GPU usage (requires additional tools)
sudo powermetrics --samplers gpu_power

# Memory pressure
vm_stat

General

# llama.cpp: verify GPU layers
./main -m model.gguf -ngl 99 --verbose

# Ollama: check what's loaded
ollama ps

Still Stuck?