Local LLM & Legacy GPU Cheatsheet

Building from Source (Legacy GPU Support)

When pre-compiled binaries drop support for older GPUs (like the Pascal-based Tesla P40), you must compile the inference engine from source targeting your specific Compute Capability.

Build llama.cpp with Specific CUDA Version

Parameters:

• compute_cap: Your GPU's Compute Capability without the decimal (e.g., 6.1 becomes 61).
• cuda_path: Path to the specific CUDA toolkit you want to build against (useful when avoiding host/server version mismatches).

git clone [https://github.com/ggerganov/llama.cpp](https://github.com/ggerganov/llama.cpp)
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=[compute_cap] -DCMAKE_CUDA_COMPILER=[cuda_path]/bin/nvcc
cmake --build build --config Release -j

Hardware Management

Multi-GPU Inference

Restrict which GPUs the AI engine can see. This is highly useful when building a budget cluster (e.g., dual Tesla P40s) to pool VRAM.

export CUDA_VISIBLE_DEVICES=[gpu_ids]
./build/bin/llama-server -m path/to/model.gguf -ngl 99

💡 Tip: Set -ngl 99 in llama.cpp to offload all possible layers to the GPUs specified in CUDA_VISIBLE_DEVICES.

Model Format Suffixes

Understanding file extensions and model suffixes is crucial for matching the right model to your specific hardware capabilities.

GGUF Quantization Levels

When downloading GGUF models, you will see different quantization suffixes. This dictates the balance between VRAM usage, speed, and output quality.

Decoding GGUF Quantization Suffixes

GGUF models use a specific naming convention to tell you exactly how the model was compressed. Understanding this helps you balance VRAM usage against the AI's "smartness."

1. Anatomy of a Quantization Name

2. Common Quantization Levels (From Best Quality to Smallest Size)

Advanced Inference: vLLM

vLLM is a high-throughput and memory-efficient LLM inference engine. It uses PagedAttention to manage attention keys and values efficiently.

When to use vLLM?

• High Concurrency: When multiple users are querying the model simultaneously.
• Modern Hardware: Best suited for GPUs with Compute Capability 7.0+ (Volta) or higher.
• Flash Attention: heavily utilizes Flash Attention hardware features (which the Tesla P40 lacks).

Run a Model via vLLM

python3 -m vllm.entrypoints.openai.api_server \
    --model unsloth/Qwen3.5-MoE-A17B \
    --quantization awq \
    --tensor-parallel-size 2

⚠️ Note for Legacy GPUs: While vLLM is the industry standard for modern production, llama.cpp remains the undisputed champion for squeezing performance out of older, non-Tensor Core GPUs like the P40.

NVIDIA CUDA Micro-Architecture & Compute Capability List

Use this table to find the target [compute_cap] for your specific hardware generation.

Resources

• llama.cpp GitHub
• vLLM Documentation
• HuggingFace (To find GGUF models)