HW, SW, Performance and Costs for Llama-2 70b and Mixtral 8x7b LLM Inference with Low...- Ivan Baldo

### Overview In this talk, Ivan Baldo, a researcher and systems administrator at NLS, shares his findings and operational experiences benchmarking large language model (LLM) inference software and designing cost-effective hardware configurations for on-premises deployment. Spurred by client demand for low-concurrency, self-managed systems driven by data sovereignty requirements (especially among governmental clients in Uruguay and abroad), the study explores how to run models like Llama-2 7B, Llama-2 70B, and Mixtral efficiently using containers and dedicated GPU hardware.

### Key Topics - **Client Requirements & Scenarios:** Addressing scenarios with low concurrency where clients prefer on-premises deployments over managed cloud services due to privacy, data sovereignty, fixed cost predictability, and independence from cloud API throttling. - **Containerized GPU Infrastructure:** Moving away from Conda environments toward standard containerization using Podman on Rocky Linux with the Container Device Interface (CDI) to achieve zero overhead direct GPU access. - **Inference Software Engines:** Evaluating and benchmarking multiple inference engines including Hugging Face Transformers, CTranslate2, MLC-LLM, Hugging Face Text Generation Inference (TGI), text-generation-webui (TGW), vLLM, Triton Inference Server with TensorRT-LLM, and PowerInfer. - **Hardware Sizing and Cloud Testing Obstacles:** Sizing VRAM and system memory requirements, navigating cloud GPU shortages across AWS and GCP, and pricing out Dell on-premises hardware.

### Technical Details - **Setup & Baseline Benchmarking:** Using Bfloat16 (`bfloat16`) to provide a greater range of values for inference without precision loss, which mandates Nvidia Ampere or later (CUDA 8+ compute capability). Testing was performed by passing GPUs into containers using Nvidia Container Toolkit and compiling kernel modules via DKMS on Rocky Linux 9. - **Engine Performance on Llama-2 7B:** Targeting GPT-4's ballpark speed of ~26 tokens per second as a usability threshold. Tested executors achieved roughly 32 tokens per second. Most engines (vLLM, TGI, MLC, CTranslate2) demonstrated comparable throughput and were primarily bottlenecked by VRAM memory bandwidth rather than GPU core compute. While PowerInfer showed high throughput by offloading cold parameters to CPU cores, it introduced model quality degradation on 70B models. - **Hardware Sizing for Llama-2 70B:** Running unquantized Llama-2 70B requires roughly 156 GB of VRAM. Baldo evaluated an 8x Nvidia L4 (24 GB each) Dell server costing ~$65,000, but noted PCIe bandwidth limitations due to the absence of NVLink. Instead, a setup with 4x Nvidia A100 (40 GB PCIe/SXM with NVLink bridges) totaling ~160 GB VRAM was configured at ~$72,000. - **Cloud Deployment Challenges:** Attempting to validate setups before purchase revealed severe global GPU shortages. GCP denied quota requests for 8x L4 instances, and AWS instances in North America faced insufficient capacity errors, ultimately requiring deployment to a Seoul, South Korea region at a 50% price premium. On this setup, CTranslate2 (run with `mpirun`) achieved 23 tokens per second across 4x A100s, with GPU compute fully utilized. - **Mixture of Experts (MoE):** Highlighting architectures like Mixtral 8x7B and Mixtral 8x22B as modern solutions where only active expert parameters (e.g., 14B active parameters) are calculated per token, enabling parallel execution across multiple GPUs.

### Takeaways - **Containerization is Viable for Inference:** Podman with CDI provides a flexible, multi-distribution setup without performance penalties. - **Economics of On-Premises vs. Cloud:** While on-premises hardware costs compute out to roughly $7/hour (favorable compared to dedicated cloud GPU instances), commercial APIs like OpenAI remain cheaper purely on cost per token for low concurrency. On-premises is justified primarily by data sovereignty, fixed cost predictability, absence of rate limits, and latency control in retrieval-augmented generation (RAG) loops. - **Deployment Planning:** Sizing server RAM to exceed GPU VRAM is necessary for model loading. When hardware constraints arise as models grow, post-training quantization (such as INT8 or NF4) can extend server longevity.