Deploying Your Own Language Models: A Use Case Demo with Open Source...

### Overview In this talk, Shaan and Surya Pathak from Red Hat's Emerging Technologies group present a comprehensive framework for building and deploying open-source large language model (LLM) applications. Focusing on a retrieval-augmented generation (RAG) system for Red Hat OpenShift on AWS (Rosa) product documentation, they illustrate how organizations can move away from proprietary, cloud-hosted APIs toward self-hosted, cloud-native deployments on OpenShift and Kubernetes. The presenters cover the end-to-end architecture, from vector embeddings and databases to supervised fine-tuning, Ray Serve deployment, and human feedback collection.

### Key Topics - **Cloud-Native RAG Architecture**: Building question-answering systems using documentation embeddings, vector storage, candidate retrieval, prompt engineering, and language model inference. - **Vector Embeddings & Databases**: Converting domain documentation into multi-dimensional embeddings (384 dimensions) and utilizing open-source vector databases (such as Qdrant) with approximate nearest neighbor search to identify top candidate answers. - **Open-Source LLM Selection & Adaptation**: Criteria for selecting open-source base models based on compute budget, licensing (e.g., Apache 2.0, MIT), and benchmarks (MMLU, TruthfulQA), followed by pre-training, supervised fine-tuning (SFT), and reinforcement learning with human feedback (RLHF). - **Scalable Model Deployment**: Leveraging Ray and Ray Serve orchestrated with the CodeFlare SDK to deploy containerized model endpoints with configurable CPU, memory, and GPU resources on Kubernetes. - **Evaluation & Feedback**: Measuring retrieval quality using Normalized Discounted Cumulative Gain (NDCG), generation quality using BLEU/ROUGE, throughput/latency benchmarks, and integrating human feedback via tools like Argilla to mitigate hallucinations.

### Technical Details - **Embedding Pipeline**: Demonstrated converting queries and Rosa documentation into 384-dimensional vector spaces. Cosine similarity confirmed semantically related Rosa text yielded higher similarity scores (~0.6) compared to unrelated text like Docker documentation (~0.2). Visualizing compressed 2D representations verified distinct clustering between unrelated topics. - **Vector Database**: Used Qdrant for both in-memory local testing and deployment on Kubernetes clusters, comparing options like Milvus, Elasticsearch, Weaviate, and PGvector. - **Prompt Engineering & Hallucinations**: Prompt templates significantly improved completeness over zero-shot queries, though challenges like hallucinated URLs persisted when querying for external documentation links. - **Model Fine-Tuning & Serving**: Highlighted supervised fine-tuning of a Flan-T5 model (783 million parameters) on a single 16GB Nvidia T4 GPU using a prompt-response dataset built from Rosa FAQs. Compared this with larger models such as LLaMA 2 (7B and 13B) and OpenAI GPT-3.5 (~170B parameters). - **Deployment Infrastructure**: Deployed Flan-T5 using the CodeFlare SDK to define a Ray cluster on Kubernetes (2 worker nodes, 2 CPUs, 8GB memory per node, 1 GPU, autoscaling instance limits) and exposed it via Ray Serve decorators specifying replica count and GPU resource allocations. - **Evaluation & Argilla Integration**: Highlighted NDCG for ranking retrieval sources (1–5). For generation without ground-truth answers, deployed an Argilla instance on Kubernetes to allow subject matter experts to rank, rate (1–5 scale), or edit model responses directly.

### Takeaways - Self-hosting open-source LLM pipelines on Kubernetes or OpenShift provides critical data privacy and security advantages, particularly for handling sensitive internal policies or documentation without sending data to third-party APIs. - RAG pipelines allow systems to scale seamlessly from small private document sets to large corporate corpora without retraining entire base models from scratch. - Achieving high-quality domain responses requires balancing base model parameter scale with compute budgets, optimizing prompt engineering, and establishing continuous evaluation loops using expert human-in-the-loop tooling.