Keynote: How LLMs Have Changed the Way Organizations Do AI

### Overview In this keynote address, Robert Nishihara, co-founder and CEO of Anyscale and co-creator of the open-source distributed framework Ray, examines how large language models (LLMs) and generative AI have transformed how organizations build, adopt, and operationalize artificial intelligence. Drawing on Anyscale's experience powering AI workloads across hundreds of organizations—ranging from OpenAI's training of GPT-4 to infrastructure at Uber, Pinterest, Spotify, and Samsara—Nishihara outlines the shifting challenges in scale, platform evolution, application lifecycles, and developer demographics.

### Key Topics - **Scale and Infrastructure Demands**: Distributed computing has shifted from an emerging need to a baseline operational requirement. Models no longer fit on single GPUs, workloads routinely consume dozens to thousands of GPUs, and upcoming transitions to multimodal image and video processing will expand data intensity far beyond text. - **Historical Precedents and Future Readiness**: Organizations previously transitioned from classical machine learning (decision trees, linear models, random forests) to deep learning, which required extensive migrations to GPU-based platforms. Teams are currently undergoing a similar architectural shift to accommodate generative AI, while simultaneously planning for multimodal data, evolving GPU architectures (predominantly Nvidia), and emerging hardware accelerators. - **Two-Phase LLM Adoption Lifecycle**: Unlike historical AI workflows that began with data curation and model training before shipping products, generative AI reverses the process. Organizations first focus on building, rapid iteration, and establishing product-market fit. In the second phase, organizations double down by scaling, customizing, fine-tuning, or pre-training models, and addressing latency, reliability, and cost. - **Changing Evaluation Priorities**: As baseline models improve (e.g., GPT-4 to GPT-5, Llama 2 to Llama 3) and become "good enough" for specific tasks, organizational focus shifts away from pure output quality toward cost efficiency, latency, privacy, and customizability. - **Expansion of AI Practitioners**: The primary audience building AI software has broadened from specialized machine learning experts to general software developers who treat models as black boxes and prioritize usability and product integration.

### Technical Details - **Distributed Frameworks**: Ray serves as the underlying distributed framework enabling compute scaling across single-node boundaries for deep learning and Python applications. - **Multi-Model Architectures and Routing**: Applications increasingly make multiple LLM calls rather than relying on a single model. Organizations deploy model routing techniques to dynamically evaluate and direct individual queries to appropriate models based on performance, cost, or specialization. - **Hybrid Deployment and Fine-Tuning**: Many organizations utilize a hybrid strategy combining proprietary frontier foundation models (such as those from OpenAI) with customized open-source models. Nishihara highlights that fine-tuned Llama models have matched or exceeded GPT-4 performance for specific tasks at a fraction of the cost. - **Evaluation Challenges**: A primary technical bottleneck in the early adoption phase is systematic application evaluation—determining whether an application or chatbot has genuinely improved when swapping one model for another.

### Takeaways - Adopting LLMs requires flexible platform infrastructure capable of accommodating multi-model pipelines, model routing, and upcoming multimodal workloads. - As foundational models mature and pass quality thresholds, competitive differentiation and operational sustainability depend on latency optimization, cost management, and fine-tuning specialized models. - The democratization of AI tooling toward standard software developers will accelerate the volume and variety of AI-powered applications across industries.