Open Source LLMs in the Cloud: Scalable Solutions

### Overview In this talk, Miley Fu and Hung-Ying Tai (Hydai) from the open-source WasmEdge project present an alternative paradigm for deploying large language models (LLMs) and AI workloads in cloud environments using WebAssembly (Wasm). Running AI in the cloud traditionally demands massive all-in-one container images crammed with Linux distributions, CUDA/ROCm runtimes, PyTorch, and heavy framework dependencies, causing major friction in scalability, image size, and dependency maintenance. The speakers outline how WebAssembly and the WasmEdge runtime offer a sandboxed, ultra-lightweight abstraction layer that decouples application business logic from underlying GPU runtimes and host drivers.

### Key Topics - **Current AI Deployment Approaches**: The speakers compare three primary deployment methods: Python-based deployment (flexible for research with LangChain/LlamaIndex, but suffering from complex dependency hell), native binaries like llama.cpp or vLLM (fast, but lacking portability across OS/hardware without full recompilation), and WebAssembly (portable, secure sandbox, near-native performance). - **WebAssembly & WasmEdge Architecture**: By compiling code written in Rust, C++, Go, or other languages into Wasm binaries, deployment artifacts shrink drastically (down to 3–20 MB). Applications access hardware acceleration via the emerging WASI-NN (WASI Neural Network) standard and WasmEdge SDKs rather than packing device libraries inside container images. - **Operational Trade-offs**: Traditional container models impose pain on the developer who must construct complex Dockerfiles, while operators simply deploy the bloated images. The Wasm approach shifts the dependency burden to the host infrastructure (operators configure the Wasm runtime, Crun, and GPU drivers), leaving container images virtually empty except for the compiled `.wasm` file. - **Real-World Use Cases & Ecosystem**: Demonstrations and case studies include WikiFunctions (Wikimedia's serverless function platform), an automated video dubbing/translation pipeline (voice-to-text with Whisper, translation via Llama, and text-to-speech), a voice-controlled robotic car, UC Berkeley educational assistant bots, and Gaianet (a decentralized Web3 network running AI inference across user nodes).

### Technical Details - **Container Construction**: In contrast to standard container images that pull Ubuntu/Debian bases and multi-gigabyte dependency trees, a Wasm container uses a three-line Dockerfile starting `FROM scratch`, copying the `.wasm` binary, and defining the execution command. - **Infrastructure Setup**: The host requires the Nvidia Container Toolkit / CLI and an OCI runtime such as `crun` built with WebAssembly support. WasmEdge plugs into crun to execute the container directly, seamlessly integrating with standard orchestration tools like `docker compose` or Kubernetes. - **WASI-NN API Flow**: Application code follows a four-step lifecycle using the WASI-NN API: loading model files with specific configurations (including splitting models across multi-GPU setups), receiving input prompts, computing inference through pluggable backends (such as llama.cpp for GGUF models, LibTorch for PyTorch, or OpenVINO), and returning outputs. - **Community Feedback & Q&A Discussion**: During the Q&A, an audience member from Docker raised challenges regarding host-side dependency management, vulnerability auditing (SBOMs), and running heterogeneous versions across hosts, suggesting multi-stage Docker builds as a potential alternative. The speakers highlighted that WasmEdge plugins can be updated directly on the host and that tooling is evolving toward automated Kubernetes operators to manage node-level dependencies.

### Takeaways WasmEdge and WebAssembly present a compelling, modular option for scaling LLM inference in cloud and edge clusters. By abstracting the hardware and runtime layers through WASI-NN, organizations can deploy tiny, portable containers that launch instantaneously, run across heterogeneous architectures (Mac, Linux, Nvidia, AMD), and integrate directly with existing container orchestration pipelines. While host configuration requires specialized setup and operator discipline, ongoing development of Kubernetes operators and OCI runtime integrations aims to streamline node lifecycle management.