Sponsored Session: Observability and Agentic AI - From Observability to Answers... Miki Matsumoto

### Overview In this sponsored session, Miki Matsumoto, the first employee in Japan for ClickHouse working in solutions architecture and technical support, explores the integration of agentic AI into observability workflows. Specifically, the presentation addresses how ClickHouse is working to transition observability from manual dashboard inspections to direct answers and automated incident remediation, with the ultimate goal of augmenting or replacing routine Site Reliability Engineering (SRE) tasks such as off-hours and weekend on-call duty.

### Key Topics - **ClickHouse Ecosystem and Observability Growth:** ClickHouse is an open-source OLAP database designed for real-time analytics on massive datasets, featuring high data compression (10x to 100x), fast SQL analytics, and horizontal scalability. Enterprise organizations (such as Anthropic and OpenAI) have migrated from platforms like Datadog to ClickHouse to control rapidly growing observability costs. Matsumoto also highlights related products in the ecosystem: HyperDX (an open-source observability platform built on ClickHouse), LibreChat (an AI chat platform), PeerDB (database change data capture synchronization), and chDB (in-process ClickHouse). - **The Quick Stack:** Comprising OpenTelemetry, ClickHouse, and HyperDX, the open-source observability stack ingests telemetry through OpenTelemetry collectors and ClickHouse exporters directly into auto-created ClickHouse tables, allowing HyperDX to visualize traces, logs, event patterns, service maps, and user session behaviors without heavy manual configuration. - **Challenges with AI SRE Agents:** When testing AI agents (via LibreChat) against ClickHouse's internal 100 petabyte / 500-billion-plus record observability store (with a 6-month TTL), two major failure modes emerged: 1. *Jitter:* Inconsistent SQL query generation where the agent issues wildly different queries across executions (even under low temperature settings). 2. *Over-Investigation:* Agents probing irrelevant, non-critical logs deeply due to an inability to prioritize errors, driving up token consumption and creating unpredictable costs ranging from $0.10 to $10 per query.

### Technical Details & Solutions To stabilize AI investigation and mitigate high token spend, Matsumoto outlines an architecture centered on modular Model Context Protocol (MCP) servers: - **Decomposed MCP Servers:** Instead of providing a single monolithic MCP server with unbounded database and query access, the system uses purpose-built, scoped MCP servers. An initial broad MCP server inspects high-level errors and time windows first, followed by secondary MCP tools that restrict query scopes to narrow down specific root causes. - **Agent Root Cause Analysis Demo:** Matsumoto demonstrates an AI agent configured locally with LibreChat connected to ClickHouse Cloud to investigate simulated errors (induced by high-concurrency inserts of 1,000 connections). The agent seeds its understanding from table structures, iteratively queries error logs, identifies cluster connection timeouts, memory limit exhaustion, and invalid partition key usage, and proposes concrete remediation actions (e.g., stopping high-concurrency inserts and optimizing partition schemes).

### Takeaways ClickHouse aims to natively embed agentic AI capabilities into HyperDX and the ClickHouse platform across three planned development phases: first, embedding the AI agent directly into the observability UI; second, enabling event-driven automation that triggers autonomous agent investigations from error notifications or customer alerts; and third, supporting a human-in-the-loop workflow where human SREs collaborate with conversational agents to diagnose complex issues across massive datasets.