Learning Event Streaming Processing with Pac-Man

### Overview In this Open Source Summit presentation, Ricardo Ferreira, a developer advocate at Confluent, demonstrates the core concepts of event stream processing through an interactive game demo called "Streaming Pac-Man." The presentation aims to bridge the gap between abstract event streaming theory and practical implementation by allowing attendees to play Pac-Man while their gameplay events are ingested and processed in near real time to update an event-driven scoreboard.

### Key Topics Ferreira contrasts traditional data storage architectures—where data is statically stored in relational or NoSQL databases before batch processing—with event stream processing, where applications act upon unbounded streams of data in motion. Event streaming is broken down into two main layers: 1. **Data Streaming (Streaming Data):** Data acquisition, ingestion, and durable storage using Apache Kafka. 2. **Streaming Analytics:** Processing, filtering, and aggregating events in real time to generate immediate value, implemented using ksqlDB.

### Technical Details & Architecture The demo's end-to-end architecture is hosted in AWS and consists of several interconnected components: - **Frontend Game & Ingestion:** The Pac-Man web client (built with HTML, JavaScript, and CSS and hosted in an S3 bucket) generates gameplay events. These events are sent to an AWS API Gateway endpoint, which triggers an AWS Lambda function to publish raw records into an inbound Apache Kafka topic called `user_game`. - **Stream Processing with ksqlDB:** - A stream is defined over the `user_game` topic, parsing structured JSON fields such as user identifier, score, level, and remaining lives. - Another stream tracks `user_losses` each time a game over occurs. - Stateful aggregations are performed via ksqlDB to build persistent tables: `stats_per_user` (tracking maximum score and level grouped by user) and `losses_per_user` (counting total deaths). - A unified table, `scoreboard`, is constructed by joining `stats_per_user` and `losses_per_user`. A separate table, `highest_score`, computes the global maximum score. - Under the hood, ksqlDB abstracts complex processing topologies executed as Kafka Streams applications. - **Downstream Data Serving & Integration:** - A scoreboard web API (AWS API Gateway backed by an endpoint reading from Kafka) exposes the computed results, which the frontend polls to display live leaderboards. - Using Kafka Connect, scoreboard updates are sinked continuously to a Redis cache. - Ferreira demonstrates querying the aggregated data via an Amazon Alexa voice assistant skill pointing at Redis, answering questions about top players in real time. - Client applications can also query ksqlDB tables directly via HTTP POST requests or the ksqlDB Java client.

### Takeaways & Q&A Ferreira emphasizes that event streaming architectures provide a continuous processing paradigm using familiar SQL-like primitives via ksqlDB. During the concluding Q&A: - **IoT Sensor Ingestion:** Asked if Kafka is suitable for collecting data from sensors, Ferreira notes that while Kafka excels at scale and processing, sensors that cannot sustain durable stateful TCP connections should first publish to an MQTT broker layer, which then relays data to Kafka. - **Client State Updates:** Client game state can be maintained and queried by writing stateful stream processors that output to dedicated topics or queryable ksqlDB tables.