"Apache Pulsar and the Legacy of Telegraphy" by Jowanza Joseph

### Overview In this presentation, Jowanza Joseph, a software engineer at Pluralsight, draws parallels between historical telegraphy—often referred to as the "Victorian Internet"—and modern distributed publish/subscribe systems, specifically Apache Pulsar. Joseph examines how early telegraph networks (both wired switchboard systems and wireless Morse code networks) grappled with foundational distributed systems challenges, including message switching, interception, cipher-based encryption, multi-point transmission, and message loss. He then details how Apache Pulsar's decoupled, layered architecture addresses these legacy challenges at cloud scale with modern durability, encryption, and stream processing mechanisms.

### Key Topics - **The Legacy of Telegraphy:** Early telegraph networks faced severe operational hurdles, including privacy risks from switchboard operators intercepting messages, weak manual ciphers easily cracked by authorities, delivery uncertainty, and difficulties handling multi-subscriber broadcasting. - **Pulsar Architecture & Decoupled Storage:** Originally developed at Yahoo and governed by the Apache Software Foundation, Apache Pulsar separates compute brokers from storage nodes via Apache BookKeeper, with coordination managed by Apache ZooKeeper. This separation allows storage and broker compute to scale independently. - **Schema Enforcement & Tiered Storage:** Pulsar manages data transmission and validation via a built-in schema registry supporting formats like Avro and JSON Schema to handle schema evolution. For cost-effective retention, it provides tiered storage offloading from BookKeeper into Amazon S3 and Glacier while maintaining a unified read API. - **Delivery Guarantees & Multi-Casting:** Pulsar supports robust publish/subscribe multi-casting and gives users configurable delivery guarantees, ranging from low-latency "at-least-once" delivery to "exactly-once" processing semantics. - **Stream Processing & Serverless Functions:** Real-time analytics are enabled via Pulsar SQL (embedding Presto across live and tiered data) and lightweight, serverless Pulsar Functions (written in Java, Python, or Go) running directly on nodes or isolated inside Docker containers.

### Technical Details - **Components:** Pulsar consists of client-facing broker nodes, Apache BookKeeper for persistent append-only immutable log storage, and Apache ZooKeeper for distributed consensus and cluster state. - **Security & Data Ingress:** Ingress data validation is performed against the schema registry using Avro schema evolution (e.g., modifying fields to accept null values). Message transmission over the wire is secured out of the box using SSL. - **Tiered Storage Architecture:** BookKeeper acts as "hot" storage; as data ages, Pulsar can automatically offload segment data to cold tiers like AWS S3 and Glacier. Client consumers continue reading via the identical Pulsar API without modifying query logic. - **Stream Processing Engine:** Pulsar SQL embeds Presto workers capable of executing distributed SQL queries spanning hot BookKeeper logs and archived S3/Glacier storage. Pulsar Functions support single-input-to-output topic event processing, with community interest in native sinks (e.g., Elasticsearch, Couchbase, CockroachDB).

### Takeaways Modern pub/sub architectures like Apache Pulsar solve historical networking and message-routing vulnerabilities through independent scaling, strong schema governance, native SSL encryption, and flexible tiering. By isolating compute from storage, systems can seamlessly handle multi-petabyte datasets, maintain unified SQL query capabilities across hot and cold archives, and run serverless stream processing workloads efficiently.