"Designing Augmented Reality Systems for Speed" by Diana Hu

### Overview In "Designing Augmented Reality Systems for Speed," Diana Hu, head of the AR engineering team at Niantic Labs (makers of Pokémon GO and acquirers of Escher Reality), examines the technical challenges of making augmented reality fast, reliable, and pervasive on commodity mobile hardware. While science fiction has long envisioned ubiquitous AR, practical AR systems must simultaneously solve interdisciplinary problems across computer vision, robotics, computer graphics, and distributed systems within tight thermal, battery, and latency budgets.

### Key Topics - **Fundamental Hurdles of AR**: Hu outlines why AR remains difficult. Computer vision requires both semantic understanding (recognizing objects via deep learning models that typically require power-hungry GPUs) and geometric perception (understanding 3D space, depth, and spatial navigation, derived from robotics). Graphics rendering must realistically integrate digital objects with complex real-world lighting variations. Additionally, developer toolchains (e.g., Unity, Unreal) still face lengthy 5- to 15-minute iteration cycles because the real world lacks fixed boundaries. - **Trends in Hardware and Telecommunications**: Projecting the future of AR, Hu highlights Koomey's law (battery requirements falling by half every 1.5 years for fixed compute loads), demonstrating that an iPhone 7 benchmarks comparably to older MacBook models. Referencing Richard Feynman's 1985 calculations on physical limits of computation and Holmes's law of telecommunications, she notes that cellular speeds inevitably converge toward Wi-Fi and wireline bandwidths (e.g., 5G targeting 10 Gbps), eventually matching human visual processing capacities. - **Treating Mobile Devices Like Distributed Servers**: As multi-core mobile processors become standard, AR architectures should adopt proven distributed system design patterns, such as colocation, caching, remoting, sharding, and concurrency without blocking locks.

### Technical Details - **Speeding Up Networking**: AR requires rendering at ~30 frames per second, necessitating response latencies within a 33 ms frame budget. Traditional client-server architectures route camera pose data from a phone through a cell tower to a remote cloud data center (e.g., in Virginia or Iceland) and back, incurring hundreds of milliseconds of latency. Niantic addressed this by eliminating cloud hops for localization updates. By establishing a direct, peer-to-peer mesh connection through local cell towers between co-located devices, multiplayer tracking achieved latency in the tens of milliseconds. A hybrid protocol approach combines UDP for low-latency peer-to-peer pose broadcasting with TCP/server connections for anti-cheat authority and larger asset transfers. - **Speeding Up Computation via the Actor Model**: Simultaneous Localization and Mapping (SLAM) and Visual Inertial Odometry (VIO) pipelines ingest high-bandwidth sensor feeds (an uncompressed 1080x720 video feed at 30 fps requires ~600 Mbps of processing) and split work across feature tracking, localization, and mapping blocks. Naive implementations rely heavily on mutexes, shared state, and busy-waiting loops, causing computation cycles to exceed 1 second on embedded architectures. Niantic decoupled the SLAM pipeline using an actor model: camera inputs, feature extraction, localization, and mapping run as isolated actors communicating through message queues and internal sequential mailboxes. Without altering underlying computer vision math or algorithms, removing lock contention and busy-waiting yielded a 5x speedup.

### Takeaways Rather than waiting for new mathematical breakthroughs or specialized hardware, significant performance gains in augmented reality can be realized immediately by applying mature distributed systems paradigms—such as peer-to-peer mesh networking and lock-free, message-passing actor architectures—directly to mobile AR software.