"Crowd dynamics, RL and Unity: A Journey" by Tomas Diaz

## Overview Tomas Diaz presents a comprehensive exploration of modeling human movement through crowd dynamics, reinforcement learning (RL), and the Unity game engine. The work aims to predict how people navigate shared spaces given desired trajectories, with applications in environmental design, simulation, and behavior analysis. The presentation is divided into four key sections: the historical evolution of crowd dynamics as a field, foundational RL concepts, the Kalman filter for state estimation, and practical implementation in Unity.

## Historical Evolution of Crowd Dynamics The field emerged from practical engineering needs. In the 1960s, Fruin's work quantified pedestrian sidewalk design using photography to analyze trajectories, introducing the concept of "level of service" (designations A–F based on density and expected flow). The 1970s brought a paradigm shift when Henderson's work demonstrated that Maxwell-Boltzmann mechanics from particle physics could model pedestrian movement, treating people like gaseous particles. The 1990s explosion of computational power enabled simulation-based validation. The social force model emerged as influential, positing that pedestrians are bombarded by attractive and repulsive forces—for instance, a burning wall repels nearby people. Recent research (the "crowd space" paper) revealed that no single algorithm captures all crowd configurations; instead, determining the specific space type and applying the appropriate algorithm yields better results.

## Reinforcement Learning and PPO Diaz pivots from physics-based models to RL, recognizing that uncovering a hidden universal human motion equation is unlikely. In RL, an agent observes a state from its environment, takes actions, and receives rewards. The policy (π) maps states to action probabilities, parameterized by neural network weights (θ). An advantage function determines whether actions outperformed or underperformed expectations, calculated as discounted cumulative rewards minus expected future rewards. The presentation introduces vanilla policy gradient before focusing on PPO (Proximal Policy Optimization by OpenAI), which prevents destructive policy updates. PPO clamps the ratio of new-to-old policy probabilities within a small range (typically ±0.2 epsilon), ensuring updates remain conservative and preventing the agent from accessing data distributions it was not trained on.

## Kalman Filter and Coleman Gain Diaz detours to explain Rudolf Kalman's work on optimal state estimation, motivated by Apollo program rocket control. The Kalman filter combines imperfect system equations with noisy sensor readings to estimate true state. The optimal weighting factor—the Kalman gain—minimizes error by dividing system error by the sum of system and sensor error. Matrix formulations account for observation matrices that collapse unmeasured variables (e.g., if only position is sensed, not acceleration). The demonstration with chaotic double pendulums shows how sensor readings "snap" the simulation back toward observed behavior. Ensemble Kalman filtering (running multiple models and averaging) improves tracking. This framework is applied to crowd dynamics: a statistical similarity measure paper uses Kalman filters to quantify how well simulations match real crowd data.

## Unity Implementation Unity ML-Agents wraps OpenAI's Gym framework, running PyTorch backend with Barracuda inference engine (supporting ONNX format). Developers override methods: OnEpisodeBegin initializes agents and environments, CollectObservations maps environment to state vectors, OnActionReceived applies actions and assigns rewards, and policy configuration specifies PPO hyperparameters. A simple example trains an agent to hit a box by controlling ball forces in x and z coordinates.

## Practical Application and Results Diaz trains agents on trajectory data from immutable crowds (20–30 trajectories, 2–3 seconds each). Agents receive no explicit reward for reaching destinations; instead, an entropy-based reward metric captures natural crowd behavior. Key insights: (1) sight information (raycasts visualizing nearby agents) is crucial but subject to diminishing returns around 40 rays; (2) reducing information to key dimensions accelerates learning; (3) data augmentation (rotation) and curriculum learning (progressively introducing trajectories) significantly improve performance. Trained agents generalize to unseen environments and different crowd densities without retraining, suggesting they capture underlying behavioral characteristics rather than memorizing trajectories.

## Takeaways The work demonstrates that RL can extract generalizable crowd behaviors from limited datasets, with potential applications in entertainment and behavioral quantification. Future directions include larger datasets, real-time camera feeds to continuously update training data, and deploying agents for diverse spatial configurations.