DevOpsDays Warsaw 2016 - Stefan Thies - Monitoring and Log Management for Docker Swarm and...

### Overview In this DevOpsDays Warsaw 2016 presentation, Stefan Thies from Sematext examines the complexities of centralized logging and performance monitoring across containerized environments, focusing specifically on Docker Swarm and Kubernetes. Thies outlines the full lifecycle of log management and metric collection, highlights differences in orchestration abstractions, discusses the advantages of interacting directly with the Docker API, and reviews lessons learned from the community-driven Swarm 3k test project.

### Key Topics - **Centralized Log Management Architecture**: Managing logs involves reconfiguring applications to emit logs to stdout/stderr rather than static files, employing log shippers (such as Fluentd, rsyslog, or Logstash) for log collection and parsing into structured JSON, and storing/indexing logs in systems like Elasticsearch alongside alerting mechanisms (e.g., PagerDuty). - **Performance Monitoring & Time-Series Data**: Monitoring containerized systems requires exposing metrics (sometimes necessitating specific web server or app configurations), shipping them to time-series databases (e.g., Prometheus, InfluxDB, Elasticsearch), aggregating data over varying intervals, and visualizing metrics in dashboards like Kibana. - **Kubernetes vs. Docker Swarm Concepts**: Kubernetes uses atomic deployment units called pods—keeping tightly coupled containers (like Elasticsearch and Kibana) together on a single host—alongside replication controllers, services, DaemonSets, Heapster, and built-in autoscalers. Docker Swarm provides familiar Docker commands, overlay networks, and scales services or containers across multiple hosts. - **Docker Logging Drivers vs. API-Driven Agents**: While Docker logging drivers (JSON-file, journald, syslog, Fluentd, Splunk) ship logs via TCP/UDP, issues like unbuffered UDP packet loss or container startup failures during syslog TCP outages can disrupt operations. In contrast, an agent communicating directly with the Docker API can read local stdout/stderr streams, capture rich metadata and labels, buffer data during network outages, and route logs intelligently. - **Application Metric Discovery**: General container metrics provide resource usage, but application-specific endpoints (such as Elasticsearch or Nginx statistics) require dynamic discovery (e.g., via etcd registrations or reading container labels and network bindings via the Docker API) to automate agent configuration. - **Large-Scale Testing (Swarm 3k)**: Experiences and telemetry from deploying monitoring agents across thousands of Docker Swarm nodes in a real-world community test.

### Technical Details - **Container Metrics & Sizing**: The Docker API exposes over 70 distinct metrics per container, including memory fail counters, out-of-memory events, and CPU throttling. Monitoring these metrics enables proper configuration of CPU limits (quotas) and Java heap memory limits to prevent out-of-memory errors. - **Docker Events**: Utilizing Docker API events allows operators to trace lifecycle actions, including image pulls, container start/stop events, volume mounting, and task statuses. - **Agent Deployment**: While early Swarm setups required custom shell scripts and loops to launch monitoring agents per node, modern orchestration features—such as Swarm global services and Kubernetes DaemonSets—simplify deploying monitoring agents universally. - **Swarm 3k Benchmark Results**: In the Swarm 3k community experiment, monitoring agents were deployed across 4,700 nodes. During the run, 28 GB of metrics were collected, scaling up to 60,000 containers (short of the 250,000 container target due to disruptions when one of the three Swarm manager nodes encountered issues and restarted). Visualizing 4,700 ungrouped hosts proved challenging, emphasizing the need for label-based grouping and filtering in dashboards.

### Takeaways - Standard container metrics are easy to capture, but gathering application-level metrics still involves considerable configuration and discovery effort. - Relying on the Docker API enables smart agents to gather logs, metrics, labels, and Docker events together, while offering disk buffering when remote destinations become unreachable. - Effective multi-node monitoring at scale requires consistent tagging, labeling, and automated agent deployment via DaemonSets or global services.