PLNOG22 - Bartosz Partyka - Reliable DNS Cluster

### Overview Bartosz Partyka, a network/systems engineer at an Internet service provider, presents the architectural evolution and implementation of a highly available and reliable recursive DNS cluster. Serving over 260,000 retail and 100,000 business customers with symmetric FTTH and other access technologies, the operator faced peak traffic of approximately 200,000 queries per second (QPS). The presentation walks through moving away from fragile single-server and centralized models to a multi-node Anycast deployment backed by Linux Virtual Server (IPVS) and automated health checking.

### Key Topics - **Client Behavior and DNS Failure Impact**: Operating systems handle DNS resolution failures differently. While Android caches aggressively, Windows waits 1 second before retrying, then escalates up to 4 seconds before querying secondary resolvers simultaneously. Default Linux resolver configurations wait up to 5 seconds. In the context of 10G connections, these resolver timeouts drastically harm perceived customer quality of experience (QoE). - **Anycast Routing Architecture**: Transitioning from centralized resolvers to multiple geographically distributed nodes using BGP Anycast allows queries to automatically route to the topologically nearest resolver node, mitigating latency and offering baseline redundancy without needing hardware appliances. - **Load Balancing and IPVS**: Within each Anycast point of presence (PoP), the team implemented Linux Virtual Server (IPVS) using Direct Routing (DR) mode. This avoids returning outbound traffic through the load balancer, modifying only destination MAC addresses so DNS response packets egress directly back to the client, conserving compute power. - **Health Checking and Automation**: IPVS lacks application-level intelligence, so Keepalived was implemented to actively probe DNS endpoints every second. If an instance fails, Keepalived dynamically updates IPVS tables. A supervisor process monitors active backend counts; if all DNS resolvers in a PoP fail, BGP Anycast announcements from that site are withdrawn to seamlessly divert queries to another node, incorporating delayed re-advertisement to avoid route flapping.

### Technical Details - **DNS Engine Selection**: The team chose PowerDNS (pdns) as the resolver engine due to its comprehensive metrics/statistics and Lua scripting capabilities. Lists of blocked domains are managed efficiently without disrupting resolver performance. - **Central Management & Dynamic Cross-Node Balancing**: A central management server monitors real-time load across nodes. By establishing cross-node Layer 2 tunnels between load balancers and backend resolvers in different sites, the orchestrator can dynamically offload queries from saturated nodes to idle nodes without breaking BGP routes. - **Security and Compliance Filtering**: The cluster integrates daily updates to block malware and satisfy legal requirements (such as the Polish register of banned gambling domains, sinkholing clients to ministerial advisory pages). On peak days, the system blocks up to nearly one million domain queries. - **Real-World Monitoring with Probes**: Beyond aggregating metrics in Zabbix and analyzing week-over-week trends, hardware probes deployed on DOCSIS, GPON, and core links simulate real user web browsing, television, and resolution requests. Measured response times average around 2–3 ms on GPON and 13 ms on DOCSIS, outperforming external public resolvers (such as 1.1.1.1).

### Takeaways - Software-defined architectures using standard open-source tools (Linux IPVS in Direct Routing mode, Keepalived, BGP Anycast, and PowerDNS) can reliably handle high-throughput recursive DNS (200k+ QPS) at low cost. - Avoiding return-path routing on load balancers drastically preserves system capacity. - Automated route withdrawal, hysteresis to prevent flapping, and synthetic end-user probing are essential to guarantee continuous service availability and high customer quality of experience.