Lessons and Lulz: The 5th Annual Black Hat USA NOC Report

## Overview The 5th Annual Black Hat USA NOC Report briefing is presented by longtime network leads "Grifter" (a threat hunter for RSA celebrating his 50th Black Hat event) and Bart (a security consultant with Optive, marking his 12th year). Together with "NOC Queen" Merisa (responsible for overall NOC operations across Informa/UBM), Chief Architect Steve Fink, and roughly 15 volunteer cybersecurity professionals spanning analyst to VP roles, they manage the entire technical operations and network infrastructure for the conference. Scaling from an initial 1,500 attendees run on Netgear switches and Cisco 2600 routers to over 20,000 attendees, the modern Black Hat NOC operates on a strict "Keep It Simple, Stupid" (KISS) philosophy supported by select technology partners rather than paid sponsors.

## Infrastructure and Architecture To retain full visibility and mitigation capabilities, the NOC team completely replaces the venue’s (Mandalay Bay) switching, routing, firewalls, and access points—utilizing only the facility's in-wall cabling. Key deployment components include: - **Core Connectivity & Firewalls:** Two 10 Gbps redundant circuits provided by CenturyLink (offering 20 Gbps total capacity), routed through a pair of Palo Alto Networks PA-5260 firewalls managed via Panorama. Overall conference bandwidth utilization hovered around 30% to 40%, with typical peaks reaching approximately 13 Gbps. - **Traffic Tapping & Monitoring:** Gigamon GigaVUE appliances tap network traffic and feed security analytics tools, notably RSA NetWitness. RSA also provides Identity Force. - **Switching & Wireless:** Ruckus ICX switches serve core and distribution roles, while Ruckus R600 and R710 wireless access points handle Wi-Fi. - **Wired-to-Wireless Migration:** Across 88 training courses serving roughly 10,000 students, the NOC transitioned almost entirely to wireless, running only a single classroom with wired drops. Across all sessions, only one class faced network degradation—traced directly to a physical in-wall cable failure.

## Monitoring, Telemetry, and Incident Response The NOC visualizes and orchestrates network activity in real time through dedicated telemetry walls: - **Visual Flow Monitoring (OIP):** Network flows are displayed visually using OIP, color-coding traffic as green (TCP), red (UDP), and white (ICMP). Thicker connection bars indicate surges or spikes in traffic. When abnormal spikes appear, analysts pull target host IPs into RSA NetWitness to distinguish between benign activity and DoS attacks. - **Subnet Mapping & Location Tracking:** Network architecture maps specific subnets directly to training rooms. During the event, an attendee running `masscan` across the internet was rapidly identified by classroom and tracked down in person. - **Centralized Orchestration:** Dashboards aggregate incidents across multi-vendor appliances into unified cases, linking alerts directly to digital facility floor plans for fast on-site verification. Infrastructure boards track core hardware status, signaling outages immediately if switches turn red.

## Physical Security and Social Engineering Exercise The physical perimeter of the NOC—protected by dedicated guards and strict access controls—was tested via a live exercise arranged with "Snow," an instructor teaching a full-scope social engineering course: - **The Challenge:** Students were given signed authorization documents ("get-out-of-jail-free" cards) and 24 hours to retrieve a designated trophy: a clipboard and folder marked *"Grifters Don't Touch"*, placed next to the NOC mascot, Lyle. - **Execution:** Within five minutes of the challenge opening, a student presented a photo of the clipboard to the security guard on duty, claiming Grifter had asked him to collect it; the guard permitted entry, and the student successfully took the folder. - **Subsequent Breach:** After the first breach prompted heightened scrutiny and badge checks, other students duplicated the visual layout of Black Hat NOC attendee badges. Two counterfeit badge holders gained entry into the NOC without prior reconnaissance, wandering into the center of the workspace before being questioned.