When Minutes Count: Speed and Accuracy During Incident Response

When Minutes Count: Speed and Accuracy During Incident Response

**Overview**

Matt Koff, VP of Cybersecurity Engineering at ExtraHop (a network detection and response platform), discusses the operational challenges of incident response in modern distributed and cloud-centric environments. The talk, framed as a "Fast Chat" interview, addresses how security teams can balance speed and accuracy when responding to cyber incidents, particularly in scenarios where data sources are diverse, distributed, and potentially unreliable. The central theme is that incident responders need visibility into both what happened (forensic accuracy) and how to respond rapidly (operational speed), requiring multiple data sources and a clear understanding of each source's constraints. Rather than relying on a single "silver bullet" detection tool, Koff emphasizes that effective incident response requires a coordinated approach across multiple data streams—network traffic, system logs, and endpoint agents—each providing complementary perspectives on attacker behavior.

**Technical Details**

Incident Response Data Sources and Constraints:

1. **The Cyber Triad Architecture**: - Three primary data sources support incident response: network visibility, system logging, and endpoint detection/response (EDR) or next-gen antivirus - Each component provides critical but complementary information - All three must work together; no single source is sufficient for complete visibility - Defenders cannot rely on any one component to detect all attacks or provide complete situational awareness

2. **System Logs and Host-Side Visibility**: - Logs are self-reported data generated by the systems themselves - Major constraint: logs are highly unstructured and require preemptive configuration - Organizations must decide in advance what to log; decisions made before the attack occurs - If logging isn't configured to capture a specific activity, it won't be available during forensics - Attackers' first action: "cut the proverbial cord on the security camera"—disable or manipulate logging facilities - Logs can be evaded, bypassed, manipulated, or deleted by attackers with system access - Endpoint detection and response solutions also rely on self-reported data from endpoints, making them similarly vulnerable to evasion

3. **Network Traffic as Observational Ground Truth**: - Network data is observational: traffic captured at the network level exists independently of whether any endpoint or log reports it - Network visibility provides behavioral ground truth—what actually happened on the wire, not what hosts claim happened - As noted in the talk: "every compromise involves two hosts and a network between them"—network data cannot be completely avoided or eliminated by attackers - Network traffic analysis provides data difficult to obtain through traditional means (host logs, endpoint agents) - Advantages: network traffic is not self-reported; attackers cannot simply delete it or disable it without ceasing communication entirely

4. **Challenges of Modern Distributed Environments**: - Pandemic accelerated shift to remote workforce: employees now distributed across home networks instead of centralized offices - Centralized access and centralized security management become more difficult with distributed workforce - Cloud migration: many organizations moved from on-premises data centers to cloud infrastructure - In cloud environments, network visibility becomes more critical because host-level access and log collection are more difficult - Projects continue with increased velocity despite distributed workforce, increasing pressure on incident response teams

5. **Network Detection and Response (NDR) Philosophy**: - Traditional NDR focuses on automated response: quarantining hosts, blocking traffic, taking actions in the network - ExtraHop's approach: NDR as supporting incident responders with data analytics and context, not just automated responses - Response means providing incident responders with: data, context, information, behavioral insights - Enables analysts to make informed decisions about containment, remediation, and investigation - Data analytics approach: convert raw network traffic into actionable intelligence about attacker behavior - Emphasis on "transactional behavior data"—understanding what hosts communicated with, what protocols were used, what data moved

6. **Incident Response Decision Workflow**: - Incident responders need speed: time is critical during active compromise - Incident responders need accuracy: false positives waste time, missing the actual attacker wastes more - Network data provides rapid access to behavioral patterns: which hosts communicated, timing of communications, protocols used - Contextual information from network analysis informs decisions about: is this normal activity, is this an attack, what systems are affected - Responders must correlate multiple data sources: what do logs say, what do endpoints report, what does network traffic show

7. **Constraints and Tradeoffs**: - Logs require preemptive configuration: you can't log what you didn't decide to log - Endpoints rely on system cooperation: if system is compromised, endpoint data may be unreliable - Network data requires network access and visibility: must be positioned to see relevant traffic - Latency vs. scale: high-volume network analysis requires processing power; can you analyze traffic at modern network speeds - Coverage vs. detail: broader network visibility shows more connections but less protocol detail than deeper packet inspection

**Defense & Takeaways**

Improving incident response speed and accuracy through layered visibility and data-driven decision making: (1) Implement comprehensive logging at the source—configure log generation on all critical systems before attacks occur; decide what to log for your threat model; understand that post-compromise log collection may be too late; (2) Accept logging constraints—logs will be incomplete, can be manipulated, and may not capture attack indicators; treat logs as one data source, not the complete picture; (3) Deploy endpoint detection and response—endpoint agents provide host-level visibility complementary to network data; understand endpoints are also manipulable by attackers with sufficient access; (4) Prioritize network visibility—network traffic analysis provides observational ground truth that attackers cannot easily erase without stopping all communications; position monitoring to see traffic between critical systems; (5) Process network data at scale—modern networks generate massive traffic volumes; invest in infrastructure that can analyze traffic in real-time or near real-time; don't let network visibility become a bottleneck; (6) Treat incident response as a data analytics problem—convert raw network traffic and logs into behavioral intelligence; surface patterns that reveal attacker actions; (7) Enable rapid analyst decisions—provide incident responders with context and data, not just automated actions; automated quarantine may be necessary but analysts need information to decide what to quarantine; (8) Correlate across data sources—single data source provides incomplete picture; cross-reference logs, endpoint data, and network traffic to build complete incident narrative; (9) Design for remote environments—in distributed workforce scenarios, traditional log collection and EDR deployment become more difficult; network-based visibility becomes relatively more important; (10) Document data source reliability—understand which data sources can be evaded and how; logs are unreliable if logging is disabled, endpoints are unreliable if endpoint agent is compromised, network data is harder to eliminate entirely; (11) Prepare forensic data collection in advance—don't wait until incident occurs to decide what data to preserve; configuring logging and retention policies must happen before compromise; (12) Train analysts on behavioral patterns—understanding normal network behavior enables faster anomaly detection; baseline what your environment looks like, then identify deviations; (13) Implement complementary monitoring at network edges—monitor traffic leaving your network (egress filtering) as well as entering (ingress filtering); attackers must exfiltrate data, requiring some network communication; (14) Use network data for timeline reconstruction—network traffic provides timestamps for all communications; reconstruct incident timeline by analyzing when each system contacted others; (15) Balance speed vs. accuracy—incident response requires rapid action but not at cost of accuracy; data-driven approach provides speed (rapid analysis of traffic patterns) and accuracy (behavioral ground truth from network). The critical insight: In distributed environments with remote workers and cloud infrastructure, traditional host-centric incident response becomes less reliable and more difficult. System logs can be disabled or manipulated by attackers; endpoint agents can be evaded; both sources require preemptive configuration. Network traffic provides the most difficult-to-evade source of behavioral data: attackers communicating requires network activity, which can be observed and analyzed. Effective incident response requires analysts to balance multiple imperfect data sources—treating incident response as a data analytics problem where the goal is to provide responders with context and information to make rapid, accurate decisions. In modern pandemic-affected remote work environments, network visibility becomes relatively more important as host-level logging and endpoint management become more difficult. Organizations must invest in capability to ingest, process, and analyze network traffic at scale, and train incident responders to correlate signals across logs, endpoints, and network data to achieve both speed and accuracy.