Automation Techniques in C++ Reverse Engineering

## Overview

Rolf (reverse engineering specialist) presented automation techniques for dramatically accelerating static reverse engineering of large C++ binaries. Traditional manual reverse engineering of C++ programs is tedious: analysts spend 85-95% of time manually recreating type information (structures, variable types, structure layouts) in disassemblers (IDA Pro, Hex-Rays). Bare assembly code with untyped pointers is nearly incomprehensible; adding accurate type information reveals semantics and makes code resembleoriginal source. Automation research goal: systematically recover type information from runtime behavior via dynamic binary instrumentation (DBI), dynamically allocate memory tracking, and post-processing analysis. Demonstrated results: recovered ~200 structures and typed ~6000 variables from single target in days—work requiring six weeks of manual analysis. Hex-Rays automatically applied tens of thousands of structure offsets after automation. Two complementary analyses: (1) track structure access patterns to infer field locations and sizes; (2) data-flow tracking from allocation sites to function arguments to infer parameter types. Implementation leverages hooked memory allocators (malloc/free), dynamic binary instrumentation callbacks on memory accesses, and post-processing of access logs to extract struct/type information.

## Memory Allocation Tracking and Instrumentation

**Memory Allocator Hooking**: First step locates memory allocators (malloc, free, realloc) in target binary. Sources: imported from standard library (interceptable via import table overwrite) or custom allocators defined in binary (requiring prologue hooking). Hooks are transparent to application—replaces original allocator with instrumentation wrapper. Malloc hook: (1) calls original malloc to obtain pointer; (2) calls remember() function to log allocation metadata; (3) returns pointer to caller. Free hook: (1) calls forget() function to remove allocation metadata; (2) calls original free; (3) returns to caller.

**Allocation Metadata Record**: Malloc hook records allocation record containing: allocated pointer value, allocation size in bytes, relative virtual address (RVA) within image where allocation originated. RVA used instead of absolute addresses to handle ASLR (Address Space Layout Randomization) and enable aggregating results across multiple program executions. Allocation records stored in data structure optimized for range queries—binary tree (AVL trees preferred for O(log n) balanced lookup performance) allowing lookups at any address within allocation range (not just allocation base).

**Dynamic Binary Instrumentation (DBI) Memory Access Logging**: Program execution under DBI framework (e.g., Intel Pin, Frida) inserts callback routines on every memory load/store instruction. Callback receives address being accessed; looks up address in allocation metadata map. If address falls within dynamically-allocated region, callback logs access details: allocation source RVA, allocation size, instruction RVA performing access, access size (1/2/4/8 bytes), offset within allocated structure, read vs. write operation. Execution traces collect millions/billions of memory access events, generating enormous log files. All done transparently; application behavior unaffected.

## Structure Recovery via Access Pattern Analysis

**Post-Processing Access Logs**: Log file processed in multiple stages. First stage: segregate all accesses by allocation site—group all accesses to memory allocated from same code location. Within allocation site, accesses reveal structure layout: instruction at offset 0x0 accessing byte 0, instruction at offset 0x8 accessing bytes 8-15, etc. reveals offsets of structure fields.

**Structure Size and Layout Recovery**: Aggregating all accesses to allocations from same site determines structure size (largest accessed offset + access width = minimum structure size) and field offsets/sizes. Accesses reveal: which instructions access which offsets; access sizes; patterns indicating arrays, nested structures, or pointer fields. Repeated accesses to same offset indicate field usage across multiple code paths.

**Field Type Inference**: Access patterns infer field types: 8-byte sequential reads/writes suggest pointers or 64-bit integers; 4-byte accesses suggest 32-bit integers or floats; 1-byte accesses suggest character/byte fields. Pointer field detection: if field value is used as base for further dereferences or compared with other pointer values, likely pointer type. Structure nesting: if field value is used to index into region with its own access pattern, likely points to nested structure.

**Cross-Site Type Merging**: If prior knowledge indicates two allocation sites allocate same data type, accesses from both sites merged to refine structure information. Information redundancy from multiple sites enables disambiguation and improves accuracy.

**Variable and Function Argument Type Inference**: Data-flow tracking follows values from allocation sites to usage points: function arguments, return values, structure field assignments. If variable receives value allocated via malloc with inferred structure type, variable is typed accordingly. Function arguments receiving allocated values are typed from allocation structure information. This propagates structure type information through program.