Comprehensive Investigation Results on Malware Code Interpretation by GPT-4

### Overview Yasik Tanaka, an analyst focused on vulnerability and malware analysis at NTT Security, presents an empirical investigation evaluating GPT-4's capability to interpret decompiled and disassembled malware code. Because LLMs can hallucinate and previous research lacked comprehensive benchmark datasets covering typical malware behaviors, the study used Mandiant's capa rules on PE files as ground-truth labels. The evaluation assessed 35 categories and 273 inspection items across decompiled code (DC) and disassembly code (DA) using Ghidra and simple prompts.

### Key Topics - **Identifiable Behaviors in Decompiled Code (RQ1):** GPT-4 successfully identified 70% of evaluated behaviors in decompiled code. It performed well on widely known malware techniques (such as process hollowing or mutex creation), recognizable Windows API specifications, and standard cryptographic algorithms (e.g., CRC32, AES, RC4 KSA). - **Limitations in Decompiled Code (RQ2):** GPT-4 struggled with obscure techniques, multi-step operations (e.g., extra window memory injection), and behaviors where overall intent is hard to infer. Notably, it mistook a callback shellcode execution via `CopyFile2` as a benign file copy. It also misidentified API arguments passed as raw hexadecimal immediate values (e.g., mistaking AES for Triple DES in `CryptGenKey`, and misjudging process mitigation policy flags). - **Identifiable Behaviors in Disassembled Code (RQ3):** Disassembly code yielded an 8% identification rate overall. However, GPT-4 could still reliably recognize specific low-level logic, such as PEB (Process Environment Block) traversal, stack strings, x86-specific debugger detection, and distinct crypto constants. - **Improvement Pipeline:** The presenter introduced an automated enhancement strategy to resolve raw immediate values into Windows API macro names using LLM prefix prediction combined with a macro database, raising the decompiled code identification rate from 70% to 86%.

### Technical Details - **Baseline Performance:** Decompiled code achieved a 70% detection rate overall, whereas raw disassembly reached only 8%. - **Cryptographic & Hash Discrepancies:** While GPT-4 recognized RC4 KSA loops and MD5 round structures, it struggled distinguishing hashes with overlapping constants or state structures (e.g., confusing SHA-512 with SHA-256, and missing SHA-224). - **Macro Name Resolution Pipeline:** Because GPT-4 frequently predicted the correct macro prefix even if the suffix was wrong, researchers used GPT-4 to generate candidate prefixes, queried a forward macro expansion database, and calculated the correct macro constants to enrich decompiled code prior to analysis.

### Takeaways While GPT-4 provides strong contextual reasoning for documented APIs and standard algorithms, raw immediate arguments and niche multi-step injection techniques cause significant interpretation failures. Augmenting decompiled code by resolving numeric flags to macro names dramatically improves LLM interpretation accuracy (from 70% to 86%), though assembly-level techniques (such as Heaven's Gate or PEB offsets) still require specialized low-level handling.