Lost in Translation: Exploiting Unicode Normalization

## Overview

In "Lost in Translation: Exploiting Unicode Normalization," Ryan Barnett and Isabella Barnett (also known as "Angel Hacker") explore vulnerabilities arising from Unicode handling and character transformations across multi-tier web architectures. Comparing web request pipelines to the party game "telephone," the speakers demonstrate how data traverses intermediaries—such as CDNs, firewalls, and application services—where it is transformed or decoded inconsistently.

A central takeaway is **CWE-180 (Incorrect Order of Processing)**: when security checks run before backend data manipulations take place, benign-looking inputs can mutate into active exploits post-inspection. The speakers correlated these behaviors from real-world bug bounty findings and created simulated challenge labs on the *YesWeHack Dojo* platform.

---

## Multibyte Awareness and Decoding Errors

Unicode allows multibyte variable-length encodings, whereas ASCII is a single-byte, 7-bit standard. Under **CWE-172 (Encoding Error)**, systems that lack multibyte awareness fail to parse continuation bytes properly:

- **Mojibake Output:** In UTF-8, leading bits specify character byte length (a leading `0` represents 1 byte, while four leading `1` bits denote 4 bytes, followed by `10` continuation bytes). If a decoder misinterprets leading bits or treats multibyte streams as individual single bytes, it separates linked bytes into mutated, garbled output known as *mojibake*. - **Tooling Observations:** - Standard **Burp Suite Decoder** historically lacked multibyte awareness, garbling inputs such as a multibyte left angle bracket (`<`). The speakers recommend the **Decoder Improved** extension from the BApp Store to properly decode Unicode via Java Swing elements. - **ActiveScan++ Update:** The speakers contributed updates utilizing left and right taint-tracking anchors. An input probe checks whether the Greek capital letter alpha (`Α`) mutates into a Latin capital letter `N` and Device Control 1 (`0x11`), serving as a signature that backend decoding is not multibyte-aware.

---

## Overlong Encodings and Security Logic Bypasses

Referencing **CAPEC-80**, the speakers illustrate overlong UTF-8 encodings (analogized to unnecessary nested packaging). Although forbidden in modern standards, overlong encodings forcefully represent characters using more bytes than necessary:

- **Mechanisms and Historical Context:** A single-byte ASCII character like `A` can be stretched into two or more bytes by distributing bits across leading and continuation byte formats. This technique was historically exploited by the **Nimda** worm (2001) to bypass security checks inspecting for `../` and `..\` directory traversal patterns before the web server normalized the input. - **SQL Injection Evasion:** Attackers utilize tamper scripts in tools like **sqlmap** to replace standard ASCII space characters with two-byte overlong equivalents. Tested against regex filters (e.g., matching `SELECT ... FROM` on *Regex101*), non-normalizing parsers treat overlong representations as invalid or replacement characters, causing security inspections to fail while downstream systems potentially normalize and execute the payload.

---

## Byte Truncation and Buffer Downcasting

When applications convert multibyte characters to smaller fixed-width storage buffers, byte truncation can occur:

- **Microsoft CRLF Injection Case Study:** An attacker bypassed Microsoft filters that blocked explicit carriage return and line feed (`CRLF` / `%0D%0A`) sequences. By submitting specific 3-byte Chinese characters, the attacker successfully injected a `Set-Cookie` header into the HTTP response. - **Root Cause & Little-Endian Buffering:** In a single-byte buffer under little-endian architecture, the system may extract only the least significant byte (the rightmost byte) and discard the rest. Replicating the flow in **CyberChef** using the `US-ASCII 7-bit` text encoding recipe demonstrates how multibyte characters truncate into `0x0D` and `0x0A`. - **Detection & Exploitation Tools:** - **ShaZaam / Shazer:** Developed by Gareth Hayes, this utility maps target hex values back to multibyte Unicode characters that truncate to those values. - **Template Injection Probing:** ActiveScan++ leverages byte truncation testing between anchors, submitting a Hangul symbol to see if it truncates to a left curly brace (`{`), indicating potential Server-Side Template Injection (SSTI).