GeeCON 2018: Jarosław Pałka - new java.io.File(`geecon.json`); is this really that simple?

## Overview

Jarosław Pałka presents a comprehensive analysis of Java file I/O operations, deconstructing the hidden complexity behind what appears to be simple file read/write code. The talk traces execution from high-level Java APIs through JNI layers to underlying operating system calls, exposing multiple buffer copies, memory allocations, and performance pitfalls.

## Key Topics

**File vs File Descriptor**: The File class in Java is not an OS file descriptor but a wrapper for path manipulation. The actual file descriptor is represented by FileDescriptor, which stores an integer representing the OS-level file descriptor. FileDescriptor implements reference counting to track multiple streams using the same descriptor; when the count reaches zero, the descriptor can be garbage collected.

**FileInputStream/FileOutputStream Implementation**: Standard streams maintain a hard-coded 8KB buffer in native code. Writes exceeding 8KB trigger additional memory allocation on the C heap. The write path involves multiple copies: Java buffer → JNI buffer → C buffer → page cache → storage. This design protects against garbage collection by copying data to stable C-allocated memory, preventing the JNI code from accessing memory that could be moved by GC.

**FileChannel as Superior Alternative**: FileChannel maintains an internal cache of buffers at multiple sizes, avoiding redundant allocation on each operation. The configurable JDK NIO max cache buffer size parameter (e.g., 128KB) allows tuning. FileChannel's stack trace shows most operations occur in Java with only one native call at the end, compared to multiple native calls in the standard stream approach. Buffer caches are thread-local and evicted via least-recently-used algorithm.

**Vectored I/O**: FileChannel supports scatter-gather operations with read/write methods accepting multiple byte buffers. Java arbitrarily limits this to 16 buffers, despite modern operating systems supporting 1,024+ buffers. This limitation stems from legacy code checking for a fixed maximum value.

**Memory-Mapped Files**: Enable file access as a byte array with random access capabilities. The operating system handles paging, allowing access to files larger than physical memory. Critical limitation: Java's ByteBuffer uses integer indexing, restricting each mapped region to 2GB maximum. Workarounds include using Unsafe to call the private native method and access the pointer, or creating multiple ByteBuffers over the same file. Memory-mapped files provide unpredictable performance due to OS-controlled flush timing similar to garbage collection pauses. Used effectively in Cassandra and Neo4j for indexes and data storage.

## Technical Details

**Linux I/O Stack**: File operations traverse the virtual file subsystem, which queues requests for drivers. The page cache mechanism avoids redundant storage reads by caching file portions in memory. The I/O scheduler merges sequential read requests and reorders them to minimize disk seek time, particularly critical for spinning drives. Direct I/O (O_DIRECT flag) bypasses the page cache, useful for databases implementing their own caching strategies.

**Zero-Copy Optimization**: The sendfile system call transfers file data directly to network sockets without copying to application memory. Java's FileChannel.transferTo and transferFrom methods expose this capability. This technique is extensively used in Apache Kafka because the log format matches network packet format exactly, allowing direct disk writes from network packets.

**Direct I/O Limitations**: While Linux supports O_DIRECT, Java lacks official API support. The sun.nio package provides unofficial access to direct I/O, but this is a private, unsupported API with no guarantee of stability. This creates a barrier to fully utilizing database-appropriate I/O modes.

## Takeaways

- For writes exceeding 8KB, prefer FileChannel over FileInputStream/FileOutputStream - Configure BufferedInputStream/BufferedOutputStream buffer sizes appropriately; default 8KB is often insufficient - Use FileChannel with vectored I/O for multi-buffer operations, accounting for the 16-buffer limitation - Memory-mapped files excel in database and index storage scenarios but are limited to 2GB per mapping - Direct I/O is feasible through sun.nio package APIs for applications requiring data integrity guarantees - Understanding the OS I/O stack (page cache, scheduler, system calls) enables informed optimization decisions - Simple file operations involve substantial hidden complexity with multiple buffer copies and system call overhead