History and Evolution of Data Lake Architecture

### Overview Presented by IT architects Masaru Dobashi and Akihiro (Takia) from NTT DATA Corporation, this session outlines the history, architectural evolution, and modern state of big data processing and data lake platforms. The talk explores how open-source software (OSS) shifted from initial distributed batch systems to SQL-on-Hadoop, columnar storage, object stores, and finally modern table formats/storage layer software (Apache Iceberg, Apache Hudi, Delta Lake) designed to bridge the gap between batch and stream processing for real-time analytics.

### Key Topics - **Hadoop Ecosystem Evolution**: Early MapReduce architectures allowed enterprises to process massive datasets across commodity hardware. To abstract complex MapReduce programming, SQL-like and domain-specific layers emerged (Hive, Pig, Cascading), alongside machine learning (Mahout) and graph frameworks (Giraph). - **Limitations of Early Hadoop**: HDFS and MapReduce struggled with high volumes of small files due to metadata overhead on master servers, as well as low-latency random reads/writes and online processing. - **Low Latency & Stream Processing**: Systems like Apache HBase introduced millisecond-level key-value lookups, while Apache Storm and Apache Kafka established scalable, near-real-time streaming pipelines. - **SQL-on-Hadoop & Columnar Formats**: Decoupling query execution from MapReduce and adopting columnar formats (Parquet, ORC) improved read efficiency via columnar projection and predicate pushdown, though conversions remained resource-intensive. - **Object Storage & Cloud Trends**: Object stores like AWS S3 and Apache Hadoop Ozone gained prominence for long-term scalability and small file handling, though early implementations faced consistency and movement overheads. - **Real-Time Analytics & Architectural Dilemmas**: Traditional architectures faced trade-offs between batch-focused pipelines (historical depth, but slow) and stream pipelines (low latency, but constrained ad-hoc query capabilities). Lambda architecture attempted to merge both but introduced high pipeline complexity and data consistency challenges.

### Technical Details The second half explores modern storage layer software operating between applications and underlying physical storage (HDFS, S3). These tools lack daemon processes and delegate raw scalability to the underlying stores while managing metadata, schema, and ACID transactions: - **Apache Iceberg**: Focuses on ACID transactional consistency and multi-format support (Parquet, ORC, Arrow). It tracks data files using snapshot-based manifest lists and manifests, enabling time travel, reproducible analytics, and predicate pruning. - **Apache Hudi**: Bridges fast ingest and analytical reading via dual file types (row-oriented append logs and columnar base files) and multiple query views: - *Read Optimized View*: Reads only columnar base files for low-latency scans. - *Real-Time View*: Merges base files and append logs on read to expose freshest updates. - *Incremental View*: Streams data change differences. - **Delta Lake**: Provides transactional guarantees, rich DML support (DELETE, UPDATE, MERGE for compliance like GDPR), and tracks file modifications via ordered JSON delta logs, which regularly compact into checkpoint files.

### Takeaways Modern storage layer technologies address traditional trade-offs by combining rapid append/write capabilities with optimized columnar formats and background file compaction. While these solutions simplify end-user query interfaces and unified batch/streaming architectures, their internal mechanisms are increasingly complex, requiring architects to carefully evaluate data access patterns and storage layout designs.