Secure and Efficient Data Sharing with Federated Cloud Storage

### Overview Masashi Ochiai from NTT Lab presented a practical open-source architecture for secure and efficient inter-company data sharing using the concept of a "virtual data lake." As demand for data sharing surges—exemplified by Snowflake reporting a 93% increase in data-sharing tasks over one year—traditional approaches aggregate data into centralized physical data lakes. However, physical consolidation leads to high resource consumption from data copying and significant complexity when managing distributed copies across organizational boundaries. The virtual data lake model solves this by leaving original data stored at each company's location while aggregating only metadata and executing on-demand data transfers.

### Key Topics - **Virtual Data Lake Concept:** Moving away from physical consolidation by keeping primary data at source locations, aggregating technical and business metadata centrally, and caching accessed data. - **Use Case – Supply Chain Carbon Accounting:** Calculating greenhouse gas (GHG) emissions across complex supply chains (such as automotive manufacturing) by querying decentralized domestic and international sources without central replication. - **Separation of Planes:** Dividing architecture into a lightweight data plane and a policy-driven control plane to simplify enforcement and streamline authorization. - **Governance and Security:** Anchoring data governance on data quality, lifecycle management, metadata tracking, and fine-grained access control adhering to standards like GDPR and CCPA.

### Technical Details To realize the virtual data lake architecture, NTT Lab combined several open-source components: - **Alluxio (Data Plane & Federation):** Functions as a distributed file system providing a unified logical namespace across heterogeneous storage systems (including Amazon S3, Google Cloud Storage, and HDFS). It provides data orchestration, in-memory caching to accelerate queries (integrating with engines like Apache Spark and Presto), and on-demand data movement. - **LinkedIn DataHub (Metadata Management & PIP):** Serves as the centralized data catalog and observability platform. DataHub tracks technical metadata, data lineage, and external pipeline integrations (e.g., Apache Airflow), while serving as the Policy Information Point (PIP) supplying resource and user attributes for authorization checks. - **Open Policy Agent (OPA as PDP):** Provides declarative, fine-grained access control using policies written in Rego. Decouples authorization logic from application code, acting as the Policy Decision Point (PDP). - **OPAL (Open Policy Administration Layer):** Solves scalability bottlenecks of centralized PDPs by serving as an administration layer that automatically distributes policies and real-time metadata changes to distributed OPA agents. - **Policy Enforcement:** Access is gated via a Policy Enforcement Point (PEP) proxy in front of Alluxio, though Alluxio Enterprise Edition can integrate directly with OPA to eliminate the proxy. - **Performance Evaluation:** Evaluated on a Kubernetes cluster under a download workload. Comparing baseline performance (Alluxio proxy without authorization) against the OPA-secured pipeline demonstrated minimal performance impact alongside linear scaling capabilities.

### Takeaways NTT Lab demonstrated that enterprise-grade, inter-company data sharing can be achieved cleanly with open-source software, eliminating data duplication while preserving strict data governance. Future work includes expanding the platform to support streaming data (which Alluxio does not manage) and integrating compatibility with the European Data Spaces Protocol.