BalCCon2k14 - Ivica Kolenkas

### Overview At BalCCon2k14, Ivica Kolenkas presented an abstract architecture designed to manage unstructured digital data using MongoDB and GridFS, introducing an academic storage system called the "V Drive". The motivation stemmed from modern educational institutions needing scalable, highly available, and disaster-resilient systems for sharing student coursework and digital library assets. Kolenkas highlighted that legacy approaches—such as traditional relational database systems or flat file-system repositories—fail to scale cost-effectively and often lack built-in geographic redundancy, citing past digital library losses from floods in India, a tornado in Texas, and earthquakes in Indonesia.

### Key Topics - **Structured vs. Unstructured Data**: Citing a 2012 study showing that over 80% of data in small and medium enterprises is unstructured (such as PDFs, movies, and emails), Kolenkas discussed the need for databases that handle schema-free binary and document formats without expensive vertical scaling. - **MongoDB Architecture**: An overview of MongoDB's document-oriented BSON format, dynamic collections, replica sets, and horizontal sharding. - **GridFS File Storage**: The specification within MongoDB for storing files exceeding the standard 16MB BSON limit by splitting files into 256KB chunks (`fs.chunks`) paired with metadata records (`fs.files`). - **The V Drive Architecture**: A plug-and-play solution providing a school-wide internal cloud system where students can upload assignments (target allocation: 1GB per student).

### Technical Details - **Base System & Deployment**: Built on a custom minimal CentOS distribution deployed automatically using kickstart files. It automatically partitions storage into boot, root, and a dynamically expandable `/var` partition dedicated to MongoDB data. - **Application & API Layer**: Uses Python with the asynchronous Tornado web framework to expose a RESTful HTTP API (supporting GET, POST, PUT, DELETE) exchanging JSON payloads. Document schemas and validation rules are defined in Python using the MongoEngine ODM library. - **High Availability & Replication**: A 3-node replica set was demonstrated live. Nodes monitor health via 2-second heartbeats. If the primary does not respond within 10 seconds, an automated election is held where majority quorum elects a new primary based on priority and the latest write timestamp. Data replication runs continuously with approximately 100ms latency, and nodes can be placed in distant geographical locations for disaster recovery. - **Test Environment & Benchmarks**: Testing was conducted using two Fujitsu RX 300 servers configured with VMware virtual machines acting as separate data centers, alongside older legacy Celeron test machines. Evaluating file uploads of 5, 25, 50, and 100 megabytes across local networks, the internet, and a throttled 2–4 Mbps VPN link demonstrated that MongoDB and GridFS streaming produced negligible overhead compared to manual file copying.

### Takeaways - Horizontal scaling with MongoDB replica sets and sharded clusters provides transparent failover and disaster recovery without proprietary, expensive enterprise storage hardware. - Replication must not be mistaken for a backup system, as deletions replicate across nodes; explicit backup routines (such as MongoDB export/dump utilities via cron) are required. - MongoDB with GridFS offers an open-source, cost-effective framework for educational institutions and small-to-medium organizations to handle growing unstructured data repositories reliably.