"Democratizing AI - Back-fitting end-to-end machine learning" by Joel Young and Bo Long

### Overview Joel Young (lead of the Machine Inference Infrastructure team) and Bo Long (lead of the Machine Learning Algorithms team) present LinkedIn's journey to completely rethink and modernize their machine learning infrastructure through the "Productive ML" initiative. As AI powers core LinkedIn products—such as People You May Know, feed ranking, job recommendations, LinkedIn Learning, Sales Navigator, Recruiter, and internal systems like anti-abuse and build failure routing—prior custom-built stacks led to substantial technical debt and "second system syndrome." With machine learning democratizing across engineering disciplines, LinkedIn needed an extensible architecture to eliminate friction, prevent ivory-tower silos, and more than double modeler productivity by accelerating experimentation iteration speed while keeping serving costs low.

### Key Topics - **Democratization of ML**: Supporting engineers across product and UI teams who have practical ML skills, moving away from reliance on dedicated ML specialists. - **Extensible Architecture**: Avoiding lock-in to single frameworks (TensorFlow, PyTorch) by structuring the system across modular layers: experimentation/authoring, training clusters, packaging, and production serving. - **Feature Marketplace**: A centralized hub enabling discoverability, metadata tracking, feature selection, and online/offline feature consistency. - **Modeling as a Directed Acyclic Graph (DAG)**: Representing ML models as DAGs consisting of feature transformations, deep embeddings, and trainable scoring components. - **Model Training Ecosystem**: Utilizing deep learning (TensorFlow), boosted trees (XGBoost), and generalized linear mixed models via the open-sourced Photon ML library. - **Health Assurance & Monitoring**: Continuous offline/online validation, automated drift tracking, anomaly detection on deployments, and threshold-triggered retraining. - **People and Organization**: Implementing a "team of teams" organizational model, establishing the AI Academy (including training for managers), and deliberately managing incremental adoption to avoid the "support trap."

### Technical Details - **Quasar DSL**: Rather than using graphical drag-and-drop tools, LinkedIn developed Quasar (Quick Ass Scoring and Ranking), an in-house declarative domain-specific language that handles complex vector operations, cosine similarity, tree-based scoring, and first-order list operations (top-k, filtering, diversity). - **Photon ML & Mixed Models**: LinkedIn employs generalized linear mixed models (GLMM) via Photon ML to scale personalized recommendations. To manage large-scale user/item interactions across 500 million members and 10 million jobs, training combines a global model (e.g., XGBoost trees over deep neural embeddings) with localized per-user and per-job linear models. - **Serving Architecture**: Instead of computing brute-force pairwise interactions (500M members × 10M jobs), serving breaks down by context: per-member coefficients are retrieved by a mid-tier service from an online key-value store, while per-job coefficients live in sharded forward indices on search nodes. The Quasar inference engine executes scoring directly on searchers using columnar modes and vector batching for real-time responsiveness. - **Frame Platform**: Enforces parity between offline training data and online inference signals through unified libraries and shared configurations, preventing data discrepancies. - **Auto-tuning**: Uses Gaussian process-based methods to automate hyperparameter tuning over vast feature and parameter spaces.

### Takeaways - Speeding up model iteration coupled with early feedback directly drives improvements in product metrics. - Modeling abstractions should separate semantic feature definitions from storage/runtime anchors to support multi-environment deployment. - Early lifecycle validation significantly reduces production maintenance overhead; proactive anomaly detection and automated retraining outperform arbitrary fixed schedules (e.g., weekly cron jobs). - To deploy large platform updates successfully, organizations should roll out components incrementally (pre-alpha to GA) to protect engineering teams from the "support trap" of overwhelming maintenance costs.