Building ML Pipelines in JupyterLab Using Elyra - Without the Need to Write Code

### Overview In this talk, Patrick Titzler, a Developer Advocate at IBM with the Center for Open Source Data & AI Technology, introduces Elyra—an open-source project extending JupyterLab with AI-centric features and curated extensions. The core focus is Elyra's visual pipeline editor, which enables data scientists and developers to build, configure, and orchestrate machine learning pipelines out of Jupyter notebooks, Python scripts, R scripts, and custom reusable components without writing glue code.

### Key Topics - **Extending JupyterLab:** Addressing limitations in vanilla JupyterLab by pre-installing valuable extensions (such as Git integration and Language Server Protocol for auto-completion and linting) alongside custom AI-oriented tools. - **Overcoming Monolithic Notebooks:** Moving from sprawling, resource-constrained local notebooks to modular workflows where processing steps (data loading, cleansing, training) can run sequentially or in parallel. - **Workflow Orchestrators:** Decoupling pipeline authoring from execution engines. Elyra compiles graphical pipelines into native workflows for Kubeflow Pipelines and Apache Airflow running on Kubernetes, as well as a local execution engine for development and testing. - **Component Reusability & Catalogs:** Moving development code into self-contained reusable custom components defined by explicit specifications and discovered via catalog connectors (such as local file systems, URLs, Apache Airflow, and the Machine Learning Exchange/MLX).

### Technical Details - **Node Configuration & Reproducibility:** Each node in an Elyra pipeline runs within a specified container image (e.g., TensorFlow 2.8) to guarantee reproducible execution over time. Users can configure node hardware constraints (such as GPU allocation), file dependencies, environment variables, mounted data volumes for large datasets, and output file declarations for downstream nodes. - **Secrets Management:** Sensitive inputs like passwords, API tokens, and user credentials can be mapped using Kubernetes Secrets, ensuring sensitive values remain concealed in pipeline definitions and editor views. - **Execution & Export:** Pipelines can be run directly from the JupyterLab UI or triggered programmatically via Elyra's CLI for CI/CD integration. Alternatively, pipelines can be exported as standalone artifacts compatible with Kubeflow Pipelines or Airflow. - **Artifact Storage:** Execution occurs remotely; logs, executed notebook snapshots, and rendered HTML output files are pushed to remote object storage, avoiding local filesystem bloat.

### Takeaways & Community Q&A - **Monitoring Delegation:** Elyra deliberately avoids implementing a native monitoring UI to focus developer resources on high-value authoring capabilities, delegating execution monitoring to Kubeflow or Airflow. - **Deployment Options:** Elyra is available via Conda Forge, PyPI, pre-built container images for Docker Desktop or JupyterHub, a custom Kubeflow notebook server image, and Red Hat's Open Data Hub. - **Testing and Security:** In the Q&A, Titzler noted that while unit testing is not built directly into the UI, pipeline execution can be validated in CI/CD using Pytest and the Elyra CLI. Regarding custom components, there is currently no built-in binary/executable signing mechanism, requiring users to rely strictly on trusted catalog sources.