Álvaro Martín Lozano
Implementing continuous delivery in a data processing pipeline
#1about 4 minutes
From research concepts to production-ready data products
The Volkswagen Data Lab shifted its focus from demonstrating proof-of-concepts to building and deploying real-world data solutions for its clients.
#2about 7 minutes
Core concepts of continuous delivery for data
Continuous delivery for data pipelines requires adapting standard CI/CD principles, where data is the deliverable, by progressing through version control, integration, and deployment stages.
#3about 11 minutes
Implementing a pipeline with immutable, versioned data
The five-step pipeline relies on treating data as immutable, creating a new versioned output for each run to enable simple rollbacks and reproducibility.
#4about 6 minutes
The challenge of orchestrating chained data jobs
Managing dependencies between jobs becomes complex when each job consumes versioned, immutable data inputs from upstream processes.
#5about 5 minutes
Pros and cons of the immutable data approach
While this method offers powerful benefits like reproducibility and instant rollbacks, it introduces challenges in orchestration complexity and increased storage costs.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
00:36 MIN
The distinct roles of CI and CD pipelines
#90DaysOfDevOps - The DevOps Learning Journey
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
06:04 MIN
Using continuous delivery to enable business agility
The Affordances of Quality
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
01:29 MIN
Overview of the data and machine learning tech stack
Empowering Retail Through Applied Machine Learning
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
09:45 MIN
Implementing an AI-in-the-loop continuous learning cycle
A solution to embed container technologies into automotive environments
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
04:09 MIN
Automating the data pipeline with multi-cloud services
Leverage Cloud Computing Benefits with Serverless Multi-Cloud ML
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
00:51 MIN
Defining continuous integration, delivery, and deployment
CI/CD with Github Actions
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
02:13 MIN
Using AI to optimize CI/CD pipelines
Navigating the AI Wave in DevOps
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
03:47 MIN
Applying software engineering practices to data pipelines
Enjoying SQL data pipelines with dbt
Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Upgrade to PRO for unlimited access to the full archive.
You have an account? Log in
Featured Partners
Related Videos
30:46Charting the Journey to Continuous Deployment with a Value Stream Map
Josh Armitage
43:47Industrializing your Data Science capabilities
Dubravko Dolic & Hüdaverdi Cakir
56:19Plan CI/CD on the Enterprise level!
Pawel Piwosz
44:40DevOps for Machine Learning
Hauke Brammer
25:24Integrating backups into your GitOps Pipeline
Florian Trieloff
41:06From Monolith Tinkering to Modern Software Development
Lars Gentsch
54:46Navigating the AI Wave in DevOps
Raz Cohen
46:06CD2CF - Continuous Deployment to Cloud Foundry
Dominik Kress
Related Articles
View all articles
From learning to earning
Jobs that call for the skills explored in this talk.
Clickhouse
Remote
Senior
Pandas
Grafana
PySpark
Apache Kafka
AARON FORMACIÓN Y CONSULTORÍA
Municipality of Madrid, Spain
Remote
€38-40K
Intermediate