Mario Valderrama

Operating etcd for Managed Kubernetes

To achieve zero-downtime migrations, we modified etcd snapshots to artificially inflate the revision number. Discover the surprising challenges of operating etcd at massive scale.

Operating etcd for Managed Kubernetes
#1about 3 minutes

The journey to managed Kubernetes at IONOS

From its first release in 2019 to managing over 20,000 clusters, IONOS scaled its Kubernetes service by building on a massive etcd foundation.

#2about 4 minutes

Evolving etcd deployment strategies over time

The team progressed from the CoreOS operator and Bitnami Helm charts to a simplified custom Helm chart for better control and stability.

#3about 3 minutes

Understanding multi-tenancy and its performance impact

Using a shared etcd with client-side prefixes reduces cost but creates noisy neighbor problems, requiring careful tuning like compaction and defragmentation.

#4about 3 minutes

Iterating on etcd cluster layouts for reliability

Initial cross-location clusters suffered from latency and revision drift, leading to a more stable single data center layout using availability zones.

#5about 3 minutes

A zero-downtime control plane migration strategy

A live migration process using `etcdctl mirror` allows moving a Kubernetes control plane to a new etcd cluster without global downtime or data loss.

#6about 3 minutes

Manipulating etcd revisions for seamless migration

By modifying an etcd snapshot to insert a high revision number, clients like kubelet continue watching for changes without needing a restart after migration.

#7about 2 minutes

Future plans for etcd management and automation

The team is working on automating the migration process, offering dedicated etcd clusters, and contributing their migration learnings to the Kaji project.

Related jobs
Jobs that call for the skills explored in this talk.

Remote

Name of

Name of

Remote
Intermediate
PHP
Java
+1

Matching moments

Understanding the challenges of scaling Kubernetes with confidence
02:45 MIN

Understanding the challenges of scaling Kubernetes with confidence

5 steps for running a Kubernetes environment at scale

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Managing containers at scale with Kubernetes
00:57 MIN

Managing containers at scale with Kubernetes

#90DaysOfDevOps - The DevOps Learning Journey

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Managing a complex mix of old and new infrastructure
05:25 MIN

Managing a complex mix of old and new infrastructure

Hosting a modern justice system

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Addressing unique data protection challenges in Kubernetes
03:37 MIN

Addressing unique data protection challenges in Kubernetes

It's all about the Data

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Building end-to-end AI solutions in European data centers
01:14 MIN

Building end-to-end AI solutions in European data centers

From foundation model to hosted AI solution in minutes

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Why teams move from monoliths to Kubernetes
02:26 MIN

Why teams move from monoliths to Kubernetes

Get ready for operations by pull requests

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Using Kubernetes as an extensible control plane
00:50 MIN

Using Kubernetes as an extensible control plane

Chaos in Containers - Unleashing Resilience

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.
Comparing managed DBaaS with databases on Kubernetes
03:27 MIN

Comparing managed DBaaS with databases on Kubernetes

Databases on Kubernetes: Why you should care

Unlock Moments
Create a free account to watch a limited number of Moments each month.
Upgrade to PRO for unlimited access to the full archive.

Featured Partners

Related Videos

See all videos
Chaos in Containers - Unleashing Resilience27:52

Chaos in Containers - Unleashing Resilience

Maish Saidel-Keesing

Kubernetes Maestro: Dive Deep into Custom Resources to Unleash Next-Level Orchestration Power!29:21

Kubernetes Maestro: Dive Deep into Custom Resources to Unleash Next-Level Orchestration Power!

Um e Habiba

Mastering Kubernetes – Beginner Edition57:24

Mastering Kubernetes – Beginner Edition

Hannes Norbert Göring

Kubernetes Security - Challenge and Opportunity42:45

Kubernetes Security - Challenge and Opportunity

Marc Nimmerrichter

How I saved 200K/yr in direct costs writing 0 code lines in K8s19:36

How I saved 200K/yr in direct costs writing 0 code lines in K8s

Igor Khokhriakov

It's all about the Data30:35

It's all about the Data

Michael Cade

What we Learned from Reading 100+ Kubernetes Post-Mortems28:36

What we Learned from Reading 100+ Kubernetes Post-Mortems

Noaa Barki

Databases on Kubernetes48:45

Databases on Kubernetes

Denis Souza Rosa

Related Articles

View all articles
Learning Kubernetes made easy with KubeCampus
Learning to use Kubernetes? KubeCampus by Kasten offers free educational content for all skill levels to get you started!Kubernetes is an open-source system for deploying, scaling and managing containerized applications. It allows you to deploy your ...
Learning Kubernetes made easy with KubeCampus
BB
Benedikt Bischof
MLOps – What’s the deal behind it?
Welcome to this issue of the WeAreDevelopers Live Talk series. This article recaps an interesting talk by Nico Axtmann who introduced us to MLOpsAbout the speaker:Nico Axtmann is a seasoned machine learning veteran. Starting back in 2014 he observed ...
MLOps – What’s the deal behind it?

From learning to earning

Jobs that call for the skills explored in this talk.