Why etcd breaks at scale in Kubernetes

What started the discussion

This r/kubernetes thread focused on Why etcd breaks at scale in Kubernetes. The original post pointed readers to learnkube.com, and the comments quickly turned into a practical discussion about the tradeoffs, surprises, and production realities behind the topic.

What stood out in the comments

Discussion point 1

This is a good article, thanks. Only 64,997 nodes to go before I need it in my homelab.

Discussion point 2

Thanks for writing this. The only part that is missing is that you can create large clusters yourself without beenig Google or AWS reimplementing the etcd api. Kubernetes supports etcd sharding natively. You can use --etcd-servers-override to use different etcd instances (or clusters) to store events, pods or services separately. For really small clusters you can also use one etcd cluster for different api-servers. Just set the --etcd-prefix different for every cluster and you can share resources and minimize maintenance.

Discussion point 3

As a person that works on Kubernetes scalability, API server caching layer and etcd (with my work being cited in the article), I can only say that this article totally misses the point and brings more disinformation about scalability than information. The recent jumps in scalability of K8s offerings are the result of years of work put into fixing inefficiencies and bottlenecks in the control plane. At GKE we tested 30k node clusters on very old v3.4 etcd release, on old n2 machine generation. It worked, meaning if you use any newer hardware you should be easily beat that number. Etcd is not the bottleneck. Want to learn more about K8s scalability? Watch KubeCon talk by people that actually work on this https://youtu.be/SdLLOcNZN5E

Discussion point 4

i mean, that's not really anything to do with "slow etcd" and everything to do with zfs slowing down etcd

Discussion point 5

a useful highly available system. what’s your deal?

Thread snapshot

• Original subreddit: r/kubernetes
• Original author: u/danielepolencic
• Reddit score: 223
• Comment count: 33
• Original thread: https://learnkube.com/etcd-breaks-at-scale