(Or: How I Learned to Stop Worrying and Love the Control Plane)
Kubernetes works great.
Then you add more teams.
Then more clusters.
Then more CRDs.
Then everything catches fire — very politely, and with excellent YAML.
This article is a survival manual.
Not a tutorial.
Not a vendor pitch.
Just a list of things I wished I knew before working with K8s in Hana Cloud teams.
Kubernetes can handle thousands of nodes.
What it cannot handle is you treating it like:
Every Pod, Job, CRD, and ConfigMap is an object. Objects live in etcd. etcd has feelings, and they are fragile.
Rule of thumb:
If your architecture diagram looks like “just add more Pods,” you are already in danger.
The control plane is not magic. It is a small number of processes trying very hard not to disappoint you.
Every controller you add:
At scale, this becomes:
“A polite but relentless DDoS you built yourself.”
Stability beats cleverness. Always.
At first, one cluster feels elegant. Later, it feels like putting every organ in one jar.
Eventually:
Multi-cluster isn’t advanced. It’s acceptance.
Design for:
Kubernetes is a control plane.
That’s it. That’s the sentence.
If your developers must understand:
…then congratulations, you’ve successfully outsourced your platform team to Stack Overflow.
Expose intent, not plumbing. Your users are humans, not controllers.
Autoscaling feels amazing right up until it:
CPU is a terrible proxy for “user pain.” Reactive scaling is always late. Unbounded scaling is just a slow outage.
Autoscaling needs:
Networking problems do not scream.
They whisper. They degrade. They wait for peak traffic.
Then everything times out simultaneously.
Be careful with:
Networking is not configuration. It is capacity planning with consequences.
The scheduler is smart. Physics is smarter.
At scale you will get:
Perfect bin-packing is impossible. Accept this. Design for interruption. Make peace with preemption.
Metrics are great. Logs are great. Traces are great.
All of them at once, at full resolution, forever — not great.
High cardinality will bankrupt you. Dashboards will lie to you. Alerts will be ignored.
At scale:
More data does not mean more understanding. Ask any Mars rover.
If governance slows people down, they will go around it. If it blocks them, they will remove it. If it surprises them, they will hate you.
Good governance feels like:
Defaults beat policies. Early failure beats runtime enforcement. Clarity beats cleverness.
Deployments generate load. They contend for resources. They fail in exciting new ways.
At scale:
Your cluster doesn’t care whether load comes from users or CI. Neither should you.
Kubernetes hides cost exceptionally well. This is both impressive and alarming.
Requests, limits, replicas, and autoscaling are economic decisions. If engineers cannot see cost, they cannot optimize it.
Efficiency is not a finance problem. It is a platform design problem.
Kubernetes will scale your systems faster than your teams.
Without:
…your platform team becomes mission control during a meteor shower.
Your platform is a product. Your users are engineers. Design like it matters — because it does.
Kubernetes at scale is not about YAML. It is not about tools. It is not about clever operators.
It is about:
If you do it right, Kubernetes becomes boring.
And boring is success.