In a Platform Roadmap Workshop, you’ll work with SUE to determine which platform is right for you and which applications should be containerized.
Kubernetes: the complete guide for IT leaders and architects
Kubernetes is the standard platform for automatically deploying, scaling, and recovering containerized applications on any infrastructure. It’s particularly useful for multiple applications that are frequently deployed and need to scale independently. There are dozens of distributions, ranging from the managed services offered by AWS, Azure, and Google to OpenShift, Rancher, and Talos Linux. Not every workload belongs in a container: for some applications, a virtual machine remains the better choice—and these can now also run on Kubernetes. The biggest challenges lie in management, security, upgrades, and costs.
Table of contents
- What is Kubernetes?
- When is Kubernetes worth it, and when isn't it?
- Virtual machine or container: When should you choose one over the other?
- A Comparison of Kubernetes Distributions and Platforms
- How do you choose the right platform?
- Architectural Choices
- Security and compliance
- Upgrades and Lifecycle
- Observability and GitOps
- How much does Kubernetes cost?
- From Virtual Machines to Containers: A Step-by-Step Guide
- Preventing cluster sprawl and Platform Engineering
- What knowledge do you need?
- Frequently asked questions
The five decisions that matter
-
To use Kubernetes or not? Only if you have multiple applications that change frequently and need to scale independently. → Section 2
-
Which workloads should be placed in a container, and which should remain in a VM? → Section 3
-
Which platform? That depends on where you operate, your compliance requirements, and your team’s expertise. → Sections 4 and 5
-
Manage it yourself or outsource it? Plan on at least two to three experienced engineers, and more for 24/7 support. → Section 13
-
How do you keep it manageable? With a platform team, established standards, and an upgrade process. → Sections 8, 10, and 12
1. What is Kubernetes?
Kubernetes (K8s) is open-source software that manages containers across a group of servers. You describe the desired state—for example, “run three instances of this application”—and Kubernetes maintains that state: it starts containers, distributes them across servers, restarts them in case of failure, and scales up during periods of high traffic. The project is part of the Cloud Native Computing Foundation (CNCF) and releases approximately three new versions per year.
Key concepts
-
Pod: the smallest unit, consisting of one or more containers that operate together.
-
Deployment: how many pods of an application are running, and how updates are deployed.
-
Service and Ingress (or Gateway API): how applications are reached.
-
Namespace: a partition within a cluster, such as one for each team.
-
Control plane: the part that manages the cluster, consisting of the API server and the etcd database.
-
Nodes: the servers on which the pods run.
-
Distribution: a Kubernetes package that includes installation, default configurations, tooling, and support, such as OpenShift, RKE2, or Talos Linux.
2. When is Kubernetes worth it, and when isn't it?
- you're running multiple applications or microservices that scale and deploy independently;
- you want to deploy frequently and automatically;
- you don't want to be tied to a single cloud or hosting provider;
- your development teams want to deploy securely on their own using an internal platform;
- you want to consolidate your virtual machines and containers onto a single platform, for example, to move away from VMware.
It usually isn't worth it if:
- you have one or a few simple applications with a stable load;
- you don't have the capacity or budget for management, whether in-house or outsourced;
- A simpler container service will suffice, such as AWS App Runner, Azure Container Apps, or a similar service from a European provider.
3. Virtual machine or container: which should you choose and when?
A container shares the operating system kernel with other containers and contains only the application and whatever it needs. This makes containers lightweight, quick to start, and easy to scale. A virtual machine (VM) has its own operating system and kernel, which makes it heavier but also more isolated and suitable for software that requires a full operating system.
The third approach: VMs on Kubernetes. With KubeVirt, you can run virtual machines as regular objects on a Kubernetes cluster, alongside your containers. Commercial options include Red Hat OpenShift Virtualization and SUSE Virtualization (based on Harvester). Sidero Labs is adding its own hypervisor to Talos Linux, with general availability scheduled for December 2026. This allows you to manage VMs and containers using the same tools, policies, and GitOps processes, and enables you to phase out your legacy virtualization platform before every application fits into a container.
Rule of thumb: containerize what you’re actively developing, run what’s stable or what a vendor requires as a VM, and bring both onto a single platform if you want to simplify management. If you need containers with the isolation of a VM, consider sandboxed runtimes such as Kata Containers or gVisor.
Container or virtual machine?
| Choose a container if | Choose a virtual machine if |
|---|---|
| the application is stateless or stores its data in a database or object storage | the application requires a specific or outdated operating system, such as older versions of Windows Server |
| you deploy frequently and want to scale automatically | the vendor only supports a VM or appliance |
| the application is actively maintained by your own development team | licenses are tied to VMs, cores, or hosts, as is the case in some Oracle and SAP environments |
| you want to break the application down into parts or modernize it | the application requires its own kernel modules, drivers, or special hardware access |
| you want to run the same application in multiple locations | you want to strictly separate software from different, untrusted parties |
| you want to manage capacity efficiently | Rewriting a large, stable monolith costs more than it yields |
4. A comparison of Kubernetes distributions and platforms
The core of Kubernetes is the same everywhere: the well-known distributions are certified by the CNCF as compliant. The differences lie in the installation, default settings, management tools, the underlying operating system, support, and the license.
Managed Kubernetes in the public cloud
| Platform | Features | Best for |
|---|---|---|
| Amazon EKS | Control plane managed by AWS; 14 months of support per version by default, with a paid extension available up to 26 months | Organizations that primarily run on AWS |
| Azure AKS | Strong integration with Entra ID; the Premium tier includes an LTS option lasting approximately two years | Microsoft-oriented organizations |
| Google GKE | Highly automated, with Autopilot for fully managed nodes; extended channel up to approximately 24 months | Teams that want maximum automation |
| European clouds | Managed Kubernetes from providers such as STACKIT, OVHcloud, IONOS, and Scaleway, partly based on Gardener | Data sovereignty requirements |
Enterprise platforms for on-premises and hybrid environments
| Platform | Features | Best for |
|---|---|---|
| Red Hat OpenShift | A comprehensive platform featuring developer tools, a registry, security, virtualization, and support; runs on RHEL CoreOS. OKD is the community edition | Regulated and hybrid environments, VMware exit |
| SUSE Rancher Prime with RKE2 | Centralized management of multiple clusters on any infrastructure; RKE2 is hardened (CIS) by default and suitable for FIPS requirements | Multicloud, on-premises, and edge |
| VMware vSphere Kubernetes Service (VKS) | Kubernetes integrated into VMware Cloud Foundation, managed from vCenter | Organizations that remain on VMware |
| Canonical Kubernetes | A Kubernetes distribution from the creator of Ubuntu; LTS versions receive security updates for up to 12 years through Ubuntu Pro | Environments that must remain on a single version for a long time |
Lightweight distributions for edge computing and small clusters
| Platform | Features | Best for |
|---|---|---|
| k3s (SUSE) | A single binary, low memory usage, quick to install; for high availability, use etcd or an external database | Edge, IoT, Testing and Development Clusters |
| k0s (Mirantis) | A single binary with no dependencies on the host OS; a CNCF sandbox project since 2025 | Edge and easily automatable clusters |
| MicroK8s (Canonical) | Installation via Snap, many add-ons | Workstations, edge computing, and small-scale environments |
Immutable Operating Systems for Kubernetes
Talos Linux from Sidero Labs is an operating system designed solely to run Kubernetes. There is no SSH, no shell, and no package manager: you manage each node via an API secured with mutual TLS, using declarative configuration in line with GitOps. The system is minimal and read-only, with CIS-hardened default settings, SELinux, secure boot, and one-step upgrades for both the operating system and Kubernetes—which you can roll back. Talos is open source (MPL-2.0). Omni is the accompanying management environment for cluster provisioning, upgrades, and backups, available as SaaS or self-hosted. In September 2026, Sidero Labs was acquired by Yardi Systems, which is allowing it to continue as an independent company.
Talos is a good fit for organizations that want to manage many identical clusters—whether on-premises or at the edge—and minimize their attack surface. It does, however, require a different way of working: troubleshooting is done via the API and tools, without logging in to the server. Similar immutable operating systems include Flatcar Container Linux and AWS Bottlerocket.
Kubernetes-as-a-Service Platforms
| Platform | Features | Best for |
|---|---|---|
| Kubermatic Kubernetes Platform | An open-source platform from Germany for managing hundreds of clusters on any infrastructure | Large organizations and service providers |
| Gardener | Originally from SAP, now part of NeoNephos at the Linux Foundation Europe; the foundation for STACKIT, among others | European and Sovereign Kubernetes Services |
| Multistax (SUE) | A ready-to-use, fully configured, and managed platform for software teams | Teams without in-house platform expertise |
Self-installed (vanilla)
With kubeadm, you install upstream Kubernetes without any add-ons. This provides maximum flexibility and no licensing costs, but you have to set everything up yourself and bear the full burden of management. This is only practical for teams with extensive in-house platform expertise.
5. How do you choose the right platform?
There is no single Kubernetes platform that is best for everyone. The well-known distributions are all certified by the CNCF, so the difference lies in where you run it and what your team is capable of.
Ask these questions in this order
-
Where are your workloads running, now and in three years? Mostly in a single public cloud: start with EKS, AKS, or GKE. On-premises or hybrid: OpenShift, Rancher, or Talos. If you’re running on VMware Cloud Foundation and plan to stay there for the time being: VKS.
-
What compliance and sovereignty requirements apply? If you need to keep data within the EU or want to reduce your dependence on non-European providers, consider European clouds, on-premises solutions, or a sovereign version of a hyperscaler. The four options are listed side by side on the sovereign cloud page.
-
Do you also need VMs on the platform? In that case, OpenShift Virtualization and SUSE Virtualization are options, and Talos will be an option once its own hypervisor is generally available (scheduled for December 2026).
-
What can your team do? A platform with many built-in features (OpenShift) requires less custom configuration; a minimal platform (Talos, kubeadm) requires more platform knowledge.
-
How many clusters and locations? Many clusters or edge locations require centralized management, for example, using Rancher, Omni, Kubermatic, or OpenShift Advanced Cluster Management.
-
What is your budget for licenses and management? Licenses are visible; management is often the largest expense.
Which platform is right for your situation?
| Situation | Often a good choice |
|---|---|
| Especially AWS, Azure, or Google Cloud | EKS, AKS, or GKE |
| Regulated, hybrid, with many built-in features desired | Red Hat OpenShift |
| Many clusters on different infrastructure | SUSE Rancher Prime with RKE2 |
| Bare metal or edge, minimal and highly secure | Talos Linux with Omni |
| Small edge locations | k3s or k0s |
| Stay on VMware for now | VMware VKS |
| Data must remain in Europe | European cloud, on-premises, or sovereign cloud |
6. Architectural choices
Choices you make at the beginning are difficult to reverse later on. So choose carefully, and keep it as simple as your requirements allow.
The most important decisions
-
One cluster or several? In any case, separate production from non-production. Additional clusters per team or application provide isolation, but also require more management. Namespaces with good policies are often sufficient.
-
Availability: three control-plane nodes, and nodes distributed across multiple zones or racks. Multiple regions only if required for business continuity.
-
Network: Choose a network plugin (CNI) that supports network policies, such as Cilium. Use the Gateway API for new ingress configurations.
-
Storage: Stateful workloads require a reliable CSI driver and a backup solution, such as Velero or Kasten. Consider using managed databases outside the cluster.
-
Operating system: A general-purpose Linux distribution (RHEL, SLES, Ubuntu) offers familiar management; an immutable OS (Talos, RHEL CoreOS, Flatcar) results in less variation and a smaller attack surface.
7. Security and compliance
- Access via central identities (SSO), with roles based on the principle of least privilege (RBAC).
- Network policies: deny all traffic by default; allow only what is needed.
- Scan images for vulnerabilities, sign them, and allow them only from trusted registries.
- Enforce policies using Kyverno or OPA Gatekeeper: no root containers, mandatory resource limits, Pod Security Standards.
- Store secrets in a secrets manager, not in code or configuration files.
- Audit logging is enabled and stored centrally.
- The control plane is not publicly accessible.
- A standardized process for upgrades and patches, including those for the nodes' operating systems.
A Cloud Security Health Check shows you how your own clusters are performing.
NIS2 and the Cybersecurity Act. The Cybersecurity Act, which implements NIS2, has been in effect in the Netherlands since 15 August 2026. If your organization is subject to this law, you must, among other things, demonstrate that you patch vulnerabilities in a timely manner, control access, detect incidents, and assess your suppliers. For a Kubernetes platform, this means: a well-maintained upgrade process, image scanning, audit logging, and formal agreements with your management partner. Have your legal counsel or compliance officer review how these requirements apply to your organization.
8. Upgrades and lifecycle
Kubernetes releases a new version approximately every four months. Each version receives about 14 months of upstream support. So plan for three minor upgrades per cluster per year, which can be consolidated into one or two maintenance windows, including testing of applications and add-ons. If you delay upgrades, you’ll end up making larger and riskier jumps later on, because you can only move up one version at a time.
| Provider | Support by Version |
|---|---|
| Upstream Kubernetes | About 14 months |
| Amazon EKS | 14 months of standard support, plus 12 months of paid extended support |
| Azure AKS | Approximately 12 months, plus one year of LTS in the Premium tier |
| Google GKE | Approximately 14 months, with the extended channel up to approximately 24 months |
| Canonical Kubernetes LTS | Up to 12 years of security updates via Ubuntu Pro |
| Red Hat OpenShift | Its own lifecycle per minor version, with Extended Update Support (EUS) for even-numbered minor versions |
Extended support provides some breathing room, but it doesn’t solve the problem: even then, you’ll eventually have to catch up on multiple versions at once. Therefore, automate upgrades as much as possible, test them first on a non-production cluster, and check in advance which APIs are being deprecated.
9. Observability and GitOps
- Observability: centrally collecting metrics, logs, and traces—for example, using Prometheus, Grafana, and OpenTelemetry, or Elastic or Dynatrace. Without observability, troubleshooting in Kubernetes is a guessing game.
- GitOps: The entire configuration of clusters and applications is stored in Git, and a tool like Argo CD or Flux automatically brings the cluster into that state. Changes become traceable, repeatable, and reversible. With an API-driven OS like Talos, this also applies to the nodes themselves.
10. How much does Kubernetes cost?
- Infrastructure: nodes, storage, network, and load balancers, plus the control plane for managed services.
- Licenses and support: for example, OpenShift, Rancher Prime, Talos Enterprise Linux, or extended support from a cloud provider.
- Management: the engineers who build, upgrade, secure, and monitor the platform 24/7. Often the largest expense.
You can save money primarily by making realistic resource requests, using autoscaling, reducing the number of clusters, and allocating costs by team—for example, with OpenCost. Learn more in the guide on reducing cloud costs.
11. From virtual machines to containers: a step-by-step guide
Not every application needs to be containerized right away. Start with something simple, and for now, let the rest run as VMs on the same platform.
In six steps
-
Take inventory of all your applications and categorize them using the rule of thumb from Section 3: containerize them, run them as VMs, or phase them out.
-
Start with stateless web applications and APIs. Complex stateful systems will come later.
-
Containerize: create an image, extract configuration and secrets from the code, and send logging to stdout.
-
Test behavior and performance in a test cluster.
-
Migrate in phases, for example, by having a temporary period during which the old and new systems run side by side.
-
Migrate the remaining VMs to Kubernetes virtualization if you want to phase out your old virtualization platform. You can then modernize on an application-by-application basis.
12. Preventing cluster sprawl and Platform Engineering
Cluster sprawl occurs when teams create clusters independently of one another: higher costs, different versions, inconsistent security, and a lack of visibility. Here’s what helps:
- a single platform team that delivers standardized clusters based on fixed templates and GitOps;
- centralized management of multiple clusters, using Rancher, Omni, Kubermatic, or OpenShift Advanced Cluster Management;
- The rule of thumb is that a new cluster should only be created for a compelling reason (compliance, isolation, location). Otherwise, a namespace is sufficient.
When multiple teams are working on Kubernetes, an Internal Developer Platform is worthwhile: a self-service layer that allows teams without in-depth Kubernetes knowledge to deploy securely, using predefined templates, pipelines, and guardrails. The platform team treats that platform as an internal product, with the development teams as its customers. Learn more in the guide on Platform Engineering.
13. What knowledge do you need?
To manage production clusters in-house, you’ll need at least two to three experienced platform engineers—and more for 24/7 coverage. They must be proficient in Linux, networking, storage, security, and automation. Relevant certifications include CKA (administration), CKAD (development), and CKS (security). Many organizations manage their applications in-house and outsource the underlying platform. To learn how to make that decision, read the guide on outsourcing IT management.
14. Frequently asked questions about Kubernetes
All frequently asked questionsIn phases, starting with the simplest applications. For each application, you decide whether to place it in a container, run it as a virtual machine, or phase it out. Next, you move on to stateless web applications and APIs. You extract configuration and secrets from the code, test in a separate cluster, and let the old and new systems run side by side for a while before switching over. Applications that don’t yet fit into a container can run as VMs on the same platform using KubeVirt. The complete step-by-step plan is in Section 11.
Start with your cloud provider’s managed Kubernetes service: Amazon EKS, Azure AKS, or Google GKE. The provider manages the control plane, and all you need to do is set up the nodes and applications. With GKE Autopilot or EKS Auto Mode, the provider also manages the nodes. To learn Kubernetes on your own laptop or a test server, k3s or MicroK8s will suffice. Installing it yourself using kubeadm or a minimal platform like Talos Linux requires more platform knowledge and is better suited for an experienced team.
- primarily in a single public cloud: EKS, AKS, or GKE;
- regulated and hybrid: Red Hat OpenShift;
- many clusters on different infrastructure: SUSE Rancher Prime with RKE2;
- on-premises or at the edge with many identical nodes and a minimal OS: Talos Linux with Omni;
- data that must remain in Europe: a European cloud provider or on-premises.
EKS, AKS, and GKE all scale automatically—both the number of pods and the number of nodes. The difference lies in how much you configure yourself. With GKE Autopilot and EKS Auto Mode, the provider fully manages the nodes. With AKS and standard EKS, you choose the node pools and the autoscaler yourself—often Karpenter for EKS. In practice, realistic resource requests and well-configured autoscalers have a greater impact on how smoothly a cluster scales than the choice of provider.
OpenShift is a comprehensive enterprise platform with extensive built-in functionality and support from Red Hat, and it is well-suited for regulated and hybrid environments. Rancher excels at centrally managing multiple clusters across different infrastructures, and it is lighter and more flexible. Existing expertise, requirements, and budget are usually the deciding factors.
Which Kubernetes platform is right for you?
Which Kubernetes platform is right for you?
In a Platform Roadmap Workshop, you’ll work with SUE to determine which platform is right for you and which applications should be containerized.
* required