What Kubernetes actually solves for a data platform
Kubernetes automates the deployment, scaling and recovery of containerised workloads. For a data platform specifically, that translates into a small number of concrete benefits: running Spark or other processing jobs as containers that scale elastically with demand, running Airflow itself on a Kubernetes executor so each task gets its own isolated pod, and giving data science and ML workloads a consistent, reproducible environment that behaves the same in development and production.
The operational cost that is easy to underestimate
- Kubernetes has a genuine learning curve — networking, RBAC, resource requests and limits, and cluster upgrades are all real, ongoing operational work.
- Someone on the team needs to own cluster health, security patching and version upgrades indefinitely, not just the initial setup.
- Debugging a failed job inside Kubernetes (pod scheduling, resource limits, node pressure) adds a layer of complexity beyond debugging the job logic itself.
- Cost visibility is harder by default — a shared cluster running many workloads needs deliberate tagging and monitoring to attribute spend accurately, unlike a dedicated managed service with its own bill.
What managed services already solve without Kubernetes
Before adopting Kubernetes for a data platform, it is worth being honest about what the managed alternatives already cover. Managed Spark (EMR, Databricks, Dataproc), managed Airflow (MWAA, Cloud Composer, Astronomer) and serverless data warehouses (BigQuery, Snowflake) all handle elastic scaling and operational management without requiring a team to run Kubernetes at all. For many data platforms, these managed services deliver the genuine benefit Kubernetes offers — elastic, reliable compute — without the operational overhead of running the orchestration layer itself.
When Kubernetes genuinely earns its cost for a data platform
- Running a genuinely diverse mix of workloads (Spark, ML training, custom microservices, streaming consumers) that benefit from sharing one consistent, elastic infrastructure layer.
- An organisation that already runs Kubernetes for application workloads and has the operational expertise in place, so adding data workloads is incremental rather than a new discipline to build.
- Very specific portability requirements across multiple clouds or on-premise, where Kubernetes provides a consistent abstraction that managed, provider-specific services do not.
- Cost sensitivity at genuinely large scale, where the efficiency of bin-packing many workloads onto shared Kubernetes infrastructure outweighs the operational overhead of running it.
- Teams building and running many custom containerised data applications that do not map cleanly onto an existing managed service.
When it is the wrong choice, even though it sounds modern
- A small team with a handful of scheduled batch jobs and no existing Kubernetes expertise — managed Airflow and a managed warehouse solve this more cheaply and reliably.
- A platform still finding its data model and pipeline structure, where the operational overhead of Kubernetes distracts from the actual data work that creates business value.
- Adopting it because it appears on job postings and feels like the expected architecture, rather than because a specific requirement demands it.
A practical adoption path
Start with managed services for the core data platform — a managed warehouse, managed orchestration — and introduce Kubernetes only for the specific workloads that genuinely need its flexibility: a custom containerised application, a diverse ML training pipeline, or a Spark workload that has outgrown a fully managed option on cost or configurability grounds. This keeps most of the platform simple while giving the small number of workloads that need Kubernetes room to use it properly, rather than running the entire platform on Kubernetes from day one because it seemed like the more capable choice.
Frequently asked questions
No. Airflow can run with local, Celery or Kubernetes executors, and managed Airflow offerings (MWAA, Cloud Composer, Astronomer) handle this choice without requiring the team to operate Kubernetes directly. The Kubernetes executor is one option among several, not a requirement.
It depends on the specific requirement. Managed Spark services (EMR, Databricks, Dataproc) handle most of the same elastic scaling with far less operational overhead. Spark on Kubernetes is worth it mainly when a team already runs Kubernetes at scale and wants Spark to share that same infrastructure and operational model.
More than most teams initially estimate. Beyond initial setup, ongoing cluster upgrades, security patching, RBAC management and capacity planning are continuous responsibilities, not one-time tasks, and typically require at least a partial dedicated role once usage grows beyond a small cluster.
Rarely as the primary platform choice. A small team is usually better served by managed services that remove the orchestration burden entirely, adopting Kubernetes later only if a specific, concrete need (custom containerised workloads, multi-cloud portability) actually arises.
It reduces lock-in for the compute and orchestration layer specifically, since Kubernetes workloads are broadly portable across clouds. It does not remove lock-in from managed data services (a specific warehouse, a specific managed database) used alongside it, which remain provider-specific regardless.
Adopting it platform-wide from the start, before the actual data pipelines and modelling are mature, which adds operational complexity at exactly the stage when the team should be focused on the data itself rather than infrastructure.
How we can help
Technology selection that weighs managed services against Kubernetes honestly.
Legacy Modernisation & BI MigrationContainerised, portable deployments as part of a modernisation programme.