← All projects

Case study. A global pharmaceutical company, 2024

Upgrading production Amazon EKS from 1.23 to 1.30

Seven sequential Kubernetes upgrades across four environments of a data platform, taking the clusters off paid extended support before AWS upgraded them automatically.

  • 7sequential version upgrades, 1.23 to 1.30
  • 4environments: production, development, test and sandbox
  • 6×the hourly cluster price on extended support ($0.60 against $0.10)
  • 42–55free IPs per control-plane subnet, against the 5 EKS needs
  • Amazon EKS
  • Kubernetes
  • AWS
  • CloudWatch Container Insights
  • IAM roles for service accounts

The problem

The company’s dbt data platform ran on Amazon EKS clusters that were still on Kubernetes 1.23. AWS ended standard support for 1.23 on 11 October 2023, and clusters that stayed on it moved to extended support, which costs $0.60 per cluster per hour instead of $0.10: an extra $0.50 an hour, or about $365 a month for every cluster.

Extended support for 1.23 itself ended on 11 October 2024. After that date AWS upgrades the control plane automatically, on its own schedule rather than the team’s, without first checking the workloads for removed APIs.

The approach

I followed the AWS EKS best-practice guide for upgrades and chose an in-place strategy: upgrade the control plane, then the add-ons, then roll the worker nodes, environment by environment.

  1. One minor version at a time. EKS only upgrades a cluster by one minor version per step, so the move from 1.23 to 1.30 was seven upgrades: 1.24, 1.25, 1.26, 1.27, 1.28, 1.29 and 1.30, with the Kubernetes and EKS release notes reviewed before each one.
  2. Backups first. Before every step, a script exported namespaces, ConfigMaps, roles, role bindings, DaemonSets, StatefulSets and Deployments, so any environment could be restored to a known state.
  3. Pre-flight checks. EKS needs up to five free IP addresses in the control-plane subnets to upgrade. I checked every subnet (each had between 42 and 55 free) and raised the dependency in the tracker before starting.
  4. Deprecated APIs. kube-no-trouble (kubent) ran inside each cluster to find resources using APIs that the next version removes, so they were migrated before the upgrade instead of failing after it.
  5. Add-ons with every version. CoreDNS, the Amazon VPC CNI, kube-proxy and the Cluster Autoscaler were moved to the versions that match each Kubernetes release.
  6. Worker nodes rolled safely. Nodes were rolled the way managed node groups do it. The runbook required PodDisruptionBudgets, so evictions respect each service’s minimum, and still set a maintenance window for each environment in case a removed API or a busy node caused trouble.
  7. Visibility afterwards. CloudWatch Container Insights was enabled on the clusters, with the CloudWatch agent and Fluentd running under IAM roles for service accounts, so node, pod and container metrics and logs were in one place.

The outcome

  • All four environments moved from Kubernetes 1.23 to 1.30, back on standard support and its $0.10 hourly price.
  • The upgrade happened on the team’s schedule, before AWS’s automatic upgrade of the control plane in October 2024.
  • Every step is written down in a runbook (backups, IP checks, kubent, add-ons, nodes), so the next upgrade does not start from scratch.

What I would do next time

Each EKS version gets 14 months of standard support. Upgrading one version at a time, a few times a year, keeps every step small and avoids extended-support charges altogether, rather than catching up seven versions at once.

Sources for the AWS dates and prices: Amazon EKS Kubernetes version lifecycle. Amazon EKS pricing. Amazon EKS extended support pricing. Amazon EKS best practices: cluster upgrades.

Talk to me about your platform