Balancing Safety and Speed in Kubernetes Rolling Updates
Optimizing deployments in Kubernetes is not merely about making things faster; it is about engineering a trade-off between safety margins and rollout velocity. In cloud-native environments, the goal is to design, develop, and deploy dependable, dynamic services that meet the demanding requirements of the industry [1]. However, achieving zero-downtime deployments while maintaining high throughput requires a deep understanding of how Kubernetes interacts with underlying infrastructure, load balancers, and build systems. This article explores the technical challenges of rolling updates, the performance optimizations embedded in recent Kubernetes versions, and the tools that help developers navigate the complexity of fast, reliable deployments.
The Challenge of Rolling Updates
A rolling update is designed to update a service's replicas without interrupting service availability. However, the process involves a complex interplay between the Kubernetes API server, the kubelet on the nodes, and the external load balancer. A critical challenge arises during the termination phase of an old pod.
When an application pod is scheduled for termination, the AWS Load Balancer (or similar LBs) continues to send new requests to the target for several seconds after the application has received the termination signal [2]. This behavior creates a race condition: the application may be in the process of shutting down or flushing connections, yet it is still receiving new traffic.
Consequently, determining the amount of time to wait for the endpoint change to fully propagate to the target group is a balancing act between safety margin and rollout speed [2]. If the wait is too short, you risk dropping in-flight requests. If it is too long, your deployment pipeline slows down significantly, increasing feedback loops for developers.
Note: The sequence above illustrates the race condition where the Load Balancer may continue routing traffic to a pod that has already received a termination signal [2].
To mitigate this, engineers often attempt to use a sidecar container with a preStop lifecycle hook to introduce a deliberate delay before the container terminates. However, this approach has limitations. Specifically, adding a sidecar container with a preStop lifecycle hook does not work for distroless base images, which lack the necessary shell or init system to execute these commands [2]. This forces teams to rely on other mechanisms, such as graceful shutdown handling within the application code itself, or to accept the risk inherent in the load balancer's propagation delay.
Kubernetes Performance and Scalability
The ability to perform rapid rolling updates is underpinned by the core performance of the Kubernetes cluster. Since Kubernetes 1.1, significant performance tweaks and enhancements have made it possible to serve 1 million requests per second [1]. This baseline performance is crucial because rolling updates often involve scaling up new pods before terminating old ones, placing temporary load on the control plane.
In Kubernetes 1.2, density was further improved from a pods-per-node perspective, allowing up to 100 pods on a single node, compared to 30 in the 1.1 release [3]. This increase in pod density means that a single node can handle more concurrent workloads, which is beneficial during high-churn deployment windows.
Two specific optimizations in Kubernetes 1.2 are particularly relevant to the stability of rolling updates:
- PLEG (Pod Lifecycle Event Generator) Optimization: The PLEG component is responsible for polling the state of containers. In previous versions, this polling could be expensive. The introduction of concurrent polling elimination in Kubernetes 1.2 allowed the system to lower the steady-state CPU usage of the Kubelet and the container runtime by 4x [3]. Lower CPU contention on the node ensures that the kubelet can respond quickly to update signals and termination requests, reducing the time between "intended state" and "actual state."
- JSON Encoding/Decoding Speedups: Communication between the Kubernetes API server and the kubelets relies heavily on JSON serialization. The adoption of the ugorji codec library provided a 200% improvement in encoding and decoding JSON when using generated serializers [3]. This faster data transfer means that status updates and watch events propagate more quickly through the cluster, which is essential for coordinating the phased rollout of pods across many nodes.
| Component | Optimization in K8s 1.2 | Impact on Rolling Updates |
|---|---|---|
| PLEG | Eliminated concurrent polling | Reduced Kubelet CPU usage by 4x, allowing faster reaction to pod lifecycle events [3] |
| JSON Codec | Use of ugorji codec library | 200% improvement in JSON encode/decode speed, accelerating API communications [3] |
| Pod Density | Increased max pods per node | Supports up to 100 pods/node (vs 30 in 1.1), allowing higher churn rates on single nodes [3] |
The Role of AWS Load Balancers
As discussed in the challenges of rolling updates, the interaction between Kubernetes and cloud-native load balancers is a primary source of deployment friction. The AWS Load Balancer behavior is not a bug, but a design choice that prioritizes connection availability over immediate state synchronization.
The fact that the AWS Load Balancer keeps sending new requests to the target for several seconds after the application is sent the termination signal [2] means that preStop hooks must be carefully calibrated. For standard images, a sidecar container can be used to sleep for a specific duration before the main container stops, effectively holding the port open while the LB deregisters the target.
However, this solution is not universal. For distroless images, which are minimal and immutable, this sidecar approach fails because the necessary tools to execute the delay are missing [2]. In these cases, the application must implement its own graceful shutdown logic, such as:
- Stopping accepting new requests.
- Waiting for active requests to complete.
- Exiting the process.
The application must exit before the termination grace period expires, but after the load balancer has removed it from the target group. Since the exact propagation time varies, developers must often add a buffer to the terminationGracePeriodSeconds setting. This buffer is the "balancing act" between safety and speed [2].
Tilt: A Faster Way to Update Kubernetes
While optimizing the cluster and load balancer behavior reduces downtime, it does not necessarily speed up the feedback loop for developers. Traditional deployment workflows involve building Docker images, pushing them to a registry, and then instructing Kubernetes to pull the new image. This process can take minutes, even for small changes.
Tilt addresses this by making Kubernetes updates fast—really fast. Instead of minutes, Tilt updates Kubernetes in seconds [4]. This is achieved through two key mechanisms:
1. Incremental Builds
In a standard Docker workflow, creating a new layer with the generated/compiled artifacts happens for every update [4]. Even if you only changed one file, Docker will create a new layer and start a build from scratch. Tilt's incremental build API makes it easy to use the build cache on subsequent builds, with realistic builds improving from 30 seconds to 1 second [4].
2. File Syncing via Synclet
Rather than always replacing the entire pod, Tilt can reuse existing pods. A "Synclet" runs on the same node as the pod. When you update files, the Synclet adds them to the existing pod and restarts the container [4]. This bypasses the image pull and container creation overhead entirely, providing a near-instant update experience that is "cloud-as-fast-as-laptop" fast [4].
| Approach | Mechanism | Typical Update Time | Use Case |
|---|---|---|---|
| Traditional Docker | New layer, full build, image pull | Minutes | Production releases, full integration tests |
| Tilt | Incremental build, file sync to pod | Seconds | Development, rapid iteration, local testing [4] |
Note: Tilt reuses existing pods by adding updated files to the existing pod and restarting the container, which significantly reduces overhead compared to standard image updates [4].
Best Practices for Rolling Updates
To balance safety and speed effectively, developers should adopt a multi-layered strategy:
- Understand LB Propagation Delays: Acknowledge that AWS Load Balancers (and similar services) continue routing traffic for several seconds after termination signals [2]. Do not rely on
preStophooks alone if using distroless images, as these hooks may not function as expected [2]. - Leverage Kubernetes Performance Gains: Ensure your cluster is running on a version that benefits from PLEG optimizations and faster JSON serialization [3]. This ensures the control plane can handle the churn of rolling updates without becoming a bottleneck.
- Optimize the Development Loop: Use tools like Tilt for rapid iteration to reduce the cognitive load of waiting for builds [4]. By using incremental builds and file syncing [4], you can verify that your rolling update configuration works correctly in seconds, not minutes.
- Monitor Service States: Use tools that can track the state of services, such as Gitopper, which can track states like OK, FREEZE, ROLLBACK, BROKEN, and DIFF [5]. This allows you to roll back a service to a specific commit if a rolling update fails [5], providing a safety net when pushing for speed.
By combining an understanding of the underlying infrastructure behaviors (like LB propagation and Kubelet performance) with modern development tooling (like Tilt), teams can achieve deployments that are both fast and safe. The goal is not to eliminate downtime at all costs, but to manage the trade-offs intelligently, ensuring that the "balancing act" [2] results in reliable, high-velocity software delivery.
References
- One million requests per second: Dependable and dynamic distributed systems at scale | Kubernetes — blog.kubernetes.io (11 November 2015)
- Zero-Downtime Kubernetes Deployments on AWS with EKS | Glasskube — glasskube.dev (10 March 2025)
- 1000 nodes and beyond: updates to Kubernetes performance and scalability in 1.2 | Kubernetes — blog.kubernetes.io (28 March 2016)
- How Tilt updates Kubernetes in Seconds, not Minutes — medium.com (7 January 2019)
- miekg/gitopper — github.com (27 August 2024)