{"id":268,"date":"2021-08-31T14:40:46","date_gmt":"2021-08-31T14:40:46","guid":{"rendered":"https:\/\/fde.cat\/?p=268"},"modified":"2021-08-31T14:40:46","modified_gmt":"2021-08-31T14:40:46","slug":"zero-downtime-node-patching-in-a-kubernetes-cluster","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/zero-downtime-node-patching-in-a-kubernetes-cluster\/","title":{"rendered":"Zero Downtime Node Patching in a Kubernetes Cluster"},"content":{"rendered":"

Authors: <\/em>Vaishnavi Galgali<\/em><\/a>, <\/em>Arpeet Kale<\/em><\/a>, <\/em>Robert\u00a0Xue<\/em><\/a><\/p>\n

Introduction<\/h3>\n

The Salesforce Einstein Vision and Language services are deployed in an AWS Elastic Kubernetes Service (EKS) cluster. One of the primary security and compliance requirements is operating system patching. The cluster nodes that the services are deployed on need to have regular operating system updates. Operating system patching mitigates vulnerabilities that may expose the virtual machines to\u00a0attacks.<\/p>\n

\"\"<\/figure>\n

Patching Process<\/h4>\n

Einstein services are deployed as Kubernetes pods on an immutable EC2 node group, also known as an AWS AutoScaling Group (ASG). The patching process involves building a new Amazon Machine Image (AMI) that contains all of the updated security patches. The new AMI is used to update the node group, which involves launching new EC2 instances, one at a time. As the new instance passes all the health checks, one of the old instances is terminated. This process continues until all of the EC2 instances in the node group are replaced. This is also known as a rolling update.<\/p>\n

However, this patching process introduces a challenge. As the old EC2 instances are terminated, the service pod running on those EC2 instances is also terminated. This may lead to failures for any user requests being processed at the time of termination unless the termination of the pod is handled gracefully. Graceful termination of the pod involves infrastructure components (the Kubernetes API and AWS ASGs) and application components (service\/app container).<\/p>\n

Graceful Application Termination<\/h3>\n

In this process, the application is first gracefully terminated. Terminating a pod may result in abruptly terminating the Docker container in the pod. This implies that the process running in the Docker container is also abruptly terminated. This may cause any requests being processed to be terminated, eventually leading to failures for any upstream service calling the application at that time.<\/p>\n

When an EC2 instance is being terminated as part of the patching process, the pods on that instance are evicted. This marks the pods for termination and the kubelet running on that EC2 instance commences the pod shutdown process. As part of the pod shutdown, kubelet issues a SIGTERM signal. If the application running in the pod isn\u2019t configured to handle the SIGTERM signal, it may result in abruptly terminating any running tasks. Therefore, you want to update your application to handle this signal and gracefully shut down.<\/p>\n

For example, in the case of a Java application, here\u2019s one way to address graceful termination (this differs from framework to framework):<\/p>\n

public static final int gracefulShutdownTimeoutSeconds = 30;<\/pre>\n
@Override
public void onApplicationEvent(@NotNull ContextClosedEvent contextClosedEvent) {
    this.connector.pause();
    Executor executor = this.connector.getProtocolHandler().getExecutor();
    if (executor instanceof ThreadPoolExecutor) {
    try {
        ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
        threadPoolExecutor.shutdown();<\/strong>
        logger.warn(\"Gracefully shutdown the service.\");
        if (!threadPoolExecutor.awaitTermination(gracefulShutdownTimeoutSeconds, TimeUnit.SECONDS)) {
        logger.warn(\"Forcefully shutdown the service after {} seconds.\", gracefulShutdownTimeoutSeconds);
        threadPoolExecutor.shutdownNow();
        }
    } catch (InterruptedException ex) {
        Thread.currentThread().interrupt();
    }
    }
}<\/pre>\n
In the above snippet, the shutdown is initiated and, after 30 seconds, the application is forcefully terminated. This gives the application 30 seconds to process any running tasks.<\/p>\n
If the pod consists of multiple containers, and the order of container termination matters, then define a container preStop hook<\/a> to ensure that the containers are terminated in the correct sequence (for example, terminating an application container before terminating a logging sidecar container).<\/p>\n
During the process of pod shutdown, the kubelet follows container lifecycle hooks, if defined. In our case, we have multiple containers in the same pod and so, for us, the order of termination matters. We define the preStop hook for our application containers as shown\u00a0below:<\/p>\n
lifecycle:
    preStop:
        exec:
          command:
          - \/bin\/sh
          - -c
          - kill -SIGTERM 1 && while ps -p 1 > \/dev\/null; do sleep 1; done;<\/pre>\n
The action defined in the preStop hook above sends a SIGTERM signal to the process running in the Docker container (PID 1) and waits in intervals of 1 second until the process is successfully terminated. This allows the process to complete any pending tasks and terminate gracefully. <\/p>\n
The default timeout of the preStop hook is 30 seconds, which. in our case, gives enough time for the process to terminate gracefully. If the default time isn\u2019t sufficient, you can specify it by using the terminationGracePeriodSeconds field in the preStop\u00a0hook.<\/p>\n
Graceful EC2 Instance Termination<\/h3>\n
As mentioned above, our services run on node groups of EC2 instances. Graceful EC2 instance termination can be achieved by using AWS ASG lifecycle hooks and an AWS lambda\u00a0service.<\/p>\n
AWS EC2 Auto Scaling Lifecycle Hooks<\/h4>\n
Lifecycle hooks help in pausing the instance state and performing custom actions before launching the new instance or terminating the old instance. Once the instance is paused, you can complete the lifecycle action by triggering a Lambda function or running commands on the instance. The instance remains in the wait state until the lifecycle action is completed.<\/p>\n
We use the Terminating:Wait <\/strong>lifecycle hook to put the instance to be terminated in the WAIT <\/strong>state. For more details on ASG lifecycle hooks, see the AWS\u00a0docs<\/a>.<\/p>\n
AWS Lambda<\/h4>\n
We use SAM<\/a> framework to deploy a Lambda function (built in-house; we call it node-drainer) that is triggered on specific ASG lifecycle hook events. The following diagram shows the sequence of events involved in gracefully terminating an EC2 instance in the node\u00a0group.<\/p>\n
\"\"<\/figure>\n