The process for migrating from Helm v2 to v3<\/a>, the latest stable major release, was pretty straightforward. However, while performing the migration, we encountered an anomaly with how one of the application charts had been deployed, thus introducing additional challenges.<\/p>\n One of our application\u2019s Helm v2 releases did not adhere to the standard naming convention when we installed it in our pre-production and later into the production environments. This had gone unnoticed (we do make mistakes!!!) <\/em>and surfaced at the time we decided to migrate from Helm v2 to v3. We have some automation capabilities in our pipeline that rely on the naming convention of the Helm release to sign off the deployment. Naturally, this started to fail for the application in scope. So our first task before we could start migrating the charts to Helm v3 was to fix the release\u00a0name.<\/p>\n A quick internet search showed us that many in the Kubernetes community had faced similar problems<\/a> and there isn\u2019t a simple solution yet<\/a>. The closest answer we got was to delete the release and install it again with the correct name. Unfortunately, this wouldn\u2019t work for us, as the service is customer-facing, serving in real-time, and we couldn\u2019t afford any downtime. Hence, we had to consider other ways to solve this\u00a0problem.<\/p>\n There were two theoretically possible solutions that would allow us to rename an existing release without causing any service disruption.<\/p>\n Let\u2019s take a closer look at the approaches:<\/p>\n Trick the datastore<\/strong><\/p>\n The first option was to modify the datastore (ie. ConfigMap in Helm v2 and Secrets in Helm v3)<\/em> that stores the resource manifest by replacing the existing (incorrect)<\/em> release name string with the desired (correct)<\/em> value. Helm v3 stores the resource manifest in a zipped, double base64 encoded secret in the namespace.<\/p>\n ## GET RELEASE INFO<\/strong>$ kubectl get secret -n <NAMESPACE><\/strong> sh.helm.release.v1.<RELEASE-NAME><\/strong>.v1 -o json | jq -r “.data.release” | base64 -D | base64 -D | gzip > release.json## REPLACE RELEASE NAME WITH DESIRED NAME & ENCODE<\/strong>DATA=`cat release.json | gzip -c | base64 | base64`## PATCH THE RELEASE<\/strong>$ kubectl patch secret -n <NAMESPACE><\/strong> sh.helm.release.v1.<RELEASE-NAME><\/strong>.v1 –type=’json’ –p=”[{“op”:”replace”,”path”:”\/data\/release”,”value”:”$DATA”}]\u201d<\/p>\n With this approach, on multiple attempts, we noticed that the decoding\/encoding was off due to the escape characters, binary data, etc., and we couldn\u2019t upgrade the release after changing the name; in another instance, we lost a release\u2019s info and had to restore from backup. The unpredictable results did not inspire confidence, so we decided to drop this approach. We would re-evaluate this as a last resort approach.<\/p>\n 2. Orphan &\u00a0Adopt<\/strong><\/p>\n The second approach that we experimented with was more deterministic and simple in a way; it didn\u2019t need us to go through the complex process of modifying the datastore. Instead, we disconnect the Kubernetes resources (orphan)<\/em> from the incorrectly named Helm release and later have the new Helm release with the correct name start managing these resources (adopt)<\/em>. Sounds simple right\u2026.voila!!!<\/p>\n Let\u2019s walk through the steps with an example. Assume that we have an incorrectly named release called \u201cworld-hello.\u201d <\/strong>We\u2019ll have to rename this to something more meaningful, such as \u201chello-world.\u201d<\/strong><\/p>\n First things first, we use Helm release names in the labelSelectors to select what backend pods the Kubernetes service (kube-proxy)<\/em> directs traffic to. Since we are renaming the release, the correctly named new release will be installed, and the Kubernetes service will immediately start proxying traffic to the new ReplicaSet pods while they are still booting. NOTE: <\/strong>Up to this point, the Kubernetes resources have been managed by the incorrectly named Helm release that we migrated from v2 to\u00a0v3.<\/p>\n ## LABELS TO BE ADDED<\/strong>app.kubernetes.io\/managed-by=Helm## ANNOTATIONS TO BE ADDED<\/strong>meta.helm.sh\/release-name=<NEW-RELEASE-NAME><\/strong> First, let’s add the release name that we initially removed back to the service\u2019s labelSelectors field. Once done, the Kubernetes service (kube-proxy) <\/em>will <\/em>start sending traffic to pods managed by the new release \u201chello-world\u201d <\/strong>only.<\/p>\n Next, delete the orphaned ReplicaSet and the incorrectly named Helm v2 and v3 releases.<\/p>\n ## ADD RELEASE LABEL<\/strong>$ git diff templates\/service.yaml Finally, redeploy the application chart one more time through the deployment pipeline you might own and verify that the upgrade went smoothly.<\/p>\n NOTE:<\/strong> Throughout this exercise, we had a traffic generator making continuous requests to the service endpoint and we didn\u2019t notice a single failure (!2XX response code), indicating a seamless and successful migration\/renaming of a Helm\u00a0release.<\/p>\n ## UPGRADE HELM v3 RELEASE WITH CORRECT NAME<\/strong>$ helm3 upgrade \u2014 install hello-world -n dev On a closing note, renaming a Helm release is not a simple task. There is a ton of prep work and experimentation that\u2019s involved. But at the same time, we got to learn interesting details on how Helm functions internally (around migrations, executions, etc)<\/em> and in the process, found out yet another way to rename a release.\u2026with documentation!<\/p>\n We hope these learnings are useful and help the community alleviate some of the problems we faced during our migration!<\/p>\n If you\u2019re interested in solving problems like these, <\/em>join our Talent Portal<\/em><\/a> to check out open roles and get periodic updates from our recruiting team!<\/em><\/p>\n How to Rename a Helm Release<\/a> was originally published in Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"Options<\/h3>\n
The service will be unavailable to our customers during this time. The application pods typically take about 20\u201330s to boot and we can\u2019t afford to have a disruption this long. To prevent this, we decided to remove the release name from the labelSelectors fields in the service\u00a0spec.Fig1. Remove the release label from the service\u2019s selector\u00a0field## REMOVE RELEASE LABEL<\/strong>$ git diff templates\/service.yaml<\/strong>app: {{ .Values.app.name }}
– release: {{ .Release.Name }}<\/strong>Next, let us follow the official steps<\/a> to migrate the release from Helm v2 to Helm v3 without correcting the name. Once done, issue an upgrade using the new client to validate that the resources are now managed by Helm v3.
The upgrade step will also add the label app.kubernetes.io\/managed-by=Helm to the resources managed by the release. Without this label on the resources, the release renaming will\u00a0fail.## MIGRATE RELEASE FROM HELM v2 TO HELM v3<\/strong>$ helm3 2to3 convert world-hello –release-versions-max 1 -n dev
<\/strong>2020\/11\/12 19:06:44 Release \u201cworld-hello\u201d will be converted from Helm v2 to Helm v3.
2020\/11\/12 19:06:44 [Helm 3] Release \u201cworld-hello\u201d will be created.
2020\/11\/12 19:06:46 [Helm 3] ReleaseVersion \u201cworld-hello.v1\u201d will be created.
2020\/11\/12 19:06:47 [Helm 3] ReleaseVersion \u201cworld-hello.v1\u201d created.
2020\/11\/12 19:06:47 [Helm 3] Release \u201cworld-hello\u201d created.
2020\/11\/12 19:06:47 Release \u201cworld-hello\u201d was converted successfully from Helm v2 to Helm v3.
2020\/11\/12 19:06:47 Note: The v2 release information still remains and should be removed to avoid conflicts with the migrated v3 release.
2020\/11\/12 19:06:47 v2 release information should only be removed using `helm 2to3` cleanup and when all releases have been migrated over## LIST HELM v3 RELEASE<\/strong>$ helm3 ls -n dev
<\/strong>NAME NAMESPACE REVISION
world-hello dev 1## UPGRADE HELM v3 RELEASE<\/strong>$ helm3 upgrade –install world-hello -n dev
<\/strong>Release \u201cworld-hello\u201d has been upgraded. Happy Helming!NAME: world-hello
LAST DEPLOYED: Thu Nov 12 20:06:02 2020
NAMESPACE: dev
STATUS: deployed
REVISION: 2
TEST SUITE: NoneNow that we\u2019ve validated that the resources can be managed by Helm v3, let\u2019s begin the process of adopting the existing resources. We need to add two annotations and a label to all the resources that need to be adopted by the new (correctly named) Helm v3 release. These annotations will indicate to Helm v3 that the new release should now start managing these resources.<\/p>\n
meta.helm.sh\/release-namespace=<NAMESPACE><\/strong>## ADD RELEASE NAME ANNOTATION<\/strong>$ for i in deploy cm sa svc role rolebinding; do kubectl annotate -n dev $i hello-world meta.helm.sh\/release-name=hello-world –overwrite; done<\/strong>deployment.extensions\/hello-world annotated
configmap\/hello-world annotated
serviceaccount\/hello-world annotated
service\/hello-world annotated
role.rbac.authorization.k8s.io\/hello-world annotated
rolebinding.rbac.authorization.k8s.io\/hello-world annotated## ADD RELEASE NAMESPACE ANNOTATION<\/strong>$ for i in deploy cm sa svc role rolebinding; do kubectl annotate -n dev $i hello-world meta.helm.sh\/release-namespace=dev –overwrite; done<\/strong>deployment.extensions\/hello-world annotated
configmap\/hello-world annotated
serviceaccount\/hello-world annotated
service\/hello-world annotated
role.rbac.authorization.k8s.io\/hello-world annotated
rolebinding.rbac.authorization.k8s.io\/hello-world annotatedOnce the annotations and labels are added to the Kubernetes resources, install the release with the correct name to sign-off on the adoption process. Once the release is upgraded, all the resources are actively managed by the correctly named release \u201chello-world.\u201d<\/strong>Because we have rolling deployments, the ReplicaSet managed by the incorrectly named release will be orphaned and hence would need to be cleaned up manually.## INSTALL HELM v3 RELEASE WITH CORRECT NAME<\/strong>$ helm3 install hello-world -n dev
<\/strong>Release \u201chello-world\u201d does not exist. Installing it now.NAME: hello-world
LAST DEPLOYED: Thu Nov 12 20:06:02 2020
NAMESPACE: dev
STATUS: deployed
REVISION: 1
TEST SUITE: None## LIST HELM v3 RELEASE<\/strong>$ helm3 ls -n dev
<\/strong>NAME NAMESPACE REVISION
world-hello dev 2
hello-world dev 1## LIST REPLICASET MANAGED BY INCORRECTLY NAMED RELEASE<\/strong>$ kubectl get rs -n dev -l release=world-hello
<\/strong>NAME DESIRED CURRENT READY AGE
hello-world-8c5959d67 2 2 2 30m## LIST REPLICASET MANAGED BY CORRECTLY NAMED RELEASE<\/strong>$ kubectl get rs -n dev -l release=hello-world
<\/strong>NAME DESIRED CURRENT READY AGE
hello-world-7f88445494 2 2 2 2mSince we also removed the release info from the service\u2019s labelSelector, the traffic is proxied toReplicaSets (pods)<\/em> managed by both the correctly named and incorrectly named releases, ie. \u201chello-world\u201d <\/strong>and \u201cworld-hello.\u201d<\/strong>Now we can start cleaning up orphaned resources and the datastore containing the incorrectly named\u00a0release.<\/p>\nCleanup<\/h3>\n
<\/strong>app: {{ .Values.app.name }}
+ release: {{ .Release.Name }}
<\/strong>## DELETE REPLICASET MANAGED BY INCORRECTLY NAMED RELEASE<\/strong>$ kubectl get rs -n dev -l release=world-hello
<\/strong>NAME DESIRED CURRENT READY AGE
hello-world-8c5959d67 2 2 2 30m$ kubectl delete rs hello-world-8c5959d67 -n dev
<\/strong>## LIST INCORRECTLY NAMED RELEASE DATASTORE (Helm v3)<\/strong>$ kubectl get secret -n dev | grep \u201csh.helm.release.v1.world-hello\u201d
<\/strong>sh.helm.release.v1.world-hello.v1
sh.helm.release.v1.world-hello.v2
## DELETE INCORRECTLY NAMED RELEASE DATASTORE (Helm v3)<\/strong>$ kubectl delete secret sh.helm.release.v1.world-hello.v1 -n dev
$ kubectl delete secret sh.helm.release.v1.world-hello.v2 -n dev
## DELETE INCORRECTLY NAMED RELEASE DATASTORE (Helm v2)<\/strong>$ helm3 2to3 cleanup –name world-hello<\/p>\n
<\/strong>Release \u201chello-world\u201d has been upgraded. Happy Helming!NAME: hello-world
LAST DEPLOYED: Thu Nov 12 20:40:06 2020
NAMESPACE: hello-world
STATUS: deployed
REVISION: 2
TEST SUITE: None<\/p>\n