{"id":279,"date":"2021-08-31T14:40:23","date_gmt":"2021-08-31T14:40:23","guid":{"rendered":"https:\/\/fde.cat\/?p=279"},"modified":"2021-08-31T14:40:23","modified_gmt":"2021-08-31T14:40:23","slug":"notary-a-certificate-lifecycle-management-controller-for-kubernetes","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/notary-a-certificate-lifecycle-management-controller-for-kubernetes\/","title":{"rendered":"Notary: A Certificate Lifecycle Management Controller for Kubernetes"},"content":{"rendered":"<p><em>Authors: <\/em><a href=\"https:\/\/salesforce.quip.com\/BLRAEA8utRZ\"><em>Vaishnavi Galgali<\/em><\/a><em>, <\/em><a href=\"https:\/\/salesforce.quip.com\/QNNAEAHBiUA\"><em>Savithru Lokanath<\/em><\/a><em>, <\/em><a href=\"https:\/\/salesforce.quip.com\/HKSAEAahHK3\"><em>Arpeet\u00a0Kale<\/em><\/a><\/p>\n<h3>Introduction<\/h3>\n<p>All services in the Einstein Vision and Language Platform use TLS\/SSL certificates to encrypt communication between microservices. The certificates are generated in <a href=\"https:\/\/docs.aws.amazon.com\/acm\/latest\/userguide\/acm-overview.html\">AWS Certificate Manager (ACM)<\/a> and stored in the <a href=\"https:\/\/docs.aws.amazon.com\/secretsmanager\/latest\/apireference\/Welcome.html\">AWS Secrets Manager<\/a> in the form of keystores and truststores (private and public keys). Certificate creation can be a manual process, especially if the permission model dictates that a certificate can only be provisioned or de-provisioned by an AWS account admin.<\/p>\n<p>Certificate management includes provisioning certificates and monitoring their expiration and renewal. AWS ACM provides the ability to auto-renew only public certs attached to AWS native resources (such as ELB and RDS). Einstein Vision and Language Platform services run on an EKS cluster with service exposure scoped to the cluster network, which means that the ACM certificates aren\u2019t attached to AWS native resources. Hence, the certificate auto-renewal feature provided by AWS ACM can\u2019t be used.<\/p>\n<p>To automate certificate provisioning and management and to provide a self-service model for carrying out these activities, we built a custom Kubernetes controller by introducing Kubernetes native resources: certificate, keystore, and truststore. Additionally, we built automation for checking certificate expiration and alert on expiration of these certificates. We call our solution\u00a0Notary.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*CRHpQutEh5LaUzv-Us-X8g.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><\/figure>\n<h3>Architecture<\/h3>\n<p>Notary is built on the core Kubernetes concepts:<\/p>\n<ul>\n<li>Controllers<\/li>\n<li>Custom Resource Definition (CRD)<\/li>\n<\/ul>\n<h3>What is a Kubernetes controller?<\/h3>\n<p>In Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state. A controller tracks at least one Kubernetes resource type. These objects have a spec field that represents the desired state. The controller for that resource is responsible for performing actions to achieve the desired state. For example, the <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/workloads\/controllers\/job\/\">job controller<\/a>, which is a built-in Kubernetes controller, helps to execute tasks in Kubernetes cluster by deploying pods as dictated by the pod spec. <\/p>\n<p>The Kubernetes API can be extended to create application-specific controllers to create, configure, and manage instances of complex, stateful applications on behalf of a Kubernetes user. The custom Kubernetes controller builds upon the Kubernetes resource and controller concepts, and includes domain or application-specific knowledge to automate common\u00a0tasks.<\/p>\n<h3>What is a\u00a0CRD?<\/h3>\n<p>CRD stands for Custom Resource Definition.<\/p>\n<p>A resource is an endpoint in the <a href=\"https:\/\/kubernetes.io\/docs\/reference\/using-api\/api-overview\/\">Kubernetes API<\/a> that stores a collection of <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/overview\/working-with-objects\/kubernetes-objects\/\">API objects<\/a> of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.<\/p>\n<p>A custom resource is an extension of the Kubernetes API that provides functionality that isn\u2019t available in a default Kubernetes installation. A custom resource represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.<\/p>\n<p>A custom resource can appear and disappear in a running cluster through dynamic registration. And cluster admins can update a custom resource independently of the cluster itself. After a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like\u00a0pods.<\/p>\n<h3>Notary<\/h3>\n<p>Notary is an extension to the Kubernetes API to satisfy an application-specific use case\u200a\u2014\u200aencryption of traffic between services running on Kubernetes cluster\u200a\u2014\u200aby managing the certificates and their lifecycle.<\/p>\n<p>Notary is a custom Kubernetes controller that watches on these custom resources: certificate, keystore, and truststore. The controller watches for create and update events, and takes defined action based on those events.<\/p>\n<p>Notary interacts with AWS Certificate Manager to request and download certificates to create keystores and truststores. After it creates keystores and truststores, Notary initiates a session with AWS Secrets Manager to upload\u00a0them.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*9sBjMZSbJ-M7_SrpLgq5bA.jpeg?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><figcaption>Notary Architecture<\/figcaption><\/figure>\n<p>The architecture diagram above depicts the functionality of Notary. The Notary controller constantly watches the custom resources\u200a\u2014\u200acertificate, keystore, and truststore\u2014for a create, update, or delete\u00a0event.<\/p>\n<ul>\n<li><strong>Certificate: <\/strong>When the create event for the certificate resource occurs, the Notary controller requests a new certificate from AWS Certificate Manager.<\/li>\n<li><strong>Keystore: <\/strong>When the create event for the keystore resource occurs, the Notary controller downloads the service\u2019s certificate bundle from AWS Certificate Manager, creates a keystore, and then uploads the keystore to AWS Secrets\u00a0Manager.<\/li>\n<li><strong>Truststore: <\/strong>When the create event for the truststore resource occurs, the Notary controller downloads the service\u2019s certificate bundle along with upstream and downstream services\u2019 certificates from AWS Certificate Manager. It then builds a truststore which includes the service certificate chain along with upstream and downstream service certificates, and uploads the truststore to AWS Secrets\u00a0Manager.<\/li>\n<\/ul>\n<h3>Components<\/h3>\n<p>Notary is built using Kubernetes objects. Let\u2019s take a look at the components of\u00a0Notary.<\/p>\n<h4>Deployment<\/h4>\n<p>Since we\u2019re dealing with sensitive data\u200a\u2014\u200acertificate, keystore, and truststore\u2014for services, it\u2019s important to secure the Notary pods. To address that, Notary is deployed in its own namespace where the namespace resources can only be accessed by cluster administrators.<\/p>\n<h4>RBAC<\/h4>\n<p>RBAC, role-based access control, is a way to manage access to resources based on roles. Notary needs access to the CRD objects certificate, keystore, and truststore which can be deployed in any service-specific namespace. Since Notary needs to watch all namespaces it needs ClusterRole privileges.<\/p>\n<p> These examples use ClusterRole and ClusterRoleBinding to give Notary access to specific resources and assign a certain level of access to resources.<\/p>\n<pre>---<br>apiVersion: rbac.authorization.k8s.io\/v1<br>kind: ClusterRole<br>metadata:<br>  labels:<br>    operator: notary<br>  name: notary<br>rules:<br>- apiGroups:<br>  - \"\"<br>  resources:<br>  - namespaces<br>  verbs:<br>  - get<br>  - list<br>- apiGroups:<br>  - infra.einstein.ai<br>  resources:<br>  - certificates<br>  - keystores<br>  - truststores<br>  verbs:<br>  - get<br>  - list<br>  - watch<br>- apiGroups:<br>  - events.k8s.io<br>  - \"\"<br>  resources:<br>  - events<br>  verbs:<br>  - create<\/pre>\n<pre>apiVersion: rbac.authorization.k8s.io\/v1<br>kind: ClusterRoleBinding<br>metadata:<br>  name: notary<br>roleRef:<br>  apiGroup: rbac.authorization.k8s.io<br>  kind: ClusterRole<br>  name: notary<br>subjects:<br>- kind: ServiceAccount<br>  name: notary<br>  namespace: notary<\/pre>\n<h4>IAM Roles<\/h4>\n<p>Notary needs access to AWS Certificate Manager to request new certificates and access to AWS Secrets Manager to upload keystores and truststores. We use AWS IAM (Identity and Access Management) roles to grant access to AWS services required for Notary. The following JSON is an example of an IAM\u00a0role.<\/p>\n<pre>{<br>    \"Version\": \"2012-10-17\",<br>    \"Statement\": [<br>        {<br>            \"Action\": [<br>                \"acm:RequestCertificate\",<br>                \"acm:GetCertificate\",<br>                \"acm:ListCertificates\",<br>                \"acm:ExportCertificate\",<br>                \"acm:UpdateCertificateOptions\",<br>                \"acm:AddTagsToCertificate\",<br>                \"acm:ListTagsForCertificate\",<br>                \"acm:DescribeCertificate\",<br>                \"acm:ResendValidationEmail\",<br>                \"acm:RemoveTagsFromCertificate\",<br>                \"acm-pca:GetCertificate\",<br>                \"acm-pca:IssueCertificate\",<br>                \"acm-pca:ListCertificateAuthorities\"<br>            ],<br>            \"Resource\": \"*\",<br>            \"Effect\": \"Allow\"<br>        },<br>        {<br>            \"Action\": [<br>                \"secretsmanager:UntagResource\",<br>                \"secretsmanager:DescribeSecret\",<br>                \"secretsmanager:PutSecretValue\",<br>                \"secretsmanager:GetSecretValue\",<br>                \"secretsmanager:CreateSecret\",<br>                \"secretsmanager:DeleteSecret\",<br>                \"secretsmanager:ListSecretVersionIds\",<br>                \"secretsmanager:GetRandomPassword\",<br>                \"secretsmanager:GetSecretValue\",<br>                \"secretsmanager:RestoreSecret\",<br>                \"secretsmanager:RotateSecret\",<br>                \"secretsmanager:CancelRotateSecret\",<br>                \"secretsmanager:UpdateSecret\",<br>                \"secretsmanager:GetResourcePolicy\",<br>                \"secretsmanager:UpdateSecretVersionStage\",<br>                \"secretsmanager:ListSecrets\",<br>                \"secretsmanager:TagResource\"<br>            ],<br>            \"Resource\": \"*\",<br>            \"Effect\": \"Allow\"<br>        }<br>    ]<br>}<\/pre>\n<h3>CRD<\/h3>\n<h4>Certificate CRD<\/h4>\n<p>This custom resource defines the metadata required to request a certificate from AWS Certificate Manager. The metadata includes the certificate FQDN and type (public or private) of certificate.<\/p>\n<pre># CUSTOM-RESOURCE-DEFINITION MANIFEST<br>---<br>apiVersion: apiextensions.k8s.io\/v1beta1<br>kind: CustomResourceDefinition<br>metadata:<br>  name: certificates.infra.einstein.ai<br>  labels:<br>    operator: {{ .Values.app.name }}<br>spec:<br>  scope: Namespaced<br>  group: infra.einstein.ai<br>  version: v1alpha1<br>  names:<br>    kind: Certificate<br>    singular: certificate<br>    plural: certificates<br>    shortNames:<br>    - cert<br>  additionalPrinterColumns:<br>    - name: Type<br>      type: string<br>      JSONPath: .spec.type<br>      description: Type of certificate<br>    - name: FQDN<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.fqdn<br>      description: FQDN of the certificate<br>    - name: AltName<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.alt<br>      description: Alternate names of the certificate<br>    - name: ARN<br>      type: string<br>      priority: 0<br>      JSONPath: .status.arn<br>      description: ARN of the certificate<br>    - name: Tags<br>      type: string<br>      priority: 0<br>      JSONPath: .status.tags<br>      description: Tags on the certificate<br>  validation:<br>    openAPIV3Schema:<br>      properties:<br>        spec:<br>          properties:<br>            type:<br>              type: string<br>              description: \"AWS Certificate Manager PEM certificate issued by either a Private of Public Root CA.\"<br>              enum: [\"Public\", \"Private\"]<br>            fqdn:<br>              type: string<br>              description: \"FQDN of the certificate\"<br>            alt:<br>              type: array<br>              description: \"Alternate name(s) for the certifcate\"<br>          required: [\"type\", \"fqdn\"]<\/pre>\n<p>When the certificate CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager for a new certificate.<\/p>\n<p> Required parameters<\/p>\n<ul>\n<li><strong>type:<\/strong> Type of certificate, either public or\u00a0private<\/li>\n<li><strong>fqdn:<\/strong> Fully qualified domain name of the\u00a0service<\/li>\n<\/ul>\n<h4>Keystore CRD<\/h4>\n<p>This custom resource defines the keystore resource. The resource spec accepts metadata to generate a new keystore for a particular service and then uploads the keystore to AWS Secrets\u00a0Manager.<\/p>\n<pre># CUSTOM-RESOURCE-DEFINITION MANIFEST<br>---<br>apiVersion: apiextensions.k8s.io\/v1beta1<br>kind: CustomResourceDefinition<br>metadata:<br>  name: keystores.infra.einstein.ai<br>  labels:<br>    operator: {{ .Values.app.name }}<br>spec:<br>  scope: Namespaced<br>  group: infra.einstein.ai<br>  version: v1alpha1<br>  names:<br>    kind: Keystore<br>    singular: keystore<br>    plural: keystores<br>    shortNames:<br>    - ks<br>  additionalPrinterColumns:<br>    - name: FQDN<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.fqdn<br>      description: FQDN of the certificate<br>    - name: Certificate Name<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.certName<br>      description: Name of the certificate that need to be included in the keystore<br>    - name: ARN<br>      type: string<br>      priority: 0<br>      JSONPath: .status.arn<br>      description: ARN of the keystore created by Notary which is uploaded as secret in AWS secrets manager<br>    - name: Tags<br>      type: string<br>      priority: 0<br>      JSONPath: .status.tags<br>      description: Tags on the certificate<br>  validation:<br>    openAPIV3Schema:<br>      properties:<br>        spec:<br>          properties:<br>            certName:<br>              type: string<br>              description: \"Name of the certificate that will be included in the keystore\"<br>            fqdn:<br>              type: string<br>              description: \"FQDN of the service of which certificate is created\"<br>          required: [\"fqdn\", \"certName\"]<\/pre>\n<p>When the keystore CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager to download the service certificate, builds the keystore, and uploads the keystore to AWS Secrets Manager.<\/p>\n<p> Required parameters<\/p>\n<ul>\n<li><strong>certName:<\/strong> Name tag of the newly requested certificate<\/li>\n<li><strong>fqdn:<\/strong> Fully qualified domain name of the\u00a0service<\/li>\n<\/ul>\n<h4>Truststore CRD<\/h4>\n<p>This custom resource defines the truststore resource. The resource spec accepts service metadata along with the service FQDN and certificate names of the upstream and downstream services.<\/p>\n<pre># CUSTOM-RESOURCE-DEFINITION MANIFEST<br>---<br>apiVersion: apiextensions.k8s.io\/v1beta1<br>kind: CustomResourceDefinition<br>metadata:<br>  name: truststores.infra.einstein.ai<br>  labels:<br>    operator: {{ .Values.app.name }}<br>spec:<br>  scope: Namespaced<br>  group: infra.einstein.ai<br>  version: v1alpha1<br>  names:<br>    kind: Truststore<br>    singular: truststore<br>    plural: truststores<br>    shortNames:<br>    - ts<br>  additionalPrinterColumns:<br>    - name: FQDN<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.fqdn<br>      description: FQDN of the certificate<br>    - name: ARN<br>      type: string<br>      priority: 0<br>      JSONPath: .status.arn<br>      description: ARN of the truststore created by Notary which is uploaded as secret in AWS secrets manager<br>    - name: Tags<br>      type: string<br>      priority: 0<br>      JSONPath: .status.tags<br>      description: Tags on the certificate<br>  validation:<br>    openAPIV3Schema:<br>      properties:<br>        spec:<br>          properties:<br>            upstream:<br>              type: array<br>              description: \"Tags of the upstream service(s) certificate that talk to this service\"<br>            downstream:<br>              type: array<br>              description: \"Tags of the downstream service(s) certificate that this service talks to\"<br>            fqdn:<br>              type: string<br>              description: \"FQDN of the service of which certificate is created\"<br>            certName:<br>              type: string<br>              description: \"Name  of the new certificate that need to be included in the truststore\"<br>          required: [\"fqdn\", \"upstream\", \"downstream\", \"certName\"]<\/pre>\n<p>Required parameters<\/p>\n<ul>\n<li><strong>upstream:<\/strong> Tags of the upstream service(s) certificate that talks to this\u00a0service<\/li>\n<li><strong>downstream:<\/strong> Tags of the downstream service(s) certificate that this service talks\u00a0to<\/li>\n<li><strong>fqdn:<\/strong> Fully qualified domain name of this\u00a0service<\/li>\n<li><strong>certName:<\/strong> Name tag of the newly-requested certificate<\/li>\n<\/ul>\n<h3>Notary In\u00a0Action<\/h3>\n<p>Now let\u2019s take a look at Notary in action. Consider a sample application, with a service named test-service. The test-service service talks to a proxy service and a database service. To have encrypted service-to-service communication, we must have certificates for each service and the resulting keystore and truststore.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*TrFTxHF0Fp9zgqBAd8g3Iw.jpeg?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><\/figure>\n<h4>Pre-requisites<\/h4>\n<p>As mentioned earlier, in addition to creating certificates, Notary also creates keystores and truststores. Both keystores and truststores are sensitive data, so they need to be protected. Notary password protects these artifacts at the time of creation so in case of unintended\/unauthorized access the keystore and truststore can\u2019t be read without their corresponding password. <\/p>\n<p>This implies there has to be a way to share the keystore and truststore credentials with the service that uses them to encrypt\/decrypt data. Currently, we handle this by creating a metadata secret for every service. This is a one-time admin-triggered automation (more on that in future blog post), part of service onboarding on our infrastructure.<\/p>\n<p>This metadata secret is an encoded JSON secret that follows the format shown\u00a0here.<\/p>\n<pre>{<br>  \"tlsStoreRootPath\": \"tls\",<br>  \"tlsKeyStoreName\": \"test-key-store\",<br>  \"tlsTrustStoreName\": \"test-trust-store\",<br>  \"tlsKeyStorePassword\": \"test-key-store-password\",<br>  \"tlsTrustStorePassword\": \"test-trust-store-password\"<br>}<\/pre>\n<p>After the service is onboarded and the metadata secret is created, Notary is ready to manage certificates, keystores, and truststores for that\u00a0service.<\/p>\n<h3>Step 1: Create the Certificate<\/h3>\n<p>To create certificates for the test service, proxy service, and database service, we must create certificate objects for all three services. Following is a Certificate custom resource YAML spec for test-service. Similarly, certificates can be created for the proxy service and database\u00a0service.<\/p>\n<pre>apiVersion: infra.einstein.ai\/v1alpha1<br>kind: Certificate<br>metadata:<br>  generation: 1<br>  labels:<br>    env: DEV<br>  name: test-service<br>  namespace: test-service<br>spec:<br>  alt:<br>  - test-service.localhost<br>  fqdn: test-service.test-service.svc.cluster.local<br>  type: Private<\/pre>\n<p>The above spec can be saved as test-service.yaml file. Let\u2019s use this file to create the test-service certificate object in the EKS cluster using the following command.<\/p>\n<pre><strong>kubectl apply -f test-service.yaml<\/strong><\/pre>\n<p>After you apply the object in the cluster, Notary requests AWS Certificate Manager to create a private certificate with the name test-service and an FQDN test-service.test-service.svc.cluster.local.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*3BQlfpQpOg0VuXm7lNJnIw.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><figcaption>AWS Certificate Manager console displaying new certificate<\/figcaption><\/figure>\n<p>We can also verify in the controller logs that the private certificate was\u00a0created.<\/p>\n<pre>[2021-03-03 21:40:45,709] kopf.objects         [INFO    ] [test-service\/test-service] Updated new certificate passphrase for service test-service<br>[2021-03-03 21:40:45,713] kopf.objects         [INFO    ] [test-service\/test-service] Handler 'processCertificate' succeeded.<br>[2021-03-03 21:40:45,713] kopf.objects         [INFO    ] [test-service\/test-service] Creation event is processed: 1 succeeded; 0 failed.<\/pre>\n<p>After we create certificates for all three services, we can also list the Kubernetes Certificate objects using this\u00a0command.<\/p>\n<pre><strong>$ kubectl get cert --all-namespaces<\/strong><br><strong>NAME               TYPE      FQDN                                                   <\/strong><br>db-new             Private   db-service.db-service.svc.cluster.local         <br>proxy-new          Private   proxy-service.proxy-service.svc.cluster.local  <br>test-service-new   Private   test-service.test-service.svc.cluster.local     <\/pre>\n<h3>Step 2: Create the\u00a0Keystore<\/h3>\n<p>The next step is to create the keystore for test-service. To create the keystore, apply the following spec in the cluster for test-service in test-service namespace.<\/p>\n<pre>---<br>apiVersion: infra.einstein.ai\/v1alpha1<br>kind: Keystore<br>metadata:<br>  name: test-service-key-store<br>  namespace: test-service<br>spec:<br>  fqdn: test-service.test-service.svc.cluster.local<br>  certName: test-service-new<\/pre>\n<p>Now let\u2019s create the test-service keystore object in the EKS cluster using this\u00a0command.<\/p>\n<pre><strong>kubectl apply -f test-service-keystore.yaml<\/strong><\/pre>\n<p>After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local and creates the keystore for the private key. The keystore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application\/service during deployment using its own IAM\u00a0role.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*jlOXyx4p8LKeqjqew4Kg9Q.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><figcaption>AWS Secret Manager Console displaying newly created\u00a0secret<\/figcaption><\/figure>\n<p>We can verify the creation of the keystore from the controller logs as shown\u00a0here.<\/p>\n<pre>[2020-10-27 23:44:43,596] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Downloaded Certificate for service test-service-new<br>[2020-10-27 23:44:43,598] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Downloaded CertificateChain for service test-service-new<br>[2020-10-27 23:44:43,599] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Downloaded PrivateKey for service test-service-new<br>[2020-10-27 23:44:43,612] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Created keystore successfully for test-service<br>[2020-10-27 23:44:43,694] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Updated secret: test-service-key-store<br>[2020-10-27 23:44:43,696] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Secret ARN: arn:aws:secretsmanager:us-west-2:&lt;a<em>ccount-id<\/em>&gt;:secret:test-service-key-store-ZimHEX<br>[2020-10-27 23:44:43,701] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Handler 'processKeystore' succeeded.<br>[2020-10-27 23:44:43,702] kopf.objects         [INFO    ] [test-service\/test-service-key-store] Creation event is processed: 1 succeeded; 0 failed.<\/pre>\n<p>After we create the keystore, we can list the Kubernetes keystore objects using this\u00a0command.<\/p>\n<pre><strong>$ kubectl get ks -n test-service<\/strong><br><strong>NAME                     FQDN                                      <\/strong><br>test-service-key-store   test-service.test-service.svc.cluster.local   <\/pre>\n<h3>Step 3: Create the Truststore<\/h3>\n<p>The next step is to create the truststore for test-service. To create the truststore, apply the following spec in the cluster in the test-service namespace. When you create the truststore, you must specify the upstream and downstream services in the spec. In our sample application, the test-service has proxy-service as an upstream service and database-service as a downstream service.<\/p>\n<pre>---<br>apiVersion: infra.einstein.ai\/v1alpha1<br>kind: Truststore<br>metadata:<br>  name: test-service-trust-store<br>  namespace: test-service<br>spec:<br>  fqdn: test-service.test-service.svc.cluster.local<br>  certName: test-service-new<br>  upstream:<br>    - tag: proxy-new<br>      fqdn: proxy-service.proxy-service.svc.cluster.local<br>  downstream:<br>    - tag: db-new<br>      fqdn: db-service.db-service.svc.cluster.local<\/pre>\n<p>Let\u2019s create the test-service truststore object in the EKS cluster using this\u00a0command.<\/p>\n<pre><strong>kubectl apply -f test-service-truststore.yaml<\/strong><\/pre>\n<p>After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local. Notary then downloads the certificates for the upstream and downstream services and creates the truststore including all downloaded certificates. The truststore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application\/service during deployment using its own IAM\u00a0role.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*IHyl0z55PFnX66sCwYOeZQ.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><figcaption>AWS Secrets Manager displaying newly created\u00a0secret<\/figcaption><\/figure>\n<p>We verify the creation of the truststore from the controller logs as shown\u00a0below.<\/p>\n<pre>[2020-10-28 00:06:32,972] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Downloaded Certificate for service test-service-new<br>[2020-10-28 00:06:32,975] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Downloaded CertificateChain for service test-service-new<br>[2020-10-28 00:06:32,976] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Downloaded PrivateKey for service test-service-new<br>[2020-10-28 00:06:33,577] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Added service proxy-new certificates to Trust store<br>[2020-10-28 00:06:34,397] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Added service db-new certificates to Trust store<br>[2020-10-28 00:06:35,191] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Added service root-ca certificates to Trust store<br>[2020-10-28 00:06:35,742] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Added service test-service-New certificates to Trust store<br>[2020-10-28 00:06:35,867] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Updated secret: test-service-trust-store<br>[2020-10-28 00:06:35,869] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Secret ARN: arn:aws:secretsmanager:us-west-2:&lt;a<em>ccount-id<\/em>&gt;:secret:test-service-trust-store-mBJAIO<br>[2020-10-28 00:06:35,875] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Handler 'processTruststore' succeeded.<br>[2020-10-28 00:06:35,876] kopf.objects         [INFO    ] [test-service\/test-service-trust-store] Creation event is processed: 1 succeeded; 0 failed.<\/pre>\n<p>After we create the truststore, we can list the Kubernetes truststore objects using the following command.<\/p>\n<pre>$ <strong>kubectl get ts -n test-service<\/strong><br><strong>NAME                     FQDN                                      <\/strong><br>test-service-trust-store   test-service.test-service.svc.cluster.local   <\/pre>\n<h3>Conclusion<\/h3>\n<p>Notary is a Kubernetes controller that bridges the gap between AWS services and Kubernetes native resources to facilitate encryption of traffic between services running on a Kubernetes cluster using AWS-generated certificates. It provides cluster administrators an easy way to create and manage certificates using AWS Certificate Manager and store them in AWS Secrets Manager for the services to use them securely.<\/p>\n<h3>References<\/h3>\n<ul>\n<li><a href=\"https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/api-extension\/custom-resources\/\">Kubernetes\u200a\u2014\u200aCustom Resources<\/a><\/li>\n<li><a href=\"https:\/\/kubernetes.io\/docs\/concepts\/architecture\/controller\">Kubernetes\u200a\u2014\u200aControllers<\/a><\/li>\n<\/ul>\n<h3>Acknowledgments<\/h3>\n<p>We would like to thank the entire Einstein Vision and Language Platform Infra team for valuable feedback during the design and implementation phase of Notary and the leadership team\u200a\u2014\u200aDaniel Tisher, Ivo Mihov, and Indira Iyer\u200a\u2014\u200afor their support and encouragement. We would also like to thank Dianne Siebold for reviewing the blog post and structuring the content. Notary was built using the <a href=\"https:\/\/kopf.readthedocs.io\/en\/latest\/\">Kopf<\/a>\u00a0project.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=a24a53e26cbd\" width=\"1\" height=\"1\" alt=\"\"><\/p>\n<hr>\n<p><a href=\"https:\/\/engineering.salesforce.com\/notary-a-certificate-lifecycle-management-controller-for-kubernetes-a24a53e26cbd\">Notary: A Certificate Lifecycle Management Controller for Kubernetes<\/a> was originally published in <a href=\"https:\/\/engineering.salesforce.com\/\">Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<p><a href=\"https:\/\/engineering.salesforce.com\/notary-a-certificate-lifecycle-management-controller-for-kubernetes-a24a53e26cbd?source=rss----cfe1120185d3---4\" target=\"_blank\" rel=\"noopener\">Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Authors: Vaishnavi Galgali, Savithru Lokanath, Arpeet\u00a0Kale Introduction All services in the Einstein Vision and Language Platform use TLS\/SSL certificates to encrypt communication between microservices. The certificates are generated in AWS Certificate Manager (ACM) and stored in the AWS Secrets Manager in the form of keystores and truststores (private and public keys). Certificate creation can be&hellip; <a class=\"more-link\" href=\"https:\/\/fde.cat\/index.php\/2021\/08\/31\/notary-a-certificate-lifecycle-management-controller-for-kubernetes\/\">Continue reading <span class=\"screen-reader-text\">Notary: A Certificate Lifecycle Management Controller for Kubernetes<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-279","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":741,"url":"https:\/\/fde.cat\/index.php\/2023\/08\/07\/using-short-lived-certificates-to-protect-tls-secrets\/","url_meta":{"origin":279,"position":0},"title":"Using short-lived certificates to protect TLS secrets","date":"August 7, 2023","format":false,"excerpt":"Short-lived certificates (SLCs) are part of our latest efforts to further secure our Transport Layer Security (TLS) private keys on our edge networks. SLCs have a very short exposure compared to traditional certificates and lower the chances of a compromised private key being abused. Implementing SLCs has required us to\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":725,"url":"https:\/\/fde.cat\/index.php\/2023\/06\/19\/sre-weekly-issue-377\/","url_meta":{"origin":279,"position":1},"title":"SRE Weekly Issue #377","date":"June 19, 2023","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Curious how companies like Figma, Tripadvisor, and 100s of others leverage Rootly to manage incidents in Slack and unlock instant best practices? Check out this lightning demo: https:\/\/www.loom.com\/share\/051c4be0425a436e888dc0c3690855ad Articles Why did AWS Support fail with US-EAST-1 again? AWS had a major\u2026","rel":"","context":"In &quot;SRE&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":287,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/optimizing-eks-networking-for-scale\/","url_meta":{"origin":279,"position":2},"title":"Optimizing EKS networking for scale","date":"August 31, 2021","format":false,"excerpt":"Authors: Savithru Lokanath, Arpeet Kale, VaishnavigalgaliElastic Kubernetes Service (EKS) is a service under the Amazon Web Services (AWS) umbrella that provides managed Kubernetes service. It significantly reduces the time to deploy, manage, and scale the infrastructure required to run production-scale Kubernetes clusters. AWS has simplified EKS networking significantly with its\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":770,"url":"https:\/\/fde.cat\/index.php\/2023\/10\/10\/revealing-the-newest-data-science-tool-speeding-ai-development-and-securing-customer-data\/","url_meta":{"origin":279,"position":3},"title":"Revealing the Newest Data Science Tool: Speeding AI Development and Securing Customer Data","date":"October 10, 2023","format":false,"excerpt":"by Chi Wang and Scott Nyberg In today\u2019s data-powered world, leveraging customer data to improve AI capabilities remains key for providing highly personalized consumer experiences. In fact, 43% of customers believe AI has improved their lives, with 54% willing to provide their anonymized data to improve AI-related products. However, more\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":268,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/zero-downtime-node-patching-in-a-kubernetes-cluster\/","url_meta":{"origin":279,"position":4},"title":"Zero Downtime Node Patching in a Kubernetes Cluster","date":"August 31, 2021","format":false,"excerpt":"Authors: Vaishnavi Galgali, Arpeet Kale, Robert\u00a0XueIntroductionThe Salesforce Einstein Vision and Language services are deployed in an AWS Elastic Kubernetes Service (EKS) cluster. One of the primary security and compliance requirements is operating system patching. The cluster nodes that the services are deployed on need to have regular operating system updates.\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":481,"url":"https:\/\/fde.cat\/index.php\/2021\/10\/04\/sre-weekly-issue-290\/","url_meta":{"origin":279,"position":5},"title":"SRE Weekly Issue #290","date":"October 4, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly \ud83d\ude92. Automate manual admin tasks like creating incident channel, Jira and Zoom, paging the right team, postmortem timeline, setting up reminders, and more. Book a demo: https:\/\/rootly.io\/?utm_source=sreweekly Articles Postmortem: Partial RavenDB Cloud outage Despite\u2026","rel":"","context":"In &quot;SRE&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/279","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=279"}],"version-history":[{"count":1,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/279\/revisions"}],"predecessor-version":[{"id":431,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/279\/revisions\/431"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=279"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=279"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=279"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}