Authors: Vaishnavi Galgali, Savithru Lokanath, Arpeet Kale
Introduction
All services in the Einstein Vision and Language Platform use TLS/SSL certificates to encrypt communication between microservices. The certificates are generated in AWS Certificate Manager (ACM) and stored in the AWS Secrets Manager in the form of keystores and truststores (private and public keys). Certificate creation can be a manual process, especially if the permission model dictates that a certificate can only be provisioned or de-provisioned by an AWS account admin.
Certificate management includes provisioning certificates and monitoring their expiration and renewal. AWS ACM provides the ability to auto-renew only public certs attached to AWS native resources (such as ELB and RDS). Einstein Vision and Language Platform services run on an EKS cluster with service exposure scoped to the cluster network, which means that the ACM certificates aren’t attached to AWS native resources. Hence, the certificate auto-renewal feature provided by AWS ACM can’t be used.
To automate certificate provisioning and management and to provide a self-service model for carrying out these activities, we built a custom Kubernetes controller by introducing Kubernetes native resources: certificate, keystore, and truststore. Additionally, we built automation for checking certificate expiration and alert on expiration of these certificates. We call our solution Notary.
Architecture
Notary is built on the core Kubernetes concepts:
- Controllers
- Custom Resource Definition (CRD)
What is a Kubernetes controller?
In Kubernetes, controllers are control loops that watch the state of your cluster, then make or request changes where needed. Each controller tries to move the current cluster state closer to the desired state. A controller tracks at least one Kubernetes resource type. These objects have a spec field that represents the desired state. The controller for that resource is responsible for performing actions to achieve the desired state. For example, the job controller, which is a built-in Kubernetes controller, helps to execute tasks in Kubernetes cluster by deploying pods as dictated by the pod spec.
The Kubernetes API can be extended to create application-specific controllers to create, configure, and manage instances of complex, stateful applications on behalf of a Kubernetes user. The custom Kubernetes controller builds upon the Kubernetes resource and controller concepts, and includes domain or application-specific knowledge to automate common tasks.
What is a CRD?
CRD stands for Custom Resource Definition.
A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.
A custom resource is an extension of the Kubernetes API that provides functionality that isn’t available in a default Kubernetes installation. A custom resource represents a customization of a particular Kubernetes installation. However, many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
A custom resource can appear and disappear in a running cluster through dynamic registration. And cluster admins can update a custom resource independently of the cluster itself. After a custom resource is installed, users can create and access its objects using kubectl, just as they do for built-in resources like pods.
Notary
Notary is an extension to the Kubernetes API to satisfy an application-specific use case — encryption of traffic between services running on Kubernetes cluster — by managing the certificates and their lifecycle.
Notary is a custom Kubernetes controller that watches on these custom resources: certificate, keystore, and truststore. The controller watches for create and update events, and takes defined action based on those events.
Notary interacts with AWS Certificate Manager to request and download certificates to create keystores and truststores. After it creates keystores and truststores, Notary initiates a session with AWS Secrets Manager to upload them.
The architecture diagram above depicts the functionality of Notary. The Notary controller constantly watches the custom resources — certificate, keystore, and truststore—for a create, update, or delete event.
- Certificate: When the create event for the certificate resource occurs, the Notary controller requests a new certificate from AWS Certificate Manager.
- Keystore: When the create event for the keystore resource occurs, the Notary controller downloads the service’s certificate bundle from AWS Certificate Manager, creates a keystore, and then uploads the keystore to AWS Secrets Manager.
- Truststore: When the create event for the truststore resource occurs, the Notary controller downloads the service’s certificate bundle along with upstream and downstream services’ certificates from AWS Certificate Manager. It then builds a truststore which includes the service certificate chain along with upstream and downstream service certificates, and uploads the truststore to AWS Secrets Manager.
Components
Notary is built using Kubernetes objects. Let’s take a look at the components of Notary.
Deployment
Since we’re dealing with sensitive data — certificate, keystore, and truststore—for services, it’s important to secure the Notary pods. To address that, Notary is deployed in its own namespace where the namespace resources can only be accessed by cluster administrators.
RBAC
RBAC, role-based access control, is a way to manage access to resources based on roles. Notary needs access to the CRD objects certificate, keystore, and truststore which can be deployed in any service-specific namespace. Since Notary needs to watch all namespaces it needs ClusterRole privileges.
These examples use ClusterRole and ClusterRoleBinding to give Notary access to specific resources and assign a certain level of access to resources.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
operator: notary
name: notary
rules:
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- list
- apiGroups:
- infra.einstein.ai
resources:
- certificates
- keystores
- truststores
verbs:
- get
- list
- watch
- apiGroups:
- events.k8s.io
- ""
resources:
- events
verbs:
- create
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: notary
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: notary
subjects:
- kind: ServiceAccount
name: notary
namespace: notary
IAM Roles
Notary needs access to AWS Certificate Manager to request new certificates and access to AWS Secrets Manager to upload keystores and truststores. We use AWS IAM (Identity and Access Management) roles to grant access to AWS services required for Notary. The following JSON is an example of an IAM role.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"acm:RequestCertificate",
"acm:GetCertificate",
"acm:ListCertificates",
"acm:ExportCertificate",
"acm:UpdateCertificateOptions",
"acm:AddTagsToCertificate",
"acm:ListTagsForCertificate",
"acm:DescribeCertificate",
"acm:ResendValidationEmail",
"acm:RemoveTagsFromCertificate",
"acm-pca:GetCertificate",
"acm-pca:IssueCertificate",
"acm-pca:ListCertificateAuthorities"
],
"Resource": "*",
"Effect": "Allow"
},
{
"Action": [
"secretsmanager:UntagResource",
"secretsmanager:DescribeSecret",
"secretsmanager:PutSecretValue",
"secretsmanager:GetSecretValue",
"secretsmanager:CreateSecret",
"secretsmanager:DeleteSecret",
"secretsmanager:ListSecretVersionIds",
"secretsmanager:GetRandomPassword",
"secretsmanager:GetSecretValue",
"secretsmanager:RestoreSecret",
"secretsmanager:RotateSecret",
"secretsmanager:CancelRotateSecret",
"secretsmanager:UpdateSecret",
"secretsmanager:GetResourcePolicy",
"secretsmanager:UpdateSecretVersionStage",
"secretsmanager:ListSecrets",
"secretsmanager:TagResource"
],
"Resource": "*",
"Effect": "Allow"
}
]
}
CRD
Certificate CRD
This custom resource defines the metadata required to request a certificate from AWS Certificate Manager. The metadata includes the certificate FQDN and type (public or private) of certificate.
# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: certificates.infra.einstein.ai
labels:
operator: {{ .Values.app.name }}
spec:
scope: Namespaced
group: infra.einstein.ai
version: v1alpha1
names:
kind: Certificate
singular: certificate
plural: certificates
shortNames:
- cert
additionalPrinterColumns:
- name: Type
type: string
JSONPath: .spec.type
description: Type of certificate
- name: FQDN
type: string
priority: 0
JSONPath: .spec.fqdn
description: FQDN of the certificate
- name: AltName
type: string
priority: 0
JSONPath: .spec.alt
description: Alternate names of the certificate
- name: ARN
type: string
priority: 0
JSONPath: .status.arn
description: ARN of the certificate
- name: Tags
type: string
priority: 0
JSONPath: .status.tags
description: Tags on the certificate
validation:
openAPIV3Schema:
properties:
spec:
properties:
type:
type: string
description: "AWS Certificate Manager PEM certificate issued by either a Private of Public Root CA."
enum: ["Public", "Private"]
fqdn:
type: string
description: "FQDN of the certificate"
alt:
type: array
description: "Alternate name(s) for the certifcate"
required: ["type", "fqdn"]
When the certificate CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager for a new certificate.
Required parameters
- type: Type of certificate, either public or private
- fqdn: Fully qualified domain name of the service
Keystore CRD
This custom resource defines the keystore resource. The resource spec accepts metadata to generate a new keystore for a particular service and then uploads the keystore to AWS Secrets Manager.
# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: keystores.infra.einstein.ai
labels:
operator: {{ .Values.app.name }}
spec:
scope: Namespaced
group: infra.einstein.ai
version: v1alpha1
names:
kind: Keystore
singular: keystore
plural: keystores
shortNames:
- ks
additionalPrinterColumns:
- name: FQDN
type: string
priority: 0
JSONPath: .spec.fqdn
description: FQDN of the certificate
- name: Certificate Name
type: string
priority: 0
JSONPath: .spec.certName
description: Name of the certificate that need to be included in the keystore
- name: ARN
type: string
priority: 0
JSONPath: .status.arn
description: ARN of the keystore created by Notary which is uploaded as secret in AWS secrets manager
- name: Tags
type: string
priority: 0
JSONPath: .status.tags
description: Tags on the certificate
validation:
openAPIV3Schema:
properties:
spec:
properties:
certName:
type: string
description: "Name of the certificate that will be included in the keystore"
fqdn:
type: string
description: "FQDN of the service of which certificate is created"
required: ["fqdn", "certName"]
When the keystore CRD is installed in the cluster, Notary monitors the create event and issues requests to AWS Certificate Manager to download the service certificate, builds the keystore, and uploads the keystore to AWS Secrets Manager.
Required parameters
- certName: Name tag of the newly requested certificate
- fqdn: Fully qualified domain name of the service
Truststore CRD
This custom resource defines the truststore resource. The resource spec accepts service metadata along with the service FQDN and certificate names of the upstream and downstream services.
# CUSTOM-RESOURCE-DEFINITION MANIFEST
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: truststores.infra.einstein.ai
labels:
operator: {{ .Values.app.name }}
spec:
scope: Namespaced
group: infra.einstein.ai
version: v1alpha1
names:
kind: Truststore
singular: truststore
plural: truststores
shortNames:
- ts
additionalPrinterColumns:
- name: FQDN
type: string
priority: 0
JSONPath: .spec.fqdn
description: FQDN of the certificate
- name: ARN
type: string
priority: 0
JSONPath: .status.arn
description: ARN of the truststore created by Notary which is uploaded as secret in AWS secrets manager
- name: Tags
type: string
priority: 0
JSONPath: .status.tags
description: Tags on the certificate
validation:
openAPIV3Schema:
properties:
spec:
properties:
upstream:
type: array
description: "Tags of the upstream service(s) certificate that talk to this service"
downstream:
type: array
description: "Tags of the downstream service(s) certificate that this service talks to"
fqdn:
type: string
description: "FQDN of the service of which certificate is created"
certName:
type: string
description: "Name of the new certificate that need to be included in the truststore"
required: ["fqdn", "upstream", "downstream", "certName"]
Required parameters
- upstream: Tags of the upstream service(s) certificate that talks to this service
- downstream: Tags of the downstream service(s) certificate that this service talks to
- fqdn: Fully qualified domain name of this service
- certName: Name tag of the newly-requested certificate
Notary In Action
Now let’s take a look at Notary in action. Consider a sample application, with a service named test-service. The test-service service talks to a proxy service and a database service. To have encrypted service-to-service communication, we must have certificates for each service and the resulting keystore and truststore.
Pre-requisites
As mentioned earlier, in addition to creating certificates, Notary also creates keystores and truststores. Both keystores and truststores are sensitive data, so they need to be protected. Notary password protects these artifacts at the time of creation so in case of unintended/unauthorized access the keystore and truststore can’t be read without their corresponding password.
This implies there has to be a way to share the keystore and truststore credentials with the service that uses them to encrypt/decrypt data. Currently, we handle this by creating a metadata secret for every service. This is a one-time admin-triggered automation (more on that in future blog post), part of service onboarding on our infrastructure.
This metadata secret is an encoded JSON secret that follows the format shown here.
{
"tlsStoreRootPath": "tls",
"tlsKeyStoreName": "test-key-store",
"tlsTrustStoreName": "test-trust-store",
"tlsKeyStorePassword": "test-key-store-password",
"tlsTrustStorePassword": "test-trust-store-password"
}
After the service is onboarded and the metadata secret is created, Notary is ready to manage certificates, keystores, and truststores for that service.
Step 1: Create the Certificate
To create certificates for the test service, proxy service, and database service, we must create certificate objects for all three services. Following is a Certificate custom resource YAML spec for test-service. Similarly, certificates can be created for the proxy service and database service.
apiVersion: infra.einstein.ai/v1alpha1
kind: Certificate
metadata:
generation: 1
labels:
env: DEV
name: test-service
namespace: test-service
spec:
alt:
- test-service.localhost
fqdn: test-service.test-service.svc.cluster.local
type: Private
The above spec can be saved as test-service.yaml file. Let’s use this file to create the test-service certificate object in the EKS cluster using the following command.
kubectl apply -f test-service.yaml
After you apply the object in the cluster, Notary requests AWS Certificate Manager to create a private certificate with the name test-service and an FQDN test-service.test-service.svc.cluster.local.
We can also verify in the controller logs that the private certificate was created.
[2021-03-03 21:40:45,709] kopf.objects [INFO ] [test-service/test-service] Updated new certificate passphrase for service test-service
[2021-03-03 21:40:45,713] kopf.objects [INFO ] [test-service/test-service] Handler 'processCertificate' succeeded.
[2021-03-03 21:40:45,713] kopf.objects [INFO ] [test-service/test-service] Creation event is processed: 1 succeeded; 0 failed.
After we create certificates for all three services, we can also list the Kubernetes Certificate objects using this command.
$ kubectl get cert --all-namespaces
NAME TYPE FQDN
db-new Private db-service.db-service.svc.cluster.local
proxy-new Private proxy-service.proxy-service.svc.cluster.local
test-service-new Private test-service.test-service.svc.cluster.local
Step 2: Create the Keystore
The next step is to create the keystore for test-service. To create the keystore, apply the following spec in the cluster for test-service in test-service namespace.
---
apiVersion: infra.einstein.ai/v1alpha1
kind: Keystore
metadata:
name: test-service-key-store
namespace: test-service
spec:
fqdn: test-service.test-service.svc.cluster.local
certName: test-service-new
Now let’s create the test-service keystore object in the EKS cluster using this command.
kubectl apply -f test-service-keystore.yaml
After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local and creates the keystore for the private key. The keystore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application/service during deployment using its own IAM role.
We can verify the creation of the keystore from the controller logs as shown here.
[2020-10-27 23:44:43,596] kopf.objects [INFO ] [test-service/test-service-key-store] Downloaded Certificate for service test-service-new
[2020-10-27 23:44:43,598] kopf.objects [INFO ] [test-service/test-service-key-store] Downloaded CertificateChain for service test-service-new
[2020-10-27 23:44:43,599] kopf.objects [INFO ] [test-service/test-service-key-store] Downloaded PrivateKey for service test-service-new
[2020-10-27 23:44:43,612] kopf.objects [INFO ] [test-service/test-service-key-store] Created keystore successfully for test-service
[2020-10-27 23:44:43,694] kopf.objects [INFO ] [test-service/test-service-key-store] Updated secret: test-service-key-store
[2020-10-27 23:44:43,696] kopf.objects [INFO ] [test-service/test-service-key-store] Secret ARN: arn:aws:secretsmanager:us-west-2:<account-id>:secret:test-service-key-store-ZimHEX
[2020-10-27 23:44:43,701] kopf.objects [INFO ] [test-service/test-service-key-store] Handler 'processKeystore' succeeded.
[2020-10-27 23:44:43,702] kopf.objects [INFO ] [test-service/test-service-key-store] Creation event is processed: 1 succeeded; 0 failed.
After we create the keystore, we can list the Kubernetes keystore objects using this command.
$ kubectl get ks -n test-service
NAME FQDN
test-service-key-store test-service.test-service.svc.cluster.local
Step 3: Create the Truststore
The next step is to create the truststore for test-service. To create the truststore, apply the following spec in the cluster in the test-service namespace. When you create the truststore, you must specify the upstream and downstream services in the spec. In our sample application, the test-service has proxy-service as an upstream service and database-service as a downstream service.
---
apiVersion: infra.einstein.ai/v1alpha1
kind: Truststore
metadata:
name: test-service-trust-store
namespace: test-service
spec:
fqdn: test-service.test-service.svc.cluster.local
certName: test-service-new
upstream:
- tag: proxy-new
fqdn: proxy-service.proxy-service.svc.cluster.local
downstream:
- tag: db-new
fqdn: db-service.db-service.svc.cluster.local
Let’s create the test-service truststore object in the EKS cluster using this command.
kubectl apply -f test-service-truststore.yaml
After we apply the above spec in the cluster, Notary downloads the certificate chain for the certificate with FQDN test-service.test-service.svc.cluster.local. Notary then downloads the certificates for the upstream and downstream services and creates the truststore including all downloaded certificates. The truststore is then uploaded to AWS Secrets Manager as a secret. This secret can later be downloaded by the application/service during deployment using its own IAM role.
We verify the creation of the truststore from the controller logs as shown below.
[2020-10-28 00:06:32,972] kopf.objects [INFO ] [test-service/test-service-trust-store] Downloaded Certificate for service test-service-new
[2020-10-28 00:06:32,975] kopf.objects [INFO ] [test-service/test-service-trust-store] Downloaded CertificateChain for service test-service-new
[2020-10-28 00:06:32,976] kopf.objects [INFO ] [test-service/test-service-trust-store] Downloaded PrivateKey for service test-service-new
[2020-10-28 00:06:33,577] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service proxy-new certificates to Trust store
[2020-10-28 00:06:34,397] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service db-new certificates to Trust store
[2020-10-28 00:06:35,191] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service root-ca certificates to Trust store
[2020-10-28 00:06:35,742] kopf.objects [INFO ] [test-service/test-service-trust-store] Added service test-service-New certificates to Trust store
[2020-10-28 00:06:35,867] kopf.objects [INFO ] [test-service/test-service-trust-store] Updated secret: test-service-trust-store
[2020-10-28 00:06:35,869] kopf.objects [INFO ] [test-service/test-service-trust-store] Secret ARN: arn:aws:secretsmanager:us-west-2:<account-id>:secret:test-service-trust-store-mBJAIO
[2020-10-28 00:06:35,875] kopf.objects [INFO ] [test-service/test-service-trust-store] Handler 'processTruststore' succeeded.
[2020-10-28 00:06:35,876] kopf.objects [INFO ] [test-service/test-service-trust-store] Creation event is processed: 1 succeeded; 0 failed.
After we create the truststore, we can list the Kubernetes truststore objects using the following command.
$ kubectl get ts -n test-service
NAME FQDN
test-service-trust-store test-service.test-service.svc.cluster.local
Conclusion
Notary is a Kubernetes controller that bridges the gap between AWS services and Kubernetes native resources to facilitate encryption of traffic between services running on a Kubernetes cluster using AWS-generated certificates. It provides cluster administrators an easy way to create and manage certificates using AWS Certificate Manager and store them in AWS Secrets Manager for the services to use them securely.
References
Acknowledgments
We would like to thank the entire Einstein Vision and Language Platform Infra team for valuable feedback during the design and implementation phase of Notary and the leadership team — Daniel Tisher, Ivo Mihov, and Indira Iyer — for their support and encouragement. We would also like to thank Dianne Siebold for reviewing the blog post and structuring the content. Notary was built using the Kopf project.
Notary: A Certificate Lifecycle Management Controller for Kubernetes was originally published in Salesforce Engineering on Medium, where people are continuing the conversation by highlighting and responding to this story.