{"id":223,"date":"2021-02-02T20:02:58","date_gmt":"2021-02-02T20:02:58","guid":{"rendered":"https:\/\/fde.cat\/?p=223"},"modified":"2021-02-02T20:02:59","modified_gmt":"2021-02-02T20:02:59","slug":"project-agumbe-share-objects-across-namespaces-in-kubernetes","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/project-agumbe-share-objects-across-namespaces-in-kubernetes\/","title":{"rendered":"Project Agumbe: Share Objects Across Namespaces in Kubernetes"},"content":{"rendered":"<p>At <a href=\"https:\/\/medium.com\/u\/f4fb2a348280\">Salesforce<\/a>, we use <a href=\"https:\/\/kubernetes.io\/\">Kubernetes<\/a> to orchestrate our services layer and recently ran into a use case where we wanted to apply and manage certain common objects across Kubernetes namespaces. Since there\u2019s no native solution to share objects across namespaces or the concept of a global object, we used Kubernetes<em>\u2019<\/em> extensibility to solve the problem. In this post, I\u2019ll shed light on how we accomplished this.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1000\/1*QKRkx1foIVB_Qo9EUqW3nw.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><\/figure>\n<p>Namespaced objects in Kubernetes aren\u2019t designed to be<em> <\/em>shared across logical boundaries. If you want to share a certain object across multiple namespaces, you must scope the object at the cluster level. The inability to scope objects at a higher level has created a gap for certain use cases that we (at Salesforce) have while running Kubernetes in production.<\/p>\n<p>When multiple namespaces (where a namespace may be mapped to a service, tenant, etc) need a reference to a common object such as a Secret or ConfigMap, there isn\u2019t an easy way to make these objects available to desired namespaces other than having to create them in each namespace. This repetitive process is time-consuming, error-prone, and boring.<\/p>\n<p>Let\u2019s take a look at an example. Enterprises rely on a private registry to store container images and application artifacts. When an application pod is deployed, the kubelet uses the sensitive information stored in a Kubernetes Secret object to authenticate against the private registry and pull a private image on behalf of the pod.<\/p>\n<p>We not only have to inject these secrets during namespace provisioning, but we must also be compliant and rotate the secrets at regular intervals. When you have hundreds and thousands of namespaces to manage, the process becomes complicated and cumbersome introducing significant operational overhead such\u00a0as:<\/p>\n<ul>\n<li>Rotate Secrets and Certificates<\/li>\n<li>Update ConfigMaps, PodDisruptionBudgets, Quotas, NetworkPolicy, and role-based access control (RBAC) resources<\/li>\n<li>Detect config drifts, manual changes,\u00a0etc<\/li>\n<\/ul>\n<p>Kubernetes is designed to be highly configurable and extensible. So we decided to make use of the<em> <\/em><a href=\"https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/operator\/\">operator<\/a> and <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/architecture\/controller\/\">controller<\/a> patterns to design a solution. We built a custom controller called Agumbe, named after a small coastal town in Karnataka, India<em>,<\/em> with the primary motive being, <strong><em>\u201cto replicate whatever is thrown at\u00a0it!!\u201d<\/em><\/strong><\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/700\/1*42SwKaT4BtRYuj7goYze9w.jpeg?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><figcaption>Agumbe: A K8s controller to replicate namespaced objects<\/figcaption><\/figure>\n<h3>Architecture<\/h3>\n<p>Let\u2019s take a look at the concepts on which Agumbe is built.<\/p>\n<p>The <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/operator\/\">operator<\/a> framework was first introduced by <a href=\"https:\/\/coreos.com\/blog\/introducing-operator-framework\">CoreOS<\/a> and the Kubernetes community in mid-2018. Operators are a software extension that follows the principle of a <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/architecture\/controller\/\">control loop<\/a>. A control loop uses the current cluster state to make intelligent decisions. Operators are clients of the Kubernetes API and act as controllers for a custom resource.<\/p>\n<ul>\n<li>A <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/extend-kubernetes\/api-extension\/custom-resources\/\">custom resource<\/a> is an extension of the Kubernetes API. It represents a customization of a Kubernetes installation, making it more\u00a0modular.<\/li>\n<li>A <a href=\"https:\/\/kubernetes.io\/docs\/concepts\/architecture\/controller\/\">controller\u2019s<\/a> responsibility is to move the current state of the cluster closer to the desired state. For example, a replication controller ensures that the desired number of pods are running at all intervals of\u00a0time.<\/li>\n<\/ul>\n<p>The most common way to deploy an operator is to add the Custom Resource Definition (CRD) which defines the structure or schema of the custom resource and its associated controller to your cluster. The CRD controller normally runs outside of the <a href=\"https:\/\/kubernetes.io\/docs\/reference\/glossary\/?all=true#term-control-plane\">control plane<\/a>, much as you would run any containerized application. The client, who might be a cluster administrator or an application developer interacts with the custom resource to accomplish a certain objective.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*H_9FVZXzZySybwk_eAz2ug.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><figcaption>Fig1. K8s operator\u00a0pattern<\/figcaption><\/figure>\n<p>Agumbe is a first-class citizen of the Kubernetes APIs and provides system administrators an abstraction of a cluster scoped object.<strong> <\/strong>The Agumbe controller introduces a custom resource object<em> <\/em>called GlobalObject which has a reference to the object that needs to be replicated. When this custom resource is created, updated, or deleted, Agumbe simply replicates the state of the object into the desired namespaces.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*szPHDXKJX9FoMTkmZw2O0g.jpeg?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><figcaption>Fig2. Agumbe replicating a\u00a0secret<\/figcaption><\/figure>\n<p>The preceding diagram shows how Agumbe works.<strong> <\/strong>The control loop<strong> <\/strong>constantly watches for any change on the GlobalObject custom resource. When an event occurs, the controller takes the corresponding action of replicating the secret (depicted as Parent Secret<em> <\/em>in the diagram) from the admin<strong> <\/strong>namespace to the other namespaces: proxy, app, and database.<\/p>\n<h3>Components<\/h3>\n<p>Let\u2019s take a look at the components constituting the Agumbe controller.<\/p>\n<ol>\n<li><strong>Deployment<\/strong><\/li>\n<\/ol>\n<ul>\n<li>Since we must transform or replicate sensitive data, it\u2019s important to run Agumbe in a namespace following the least permissive model. We used an admin namespace that can only be accessed by cluster administrators.<\/li>\n<li>In production clusters, we run multiple replicas of the controller and elect a leader to avoid running into a race condition. The election process is triggered during pod:Init and the pod with the lowest priority wins the election and becomes the\u00a0leader.<\/li>\n<li>Agumbe is designed to be modular and pluggable, and hence it\u2019s possible to replicate any namespace scoped object through a simple config file\u00a0change.<\/li>\n<\/ul>\n<pre><strong># MODULAR AND PLUGGABLE CONFIG<br><\/strong><br>resources.yaml: |-<br>  ---<br>  core:<br>  - name: namespace<br>    api: CoreV1Api<br>  - name: sanitize<br>    api: ApiClient<br>  scoped:<br>  - name: secret<br>    api: CoreV1Api<br>    convention: secret<br>  - name: configmap<br>    api: CoreV1Api<br>    convention: config_map<\/pre>\n<p><strong>2. Role-Based Access Control\u00a0(RBAC)<\/strong><\/p>\n<p>Agumbe needs access permissions to monitor the GlobalObject custom resource object changes and to create, read, update, delete<em> <\/em>objects<em> <\/em>in the namespace. Hence the controller needs ClusterRole privileges.<\/p>\n<p>The following ClusterRole example has a set of rules, with each rule defining what level of permissions one would like to grant on a specific resource. In the below example, we give Agumbe the permission to replicate Kubernetes Secrets and ConfigMaps to the target namespace (but is not limited to these two resources only).<\/p>\n<pre><strong># CLUSTER-ROLE MANIFEST<\/strong><br><br>rules:<br>- apiGroups: [\"\"]<br>  resources: [\"secrets\", \"configmaps\"]<br>  verbs: [\"get\", \"create\", \"update\", \"list\"]<br>- apiGroups: [\"\"]<br>  resources: [\"namespaces\"]<br>  verbs: [\"get\", \"list\"]<br>- apiGroups: [\"infra.einstein.ai\"]<br>  resources: [\"globalobjects\"]<br>  verbs: [\"get\", \"watch\"]<br>- apiGroups: [\"events.k8s.io\"]<br>  resources: [\"events\"]<br>  verbs: [\"create\"]<\/pre>\n<p><strong>3. Custom Resource Definition (CRD)<\/strong><\/p>\n<p>The CRD for globalsecrets.einstein.ai defines the schema for the custom resources that will be created by Kubernetes clients.<\/p>\n<pre><strong># CUSTOM-RESOURCE-DEFINITION MANIFEST<br><\/strong><br>---<br>apiVersion: apiextensions.k8s.io\/v1beta1<br>kind: CustomResourceDefinition<br>metadata:<br>  name: globalobjects.infra.einstein.ai<br>  labels:<br>    app: agumbe<br>spec:<br>  scope: Namespaced<br>  group: infra.einstein.ai<br>  version: v1beta<br>  names:<br>    kind: GlobalObject<br>    singular: globalobject<br>    plural: globalobjects<br>    shortNames:<br>    - go<br>  additionalPrinterColumns:<br>    - name: Object Type<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.type<br>      description: Type of the object to replicate into namespace(s)<br>    - name: Object Name<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.name<br>      description: Name of the object to replicate into namespace(s)<br>    - name: Object Target Name<br>      type: string<br>      priority: 0<br>      JSONPath: .spec.targetName<br>      description: Name of the object in the target namespace(s)<br>  validation:<br>    openAPIV3Schema:<br>      properties:<br>        spec:<br>          properties:<br>            type:<br>              type: string<br>              description: \"Type of the object to replicate\"<br>            name:<br>              type: string<br>              description: \"Name of the object to replicate\"<br>            targetName:<br>              type: string<br>              description: \"Name of the object at the target\"<br>            targetNamespaces:<br>              type: array<br>              description: \"Target namespace(s)\"<br>            matchLabels:<br>              type: array<br>              description: \"Namespace(s) labels to match\"<br>          required: [\"type\", \"name\"]<\/pre>\n<p><strong>4.<\/strong> <strong>Parent\u00a0Objects<\/strong><\/p>\n<p>These are the objects that are replicated to the namespaces. The GlobalObject custom<em> <\/em>resource specification has a reference to Kubernetes native objects such as a Secret or ConfigMap. Therefore, it\u2019s required that the parent object exist before you create the GlobalObject custom resource<em>.<\/em><\/p>\n<p><strong>5. GlobalObject Custom\u00a0Resource<\/strong><\/p>\n<p>A custom resource is an extension of the Kubernetes API. After the custom resource is installed, users can create and access its objects using the <a href=\"https:\/\/kubernetes.io\/docs\/user-guide\/kubectl-overview\/\">kubectl<\/a>\u00a0client.<\/p>\n<h3>In Action<\/h3>\n<p>Let\u2019s see how the object replication occurs using Agumbe<strong>.<\/strong> Before the GlobalObject custom resource is created, let\u2019s verify the labels on the target namespaces and ensure there are no objects of the type we want to replicate.<\/p>\n<p>We shall see why labels on a namespace are important in the subsequent steps.<\/p>\n<pre><strong># GET LABELS ON NAMESPACES<\/strong><br><br><strong>$ for namespace in proxy app database logging metrics; do kubectl get ns $namespace -o=custom-columns=NAMESPACE:.metadata.name,LABELS:.metadata.labels; done<br>NAMESPACE   LABELS<\/strong><br>proxy       map[app=proxy]<br><strong>NAMESPACE   LABELS<\/strong><br>app         map[app=app]<br><strong>NAMESPACE   LABELS<\/strong><br>database    map[app=database]<br><strong>NAMESPACE   LABELS<\/strong><br>logging     map[infra.einstein.ai\/namespace:monitoring]<br><strong>NAMESPACE   LABELS<\/strong><br>metrics     map[infra.einstein.ai\/namespace:monitoring]<br><br><br><strong># GET SECRETS IN NAMESPACES<br><\/strong><br><strong>$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace; done<br><\/strong>NAME                      NAMESPACE<br>NAME                      NAMESPACE<br>NAME                      NAMESPACE<br>NAME                      NAMESPACE<br>NAME                      NAMESPACE<\/pre>\n<p>From the above results, the namespaces proxy, app, and database have a label key equal to<em> <\/em><strong><em>\u201capp\u201d<\/em> <\/strong>and the label value equal to their function; namespaces logging and metrics have a label key, value pair of <strong><em>\u201cinfra.einstein.ai\/namespace\u201d <\/em><\/strong>and<strong> <em>\u201cmonitoring\u201d<\/em> <\/strong>respectively.<\/p>\n<p>Now let\u2019s create a Secret object that we want to replicate in the admin namespace. Let\u2019s name it secret-sep-01-2020<strong> <\/strong>with the data key <strong><em>\u201chello\u201d<\/em> <\/strong>and the value set to the base64 encoded result of the string\u00a0<strong><em>\u201cworld\u201d<\/em><\/strong>.<\/p>\n<pre><strong># PARENT SECRETS MANIFEST<br><\/strong><br>---<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: secret-sep-01-2020<br>  namespace: admin<br>type: Opaque<br>data:<br>  hello: d29ybGQ=<\/pre>\n<p>On creating the secret, verify that it exists in the admin namespace.<\/p>\n<pre><strong># GET SECRET IN ADMIN NAMESPACE<br><br>$ kubectl get secret -n admin -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace<br>NAME                                          NAMESPACE<\/strong><br>secret-sep-01-2020                            admin<\/pre>\n<p>In this step, let\u2019s create the GlobalObject custom resource with the object reference as the Secret created in the previous step (secret-sep-01-2020).<strong> <\/strong>To scope the target namespaces, we use both specifications that are available to\u00a0us:<\/p>\n<ul>\n<li>.spec.matchLabels:<strong> <\/strong>Match all namespaces that have a matching label (key\/value pair)<\/li>\n<li>.spec.targetNamespaces: Explicitly specify the namespaces<\/li>\n<\/ul>\n<p>The GlobalObject spec shown below replicates the secret called secret-sep-01-2020<strong> <\/strong>as<strong> <\/strong>my-secret<strong> <\/strong>into,<\/p>\n<ul>\n<li>namespaces proxy, app, and database since they are described under\u00a0.spec.targetNamespaces<\/li>\n<li>namespaces logging and metrics\u00a0, since they match the key\/value pair specified under\u00a0.spec.matchLabels<\/li>\n<\/ul>\n<pre><strong># CUSTOM-RESOURCE MANIFEST<br><\/strong><br>---<br>apiVersion: infra.einstein.ai\/v1beta<br>kind: GlobalObject<br>metadata:<br>  name: global-secret<br>  namespace: admin<br>spec:<br>  type: Secret<br>  name: secret-sep-01-2020<br>  matchLabels:<br>  - key: infra.einstein.ai\/namespace<br>    value: monitoring<br>  targetName: my-secret<br>  targetNamespaces:<br>  - proxy<br>  - app<br>  - database<\/pre>\n<p>After we create the GlobalObject custom resource, verify that the secret has been replicated to the target namespaces. This can be accomplished by making a GET operation call on the object and using output filters to display the desired\u00a0fields.<\/p>\n<pre><strong># GET GLOBALOBJECT CUSTOM RESOURCE IN ADMIN NAMESPACE<\/strong><br><br><strong>$ kubectl get globalsecret -n admin<\/strong><br><strong>NAME              SECRET                 TARGET SECRET NAME<\/strong><br>global-secret     secret-sep-01-2020     my-secret<br><br><br><strong># GET SECRETS IN NAMESPACES<\/strong><br><br><strong>$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace; done<\/strong><br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               proxy<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               app<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               database<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               logging<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               metrics<br><br><br><strong># GET SECRETS (KEY, VALUE) IN NAMESPACES<\/strong><br><br><strong>$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,VALUE:.data; done<\/strong><br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             proxy       map[hello:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             app         map[hello:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             database    map[hello:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             logging     map[hello:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             metrics     map[hello:d29ybGQ=]<\/pre>\n<p>We can verify the object replication by observing the controller logs. The command displays the application logs from the Agumbe controller running in admin namespace.<\/p>\n<pre><strong>$ kubectl logs deployment\/agumbe -n admin -c logger<\/strong><br>[2020-09-14 22:21:59,912] [INFO    ] Controller priority 0, setting as elected leader<br>[2020-09-14 22:23:14,242] [INFO    ] [admin\/global-secret] CREATE: GlobalObject \"admin\/Secret\/global-secret\" created<br>[2020-09-14 22:23:14,300] [INFO    ] [admin\/global-secret] CREATE: Namespaces ['logging', 'metrics'] matched for label \"infra.einstein.ai\/namespace=monitoring\"<br>[2020-09-14 22:23:14,345] [INFO    ] [admin\/global-secret] CREATE: Secret admin\/secret-sep-01-2020 duped to proxy\/my-secret<br>[2020-09-14 22:23:14,351] [INFO    ] [admin\/global-secret] CREATE: Secret admin\/secret-sep-01-2020 duped to app\/my-secret<br>[2020-09-14 22:23:14,358] [INFO    ] [admin\/global-secret] CREATE: Secret admin\/secret-sep-01-2020 duped to database\/my-secret<br>[2020-09-14 22:23:14,363] [INFO    ] [admin\/global-secret] CREATE: Secret admin\/secret-sep-01-2020 duped to logging\/my-secret<br>[2020-09-14 22:23:14,368] [INFO    ] [admin\/global-secret] CREATE: Secret admin\/secret-sep-01-2020 duped to metrics\/my-secret<\/pre>\n<p>Next, let\u2019s rotate\/update the data stored in the Secret. Create a new Secret called secret-oct-01-2020<strong>.<\/strong> Let&#8217;s change the data key from <strong><em>\u201chello\u201d<\/em><\/strong> to <strong><em>\u201cworld\u201d<\/em><\/strong> and leave everything else the\u00a0same.<\/p>\n<pre><strong># PARENT SECRETS MANIFEST (UPDATED)<br><\/strong><br>---<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: secret-oct-01-2020<br>  namespace: admin<br>type: Opaque<br>data:<br>  world: d29ybGQ=<\/pre>\n<p>After you verify that the new secret is created, edit the GlobalObject custom resource<em> <\/em>and change the object reference from secret-sep-01-2020 to secret-oct-01-2020.<\/p>\n<pre><strong># CUSTOM-RESOURCE MANIFEST (UPDATED)<\/strong><br><br>-  name: secret-sep-01-2020<br>+  name: secret-oct-01-2020<\/pre>\n<p>After you apply the change, notice how the Secret data key got updated from <em>\u201c<\/em><strong><em>hello\u201d<\/em><\/strong> to <strong><em>\u201cworld\u201d<\/em><\/strong><em>.<\/em> There are no other changes to either the data value or the target object name. This confirms that the new Secret value was replicated into the namespaces through a PUT operation.<\/p>\n<pre><strong># GET GLOBALOBJECT CUSTOM RESOURCE IN ADMIN NAMESPACE<\/strong><br><br><strong>$ kubectl get globalsecret -n admin<\/strong><br><strong>NAME              SECRET                 TARGET SECRET NAME<\/strong><br>global-secret     secret-oct-01-2020     my-secret<br><br><br><strong># GET SECRETS IN NAMESPACES<\/strong><br><br><strong>$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace; done<\/strong><br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               proxy<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               app<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               database<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               logging<br><strong>NAME                    NAMESPACE<\/strong><br>my-secret               metrics<br><br><br><strong># GET SECRETS (KEY, VALUE) IN NAMESPACES<\/strong><br><br><strong>$ for namespace in proxy app database logging metrics; do kubectl get secret -n $namespace -o=custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,VALUE:.data; done<\/strong><br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             proxy       map[world:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             app         map[world:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             database    map[world:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             logging     map[world:d29ybGQ=]<br><strong>NAME                  NAMESPACE   VALUE<\/strong><br>my-secret             metrics     map[world:d29ybGQ=]<\/pre>\n<p>Verify that the change was replicated into the target namespaces by tailing the controller logs. Since we are modifying an existing object, notice that it\u2019s an UPDATE event and not a CREATE\u00a0event.<\/p>\n<pre><strong>$ kubectl logs deploy\/agumbe-57b98984bb-kc7qn -n admin -c logger<\/strong><br>[2020-09-14 22:49:06,372] [INFO    ] [admin\/global-secret] UPDATE: GlobalObject \"admin\/Secret\/global-secret\" updated<br>[2020-09-14 22:49:06,379] [INFO    ] [admin\/global-secret] UPDATE: Namespaces ['logging', 'metrics'] matched for label \"infra.einstein.ai\/namespace=monitoring\"<br>[2020-09-14 22:49:06,394] [INFO    ] [admin\/global-secret] UPDATE: Secret admin\/secret-sep-01-2020 duped to proxy\/my-secret<br>[2020-09-14 22:49:06,407] [INFO    ] [admin\/global-secret] UPDATE: Secret admin\/secret-sep-01-2020 duped to app\/my-secret<br>[2020-09-14 22:49:06,418] [INFO    ] [admin\/global-secret] UPDATE: Secret admin\/secret-sep-01-2020 duped to database\/my-secret<br>[2020-09-14 22:49:06,430] [INFO    ] [admin\/global-secret] UPDATE: Secret admin\/secret-sep-01-2020 duped to logging\/my-secret<br>[2020-09-14 22:49:06,438] [INFO    ] [admin\/global-secret] UPDATE: Secret admin\/secret-sep-01-2020 duped to metrics\/my-secret<\/pre>\n<h3>Conclusion<\/h3>\n<p>To summarize, Agumbe<strong> <\/strong>is a custom Kubernetes controller that provides cluster administrators an abstracted global view of objects scoped to the namespace. The controller enables us to share certain common objects across logical boundaries while being modular and cloud-agnostic at the same\u00a0time.<\/p>\n<h3>Acknowledgments<\/h3>\n<p>Agumbe was inspired by the <a href=\"https:\/\/kopf.readthedocs.io\/en\/latest\/\">Kopf<\/a> project. A huge shoutout to the amazingly passionate people behind this open-source project. I would also like to thank <a href=\"https:\/\/www.linkedin.com\/in\/arpeetkale\">Arpeet Kale<\/a> for his valuable feedback during the design and implementation phase, <a href=\"https:\/\/www.linkedin.com\/in\/dsiebold\">Dianne Siebold<\/a> for making this document look pretty, and the entire <a href=\"https:\/\/einstein.ai\/\">Einstein AI<\/a> engineering and leadership team (<a href=\"https:\/\/www.linkedin.com\/in\/dtisher\">Daniel Tisher<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/ivo\">Ivo Mihov<\/a>, and <a href=\"https:\/\/www.linkedin.com\/in\/indira-iyer-5604632\">Indira Iyer<\/a>) for the constant support and encouragement that we\u2019ve received from day\u00a0one.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=1fc2e1ddb3eb\" width=\"1\" height=\"1\" alt=\"\"><\/p>\n<hr>\n<p><a href=\"https:\/\/engineering.salesforce.com\/project-agumbe-share-objects-across-namespaces-in-kubernetes-1fc2e1ddb3eb\">Project Agumbe: Share Objects Across Namespaces in Kubernetes<\/a> was originally published in <a href=\"https:\/\/engineering.salesforce.com\/\">Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<p><a href=\"https:\/\/engineering.salesforce.com\/project-agumbe-share-objects-across-namespaces-in-kubernetes-1fc2e1ddb3eb?source=rss----cfe1120185d3---4\" target=\"_blank\" rel=\"noopener\">Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>At Salesforce, we use Kubernetes to orchestrate our services layer and recently ran into a use case where we wanted to apply and manage certain common objects across Kubernetes namespaces. Since there\u2019s no native solution to share objects across namespaces or the concept of a global object, we used Kubernetes\u2019 extensibility to solve the problem.&hellip; <a class=\"more-link\" href=\"https:\/\/fde.cat\/index.php\/2021\/02\/02\/project-agumbe-share-objects-across-namespaces-in-kubernetes\/\">Continue reading <span class=\"screen-reader-text\">Project Agumbe: Share Objects Across Namespaces in Kubernetes<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-223","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":454,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/looking-at-the-kubernetes-control-plane-for-multi-tenancy\/","url_meta":{"origin":223,"position":0},"title":"Looking at the Kubernetes Control Plane for Multi-Tenancy","date":"August 31, 2021","format":false,"excerpt":"The Salesforce Platform-as-a-Service Security Assurance team is constantly assessing modern compute platforms for security level and features. We use the insights from these research efforts to provide fast and comprehensive support to engineering teams who explore platform options that adequately support their security requirements. Unsurprisingly, Kubernetes is one of the\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":279,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/notary-a-certificate-lifecycle-management-controller-for-kubernetes\/","url_meta":{"origin":223,"position":1},"title":"Notary: A Certificate Lifecycle Management Controller for Kubernetes","date":"August 31, 2021","format":false,"excerpt":"Authors: Vaishnavi Galgali, Savithru Lokanath, Arpeet\u00a0KaleIntroductionAll services in the Einstein Vision and Language Platform use TLS\/SSL certificates to encrypt communication between microservices. The certificates are generated in AWS Certificate Manager (ACM) and stored in the AWS Secrets Manager in the form of keystores and truststores (private and public keys). Certificate\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":284,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/hadoop-hbase-on-kubernetes-and-public-cloud-part-i\/","url_meta":{"origin":223,"position":2},"title":"Hadoop\/HBase on Kubernetes and Public Cloud (Part I)","date":"August 31, 2021","format":false,"excerpt":"Authors: Dhiraj Hegde, Ashutosh Parekh, and Prashant\u00a0MurthyAt Salesforce, we run a large number of HBase and HDFS clusters in our own data centers. More recently, we have started deploying our clusters on Public Cloud infrastructure to take advantage of the on-demand scalability available there. As part of this foray onto\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":900,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/22\/data-clouds-lightning-fast-migration-from-amazon-ec2-to-kubernetes-in-6-months\/","url_meta":{"origin":223,"position":3},"title":"Data Cloud\u2019s Lightning-Fast Migration: From Amazon EC2 to Kubernetes in 6 Months","date":"July 22, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we delve into the journeys of distinguished engineering leaders. Today, we feature Archana Kumari, Director of Software Engineering at Salesforce. Archana leads our India-based Data Cloud Compute Layer team, which played a pivotal role in a recent transition from Amazon EC2 to Kubernetes for\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":644,"url":"https:\/\/fde.cat\/index.php\/2022\/10\/24\/how-salesforce-built-a-cloud-native-task-execution-service\/","url_meta":{"origin":223,"position":4},"title":"How Salesforce Built a Cloud-Native Task Execution Service","date":"October 24, 2022","format":false,"excerpt":"If you\u2019re paying attention to Salesforce technology, you\u2019ve no doubt heard about\u00a0Hyperforce, our new approach to deploying Salesforce on public cloud providers. Start with\u00a0a look at Hyperforce\u2019s architecture. There are many compelling reasons to move to Hyperforce, both for us and our customers. We\u2019re excited to do it in the\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":285,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/hadoop-hbase-on-kubernetes-and-public-cloud-part-ii\/","url_meta":{"origin":223,"position":5},"title":"Hadoop\/HBase on Kubernetes and Public Cloud (Part II)","date":"August 31, 2021","format":false,"excerpt":"The first part of this two part blog provided an introduction to concepts in Kubernetes and Public Cloud that are relevant to stateful application management. We also covered how Kubernetes and Hadoop features were leveraged to provide a highly available service. In this second part of the blog we will\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/223","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=223"}],"version-history":[{"count":1,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/223\/revisions"}],"predecessor-version":[{"id":244,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/223\/revisions\/244"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=223"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=223"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=223"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}