{"id":289,"date":"2021-08-31T14:40:23","date_gmt":"2021-08-31T14:40:23","guid":{"rendered":"https:\/\/fde.cat\/?p=289"},"modified":"2021-08-31T14:40:23","modified_gmt":"2021-08-31T14:40:23","slug":"journey-to-spinnaker-deployment-orchestration","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/journey-to-spinnaker-deployment-orchestration\/","title":{"rendered":"Journey to Spinnaker Deployment Orchestration"},"content":{"rendered":"<h3><strong>Introduction<\/strong><\/h3>\n<p>Spinnaker has been gaining popularity as a Continuous Deployment (CD) solution. It certainly offers many useful features supporting deployment pipelines including but not limited to access permission control, automatic and manual gated deployment configurations when moving from one phase to the next. Having said that, if you are operating a complex legacy CI\/CD system, what might you need to consider before a migration to Spinnaker? How can you make sure the process is smooth for all stakeholders?<\/p>\n<p>This post is a recap of my <a href=\"https:\/\/player.vimeo.com\/video\/475918473\">presentation at Spinnaker Summit 2020<\/a> in which I discussed the challenges we encountered in our legacy deployment orchestration system and how we migrated to Spinnaker.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1000\/1*kPlC462VrHXzmtGgD-wF3g.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><\/figure>\n<h3>Background<\/h3>\n<p>We (build and release team) previously used TeamCity as both the CI and CD solutions for many of our platform services. However, as our build and release activities expanded over the years, the challenges facing CI\/CD got more complex. <\/p>\n<p>For example, TeamCity was able to access production environments, thus failing the least privilege security principle. Build and deployment projects were configured separately to accommodate increasing engineering activities, making it difficult to enforce consistent permission and deployment policies. When it came to release management, gating deployments from dev to staging to production required constant monitoring, manual review and approval. As a result, deployment orchestration enhancement was becoming a much higher priority.<\/p>\n<p> <strong>Before<\/strong><\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*0DTcDQtb5qoFbfpWpH-DKQ.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><\/figure>\n<p><strong>After<\/strong><\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*mGTUCyeJLhe7vdtP7Oxz_A.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><\/figure>\n<ul>\n<li>TeamCity remains a CI tool while Spinnaker manages\u00a0CD.<\/li>\n<\/ul>\n<h3>How<\/h3>\n<p>Once we decided we needed to move forward with migrating legacy CI\/CD to Spinnaker, the question became, \u201cWhere to start?!\u201d There are a few phases we went through to ensure a smooth transition experience for all stakeholders including developers, SRE, quality engineers, performance engineers, and project managers. These phases include Spinnaker installation, preparing deployment configuration changes, and constructing new deployment pipelines.<\/p>\n<h3>Spinnaker Installation Key\u00a0Features<\/h3>\n<p>The core features of Spinnaker, including Application Management, Application Deployment, Pipelines and Kubernetes Provider, are configured by infrastructure engineering.<\/p>\n<p>In addition, we added Authz, Role Based Access Control (RBAC). This feature enables access restriction against applications, pipelines, and accounts and provides custom permissions for automatic and manual pipeline triggers. One key reason we adopted this feature was to fulfill our security requirements.<\/p>\n<p> In order to meet our change control requirements, we added Pipeline as Code as well. This feature allows defining deployment pipeline configuration in Github. As a result, pipeline changes require pull request reviews and validation. Plus, we are able to restrict who can apply a pipeline code\u00a0change.<\/p>\n<h3>Preparation<\/h3>\n<p>Given that many deployment configurations were operating in TeamCity for years, how did we move them over to Spinnaker while minimizing interruption? <\/p>\n<p> First, we identified all active deployment jobs in TeamCity. It had deployment configurations for 20+ applications\/services in 5+ environments from dev, staging, performance to production. Among deployment jobs, we looked for configurations that included build steps that called Helm commands. From these configurations, we identified Helm chart paths for services to deploy. These same Helm charts are now used by Spinnaker.<\/p>\n<p> Once we had a solid understanding of the actual TeamCity configurations we needed to migrate, the next step was a very important aspect of migration: communicating and collaborating with all stakeholders.<\/p>\n<p>We<strong> <\/strong>presented a proposal to all stakeholders about changes in architecture, policies, and process and engaged stakeholders about detailed configuration changes and expectations. We emphasized the following key points in communications:<\/p>\n<ul>\n<li><strong>Standardization<\/strong>: Build and Release team owns pipeline templates, making it easier to support org, onboarding new services and\u00a0audit.<\/li>\n<li><strong>Opinionated Permissions<\/strong>: We provide well-defined user groups &amp; access control list (ACL) by configuring opinionated permissions.<\/li>\n<li><strong>Pipelines as Code<\/strong>: Changes to configurations are reviewed through the Pull Request\u00a0process.<\/li>\n<li><strong>Separate CI &amp; CD<\/strong>: Use TeamCity for build &amp; test only; remove the ability to deploy. CD is only managed in Spinnaker, drastically reducing the chances of an unintended change against production.<\/li>\n<li><strong>Automate Releases<\/strong>: We integrate with End-to-End tests in pipelines, coupled with roll-backs and issue tracking integration.<\/li>\n<\/ul>\n<h3>Construction<\/h3>\n<p>Once the migration proposal went through discussion, feedbacks, and revision, we were ready to work on constructing our matching deployment pipelines in Spinnaker. Pipelines in Spinnaker must meet certain conditions.<\/p>\n<p>For each active application\/service in production, pipelines are constructed from standardized templates. They must carry forward and enforce the same release process as before migration. The move to Spinnaker made this much easier to enforce. <\/p>\n<p> Generally speaking, pipeline triggers are implemented as\u00a0follows:<\/p>\n<ul>\n<li>Dev pipelines auto deploy when a new dev build is published.<\/li>\n<li>Staging pipelines trigger if dev deployment tests\u00a0pass.<\/li>\n<li>Prod pipeline trigger conditions include passing staging deployment tests and manual judgment by release manager or authorized personnel.<\/li>\n<\/ul>\n<p>In order to enforce pipeline change control, we configured Pipeline as Code and dinghy service. Pipeline code is defined in Github. New pipeline commits automatically apply changes in Spinnaker pipelines. To safeguard pipeline code, we enabled Pull Request (PR) review with validation tests. Only authorized personnel are allowed to approve and merge PRs.<\/p>\n<p>To help standardize pipelines, we defined common pipeline stage modules including Bake, Create Namespace, Deploy Manifest, Restricted Window, Integration Tests. These modules can be shared between applications and pipelines.<\/p>\n<p>Here is an example of a typical deployment pipeline. It includes common stages such as Bake, Manual Judgment, Deploy Manifest, Post deployment testing, and Datadog notification. These stages are generally similar among pipelines except for pipeline-specific artifacts and accounts.<\/p>\n<figure><img decoding=\"async\" alt=\"\" src=\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*hXvtAp_TDfAf76qcoriomg.png?w=750&#038;ssl=1\" data-recalc-dims=\"1\"><\/figure>\n<p>Next, we coordinated moving deployment jobs from TeamCity to Spinnaker. Our general approach was to start by testing a Spinnaker pipeline. Once that worked, we disabled its corresponding deployment job in TeamCity and informed the service dev team about pipeline update. We started with a pilot program covering a few services. Once the pilot program succeeded, we started implementing Spinnaker pipelines for the rest of the services under our purview. During the migration, we announced our progress to all stakeholders on a regular basis.<\/p>\n<p>While working on pipeline code, we noticed varied complexity due to service dependencies and requirements. Therefore, we had to revisit and improve our shareable modules to allow easier module re-use.<\/p>\n<p>While communicating progress with all dev teams, we also updated them with instructions about how pipelines work in Spinnaker compared to TeamCity. In particular, we highlighted the benefits offered by Spinnaker deployments including release cadence improvement, streamlining processes for ad hoc and hot fix releases, automatic post deployment validation tests, and preventing unexpected release failure due to lack of change control in legacy CI\/CD. As a result, we are able to achieve more reliable and repeatable deployments.<\/p>\n<h3>Conclusion<\/h3>\n<p>The main goal of this journey, which was a success, was to separate CI and CD. TeamCity remains CI while CD is managed by Spinnaker. This allows for easier enforcement of deployment security policies, better change control, and standardized pipelines, all of which streamlines the release process for both scheduled and unscheduled deployments.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=743bbda6c01c\" width=\"1\" height=\"1\" alt=\"\"><\/p>\n<hr>\n<p><a href=\"https:\/\/engineering.salesforce.com\/journey-to-spinnaker-deployment-orchestration-743bbda6c01c\">Journey to Spinnaker Deployment Orchestration<\/a> was originally published in <a href=\"https:\/\/engineering.salesforce.com\/\">Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<p><a href=\"https:\/\/engineering.salesforce.com\/journey-to-spinnaker-deployment-orchestration-743bbda6c01c?source=rss----cfe1120185d3---4\" target=\"_blank\" rel=\"noopener\">Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Spinnaker has been gaining popularity as a Continuous Deployment (CD) solution. It certainly offers many useful features supporting deployment pipelines including but not limited to access permission control, automatic and manual gated deployment configurations when moving from one phase to the next. Having said that, if you are operating a complex legacy CI\/CD system,&hellip; <a class=\"more-link\" href=\"https:\/\/fde.cat\/index.php\/2021\/08\/31\/journey-to-spinnaker-deployment-orchestration\/\">Continue reading <span class=\"screen-reader-text\">Journey to Spinnaker Deployment Orchestration<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-289","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":662,"url":"https:\/\/fde.cat\/index.php\/2022\/12\/14\/how-salesforce-uses-immutable-infrastructure-in-hyperforce\/","url_meta":{"origin":289,"position":0},"title":"How Salesforce uses Immutable Infrastructure in Hyperforce","date":"December 14, 2022","format":false,"excerpt":"Credits go to: Armin Bahramshahry, Software Engineering Principal Architect @ Salesforce\u00a0&\u00a0Shan Appajodu, VP, Software Engineering for Developer Productivity Experiences @ Salesforce. To leverage the scale and agility of the world\u2019s leading public cloud platforms, our Technology and Products team at Salesforce has worked together over the past few years to\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":890,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/02\/hyperforces-template-for-enhancing-developer-workflow-inside-the-7-pillars-of-agile-development\/","url_meta":{"origin":289,"position":1},"title":"Hyperforce\u2019s Template for Enhancing Developer Workflow: Inside the 7 Pillars of Agile Development","date":"July 2, 2024","format":false,"excerpt":"Written by Armin Bahramshahry and Shan Appajodu. Hyperforce is a pivotal infrastructure platform for Salesforce, enhancing global service delivery through top public cloud platforms for increased safety, scalability, and agility. Hyperforce enabled rollout of new innovations like Data Cloud and boosted the global scalability of Salesforce\u2019s Core CRM. To help\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":830,"url":"https:\/\/fde.cat\/index.php\/2024\/02\/26\/how-dotslash-makes-executable-deployment-simpler\/","url_meta":{"origin":289,"position":2},"title":"How DotSlash makes executable deployment simpler","date":"February 26, 2024","format":false,"excerpt":"Andres Suarez and Michael Bolin, two software engineers at Meta, join Pascal Hartig (@passy) on the Meta Tech Podcast to discuss the ins and outs of DotSlash, a new open source tool from Meta. DotSlash takes the pain out of distributing binaries and toolchains to developers. Instead of committing large,\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":619,"url":"https:\/\/fde.cat\/index.php\/2022\/08\/10\/architectural-principles-for-high-availability-on-hyperforce\/","url_meta":{"origin":289,"position":3},"title":"Architectural Principles for High Availability on Hyperforce","date":"August 10, 2022","format":false,"excerpt":"Infrastructure and software failures will happen. We idolize four 9s (99.99%) availability. We know we need to optimize and improve Recovery-Time-Objective (RTO, the time it takes to restore service after a service disruption) and Recovery-Point-Objective (RPO, the acceptable data loss measured in time). But how can we actually deliver high\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":282,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/sre-weekly-issue-261\/","url_meta":{"origin":289,"position":4},"title":"SRE Weekly Issue #261","date":"August 31, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, StackHawk: Join Snyk and StackHawk on March 18 as they walk through how to use Software Composition Analysis (SCA) and Dynamic Application Security Testing (DAST) in CI\/CD to ship more secure applications. http:\/\/sthwk.com\/snyk-stackhawk-webinar Articles What Do Fighter Pilots and Incident Management Have\u2026","rel":"","context":"In &quot;SRE&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":688,"url":"https:\/\/fde.cat\/index.php\/2023\/03\/07\/automated-environment-build-salesforces-secret-sauce-for-rapid-cloud-expansion\/","url_meta":{"origin":289,"position":5},"title":"Automated Environment Build: Salesforce\u2019s Secret Sauce for Rapid Cloud Expansion","date":"March 7, 2023","format":false,"excerpt":"Around the world, companies must satisfy global compliance regulations or face pricey fines, where failure to comply results in 2.71 higher costs than the cost to comply. For example, Fortune 500 companies are projected to lose $8 billion per year as a result of GDPR non-compliance. In response, Salesforce created\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/289","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=289"}],"version-history":[{"count":1,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/289\/revisions"}],"predecessor-version":[{"id":421,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/289\/revisions\/421"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=289"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=289"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=289"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}