{"id":662,"date":"2022-12-14T17:17:05","date_gmt":"2022-12-14T17:17:05","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/12\/14\/how-salesforce-uses-immutable-infrastructure-in-hyperforce\/"},"modified":"2022-12-14T17:17:05","modified_gmt":"2022-12-14T17:17:05","slug":"how-salesforce-uses-immutable-infrastructure-in-hyperforce","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/12\/14\/how-salesforce-uses-immutable-infrastructure-in-hyperforce\/","title":{"rendered":"How Salesforce uses Immutable Infrastructure in Hyperforce"},"content":{"rendered":"

Credits go to: Armin Bahramshahry<\/a>, Software Engineering Principal Architect @ Salesforce\u00a0&\u00a0Shan Appajodu<\/a>, VP, Software Engineering for Developer Productivity Experiences<\/em> @ Salesforce. <\/em><\/p>\n

To leverage the scale and agility of the world\u2019s leading public cloud platforms, our Technology and Products team at Salesforce has worked together over the past few years to build a new generation of infrastructure platform for Salesforce \u2014 one that uses cloud-native tools, deployment patterns, security, and processes. We call this architecture Hyperforce<\/a>, and it\u2019s available across all of our product lines. Here is the overview blog on Hyperforce<\/a>.<\/p>\n

Software is not permanent; it changes\u00a0continuously<\/em>. At the same time, we have an obvious responsibility to keep our products up and running at all times and minimize the risk of these changes impacting our customers. Being comfortable with constant change requires\u00a0consistent<\/em>\u00a0quality. Instead of stopping the changes or accumulating them for longer, we opt to release them safely in smaller increments. This makes it easier to validate and maintain high trust for our customers and agility for our developers.\u00a0<\/p>\n

In this series of blogs, we would like to delve into how teams release changes safely with 1. Immutable infrastructure, 2. Infrastructure as Code, 3. Safe Releases, and 4. Developer experience in building secure and compliant services on Hyperforce.<\/p>\n

Since its inception in 1999, Salesforce has been running services on a static set of servers in our data centers. Changes (like operating system updates and service updates) on these hosts were managed by operators using tools such as Puppet, Apache Ambari, etc. These tools were geared towards\u00a0mutable<\/strong>\u00a0infrastructure\u2014that is, infrastructure that you modify \u201cin place,\u201d changing binaries and configuration on the hosts over time. Upgrade processes like these are difficult to reason with because failures can result in partial changes to hosts, making the recovery very complicated. The mutable nature of such deployments also results in a constant temptation for engineers to apply manual fixes for urgent issues. Unfortunately, these fixes are often forgotten, resulting in lingering drift in configuration.<\/p>\n

Instead, the idea of immutable infrastructure changes this by making our deployment mechanisms more idempotent and robust so that we can overcome issues like this. Hyperforce provides capabilities that enable us to roll out changes safely and immutably, like:<\/p>\n

Infrastructure as a Service:<\/strong> Software-driven Virtualized infrastructure, where every part of the infrastructure, compute, network, and storage can be provisioned and managed dynamically via API calls.Elasticity: <\/strong>The ability to elastically add or remove infrastructure, based on demand.<\/p>\n

Combined with the above capabilities, Virtual Machines (VMs) and Containers allow us to embrace a new immutable form of deployment. If a change is made to the code or configuration, a new image for the entire VM or Container is built and deployed as a unit, replacing older VMs and Containers (rather than changing them in place).<\/p>\n

The phrase \u201cimmutable infrastructure\u201d sometimes causes confusion with our customers, so to be perfectly clear, \u201cImmutable\u201d doesn\u2019t refer to the contents of our services. Obviously, the data you enter into Salesforce is\u00a0highly<\/em>\u00a0mutable (you can change it whenever you want!).<\/p>\n

Instead, \u201cimmutable\u201d refers to the resources (servers, containers, services, networks, and their respective code or configuration) that never change after deployment. This means that once the resource is in place, we replace it wholesale with an updated version rather than making patches or changes to it directly in our production environment. Immutable deployments are a way of managing infrastructure that moves the unit of update from an individual\/set of binaries to an entire compute unit.\u00a0<\/p>\n

Immutable deployments require that:<\/p>\n

Setup and deployment for every part and layer of your infrastructure are automated. This is made possible with the public cloud Infrastructure-as-a-Service capabilities.You make zero manual changes to any part of a system once it\u2019s deployed. All changes to code or configuration are applied by deploying a new system and tearing down the old one. This is made possible by the Public cloud elasticity (you don\u2019t have to pay the cost of keeping twice as many servers around all the time for this).<\/p>\n

Immutable Infrastructure and deployments have several significant advantages:<\/p>\n

Replacing a system at the lowest level forces you to depend on automation at every step of your deployments. This enforces repeatability and ensures that environments can be managed with minimal human intervention.Completely replacing, instead of updating, an existing part of your infrastructure makes deployments less complex. As the desired state of the world is known, edge cases are reduced.Immutable deployments are safer. An immutable deployment unit can be entirely tested in test and staging environments and then gradually released to Salesforce customers.Immutable deployments also make patching far easier. The patch process is built into the base Operating System image baking pipeline and its deployment, using the same automation for code or configuration changes and related safety measures. Immutable deployments completely\u00a0replace<\/em>\u00a0patching.Immutable deployments result in a more secure environment. If we are rebuilding the system for each deployment, we are constantly erasing any foothold an attacker may have gained and requiring them to try to regain that surface.<\/p>\n

In Hyperforce, we rely heavily on all of these benefits:<\/p>\n

Infrastructure-as-Code (IaC):<\/strong>\u00a0All aspects of the service are managed via IaC, from build to deployment of the service and its related resources.\u00a0Terraform<\/a>\u00a0is used via\u00a0Spinnaker<\/a>\u00a0pipelines, where the state of infrastructure is managed in the Terraform service for each Hyperforce Instance. Managing infrastructure as code enables the safety we require. (Stay tuned for a future blog post on Hyperforce IaC!)<\/p>\n

VM Deployments:<\/strong>\u00a0When any part of the software needs to be updated, a new machine image is baked with the changes. Instead of deploying an updated binary into an existing EC2 instance, a new EC2 Instance is started with the new machine image, and the load balancer is pointed to the new server. The old server is then removed. Patching of existing servers is\u00a0never<\/em>\u00a0permitted.<\/p>\n

Container Deployments:<\/strong>\u00a0Leveraging\u00a0Kubernetes<\/a>, container deployments are immutable by default. The continuous integration pipelines build new container images, which are then deployed to Kubernetes via Spinnaker pipelines. The Kubernetes nodes, being EC2 instances, are replaced with new EC2 Instances running updated versions of the Base Operating Systems Image.\u00a0<\/p>\n

Infrastructure Configuration:<\/strong>\u00a0Leveraging Terraform and Kubernetes, the infrastructure configuration is also maintained in code. Environment-specific configuration is declared in code and wired via Spinnaker pipelines.<\/p>\n

Zero Downtime Deployments:\u00a0<\/strong>In Hyperforce, teams adopt safe Zero Downtime deployment practices such as\u00a0Blue\/Green Deployments<\/a>\u00a0to ensure the changes are tested on the new service instance before switching customer traffic. More advanced services perform \u201cCanary\u201d deployments, where a small percentage of customer traffic is routed to new service instances and observed for any regressions before opening to the entire traffic stream. This decreases the blast radius and impact of any breaking change. Enabling Blue\/green for stateless services might be trivial, but doing the same for stateful services (such as data stores) requires additional work in coordinating the state changes, preserving the ability to roll back, etc.<\/p>\n

Capacity Awareness:<\/strong>\u00a0Planning for sufficient capacity when using strategies such as blue\/green deployment is important. It may not seem like a problem when you\u2019re doing this for a single service, but capacity planning is required when you\u2019re doing it at scale across multiple services simultaneously. At Hyperforce scale, efficient capacity planning and reservation are critical for the cost-to-serve and availability of our services.<\/p>\n

Feature Flags:\u00a0<\/strong>Enabling teams to release changes that are conditionally made available to the customers provides another level of safety. This allows teams to release the changes early without impacting the customers. One might argue that Feature flags are an \u201canti-pattern\u201d to immutability. However, when feature flags are documented and tested in all environments and follow proper change control processes, they prove to be very useful in releasing changes frequently (enabling early feedback) and safely, minimizing customer impact.<\/p>\n

As you can see, an immense amount is involved in the immutable infrastructure concept. Still, it\u2019s been a massive leg up for our ability to deliver secure, highly available software for our customers.<\/p>\n

In the next part, we will discuss how Infrastructure-as-Code enables changes to our infrastructure to follow the same lifecycle as any other part of our software system \u2014 validation, peer review, automated testing, staging, and gradual rollout. Stay tuned.<\/p>\n

The post How Salesforce uses Immutable Infrastructure in Hyperforce<\/a> appeared first on Salesforce Engineering Blog<\/a>.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

Credits go to: Armin Bahramshahry, Software Engineering Principal Architect @ Salesforce\u00a0&\u00a0Shan Appajodu, VP, Software Engineering for Developer Productivity Experiences @ Salesforce. To leverage the scale and agility of the world\u2019s leading public cloud platforms, our Technology and Products team at Salesforce has worked together over the past few years to build a new generation of… Continue reading How Salesforce uses Immutable Infrastructure in Hyperforce<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-662","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":538,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/01\/behind-the-scenes-of-hyperforce-salesforces-infrastructure-for-the-public-cloud\/","url_meta":{"origin":662,"position":0},"title":"Behind the Scenes of Hyperforce: Salesforce\u2019s Infrastructure for the Public Cloud","date":"February 1, 2022","format":false,"excerpt":"Salesforce has been running cloud infrastructure for over two decades, bringing companies and their customers together. When Salesforce first started out in 1999, the world was very different; back then, the only practical way to provide our brand of Software-As-A-Service was to run everything yourself\u200a\u2014\u200anot just the software, but the\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":870,"url":"https:\/\/fde.cat\/index.php\/2024\/05\/23\/hyperforce-behind-the-scenes-ushering-in-a-new-age-of-ai-driven-cloud-scalability\/","url_meta":{"origin":662,"position":1},"title":"Hyperforce Behind the Scenes: Ushering in a New Age of AI-Driven Cloud Scalability","date":"May 23, 2024","format":false,"excerpt":"In our latest edition of our \u201cEngineering Energizers\u201d Q&A series, we meet Paul Constantinides, Executive Vice President of Engineering. With an extensive history in the technology industry, and a 20-year career at Salesforce, Paul leads the Hyperforce Platform Services team and is responsible for developing Hyperforce, Salesforce\u2019s public cloud-native architecture\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":544,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/22\/the-unified-infrastructure-platform-behind-salesforce-hyperforce\/","url_meta":{"origin":662,"position":2},"title":"The Unified Infrastructure Platform Behind Salesforce Hyperforce","date":"February 22, 2022","format":false,"excerpt":"If you\u2019re paying attention to Salesforce technology at all, you\u2019ve no doubt heard about Hyperforce, our new approach to deploying Salesforce on public cloud providers. As with any big announcement, it can be a little hard to cut through the hyperbolic language and understand what\u2019s going\u00a0on. In this blog series,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":625,"url":"https:\/\/fde.cat\/index.php\/2022\/08\/30\/hyperpacks-using-buildpacks-to-build-hyperforce\/","url_meta":{"origin":662,"position":3},"title":"Hyperpacks: Using Buildpacks to Build Hyperforce","date":"August 30, 2022","format":false,"excerpt":"At Salesforce we regularly use our products and services to scale our own business. One example is Buildpacks, which we created nearly a decade ago and is now a part of Hyperforce. Hyperpacks are an innovative new way of using Cloud Native Buildpacks (CNB) to manage our public cloud infrastructure.\u00a0\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":585,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/22\/the-unified-infrastructure-platform-behind-salesforce-hyperforce-2\/","url_meta":{"origin":662,"position":4},"title":"The Unified Infrastructure Platform Behind Salesforce Hyperforce","date":"February 22, 2022","format":false,"excerpt":"If you\u2019re paying attention to Salesforce technology at all, you\u2019ve no doubt heard about\u00a0Hyperforce, our new approach to deploying Salesforce on public cloud providers. As with any big announcement, it can be a little hard to cut through the\u00a0hyperbolic language and understand what\u2019s going on. In this blog series, we\u2019ll\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":688,"url":"https:\/\/fde.cat\/index.php\/2023\/03\/07\/automated-environment-build-salesforces-secret-sauce-for-rapid-cloud-expansion\/","url_meta":{"origin":662,"position":5},"title":"Automated Environment Build: Salesforce\u2019s Secret Sauce for Rapid Cloud Expansion","date":"March 7, 2023","format":false,"excerpt":"Around the world, companies must satisfy global compliance regulations or face pricey fines, where failure to comply results in 2.71 higher costs than the cost to comply. For example, Fortune 500 companies are projected to lose $8 billion per year as a result of GDPR non-compliance. In response, Salesforce created\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/662","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=662"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/662\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=662"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=662"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=662"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}