{"id":719,"date":"2023-05-23T22:49:51","date_gmt":"2023-05-23T22:49:51","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/05\/23\/automation-at-scale-migrating-200000-machines-from-centos-7-to-rhel-9\/"},"modified":"2023-05-23T22:49:51","modified_gmt":"2023-05-23T22:49:51","slug":"automation-at-scale-migrating-200000-machines-from-centos-7-to-rhel-9","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/05\/23\/automation-at-scale-migrating-200000-machines-from-centos-7-to-rhel-9\/","title":{"rendered":"Automation at Scale: Migrating 200,000 Machines from CentOS 7 to RHEL 9"},"content":{"rendered":"

When a legacy operating system (OS) approaches its end-of-support date, some organizations will upgrade their OS as fast as possible. Others may kick the can down the road, delaying any headaches they might encounter during the upgrade process.<\/a><\/p>\n

Six years ago, Salesforce Engineering put the pedal to the metal, migrating to CentOS 7, an open-source operating system based on the Red Hat Enterprise Linux (RHEL) source code. Back then, the team\u2019s biggest challenge for upgrading was the tedious and time-consuming manual labor involved, ranging from determining the health of machines to scheduling their OS upgrade.<\/p>\n

As CentOS 7 races to the end of its operational runway,<\/a> Salesforce Engineering tackles its new OS upgrade task head-on. This time around, the team faces a much bigger hurdle: migrating 200,000 machines from the current OS to RHEL 9<\/a>, an advanced OS that delivers enhanced performance, boosts security, and drives integration of next-generation hardware.<\/p>\n

Due to the sheer number of systems the team must migrate, a manual conversion is not tenable. Instead, the team will use automation \u2014 enabling them to eliminate downtime, ensure machine health, improve visibility, and power parallelization so that machines are efficiently and reliably ported to RHEL 9 faster \u2014 and more reliably \u2014 than ever.<\/p>\n

Tyson Lutz, Sr. Vice President of Software Engineering (foreground), leads the RHEL 9 implementation team (background).<\/em><\/p>\n

How does migrating to RHEL 9 deliver an improved OS capability?<\/strong><\/p>\n

From integrating cutting-edge processors to stopping bugs in their tracks to boosting security, upgrading Salesforce\u2019s OS to RHEL 9 provides a durable enterprise-grade OS platform and unlocks many concrete benefits for Salesforce Engineering and our customers.<\/p>\n

Enables essential technology.<\/strong> Salesforce engineers require the latest hardware to harness new software innovations for our customers. CentOS 7 cannot sustain highly advanced processors, however, RHEL 9 has first-class support for next-generation ARM-based architectures, delivering 20-30% in savings, while providing the same level of performance.<\/p>\n

Provides for every use case<\/strong>. Salesforce customers may have highly specific workloads that require significant computing power. Other customers run workloads that require less processing needs. RHEL 9 now backs both use cases \u2014 enabling customers to select the environment that best fits their needs.<\/p>\n

Finds and fixes bugs faster. <\/strong>Historically, the team may have spent a week working to determine the root cause of a unique problem. As the team learns about that bug and fixes it, they cannot apply their knowledge to fix it again because the bug does not reappear. Moving to RHEL 9 provides a new level of customer support, whereby Red Hat engineers can help Salesforce Engineering pinpoint issues in mere minutes \u2014<\/em> enabling rapid fixes.<\/p>\n

Improves security posture<\/strong>. Outdated technology may lead to compromised cybersecurity \u2014<\/em> potentially leading to ransomware or damaging malware attacks that require costly rebuilds. RHEL 9 takes security to the next level<\/a> for Salesforce, leveraging technology that governments around the world use to ensure heightened levels of security and satisfy stringent security compliance requirements.<\/p>\n

Under the hood: How does automation help drive the OS migration?<\/strong><\/p>\n

As they plan their transition to RHEL 9, the conversion team uses four key automation-driven tools:<\/p>\n

The first is their conversion playbook<\/em>, which defines an automated schedule, detailing when machines should be converted.<\/p>\n

Next, the team\u2019s graph database<\/em> \u2014 a fleetwide management and control system \u2014 kicks off the migration process.<\/p>\n

Together, the conversion playbook and graph database inform the conversion orchestrator system \u2014 <\/em>which determines the machines that should move over and when. The orchestrator then scales the migration across Salesforce\u2019s 200,000 systems at a measured rate. By staggering conversions \u2014<\/em> about 5,000 machines daily \u2014 the vast majority of machines remain active, ensuring a seamless and transparent experience for Salesforce customers.<\/p>\n

After each batch is converted, an automated configuration management system<\/em> ensures that the machines satisfy their normal specifications on the new OS and performs upgrades as needed.<\/p>\n

During off-peak system usage hours, automation kicks into high-gear, rapidly converting systems to RHEL 9 until all hosts are migrated.<\/em><\/p>\n

What are automation\u2019s biggest benefits?<\/strong><\/p>\n

In 2017, Salesforce Engineering migrated to CentOS manually, a challenging experience that required the conversion team to navigate numerous productivity roadblocks. Automation\u2019s many benefits alleviates those issues, enabling the team to pave a much smoother path for a RHEL 9 migration, ahead of CentOS 7\u2019s end-of-support date, with time to spare.<\/p>\n

Eliminates downtime.<\/strong> During the previous OS conversion, teams of engineers needed to coordinate efforts across time zones to manually solve machine issues. Automation eliminates that productivity lag \u2014 delivering an always-on capability that instantaneously remediates issues \u2014 so machines can smoothly onboard to RHEL 9. <\/p>\n

Ensures system-wide health<\/strong>. During the manual conversion to CentOS 7, the team required significant communication and hands-on coordination to define the schedule for machine migration and move the hardware over. Any missteps could have disrupted machine health \u2014 potentially impacting Salesforce customers\u2019 productivity. <\/p>\n

After confirming the machines\u2019 health and readiness, the automation system converts them to the new OS on a scheduled basis. Should one of the machines need a software update, the system performs the fix, unless a human technician must fix a physical anomaly. Once repaired, the machine automatically migrates to the new OS and the process repeats with the next 5,000 machines until all 200,000 machines successfully convert to RHEL 9.<\/p>\n

Enables visibility. <\/strong>Manual conversions have historically introduced visibility challenges, where the team lacked insights on machine fleet size, which machines required migration, and if machines were operational. Following the CentOS conversion, the team scrambled to find and fix machines to avoid any outages. <\/p>\n

Now, using automation, real-time monitoring and health metrics, the team has complete visibility of the machine fleet, from its size to its health to which machines may be converted.<\/p>\n

Powers parallelization. <\/strong>During a manual migration, a human technician can only perform one task at a time, such as repairing a system or migrating it to RHEL 9. <\/p>\n

Conversely, automation allows engineers to \u201cset and forget\u201d the system, where it performs infinite parallelization of OS migration tasks \u2014 operating at a scale that human engineers cannot match. For example, the system may be tasked with scheduling 50 machines for migration to RHEL 9. After scanning them, it could learn that 25 require repair. As it performs the fixes, it simultaneously converts the remaining 25 machines.<\/p>\n

Learn more<\/strong><\/h4>\n

Hungry for more automation stories? Read this blog<\/a> to explore how India\u2019s Salesforce Engineering team uses automation to slash customer processing time from hours to seconds.<\/p>\n

Stay connected \u2014 join our Talent Community<\/a>!<\/p>\n

Check out our Technology and Product teams<\/a> to learn how you can get involved.<\/p>\n

The post Automation at Scale: Migrating 200,000 Machines from CentOS 7 to RHEL 9<\/a> appeared first on Salesforce Engineering Blog<\/a>.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

When a legacy operating system (OS) approaches its end-of-support date, some organizations will upgrade their OS as fast as possible. Others may kick the can down the road, delaying any headaches they might encounter during the upgrade process. Six years ago, Salesforce Engineering put the pedal to the metal, migrating to CentOS 7, an open-source… Continue reading Automation at Scale: Migrating 200,000 Machines from CentOS 7 to RHEL 9<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-719","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":799,"url":"https:\/\/fde.cat\/index.php\/2023\/12\/05\/explaining-salesforces-large-scale-migration-to-git-how-we-enhanced-developer-productivity\/","url_meta":{"origin":719,"position":0},"title":"Explaining Salesforce\u2019s Large-Scale Migration to Git: How We Enhanced Developer Productivity","date":"December 5, 2023","format":false,"excerpt":"By Patrick Calahan and Scott Nyberg As new developer productivity technologies emerge, small and nimble enterprises with newer codebases swiftly embrace innovation. Conversely, larger organizations, rooted in larger and aging codebases, face obstacles replacing legacy technologies. Salesforce faced such a challenge with its primary Source Code Management (SCM) system. For\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":828,"url":"https:\/\/fde.cat\/index.php\/2024\/02\/20\/unlocking-hyperforce-migration-innovative-solutions-for-a-smooth-transition-to-the-cloud\/","url_meta":{"origin":719,"position":1},"title":"Unlocking Hyperforce Migration: Innovative Solutions for a Smooth Transition to the Cloud","date":"February 20, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we delve into the experiences and expertise of Salesforce Engineering leaders. Today, we\u2019re meeting Mahamadou Sylla, a Senior Member of the Technical Staff at Salesforce Engineering. Mahamadou is a key member of our Hyperforce\u2019s Bill of Materials (BOM) team, which assists internal teams in\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":810,"url":"https:\/\/fde.cat\/index.php\/2024\/01\/09\/implementing-salesforces-largest-database-upgrade-inside-the-migration-to-hbase-2\/","url_meta":{"origin":719,"position":2},"title":"Implementing Salesforce\u2019s Largest Database Upgrade: Inside the Migration to HBase 2","date":"January 9, 2024","format":false,"excerpt":"Written by Viraj Jasani and Andrew Purtell Data is the engine behind Salesforce operations, helping our customers make better decisions on a daily basis. The Big Data Storage (BDS) team, a key part of Salesforce\u2019s engineering organization, deploys arguably one of the largest distributed database production footprints. This infrastructure is\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":306,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/blazing-the-trail-one-year-with-openjdk-11\/","url_meta":{"origin":719,"position":3},"title":"Blazing the Trail: One Year with OpenJDK 11","date":"August 31, 2021","format":false,"excerpt":"Early Adoption of Java Runtime Innovations in Production at\u00a0ScaleCo-written by Donna\u00a0ThomasIntroductionSalesforce was one of the first major enterprises to adopt OpenJDK 11 at scale in production, starting our adoption journey shortly after its release in late 2018. Cutting edge? Sure. Safe? Absolutely. You might not know this, but Salesforce has\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":866,"url":"https:\/\/fde.cat\/index.php\/2024\/05\/15\/revealing-einsteins-blueprint-for-creating-the-new-unified-ai-platform-from-siloed-legacy-stacks\/","url_meta":{"origin":719,"position":4},"title":"Revealing Einstein\u2019s Blueprint for Creating the New, Unified AI Platform from Siloed Legacy Stacks","date":"May 15, 2024","format":false,"excerpt":"In our insightful \u201cEngineering Energizers\u201d Q&A series, we delve into the inspiring journeys of engineering leaders who have achieved remarkable success in their specific domains. Today, we meet Indira Iyer, Senior Vice President of Salesforce Engineering, leading Salesforce Einstein development. Her team\u2019s mission is to build Salesforce\u2019s next-gen AI Platform,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":683,"url":"https:\/\/fde.cat\/index.php\/2023\/02\/21\/what-is-the-secret-behind-increasing-salesforces-developer-velocity\/","url_meta":{"origin":719,"position":5},"title":"What is the Secret Behind Increasing Salesforce\u2019s Developer Velocity?","date":"February 21, 2023","format":false,"excerpt":"From retail to healthcare to IT and beyond, countless industries rely on software development to enhance business performance. However, to optimize software innovation and performance, companies must create enhanced environments that remove productivity blockers and deliver great experiences for developers. By empowering engineers to focus more on building new features\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/719","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=719"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/719\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=719"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=719"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=719"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}