{"id":810,"date":"2024-01-09T16:26:15","date_gmt":"2024-01-09T16:26:15","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2024\/01\/09\/implementing-salesforces-largest-database-upgrade-inside-the-migration-to-hbase-2\/"},"modified":"2024-01-09T16:26:15","modified_gmt":"2024-01-09T16:26:15","slug":"implementing-salesforces-largest-database-upgrade-inside-the-migration-to-hbase-2","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2024\/01\/09\/implementing-salesforces-largest-database-upgrade-inside-the-migration-to-hbase-2\/","title":{"rendered":"Implementing Salesforce\u2019s Largest Database Upgrade: Inside the Migration to HBase 2"},"content":{"rendered":"

Written by Viraj Jasani and Andrew Purtell<\/em><\/p>\n

Data is the engine behind Salesforce operations, helping our customers make better decisions on a daily basis. The Big Data Storage (BDS) team, a key part of Salesforce\u2019s engineering organization, deploys arguably one of the largest distributed database production footprints. This infrastructure is based on the combination of open-source HBase<\/a> and Phoenix<\/a>, designed to empower high scale, high throughput, low latency, real-time random read-write access to petabytes of data. This is a capability not achievable with off-the-shelf traditional relational databases.<\/p>\n

In 2014,<\/a> BDS described its motivations in detail, encapsulating how HBase and Phoenix solved many technical problems during the team\u2019s ongoing 10+ year journey. An important part of this journey involved developing a strategy for upgrading infrastructure. At Salesforce\u2019s scale, where customers demand non-stop, disruption-free 24\/7 operation, significant challenges arose. <\/p>\n

Read on to explore how BDS drove the upgrade from HBase 1 to HBase 2 with minimal impact to Salesforce customers.<\/p>\n

Why upgrade to HBase 2?<\/h4>\n
\n

For many years, running HBase 1 and Phoenix 4 presented minimal issues. Nonetheless, a major upgrade to HBase 2 and Phoenix 5 was pursued for several reasons such as:<\/p>\n

Availability<\/strong>. HBase, a strongly consistent (CP of CAP theorem<\/a>) distributed database, lacked high availability during table region transitions (see our existing post<\/a> on details of region transitions). HBase 2 significantly rearchitected region assignment workflows<\/a> by eliminating its dependency on Zookeeper<\/a>. This increased overall datastore availability and boosted infrastructure stability while maintaining customer trust.<\/p>\n

Need for speed<\/strong>. HBase 2 enhanced write request throughput with a design that writes to multiple replicas of the Write-Ahead-Log (WAL) files asynchronously \u2014 key for applications utilizing a database, delivering a high throughput.<\/p>\n

Low latency<\/strong>. Being a latency sensitive database, BDS encountered issues from long GC pauses at times<\/a>. HBase 2 and Phoenix 5 provide compile and runtime support for Java 11. Using Java together with Shenandoah GC in production<\/a> helped reduce latency.<\/p>\n

Less idle threads.<\/strong> During production, BDS regularly reported a higher number of idle threads that were intended to initiate Remote-Procedure Calls (or RPC<\/a>). HBase 2 supports an asynchronous, event-driven, non-blocking socket-based Netty framework<\/a> for reduced resource consumption and improved throughput during RPC calls.<\/p>\n

Quick security fixes<\/strong>. Staying close to the latest major open-source releases drives fast and effective resolution of third-party security vulnerabilities. HBase 1\u2019s release line end-of-life-status further justified the move to HBase 2.<\/p>\n<\/div>\n

What challenges did BDS face?<\/h4>\n

Regardless of the upgrade, maximizing availability for customers is a key focus. Maintaining 99.999% of availability is critical, making any bugs and behavioral changes a priority for solving. Consequently, any major, minor, or patch upgrade of HBase\/Phoenix meant that BDS had to maintain availability both during and after the completion of the rolling upgrade.<\/p>\n

Salesforce has been engaged in a multi-year project to migrate from its own private datacenters into the public cloud infrastructure (more details here<\/a>), and the flexibility of the public cloud infrastructure can greatly improve BDS team\u2019s capability to perform a seamless migration by leveraging additional resources to build an exact replica of the given cluster, followed by a switchover of client traffic from the old version running cluster to the new cluster.<\/p>\n

However, as of this moment, a significant number of clusters are still running on first-party data centers. Therefore, the upgrade from HBase 1 to HBase 2, along with Phoenix 4 to Phoenix 5, comes with a critical challenge: it had to be performed without any downtime regardless of the underlying infrastructure.<\/p>\n

As an example of scale, production clusters with approximately 300 nodes take more than 5 hours to complete the rolling upgrade of all components. The significant re-architecture with HBase 2 makes performing the upgrade without downtime impossible. BDS has worked with the open-source community to find solutions for this as it\u2019s a fairly unique challenge for organizations at Salesforce\u2019s scale.<\/p>\n

How did BDS prepare?<\/h4>\n

With any complex goal comes significant trade-offs. BDS methodically evaluated core functionalities in HBase\/Phoenix, determining whether non-critical background activities could be temporarily disabled during the upgrade. This journey occurred over a two-year period, marked by unwavering dedication to delivering a seamless production rollout, which involved several key steps.<\/p>\n

Stabilizing builds was an initial priority, which ensured stability for HBase and Phoenix downstream projects. This encompassed in-depth verification of code compilation and successfully testing the entire CI build.<\/p>\n

The toughest challenge emerged when BDS needed to evaluate various tradeoffs and develop a strategy to execute a seamless rolling upgrade while eliminating downtime. This required significant time, developing an exact sequence of steps that achieved the goals while maintaining the service, while also working around compatibility issues in the intermediate states.<\/p>\n

As the team encountered issues related to system table schema differences between the two versions, they recognized the possible unattainability of the task. However, BDS successfully resolved this by implementing a procedure that could update schemas without disturbing compatibility between old clients and new servers.<\/p>\n

Additionally, as they encountered other hurdles, BDS contributed their fixes to the open-source codebase.<\/p>\n

And as they used HBase and Phoenix, the team ensured that clients using JDBC connections for querying data could continue performing CRUD operations with minimal retries during the upgrade.<\/p>\n

Finally, to manage the zero-downtime upgrade, it was quite crucial for BDS to emphasize building a procedure facilitating the transition from \u201cZookeeper based\u201d to \u201cZookeeper-less\u201d table region assignments \u2014 a critical prerequisite for their upgrade.<\/p>\n

The team also performed rigorous performance testing throughout the lifecycle of the project. Being one of the active members of the open-source community also helped them get better traction on critical bug-fixes that they contributed.<\/p>\n

A look at HBase 2 pre-requisite + upgrade procedural steps. (details here<\/a>)<\/p>\n

\n

Learn more<\/strong><\/h4>\n

For a deeper dive, check out the in-depth technical details for this project here<\/a>.<\/p>\n

Stay connected \u2014 join our Talent Community<\/a>!<\/p>\n

Check out our Technology and Product teams<\/a> to learn how you can get involved.<\/p>\n<\/div>\n

The post Implementing Salesforce\u2019s Largest Database Upgrade: Inside the Migration to HBase 2<\/a> appeared first on Salesforce Engineering Blog<\/a>.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

Written by Viraj Jasani and Andrew Purtell Data is the engine behind Salesforce operations, helping our customers make better decisions on a daily basis. The Big Data Storage (BDS) team, a key part of Salesforce\u2019s engineering organization, deploys arguably one of the largest distributed database production footprints. This infrastructure is based on the combination of… Continue reading Implementing Salesforce\u2019s Largest Database Upgrade: Inside the Migration to HBase 2<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-810","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":456,"url":"https:\/\/fde.cat\/index.php\/2021\/09\/01\/evolution-of-region-assignment-in-the-apache-hbase-architecture%e2%80%8a-%e2%80%8apart-1\/","url_meta":{"origin":810,"position":0},"title":"Evolution of Region Assignment in the Apache HBase Architecture\u200a\u2014\u200aPart 1","date":"September 1, 2021","format":false,"excerpt":"Evolution of Region Assignment in the Apache HBase Architecture\u200a\u2014\u200aPart\u00a01 Written by Viraj Jasani and Andrew\u00a0Purtell At Salesforce, we run a large number of Apache HBase clusters in our own data centers as well as in public cloud infrastructure. This post outlines some important design aspects of Apache HBase and how\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":294,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/the-design-of-strongly-consistent-global-secondary-indexes-in-apache-phoenix%e2%80%8a-%e2%80%8apart-1\/","url_meta":{"origin":810,"position":1},"title":"The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix\u200a\u2014\u200aPart 1","date":"August 31, 2021","format":false,"excerpt":"The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix\u200a\u2014\u200aPart\u00a01Phoenix is a relational database with a SQL interface that uses HBase as its backing store. This combination allows it to leverage the flexibility and scalability of HBase, which is a distributed key-value store. Phoenix provides additional functionality on top\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":457,"url":"https:\/\/fde.cat\/index.php\/2021\/09\/02\/evolution-of-region-assignment-in-the-apache-hbase-architecture%e2%80%8a-%e2%80%8apart-2\/","url_meta":{"origin":810,"position":2},"title":"Evolution of Region Assignment in the Apache HBase Architecture\u200a\u2014\u200aPart 2","date":"September 2, 2021","format":false,"excerpt":"Evolution of Region Assignment in the Apache HBase Architecture\u200a\u2014\u200aPart\u00a02 The first part of this two-part series of blog posts provided an introduction to some of the important design aspects of Apache HBase. We introduced the concept of the AssignmentManager and the importance of its role in the HBase architecture. In\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":485,"url":"https:\/\/fde.cat\/index.php\/2021\/10\/07\/evolution-of-region-assignment-in-the-apache-hbase-architecture%e2%80%8a-%e2%80%8apart-3\/","url_meta":{"origin":810,"position":3},"title":"Evolution of Region Assignment in the Apache HBase Architecture\u200a\u2014\u200aPart 3","date":"October 7, 2021","format":false,"excerpt":"Evolution of Region Assignment in the Apache HBase Architecture\u200a\u2014\u200aPart\u00a03 Authors: Viraj Jasani, Andrew Purtell, \u5f20\u94ce(Duo\u00a0Zhang) In the second part of this blog post series, we provided an overview of how the redesigned AssignmentManager in HBase 2 efficiently and reliably manages the process of region assignment. In this third entry in\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":284,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/hadoop-hbase-on-kubernetes-and-public-cloud-part-i\/","url_meta":{"origin":810,"position":4},"title":"Hadoop\/HBase on Kubernetes and Public Cloud (Part I)","date":"August 31, 2021","format":false,"excerpt":"Authors: Dhiraj Hegde, Ashutosh Parekh, and Prashant\u00a0MurthyAt Salesforce, we run a large number of HBase and HDFS clusters in our own data centers. More recently, we have started deploying our clusters on Public Cloud infrastructure to take advantage of the on-demand scalability available there. As part of this foray onto\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":285,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/hadoop-hbase-on-kubernetes-and-public-cloud-part-ii\/","url_meta":{"origin":810,"position":5},"title":"Hadoop\/HBase on Kubernetes and Public Cloud (Part II)","date":"August 31, 2021","format":false,"excerpt":"The first part of this two part blog provided an introduction to concepts in Kubernetes and Public Cloud that are relevant to stateful application management. We also covered how Kubernetes and Hadoop features were leveraged to provide a highly available service. In this second part of the blog we will\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/810","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=810"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/810\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=810"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=810"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=810"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}