{"id":335,"date":"2021-08-31T14:39:28","date_gmt":"2021-08-31T14:39:28","guid":{"rendered":"https:\/\/fde.cat\/?p=335"},"modified":"2021-08-31T14:39:28","modified_gmt":"2021-08-31T14:39:28","slug":"migrating-facebook-to-mysql-8-0","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/migrating-facebook-to-mysql-8-0\/","title":{"rendered":"Migrating Facebook to MySQL 8.0"},"content":{"rendered":"

MySQL<\/span><\/a>, an open source database developed by Oracle, powers some of Facebook\u2019s most important workloads. We actively develop new features in MySQL to support our evolving requirements. These features change many different areas of MySQL, including client connectors, storage engine, optimizer, and replication. Each new major version of MySQL requires significant time and effort to migrate our workloads. The challenges\u00a0 include:<\/span><\/p>\n

Porting our custom features to the new version<\/span>
\nEnsuring replication is compatible between the major versions<\/span>
\nMinimizing changes needed for existing application queries<\/span>
\nFixing performance regressions that prevent the server from supporting our workloads<\/span><\/p>\n

Our last major version upgrade, to MySQL 5.6, took more than a year to roll out. When version 5.7 was released, we were still in the midst of developing our LSM-Tree storage engine, <\/span>MyRocks<\/span><\/a>, on version 5.6. Since upgrading to 5.7 while simultaneously building a new storage engine would have significantly slowed the progress on MyRocks, we opted to stay with 5.6 until MyRocks was complete. MySQL 8.0 was announced as we were finishing the rollout of MyRocks to our user database (UDB) service tier.\u00a0<\/span><\/p>\n

That version included compelling features like writeset-based parallel replication and a transactional data dictionary that provided atomic DDL support. For us, moving to 8.0 would also bring in the 5.7 features we had missed, including Document Store. Version 5.6 was approaching end of life, and we wanted to stay active within the MySQL community, especially with our work on the MyRocks storage engine. Enhancements in 8.0, like instant DDL, could speed up MyRocks schema changes, but we needed to be on the 8.0 codebase to use it. Given the benefits of the code update, we decided to migrate to 8.0. We\u2019re sharing how we tackled our 8.0 migration project \u2014 and some of the surprises we discovered in the process. When we initially scoped out the project, it was clear that moving to 8.0 would be even more difficult than migrating to 5.6 or MyRocks.<\/span><\/p>\n

At the time, our customized 5.6 branch had over 1,700 code patches to port to 8.0. As we were porting those changes, new Facebook MySQL features and fixes were added to the 5.6 codebase that moved the goalpost further away.<\/span>
\nWe have many MySQL servers running in production, serving a large number of disparate applications. We also have extensive software infrastructure for managing MySQL instances. These applications perform operations like gathering statistics and managing server backups.<\/span>
\nUpgrading from 5.6 to 8.0 skipped over 5.7 entirely. APIs that were active in 5.6 would have been deprecated in 5.7 and possibly removed in 8.0, requiring us to update any application using the now-removed APIs.<\/span>
\nA number of Facebook features were not forward-compatible with similar ones in 8.0 and required a deprecation and migration path forward.<\/span>
\nMyRocks enhancements were needed to run in 8.0, including native partitioning and crash recovery.<\/span><\/p>\n

Code patches<\/span><\/h2>\n

We first set up the 8.0 branch for building and testing in our development environments. We then began the long journey to port the patches from our 5.6 branch. There were more than 1,700 patches when we started, but we were able to organize them into a few major categories. Most of our custom code had good comments and descriptions so we could easily determine whether it was still needed by the applications or if it could be dropped. Features that were enabled by special keywords or unique variable names also made it easy to determine relevance because we could search through our application codebases to find their use cases. A few patches were very obscure and required detective work \u2014 digging through old design documents, posts, and\/or code review comments \u2014\u00a0to understand their history.<\/span><\/p>\n

We sorted each patch into one of four buckets:<\/span><\/p>\n

Drop: Features that were no longer used, or had equivalent functionality in 8.0, did not need to be ported.<\/span>
\nBuild\/Client: Non-server features that supported our build environment and modified MySQL tools like mysqlbinlog, or added functionality like the async client API, were ported.<\/span>
\nNon-MyRocks Server: Features in the mysqld server that were not related to our MyRocks storage engine were ported.<\/span>
\nMyRocks Server: Features that supported the MyRocks storage engine were ported.<\/span><\/p>\n

We tracked the status and relevant historical information of each patch using spreadsheets, and recorded our reasoning when dropping a patch. Multiple patches that updated the same feature were grouped together for porting. Patches ported and committed to the 8.0 branch were annotated with the 5.6 commit information. Discrepancies on porting status would inevitably arise due to the large number of patches we needed to sift through and these notes helped us resolve them.<\/span><\/p>\n

Each of the client and server categories naturally became a software release milestone. With all client-related changes ported, we were able to update our client tooling and connector code to 8.0. Once all of the non-MyRocks server features were ported, we were able to deploy 8.0 mysqld for InnoDB servers. Finishing up the MyRocks server features enabled us to update MyRocks installations.<\/span><\/p>\n

Some of the most complex features required significant changes for 8.0, and a few areas had major compatibility problems. For example, upstream 8.0 binlog event formats were incompatible with some of our custom 5.6 modifications. Error codes used by Facebook 5.6 features conflicted with those assigned to new features by upstream 8.0. We ultimately needed to patch our 5.6 server to be forward-compatible with 8.0.<\/span><\/p>\n

It took a couple of years to complete porting all of these features. By the time we got to the end, we had evaluated more than 2,300 patches and ported 1,500 of those to 8.0.<\/span><\/p>\n

The migration path<\/span><\/h2>\n

We group together multiple mysqld instances into a single MySQL replica set. Each instance in a replica set contains the same data but is geographically distributed to a different data center to provide data availability and failover support. Each replica set has one primary instance. The remaining instances are all secondaries. The primary handles all write traffic and replicates the data asynchronously to all secondaries.<\/span><\/p>\n<\/p>\n

We started with replica sets consisting of 5.6 primary\/5.6 secondaries and the end goal was replica sets with 8.0 primary\/8.0 secondaries. We followed a plan similar to the<\/span> UDB MyRocks migration plan<\/span><\/a>.<\/span><\/p>\n

For each replica set, create and add 8.0 secondaries via a logical copy using mysqldump. These secondaries do not serve any application read traffic.<\/span>
\nEnable read traffic on the 8.0 secondaries.<\/span>
\nAllow the 8.0 instance to be promoted to primary.<\/span>
\nDisable the 5.6 instances for read traffic.<\/span>
\nRemove all the 5.6 instances.<\/span><\/p>\n

Each replica set could transition through each of the steps above independently and stay on a step as long as needed. We separated replica sets into much smaller groups, which we shepherded through each transition. If we found problems, we could rollback to the previous step. In some cases, replica sets were able to reach the last step before others started.<\/span><\/p>\n

To automate the transition of a large number of replica sets, we needed to build new software infrastructure. We could group replica sets together and move them through each stage by simply changing a line in a configuration file. Any replica set that encountered problems could then be individually rolled back.<\/span><\/p>\n

Row-based replication<\/span><\/h3>\n

As part of the 8.0 migration effort, we decided to standardize on using row-based replication (RBR). Some 8.0 features required RBR, and it simplified our MyRocks porting efforts. While most of our MySQL replica sets were already using RBR, those still running statement-based replication (SBR) could not be easily converted. These replica sets usually had tables without any high cardinality keys. Switching completely to RBR had been a goal, but the long tail of work needed to add primary keys was often prioritized lower than other projects.<\/span><\/p>\n

Hence, we made RBR a requirement for 8.0. After evaluating and adding primary keys to every table, we switched over the last SBR replica set this year. Using RBR also gave us an alternative solution for resolving an application issue that we encountered when we moved some replica sets to 8.0 primaries, which will be discussed later.<\/span><\/p>\n

Automation validation<\/span><\/h2>\n

Most of the 8.0 migration process involved testing and verifying the mysqld server with our automation infrastructure and application queries.<\/span><\/p>\n

As our MySQL fleet grew, so did the automation infrastructure we use to manage the servers. In order to ensure all of our MySQL automation was compatible with the 8.0 version, we invested in building a test environment, which leveraged test replica sets with virtual machines to verify the behaviors. We wrote integration tests to canary each piece of automation to run on both the 5.6 version and the 8.0 version and verified their correctness. We found several bugs and behavior differences as we went through this exercise.<\/span><\/p>\n

As each piece of MySQL infrastructure was validated against our 8.0 server, we found and fixed (or worked around) a number of interesting issues:<\/span><\/p>\n

Software that parsed text output from error log, mysqldump output, or server show commands easily broke. Slight changes in the server output often revealed bugs in a tool\u2019s parsing logic.<\/span>
\nThe 8.0\u2019s default <\/span>utf8mb4<\/span> collation settings resulted in collation mismatches between our 5.6 and 8.0 instances. 8.0 tables may use the new <\/span>utf8mb4_0900<\/span> collations even for create statements generated by 5.6\u2019s show create table because the 5.6 schemas using <\/span>utf8mb4_general_ci<\/span> do not explicitly specify collation. These table differences often caused problems with replication and schema verification tools.<\/span>
\nThe error codes for certain replication failures changed and we had to fix our automation to handle them correctly.<\/span>
\nThe 8.0 version\u2019s data dictionary obsoleted table .frm files, but some of our automation used them to detect table schema modifications.<\/span>
\nWe had to update our automation to support the dynamic privs introduced in 8.0.<\/span><\/p>\n

Application validation<\/span><\/h3>\n

We wanted the transition for applications to be as transparent as possible, but some application queries hit performance regressions or would fail on 8.0.<\/span><\/p>\n

For the MyRocks migration, we built a MySQL shadow testing framework that captured production traffic and replayed them to test instances. For each application workload, we constructed test instances on 8.0 and replayed shadow traffic queries to them. We captured and logged the errors returning from the 8.0 server and found some interesting problems. Unfortunately, not all of these problems were found during testing. For example, the transaction deadlock was discovered by applications during the migration. We were able to roll back these applications to 5.6 temporarily while we researched different solutions.<\/span><\/p>\n

New reserved keywords were introduced in 8.0 and a few, such as groups and rank, conflicted with popular table column names and aliases used in application queries. These queries did not escape the names via backquotes, leading to parsing errors. Applications using software libraries that automatically escaped the column names in queries did not hit these issues, but not all applications used them. Fixing the problem was simple, but it took time to track down application owners and codebases generating these queries.<\/span>
\nA few REGEXP incompatibilities were also found between 5.6 and 8.0.<\/span>
\nA few applications hit <\/span>repeatable-read transaction deadlocks<\/span><\/a> involving <\/span>insert \u2026 on duplicate key<\/span> queries on InnoDB. 5.6 had a bug which was corrected in 8.0, but the fix increased the likelihood of transaction deadlocks. After analyzing our queries, we were able to resolve them by lowering the isolation level. This option was available to us since we had made the switch to row-based replication.<\/span>
\nOur custom 5.6 Document Store and JSON functions were not compatible with 8.0\u2019s. Applications using Document Store needed to convert the document type to text for the migration. For the JSON functions, we added 5.6-compatible versions to the 8.0 server so that applications could migrate to the 8.0 API at a later time.<\/span><\/p>\n

Our query and performance testing of the 8.0 server uncovered a few problems that needed to be addressed almost immediately.<\/span><\/p>\n

We found new mutex contention hotspots around the ACL cache. When a large number of connections were opened simultaneously, they could all block on checking ACLs.<\/span>
\nSimilar contention was found with binlog index access when many binlog files are present and high binlog write rates rotate files frequently.<\/span>
\nSeveral queries involving temp tables were broken. The queries would return unexpected errors or take so long to run that they would time out.<\/span><\/p>\n

Memory usage compared with 5.6 had increased, especially for our MyRocks instances, because InnoDB in 8.0 must be loaded. The default performance_schema settings enabled all instruments and consumed significant memory. We limited the memory usage by only enabling a small number of instruments and making code changes to disable tables that could not be manually turned off. However, not all the increased memory was being allocated by performance_schema. We needed to examine and modify various InnoDB internal data structures to reduce the memory footprint further. This effort brought 8.0\u2019s memory usage down to acceptable levels.\u00a0<\/span><\/p>\n

What\u2019s next<\/span><\/h2>\n

The 8.0 migration has taken a few years so far. We have converted many of our InnoDB replica sets to running entirely on 8.0. Most of the remaining ones are at various stages along the migration path. Now that most of our custom features have been ported to 8.0, updating to Oracle\u2019s minor releases has been comparatively easier and we plan to keep pace with the latest versions.<\/span><\/p>\n

Skipping a major version like 5.7 introduced problems, which our migration needed to solve.<\/span><\/p>\n

First, we could not upgrade servers in place and needed to use logical dump and restore to build a new server. However, for very large mysqld instances, this can take many days on a live production server and this fragile process will likely be interrupted before it can complete. For these large instances, we had to modify our backup and restore systems to handle the rebuild.<\/span><\/p>\n

Second, it is much harder to detect API changes because 5.7 could have provided deprecation warnings to our application clients to fix potential issues. Instead, we needed to run additional shadow tests to find failures before we could migrate the production workloads. Using mysql client software that automatically escaped schema object names helps reduce the number of compatibility issues.<\/span><\/p>\n

Supporting two major versions within a replica set is hard. Once a replica set promotes its primary to be an 8.0 instance, it is best to disable and remove the 5.6 ones as soon as possible. Application users tend to discover new features that are supported only by 8.0, like <\/span>utf8mb4_0900<\/span> collations, and using these can break the replication stream between 8.0 and 5.6 instances.<\/span><\/p>\n

Despite all the hurdles in our migration path, we have already seen the benefits of running 8.0. Some applications have opted for early conversion to 8.0 to utilize features like Document Store and improved datetime support. We have been considering how to support storage engine features like Instant DDL on MyRocks. Overall, the new version greatly expands on what we can do with MySQL @ Facebook.<\/span><\/p>\n

The post Migrating Facebook to MySQL 8.0<\/a> appeared first on Facebook Engineering<\/a>.<\/p>\n

Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

MySQL, an open source database developed by Oracle, powers some of Facebook\u2019s most important workloads. We actively develop new features in MySQL to support our evolving requirements. These features change many different areas of MySQL, including client connectors, storage engine, optimizer, and replication. Each new major version of MySQL requires significant time and effort to… Continue reading Migrating Facebook to MySQL 8.0<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-335","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":714,"url":"https:\/\/fde.cat\/index.php\/2023\/05\/16\/building-and-deploying-mysql-raft-at-meta\/","url_meta":{"origin":335,"position":0},"title":"Building and deploying MySQL Raft at Meta","date":"May 16, 2023","format":false,"excerpt":"\u00a0 We\u2019re rolling out MySQL Raft with the aim to eventually replace our current MySQL semisynchronous databases.\u00a0 The biggest win of MySQL Raft was simplification of the operation and making MySQL servers take care of promotions and membership. This gave the provable safety of Raft and reduced significant operational pain.\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":274,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/foqs-scaling-a-distributed-priority-queue\/","url_meta":{"origin":335,"position":1},"title":"FOQS: Scaling a distributed priority queue","date":"August 31, 2021","format":false,"excerpt":"We will be hosting a talk about our work on Scaling a Distributed Priority Queue during our virtual Systems @Scale event at 11 am PT on Wednesday, February 24, followed by a live Q&A session. Please submit any questions to systemsatscale@fb.com before the event. The entire Facebook ecosystem is powered\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":533,"url":"https:\/\/fde.cat\/index.php\/2022\/01\/18\/foqs-making-a-distributed-priority-queue-disaster-ready\/","url_meta":{"origin":335,"position":2},"title":"FOQS: Making a distributed priority queue disaster-ready","date":"January 18, 2022","format":false,"excerpt":"Facebook Ordered Queueing Service (FOQS) is a fully managed, distributed priority queueing service used for reliable message delivery among many services. FOQS has evolved from a regional deployment into a geo-distributed, global deployment to ensure that data stored within logical queues is highly available, even through large-scale disaster scenarios. Migrating\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":255,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/sre-weekly-issue-252\/","url_meta":{"origin":335,"position":3},"title":"SRE Weekly Issue #252","date":"August 31, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, StackHawk: Interested in how you can automate application security testing with GitHub Actions? Check out this on demand webinar from StackHawk and Snyk and see how simple it is to get started. https:\/\/sthwk.com\/stackhawk-snyk Articles Building On-Call Culture at GitHub Their on-call started\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":518,"url":"https:\/\/fde.cat\/index.php\/2021\/12\/16\/power-loss-siren-making-meta-resilient-to-power-loss-events\/","url_meta":{"origin":335,"position":4},"title":"Power Loss Siren: Making Meta resilient to power loss events","date":"December 16, 2021","format":false,"excerpt":"There are thousands of distributed services running on millions of servers in Meta\u2019s data centers. Part of ensuring the reliability of those services means making them resilient to power loss events as our data center fleet grows. To help increase resiliency, we built the Power Loss Siren (PLS) \u2014 a\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":672,"url":"https:\/\/fde.cat\/index.php\/2023\/01\/31\/asynchronous-computing-at-meta-overview-and-learnings\/","url_meta":{"origin":335,"position":5},"title":"Asynchronous computing at Meta: Overview and learnings","date":"January 31, 2023","format":false,"excerpt":"We\u2019ve made architecture changes to Meta\u2019s event driven asynchronous computing platform that have\u00a0 enabled easy integration with multiple event-sources.\u00a0 We\u2019re sharing our learnings from handling various workloads and how to tackle trade offs made with certain design choices in building the platform. Asynchronous computing is a paradigm where the user\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/335","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=335"}],"version-history":[{"count":1,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/335\/revisions"}],"predecessor-version":[{"id":376,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/335\/revisions\/376"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=335"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=335"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=335"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}