{"id":649,"date":"2022-11-09T17:30:42","date_gmt":"2022-11-09T17:30:42","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/11\/09\/tulip-schematizing-metas-data-platform\/"},"modified":"2022-11-09T17:30:42","modified_gmt":"2022-11-09T17:30:42","slug":"tulip-schematizing-metas-data-platform","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/11\/09\/tulip-schematizing-metas-data-platform\/","title":{"rendered":"Tulip: Schematizing Meta\u2019s data platform"},"content":{"rendered":"

We\u2019re sharing Tulip, a binary serialization protocol supporting schema evolution.\u00a0<\/span>
\nTulip assists with data schematization by addressing protocol reliability and other issues simultaneously.\u00a0<\/span>
\nIt replaces multiple legacy formats used in Meta\u2019s data platform and has achieved significant performance and efficiency gains.<\/span><\/p>\n

There are numerous heterogeneous services, such as warehouse data storage and various real-time systems, that make up Meta\u2019s data platform \u2014 all exchanging large amounts of data among themselves as they communicate via service APIs. As we continue to grow the number of AI- and machine learning (ML)\u2013related workloads in our systems that leverage data for tasks such as training ML models, we\u2019re continually working to make our data logging systems more efficient.<\/span><\/p>\n

Schematization of data plays an important role in a data platform at Meta\u2019s scale. These systems are designed with the knowledge that every decision and trade-off can impact the reliability, performance, and efficiency of data processing, as well as our engineers\u2019 developer experience.\u00a0<\/span><\/p>\n

Making huge bets, like changing serialization formats for the entire data infrastructure, is challenging in the short term, but offers greater long-term benefits that help the platform evolve over time.<\/span><\/p>\n

The challenge of a data platform at exabyte scale\u00a0<\/span><\/h2>\n

The <\/span>data analytics logging library is present in the web tier as well as in internal services. It is responsible for logging analytical and operational data via <\/span>Scribe<\/span><\/a> (Meta\u2019s persistent and durable message queuing system). Various services read and ingest data from Scribe, including (but not limited to) the data platform Ingestion Service, and real-time processing systems, such as<\/span> Puma<\/span><\/a>,<\/span> Stylus<\/span><\/a>, and<\/span> XStream<\/span><\/a>. The <\/span>data analytics reading library correspondingly assists in deserializing data and rehydrating it into a structured payload. While this article will focus on only the logging library, the narrative applies to both.<\/span><\/p>\n

Figure 1: High-level system diagram for analytics-logging data flow at Meta.<\/p>\n

At the scale at which Meta\u2019s data platform operates, thousands of engineers create, update, and delete logging schemas every month. These logging schemas see petabytes of data flowing through them every day over Scribe.<\/span><\/p>\n

Schematization is important to ensure that any message logged in the present, past, or future, relative to the version of (de)serializer, can be (de)serialized reliably at any point in time with the highest fidelity and no loss of data. This property is called safe schema evolution via forward and backward compatibility.<\/span><\/p>\n

This article will focus on the<\/span> on-wire serialization format chosen to encode data that is finally processed by the data platform. We motivate the evolution of this design, the trade-offs considered, and the resulting improvements. From an efficiency point of view, the new encoding format needs between\u00a0 <\/span>40 percent to 85 percent fewer bytes, and uses <\/span>50 percent to 90 percent fewer CPU cycles to (de)serialize data compared with the previously used serialization formats, namely <\/span>Hive Text Delimited<\/span><\/a> and <\/span>JSON<\/span><\/a> serialization.<\/span><\/p>\n

How we developed Tulip<\/span><\/h2>\n

An overview of the data analytics logging library\u00a0<\/span><\/h3>\n

The logging library is used by applications written in various languages (such as Hack, C++, Java, Python, and Haskell) to serialize a payload according to a logging schema. Engineers define logging schemas in accordance with business needs. These serialized payloads are written to Scribe for durable delivery.<\/span><\/p>\n

The logging library itself comes in two flavors:<\/span><\/p>\n

Code-generated: In this flavor, statically typed setters for each field are generated for type-safe usage. Additionally, post-processing and serialization code are also code-generated (where applicable) for maximum efficiency. For example, Hack\u2019s thrift serializer uses a <\/span>C++ accelerator<\/span><\/a>, where code generation is partially employed.<\/span>
\nGeneric: A C++ library called Tulib (not to be confused with Tulip) to perform (de)serialization of dynamically typed payloads is provided. In this flavor, a dynamically typed message is serialized according to a logging schema. This mode is more versatile than the code-generated mode because it allows (de)serialization of messages without rebuilding and redeploying the application binary.<\/span><\/p>\n

Legacy serialization format<\/span><\/h3>\n

The logging library writes data to multiple back-end systems that have historically dictated their own serialization mechanisms. For example, warehouse ingestion uses <\/span>Hive Text Delimiters<\/span><\/a> during serialization, whereas other systems use <\/span>JSON serialization<\/span><\/a>. There are many problems when using one or both of these formats for serializing payloads.<\/span><\/p>\n

Standardization: Previously, each downstream system had its own format, and there was <\/span>no standardization of serialization formats. This increased development and maintenance costs.<\/span>
\nReliability: The Hive Text Delimited format is positional in nature. To maintain deserialization reliability, new columns can be added only at the end. Any attempt to add fields in the middle of a column or delete columns will shift all the columns after it, making the row impossible to deserialize (since a row is not self-describing, unlike in JSON). We distribute the updated schema to readers in real time.<\/span>
\nEfficiency: Both the Hive Text Delimited and JSON protocol are text-based and inefficient in comparison with binary (de)serialization.<\/span>
\nCorrectness: Text-based protocols such as Hive Text require escaping and unescaping of control characters field delimiters and line delimiters. This is done by every writer\/reader and puts additional burden on library authors. It\u2019s challenging to deal with legacy\/buggy implementations that only check for the presence of such characters and disallow the entire message instead of escaping the problematic characters.<\/span>
\nForward and backward compatibility: It\u2019s desirable for consumers to be able to consume payloads that were serialized by a serialization schema both before and after the version that the consumer sees. The Hive Text Protocol doesn\u2019t provide this guarantee.<\/span>
\nMetadata: Hive Text Serialization doesn\u2019t trivially permit the addition of metadata to the payload. Propagation of <\/span>metadata for downstream systems is critical to implement features that benefit from its presence. For example, certain debugging workflows benefit from having a hostname or a checksum transferred along with the serialized payload.<\/span><\/p>\n

The fundamental problem that Tulip solved is the <\/span>reliability issue, by ensuring a safe schema evolution format with forward and backward compatibility across services that have their own deployment schedules.\u00a0<\/span><\/p>\n

One could have imagined solving the others independently by pursuing a different strategy, but the fact that Tulip was able to solve all of these problems at once made it a much more compelling investment than other options.<\/span><\/p>\n

Tulip serialization<\/span><\/h2>\n

The Tulip serialization protocol is a binary serialization protocol that uses Thrift\u2019s <\/span>TCompactProtocol<\/span><\/a> for serializing a payload. It follows the same rules for numbering fields with IDs as one would expect an engineer to use when updating IDs in a Thrift struct.<\/span><\/p>\n

When engineers author a logging schema, they specify <\/span>a list of field names and types. Field IDs are not specified by engineers<\/span>, but are instead assigned by the <\/span>data platform management module.<\/span><\/p>\n

Figure 2: Logging schema authoring flow.<\/p>\n

This figure shows user-facing workflow when an engineer creates\/updates a logging schema. Once validation succeeds, the changes to the logging schema are published to various systems in the data platform.<\/span><\/p>\n

The logging schema is translated into a serialization schema and stored in the <\/span>serialization schema repository. A serialization config holds lists of (field name, field type, field ID) for a corresponding logging schema as well as the field history. A transactional operation is performed on the serialization schema when an engineer wishes to update a logging schema.<\/span><\/p>\n

Figure 3: Tulip serialization schema evolution<\/p>\n

The example above shows the creation and updation of a logging schema and its impact on the serialization schema over time.<\/span><\/p>\n

Field addition: When a new field named \u201cauthors\u201d is added to the logging schema, a new ID is assigned in the serialization schema.<\/span>
\nField type change: Similarly, when the type of the field \u201cisbn\u201d is changed from \u201ci64\u201d to \u201cstring\u201d, a new ID is associated with the new field, but the ID of the original \u201ci64\u201d typed \u201cisbn\u201d field is retained in the serialization schema. When the underlying data store doesn\u2019t allow field type changes, the logging library disallows this change.<\/span>
\nField deletion: IDs are never removed from the serialization schema, allowing complete backward compatibility with already serialized payloads. The field in a serialization schema for a logging schema is indelible even if fields in the logging schema are added\/removed.<\/span>
\nField rename: There\u2019s no concept of a field rename, and this operation is treated as a field deletion followed by a field addition.<\/span><\/p>\n

Acknowledgements<\/span><\/h2>\n

We would like to thank all the members of the data platform team who helped make this project a success. Without the XFN-support of these teams and engineers at Meta, this project would not have been possible.<\/span><\/p>\n

A special thank-you to Sriguru Chakravarthi, Sushil Dhaundiyal, Hung Duong, Stefan Filip, Manski Fransazov, Alexander Gugel, Paul Harrington, Manos Karpathiotakis, Thomas Lento, Harani Mukkala, Pramod Nayak, David Pletcher, Lin Qiao, Milos Stojanovic, Ezra Stuetzel, Huseyin Tan, Bharat Vaidhyanathan, Dino Wernli, Kevin Wilfong, Chong Xie, Jingjing Zhang, and Zhenyuan Zhao.<\/span><\/p>\n

The post Tulip: Schematizing Meta\u2019s data platform<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

We\u2019re sharing Tulip, a binary serialization protocol supporting schema evolution.\u00a0 Tulip assists with data schematization by addressing protocol reliability and other issues simultaneously.\u00a0 It replaces multiple legacy formats used in Meta\u2019s data platform and has achieved significant performance and efficiency gains. There are numerous heterogeneous services, such as warehouse data storage and various real-time systems,… Continue reading Tulip: Schematizing Meta\u2019s data platform<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-649","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":669,"url":"https:\/\/fde.cat\/index.php\/2023\/01\/26\/tulip-modernizing-metas-data-platform\/","url_meta":{"origin":649,"position":0},"title":"Tulip: Modernizing Meta\u2019s data platform","date":"January 26, 2023","format":false,"excerpt":"The technical journey discusses the motivations, challenges, and technical solutions employed for warehouse schematization, especially a change to the wire serialization format employed in Meta\u2019s data platform for data interchange related to Warehouse Analytics Logging. Here, we discuss the engineering, scaling, and nontechnical challenges of modernizing\u00a0 Meta\u2019s exabyte-scale data platform\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":893,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/10\/metas-approach-to-machine-learning-prediction-robustness\/","url_meta":{"origin":649,"position":1},"title":"Meta\u2019s approach to machine learning prediction robustness","date":"July 10, 2024","format":false,"excerpt":"Meta\u2019s advertising business leverages large-scale machine learning (ML) recommendation models that power millions of ads recommendations per second across Meta\u2019s family of apps. Maintaining reliability of these ML systems helps ensure the highest level of service and uninterrupted benefit delivery to our users and advertisers. To minimize disruptions and ensure\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":604,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/06\/watch-metas-engineers-discuss-quic-and-tcp-innovations-for-our-network\/","url_meta":{"origin":649,"position":2},"title":"Watch Meta\u2019s engineers discuss QUIC and TCP innovations for our network","date":"July 6, 2022","format":false,"excerpt":"With more than 75 percent of our internet traffic set to use QUIC and HTTP\/3 together, QUIC is slowly moving to become the de facto protocol used for internet communication at Meta. For Meta\u2019s data center network, TCP remains the primary network transport protocol that supports thousands of services on\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":501,"url":"https:\/\/fde.cat\/index.php\/2021\/11\/09\/ocp-summit-2021-open-networking-hardware-lays-the-groundwork-for-the-metaverse\/","url_meta":{"origin":649,"position":3},"title":"OCP Summit 2021: Open networking hardware lays the groundwork for the metaverse","date":"November 9, 2021","format":false,"excerpt":"Open infrastructure technologies and networking hardware will play an important role as we build new technologies for the metaverse, where billions of people will someday come together in virtual spaces. As we head toward the next major computing platform with a continued spirit of embracing openness and disaggregation, we\u2019re announcing\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":779,"url":"https:\/\/fde.cat\/index.php\/2023\/10\/31\/automating-data-removal\/","url_meta":{"origin":649,"position":4},"title":"Automating data removal","date":"October 31, 2023","format":false,"excerpt":"Meta\u2019s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. SCARF avoids tedious manual work and ensures that product data is correctly removed when a\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":753,"url":"https:\/\/fde.cat\/index.php\/2023\/08\/29\/scheduling-jupyter-notebooks-at-meta\/","url_meta":{"origin":649,"position":5},"title":"Scheduling Jupyter Notebooks at Meta","date":"August 29, 2023","format":false,"excerpt":"At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals. Users with such notebooks would have to remember to manually\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/649","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=649"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/649\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=649"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=649"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=649"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}