{"id":776,"date":"2023-10-23T16:00:10","date_gmt":"2023-10-23T16:00:10","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/10\/23\/5-things-you-didnt-know-about-buck2\/"},"modified":"2023-10-23T16:00:10","modified_gmt":"2023-10-23T16:00:10","slug":"5-things-you-didnt-know-about-buck2","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/10\/23\/5-things-you-didnt-know-about-buck2\/","title":{"rendered":"5 Things you didn\u2019t know about Buck2"},"content":{"rendered":"

Meta has a very large monorepo, with many\u00a0 different programming languages. To optimize build and performance, we developed our own build system called <\/span>Buck<\/span><\/a>, which was first open-sourced in 2013.\u00a0<\/span><\/p>\n

Buck2<\/span><\/a> is the <\/span>recently open-sourced<\/span><\/a> successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Below are five interesting facts you might not have known about Buck2.<\/span><\/p>\n

Fact1: Buck2 is written in Rust<\/span><\/h2>\n

The core of Buck2 is written in Rust, with the rules written in Starlark (a Python-like language) and interpreted by our <\/span>open-source Starlark library<\/span><\/a>. Using Starlark was a natural choice since Buck1 and Bazel both already use it. We debated between using Java (like Buck1), Haskell (like the <\/span>Shake build system<\/span><\/a>) or Go for the core programming language. In the end we chose<\/span> Rust<\/span><\/a> because of its high performance, control over memory usage (garbage collection isn\u2019t a great fit for big graphs), and its focus on correct concurrency (important for a highly parallel build system).<\/span><\/p>\n

Fact2: Buck2 can avoid downloading intermediate outputs\u00a0<\/span><\/h2>\n

When configured using remote execution, Buck2 can run actions remotely. If the output of those actions is then used as the input for a further remote action, we avoid ever downloading the intermediate output and just remember the resulting hash. In some cases a build will proceed entirely remotely, with only minimal local activity, followed by downloading the final results.\u00a0 This technique can save both bandwidth and disk space.<\/span><\/p>\n

Fact3: Buck2 has 11 different types of file path internally<\/span><\/h2>\n

In addition to the <\/span>built-in Rust Path type<\/span><\/a>, and the <\/span>relative-path crate\u2019s type RelativePath<\/span><\/a>, we have further defined nine additional path types. We use these distinct types to identify which paths are absolute and, for relative paths, what they are relative to (e.g., relative to project root, enclosing cell, package, a given label, etc). While it was fairly tedious to write out so many types of path, the bugs it prevented made it worth it. We take our type safety very seriously!<\/span><\/p>\n

Fact4: Buck2 has no knowledge of any programming language built in\u00a0<\/span><\/h2>\n

Everything language-specific in Buck2 is described in Starlark. Furthermore, while Buck2 comes with a <\/span>Starlark \u201cprelude\u201d<\/span><\/a> that supports a number of programming languages (e.g., C++, Rust, Python, OCaml, etc.) you can write your own prelude from scratch (several open source users have <\/span>done<\/span><\/a> so<\/span><\/a>).\u00a0<\/span><\/p>\n

Because we kept the core of Buck2 free from language concepts, we had to make the Starlark APIs powerful enough to write things like <\/span>dep files<\/span><\/a> and conditional relinking\u2013 important performance optimisations in C++. Happily, the APIs required for those advanced features tend to be useful in other languages for purposes we hadn\u2019t even imagined (e.g. Erlang dependency trimming, Android package grouping).\u00a0<\/span><\/p>\n

Fact5: Buck2 provides <\/span>BXL<\/span><\/a> (Buck2 Extension Language) for inspecting and interacting with the Buck graph\u00a0<\/span><\/h2>\n

This feature, which is unique in the build system space as far as we are aware, gives access to the graph with Starlark API, and also lets you define new build actions native to BXL. The build graph often serves as the source of truth for a project, and BXL lets that information be easily reused. We have used BXL to generate IDE projects but think this could have lots of future opportunities.<\/span><\/p>\n

Further resources:<\/span><\/h2>\n

Build faster with Buck2: Our open source build system<\/span><\/a> (Buck2 announcement blog)\u00a0<\/span>
\nBuck2 website<\/span><\/a>
\nBuck2 GitHub<\/span><\/a><\/p>\n

We hope you found this article insightful to show you what Buck2 is capable of and where you can use it for your own workflow. Stay tuned for our upcoming blog posts, where we will explore other projects from Meta\u2019s DevInfra teams, and interesting facts on how you can use them in your own projects.\u00a0<\/span><\/p>\n

About this series<\/span><\/h3>\n

This blog is a part of our <\/span>5 Things you didn\u2019t know series, where we go over the top five interesting facts about a DevInfra project from Meta that can help you simplify your projects or workflows. Look out for more of these blogs where we discuss other DevInfra projects and interesting facts to help you get started with using them in your own projects.<\/span><\/p>\n

To learn more about <\/span>Meta Open Source<\/span><\/a>, visit our <\/span>website<\/span><\/a>, subscribe to our <\/span>YouTube channel<\/span><\/a>, or follow us on <\/span>X<\/span><\/a> and <\/span>Facebook<\/span><\/a>.<\/span><\/p>\n

The post 5 Things you didn\u2019t know about Buck2<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

Meta has a very large monorepo, with many\u00a0 different programming languages. To optimize build and performance, we developed our own build system called Buck, which was first open-sourced in 2013.\u00a0 Buck2 is the recently open-sourced successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x as fast as Buck1. Below… Continue reading 5 Things you didn\u2019t know about Buck2<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-776","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":697,"url":"https:\/\/fde.cat\/index.php\/2023\/04\/06\/build-faster-with-buck2-our-open-source-build-system\/","url_meta":{"origin":776,"position":0},"title":"Build faster with Buck2: Our open source build system","date":"April 6, 2023","format":false,"excerpt":"Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient.\u00a0 In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":728,"url":"https:\/\/fde.cat\/index.php\/2023\/06\/27\/meta-developer-tools-working-at-scale\/","url_meta":{"origin":776,"position":1},"title":"Meta developer tools: Working at scale","date":"June 27, 2023","format":false,"excerpt":"Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we\u2019ll go through a few of the tools in the development process. And, as\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":820,"url":"https:\/\/fde.cat\/index.php\/2024\/02\/06\/dotslash-simplified-executable-deployment\/","url_meta":{"origin":776,"position":2},"title":"DotSlash: Simplified executable deployment","date":"February 6, 2024","format":false,"excerpt":"We\u2019ve open sourced DotSlash, a tool that makes large executables available in source control with a negligible impact on repository size, thus avoiding I\/O-heavy clone operations. With DotSlash, a set of platform-specific executables is replaced with a single script containing descriptors for the supported platforms. DotSlash handles transparently fetching, decompressing,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":818,"url":"https:\/\/fde.cat\/index.php\/2024\/01\/29\/improving-machine-learning-iteration-speed-with-faster-application-build-and-packaging\/","url_meta":{"origin":776,"position":3},"title":"Improving machine learning iteration speed with faster application build and packaging","date":"January 29, 2024","format":false,"excerpt":"Slow build times and inefficiencies in packaging and distributing execution files were costing our ML\/AI engineers a significant amount of time while working on our training stack. By addressing these issues head-on, we were able to reduce this overhead by double-digit percentages.\u00a0 In the fast-paced world of AI\/ML development, it\u2019s\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":302,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/a-brief-history-of-rust-at-facebook\/","url_meta":{"origin":776,"position":4},"title":"A brief history of Rust at Facebook","date":"August 31, 2021","format":false,"excerpt":"Facebook is embracing Rust, one of the most loved and fastest-growing programming languages available today. In addition to bringing new talent to its Rust team, Facebook has announced that it is officially joining the nonprofit Rust Foundation. Alongside fellow members including Mozilla (the creators of Rust), AWS, Microsoft, and Google,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":827,"url":"https:\/\/fde.cat\/index.php\/2024\/02\/20\/aligning-velox-and-apache-arrow-towards-composable-data-management\/","url_meta":{"origin":776,"position":5},"title":"Aligning Velox and Apache Arrow: Towards composable data management","date":"February 20, 2024","format":false,"excerpt":"We\u2019ve partnered with Voltron Data and the Arrow community to align and converge Apache Arrow with Velox, Meta\u2019s open source execution engine. Apache Arrow 15 includes three new format layouts developed through this partnership: StringView, ListView, and Run-End-Encoding (REE). This new convergence helps Meta and the larger community build data\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/776","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=776"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/776\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=776"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=776"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}