{"id":697,"date":"2023-04-06T16:00:52","date_gmt":"2023-04-06T16:00:52","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/04\/06\/build-faster-with-buck2-our-open-source-build-system\/"},"modified":"2023-04-06T16:00:52","modified_gmt":"2023-04-06T16:00:52","slug":"build-faster-with-buck2-our-open-source-build-system","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/04\/06\/build-faster-with-buck2-our-open-source-build-system\/","title":{"rendered":"Build faster with Buck2: Our open source build system"},"content":{"rendered":"

Buck2, our <\/span>new open source, large-scale build system<\/span><\/a>, is now available on GitHub.<\/span>
\nBuck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient.\u00a0<\/span>
\nIn our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1.<\/span><\/p>\n

Buck2, Meta\u2019s open source large-scale build system, is now publicly available via the<\/span> Buck2 website<\/span><\/a> and<\/span> the Buck2 GitHub repository<\/span><\/a>. While it shares some commonalities with other build systems (like Buck1<\/a> and Bazel<\/a>), Buck2 is a from-scratch rewrite. Buck2 features a complete separation of the core and language-specific rules, with increased parallelism, integration with remote execution and virtual file systems, and a redesigned console output. All of these changes are aimed at helping engineers and developers spend less time waiting, and more time iterating on their code.<\/span><\/p>\n

Thousands of developers at Meta are already using Buck2 and performing millions of builds per day, with builds completing twice as fast as with Buck1. Our own internal analysis has shown that engineers were able to produce meaningfully more code when their builds were executed by Buck2, and we hope the wider industry will also see benefits.<\/span><\/p>\n

Why rebuild Buck?<\/span><\/h2>\n

Build systems stand between a programmer and running their code, so anything we can do to make the experience quicker or more productive directly impacts how effective a developer can be. The goal of Buck2 was to keep what we liked about Buck1 (the core concepts and workflows), draw inspiration from innovations after Buck1 (including <\/span>Bazel<\/span><\/a>, <\/span>Adapton<\/span><\/a>, and <\/span>Shake<\/span><\/a>), and focus on speed and enabling new experiences.<\/span><\/p>\n

Buck2\u2019s design is based on the following principles:<\/span><\/p>\n

The core build system has no knowledge of any language-specific rules. Having the rules separated from the core means that the rules are easier to change and understand. The core of Buck2 is written in Rust, and its language rules (like how to build C++) are written in <\/span>Starlark<\/span><\/a>. This separation is in contrast to Buck1 (where all rules are written in the core) and Bazel (where C++\/Java are written in the core). <\/span>
\nThe build system is powered by a single incremental dependency graph<\/a>, avoiding any phases (in contrast to Buck1 or Bazel). This decision eliminates many types of bugs and increases parallelism.<\/span>
\nThe rules API is designed to contain advanced features for performance, along with dynamic (or monadic) dependency features for expressibility. At the same time, these features are carefully restricted to ensure other properties (for example, fast queries or hermeticity) are not harmed.<\/span>
\nThe open source release is almost identical to our internal version. The only pieces swapped out are the toolchains (which point at the internal copies of our compilers) and remote execution (which points at our internal servers) <\/span>\u2014 <\/span>both have open source alternatives supplied. We are also releasing all the rules exactly as they are used internally. Furthermore, we have separated some of the logical components into separate crates (e.g. <\/span>Starlark<\/span><\/a>, <\/span>Superconsole<\/span><\/a>, <\/span>Allocative<\/span><\/a>, <\/span>Gazebo<\/span><\/a>) so that they can be used outside Buck2.<\/span>
\nBuck2 is written to integrate with remote execution, with the ability to run actions on remote machines. We use the same <\/span>API as Bazel<\/span><\/a>, and have been testing remote execution with <\/span>Buildbarn<\/span><\/a> and <\/span>EngFlow<\/span><\/a>. While not required (and not really expected for people starting out with the open source version), we are able to efficiently compute recursive digests and send them to remote execution efficiently.<\/span>
\nBuck2 is written to integrate with virtual file systems, where the entire repository is not all checked out, but fetched on demand as the files are accessed. In particular, we support <\/span>Sapling-based file systems<\/span><\/a>. To integrate well, we watch for file notifications (with <\/span>Watchman<\/span><\/a>) and request both files and file-digests without direct file operations. The benefit is that we can make virtual file systems as fast as a full checkout, but with the benefits of much faster checkout and much lower disk usage.<\/span><\/p>\n

The key takeaway from all these improvements is that we have designed Buck2 to be fast. In real world usage, depending on the build, Buck2 is significantly faster than Buck1. If there are no source code changes, Buck2 is almost instant on subsequent builds. If there is a lot of work to do, Buck2 starts executing faster and has greater parallelism. This increase in speed is both a consequence of many of the factors above, but also care and attention.<\/span><\/p>\n

The user view<\/span><\/h2>\n

For end users, Buck2 works mostly the same as Buck1 (which, to a first approximation, is fairly similar to Bazel). A user defines targets in a <\/span>BUCK<\/span> file:<\/span><\/p>\n

rust_binary(
\n\u00a0\u00a0\u00a0\u00a0name = \u201cmy_binary\u201d,
\n\u00a0\u00a0\u00a0\u00a0srcs = [\u201cmain.rs\u201d],
\n\u00a0\u00a0\u00a0\u00a0deps = [\u201c:my_library\u201d],
\n)<\/p>\n

A user can then build with <\/span>buck2 build \/\/:my_binary<\/span><\/span>. The value <\/span>main.rs<\/span> is a source file, and <\/span>:my_library<\/span> is a dependency defined in the same <\/span>BUCK<\/span> file. It\u2019s worth noting that Buck2 is mostly compatible with the <\/span>BUCK<\/span> files of Buck1.\u00a0<\/span><\/p>\n

As well as the increase in speed, there are two major additional user-visible differences compared to Buck1.<\/span><\/p>\n

First, the console output has been redesigned on top of the <\/span>Superconsole library<\/span><\/a>, which we specifically developed for Buck2. The console shows a few more details and feels a lot nicer to use:<\/span><\/p>\n\n

Second, there is a persistent daemon that maintains a single dependency graph. When you change a <\/span>BUCK<\/span> file, a dependency, or a source file, we invalidate the appropriate things on the dependency graph, then request the output artifacts per the command line. In Buck1 there are multiple distinct dependency graphs, which result in phases like target graph construction, action graph construction, and then action graph execution. There are also some operations that aren\u2019t performed on the graph. If certain things change in Buck1, then entire graphs are thrown away, rather than the minimum pieces being invalidated. With a single dependency graph, Buck2 is simpler, avoids more redundant work, and avoids explicit phases. Everything on the dependency graph has a key (how it is identified) and a value, along with a function to compute the value from the key and other related keys (following the model in the paper, \u201c<\/span>Build Systems a la Carte\u201d<\/span><\/a>).<\/span><\/p>\n

The rule author view<\/span><\/h2>\n

While the user model follows Buck1 very closely, the approach for rules is completely different. There are lots of rules in Buck, for example <\/span>rust_binary<\/span> used above. While a rule in Buck1 was a Java class, baked into Buck1, a rule in Buck2 is entirely decoupled. Buck2 also ships with a \u201cprelude\u201d of rules that implement most of the Buck1 rules.\u00a0<\/span><\/p>\n

Buck1 rules were tuned over time, had lots of performance optimizations and powerful features like graph traversal, but those rules were also expected to obey a lot of complex invariants<\/span>\u2014<\/span>sometimes breaking those rules. For Buck2, the rule API is entirely in Starlark, which forced us to abstract those features as generically reusable APIs, aiming to make them safe, expressive, and performant<\/span>\u2014<\/span>a tricky balance. We\u2019ll touch on two such examples.<\/span><\/p>\n

OCaml dependencies<\/span><\/h3>\n

The dependency structure of the OCaml library is hard to express in Buck1. An OCaml library consists of a number of OCaml files. These must be compiled in dependency order<\/span>\u2014<\/span>so if <\/span>A.ml<\/span> uses <\/span>B.ml<\/span>, you must compile <\/span>B.ml<\/span> first. Bazel requires the dependency of <\/span>A.ml<\/span> on <\/span>B.ml<\/span> to be written explicitly in the <\/span>BUCK<\/span> file. Buck1 and Buck2 both leave that internal dependency implicit and run the tool <\/span>ocamldep<\/span> to infer it, which requires less maintenance as the structure changes.\u00a0 What Buck1 did is run <\/span>ocamldep<\/span> just after parsing the <\/span>BUCK<\/span> file, which wasn\u2019t really allowed, and it didn\u2019t track dependencies, so if you changed the imports too much Buck1 gave spurious compilation failures. With Buck2, we can use the <\/span>new primitive <\/span>dynamic_output<\/span><\/a>, which lets you run a command, read the output of the file, then wire up the rest of the graph<\/span>\u2014<\/span>putting in the correct dependencies between the <\/span>.ml<\/span> files automatically.<\/span><\/p>\n

C++ link dependencies<\/span><\/h3>\n

Consider the C++ linking model:<\/span> To produce a library, you usually need to link together its build output, along with the transitive closure of the build output of its dependencies. If you simply duplicate the set of dependencies at each layer as you move up the graph, you end up with <\/span>O(n<\/span>2<\/span>)<\/span> memory usage. In Buck1, there was custom code in many rules to capture this pattern, relying on the ability to share Java values in memory and for the dependencies to be represented in place within the rule structure (as there was no reified dependency graph). In Buck2, there are much stronger abstraction boundaries, so such reuse has to be made more explicit. Therefore, we <\/span>introduced transitive-sets (tsets)<\/span><\/a> to capture this pattern of sets representing a transitive closure. By making tsets more abstract, we were also able to wire the tset directly into the underlying dependency graph, meaning this representation is efficient in both memory and computation time.<\/span><\/p>\n

Try Buck2 now<\/span><\/h2>\n

We\u2019re keen for people to give Buck2 a try, and we would be happy to hear any feedback (<\/span>GitHub issues<\/span><\/a> are the best way). We expect Buck2 will be most interesting to moderately sized multi-language projects. Visit the <\/span>Buck2 getting started page<\/span><\/a> for more information.<\/span><\/p>\n

The post Build faster with Buck2: Our open source build system<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient.\u00a0 In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast as Buck1. Buck2, Meta\u2019s open… Continue reading Build faster with Buck2: Our open source build system<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-697","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":776,"url":"https:\/\/fde.cat\/index.php\/2023\/10\/23\/5-things-you-didnt-know-about-buck2\/","url_meta":{"origin":697,"position":0},"title":"5 Things you didn\u2019t know about Buck2","date":"October 23, 2023","format":false,"excerpt":"Meta has a very large monorepo, with many\u00a0 different programming languages. To optimize build and performance, we developed our own build system called Buck, which was first open-sourced in 2013.\u00a0 Buck2 is the recently open-sourced successor. In our internal tests at Meta, we observed that Buck2 completed builds approximately 2x\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":728,"url":"https:\/\/fde.cat\/index.php\/2023\/06\/27\/meta-developer-tools-working-at-scale\/","url_meta":{"origin":697,"position":1},"title":"Meta developer tools: Working at scale","date":"June 27, 2023","format":false,"excerpt":"Every day, thousands of developers at Meta are working in repositories with millions of files. Those developers need tools that help them at every stage of the workflow while working at extreme scale. In this article we\u2019ll go through a few of the tools in the development process. And, as\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":820,"url":"https:\/\/fde.cat\/index.php\/2024\/02\/06\/dotslash-simplified-executable-deployment\/","url_meta":{"origin":697,"position":2},"title":"DotSlash: Simplified executable deployment","date":"February 6, 2024","format":false,"excerpt":"We\u2019ve open sourced DotSlash, a tool that makes large executables available in source control with a negligible impact on repository size, thus avoiding I\/O-heavy clone operations. With DotSlash, a set of platform-specific executables is replaced with a single script containing descriptors for the supported platforms. DotSlash handles transparently fetching, decompressing,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":818,"url":"https:\/\/fde.cat\/index.php\/2024\/01\/29\/improving-machine-learning-iteration-speed-with-faster-application-build-and-packaging\/","url_meta":{"origin":697,"position":3},"title":"Improving machine learning iteration speed with faster application build and packaging","date":"January 29, 2024","format":false,"excerpt":"Slow build times and inefficiencies in packaging and distributing execution files were costing our ML\/AI engineers a significant amount of time while working on our training stack. By addressing these issues head-on, we were able to reduce this overhead by double-digit percentages.\u00a0 In the fast-paced world of AI\/ML development, it\u2019s\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":651,"url":"https:\/\/fde.cat\/index.php\/2022\/11\/15\/sapling-source-control-thats-user-friendly-and-scalable\/","url_meta":{"origin":697,"position":4},"title":"Sapling: Source control that\u2019s user-friendly and scalable","date":"November 15, 2022","format":false,"excerpt":"Sapling is a new Git-compatible source control client. Sapling emphasizes usability while also scaling to the largest repositories in the world. ReviewStack is a demonstration code review UI for GitHub pull requests that integrates with Sapling to make reviewing stacks of commits easy. You can get started using Sapling today.\u00a0\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":258,"url":"https:\/\/fde.cat\/index.php\/2021\/02\/07\/open-sourcing-thrift-for-haskell\/","url_meta":{"origin":697,"position":5},"title":"Open-sourcing Thrift for Haskell","date":"February 7, 2021","format":false,"excerpt":"What it is: Thrift is a serialization and remote procedure call (RPC) framework used for cross-service communication. Most services at Facebook communicate via Thrift because it provides a simple, language-agnostic protocol for communicating with structured data. Thrift can already be used in programming languages such as C++, Python, and Java\u2026","rel":"","context":"In "External"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/697","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=697"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/697\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=697"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=697"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=697"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}