{"id":875,"date":"2024-06-10T16:30:55","date_gmt":"2024-06-10T16:30:55","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2024\/06\/10\/serverless-jupyter-notebooks-at-meta\/"},"modified":"2024-06-10T16:30:55","modified_gmt":"2024-06-10T16:30:55","slug":"serverless-jupyter-notebooks-at-meta","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2024\/06\/10\/serverless-jupyter-notebooks-at-meta\/","title":{"rendered":"Serverless Jupyter Notebooks at Meta"},"content":{"rendered":"

At Meta, <\/span>Bento<\/span><\/a>, our internal <\/span>Jupyter<\/span><\/a> notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from what we call \u201clite\u201d workloads that involve simple prototyping to heavier and more complex machine learning workflows. However, even though the lite workflows require limited compute, users still have to go through the same process of reserving and provisioning remote compute \u2013 a process that takes time \u2013 before the notebook is ready for any code execution.<\/span><\/p>\n

To address this problem, we have invested in building infrastructure that allows for code execution directly in the browser, removing the need to provision remote compute for some lite workloads. This infrastructure leverages a library called <\/span>Pyodide<\/span><\/a> that sits on top of <\/span>WebAssembly<\/span><\/a> (<\/span>Wasm<\/span>)<\/span><\/p>\n

Here\u2019s how we married Bento with this in-browser, serverless code execution technology to power our notebooks platform for these lite workloads.<\/span><\/p>\n

The motivation for supporting lite workloads<\/span><\/h2>\n

We define lite workloads as workloads that only consume data from upstream systems, do not have side effects to our underlying systems, and use up to the maximum Chrome tab memory limit. We frequently get internal feedback from the owners of these lite workloads that the time and complexity in getting started is not proportionate to what they want to use Bento for.\u00a0<\/span><\/p>\n

The requirements can be summarized as follows:<\/span><\/p>\n

An intuitive startup process that works right out of the box\u00a0<\/span>
\nA startup process that is very quick and has the notebook immediately ready for execution\u00a0<\/span>
\nA startup process that does not include the complex remote compute reservation process\u00a0<\/span>
\nAn execution environment that supports the majority of the lite workloads<\/span><\/p>\n

How we put the pieces together<\/span><\/h2>\n\n

How this all works<\/span><\/h3>\n

Pyodide<\/span><\/a> (a Python distribution for the browser that runs on <\/span>WebAssembly<\/span><\/a>) is an important ingredient for this work. We\u2019ve built a kernel abstraction around this which, when called from Bento, will just work as any of the classic kernels we have (with some limitations) and perform message passing using the <\/span>Jupyter Protocol<\/span><\/a>.<\/span>
\n<\/span><\/p>\n

Kernel bridge\u00a0<\/span><\/h4>\n

This is just an abstraction that allows Bento to work with both traditional server-based kernels and this new browser-based kernel with no changes whatsoever to the rest of the system. The visible manifestation of this is just a selector in the notebook that toggles between server-based kernels and serverless.<\/span><\/p>\n\n

Magics<\/span><\/h4>\n

Cell magics<\/span><\/a> are an important\u00a0 component of the Bento extension platform. In order to allow existing custom cells to work with no changes, we built middleware to capture these cell magics, process them directly in the context of javascript, and then just inject the expected results back into the Python kernel. A good example of this pattern is around <\/span>%%sql<\/span>, which we use to power our custom SQL <\/span>cell.\u00a0<\/span><\/p>\n

We\u2019ll showcase a few more examples in the section below on \u201cMeta-specific\u201d integrations.<\/span><\/p>\n

Why we need a webworker<\/span><\/h4>\n

Since J<\/span>avaScript<\/span> is single-threaded, in the absence of a webworker, the entire browser would just lock up when we have \u201cexpensive\u201d kernel operations. Having kernel operations run in a webworker with just the results being passed to the main thread helps mitigate this.<\/span><\/p>\n\n

Meta-specific integrations\u00a0<\/span><\/h2>\n

In order to unlock additional utility and have a coherent story around the extract, transform, and load (ETL) narrative, we built integrations with an initial set of existing extensions. These represent a relatively popular set of extensions that users leverage to perform data operations.<\/span><\/p>\n

SQL Cell<\/span><\/h3>\n

This leverages the <\/span>%%sql<\/span> magic to fetch data from the warehouse and make it available for further processing in the <\/span>Pyodide<\/span> kernel.<\/span><\/p>\n\n

Google Sheets\u00a0<\/span><\/h3>\n

Here, we leverage the <\/span>%%googlesheet<\/span> magic to fetch data from a Google sheet and make it available for further processing in the notebook.<\/span><\/p>\n\n

GraphQL<\/span><\/h3>\n

Here, we leverage <\/span>%%graphql<\/span> magic, which powers the GraphQL cell to make data fetches and then inject the result back into the kernel for further processing.<\/span><\/p>\n\n

Dataframe uploads<\/span><\/h3>\n

Data uploads are a bit trickier to pull off as compared to the data reads we showcased above. We instead achieve this functionality by:<\/span><\/p>\n

Leveraging the <\/span>%%dataframe<\/span> magic that powers the upload custom cell in order to fetch the arguments in a structured way.<\/span>
\nWe then kick off an async job using <\/span>Tupperware<\/span><\/a> (Meta\u2019s async tier compute platform) and show the status of the associated tupperware job in the cell output.<\/span><\/span><\/p>\n

What\u2019s next for serverless notebooks<\/span><\/h2>\n

While we\u2019ve addressed the initial set of challenges to bring this product online, there is still a lot of work to be done to improve the developer experience for users. Firstly, we\u2019re planning on improving the lite workloads heuristic. Once we have this figured out, the next step will involve defaulting all new workloads to start as serverless. Then we can quickly autodetect (based on memory requirements, data volumes, or libraries in use) whether the workload is lite enough. If not, we can automatically switch that notebook to leverage a server-based kernel with minimal interruption to the user flow.<\/span><\/p>\n

After this, we plan to integrate with more existing cell extensions built on top of the Bento platform and thus expand the scope of what\u2019s possible when running \u201cserverless.\u201d<\/span><\/p>\n

The biggest limitation with this approach at Meta is that homegrown libraries that have not been ported to WebAssembly will be unavailable. Given this, we\u2019re also planning to explore whether we can farm out the execution of specific \u201cnon-lite\u201d cells to our remote execution infrastructure while making this work seamlessly with Pyodide.<\/span><\/p>\n

Once these have been addressed, \u201cserverless\u201d notebooks will become the de facto landing experience in Bento.<\/span><\/p>\n

Acknowledgments\u00a0<\/span><\/h2>\n

Some of the approaches we took were directly inspired by the work done on <\/span>JupyterLite<\/span><\/a> and directly leverages the <\/span>Pyodide<\/span><\/a> library without which this project would not have been possible. I\u2019d also like to thank all the engineers at Meta I collaborated with to make this project a reality.<\/span><\/p>\n

The post Serverless Jupyter Notebooks at Meta<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

At Meta, Bento, our internal Jupyter notebooks platform, is a popular tool that allows our engineers to mix code, text, and multimedia in a single document. Use cases run the entire spectrum from what we call \u201clite\u201d workloads that involve simple prototyping to heavier and more complex machine learning workflows. However, even though the lite… Continue reading Serverless Jupyter Notebooks at Meta<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-875","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":753,"url":"https:\/\/fde.cat\/index.php\/2023\/08\/29\/scheduling-jupyter-notebooks-at-meta\/","url_meta":{"origin":875,"position":0},"title":"Scheduling Jupyter Notebooks at Meta","date":"August 29, 2023","format":false,"excerpt":"At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals. Users with such notebooks would have to remember to manually\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":566,"url":"https:\/\/fde.cat\/index.php\/2022\/04\/26\/sql-notebooks-combining-the-power-of-jupyter-and-sql-editors-for-data-analytics\/","url_meta":{"origin":875,"position":1},"title":"SQL Notebooks: Combining the power of Jupyter and SQL editors for data analytics","date":"April 26, 2022","format":false,"excerpt":"At Meta, our internal data tools are the main channel from our data scientists to our production engineers. As such, it\u2019s important for us to empower our scientists and engineers not only to use data to make decisions, but also to do so in a secure and compliant way. We\u2019ve\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":758,"url":"https:\/\/fde.cat\/index.php\/2023\/09\/07\/using-chakra-execution-traces-for-benchmarking-and-network-performance-optimization\/","url_meta":{"origin":875,"position":2},"title":"Using Chakra execution traces for benchmarking and network performance optimization","date":"September 7, 2023","format":false,"excerpt":"Meta presents Chakra execution traces, an open graph-based representation of AI\/ML workload execution, laying the foundation for benchmarking and network performance optimization. Chakra execution traces represent key operations, such as compute, memory, and communication, data and control dependencies, timing, and resource constraints. In collaboration with MLCommons, we are seeking industry-wide\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":697,"url":"https:\/\/fde.cat\/index.php\/2023\/04\/06\/build-faster-with-buck2-our-open-source-build-system\/","url_meta":{"origin":875,"position":3},"title":"Build faster with Buck2: Our open source build system","date":"April 6, 2023","format":false,"excerpt":"Buck2, our new open source, large-scale build system, is now available on GitHub. Buck2 is an extensible and performant build system written in Rust and designed to make your build experience faster and more efficient.\u00a0 In our internal tests at Meta, we observed that Buck2 completed builds 2x as fast\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":759,"url":"https:\/\/fde.cat\/index.php\/2023\/09\/07\/arcadia-an-end-to-end-ai-system-performance-simulator\/","url_meta":{"origin":875,"position":4},"title":"Arcadia: An end-to-end AI system performance simulator","date":"September 7, 2023","format":false,"excerpt":"We\u2019re introducing Arcadia, Meta\u2019s unified system that simulates the compute, memory, and network performance of AI training clusters. Extracting maximum performance from an AI cluster and increasing overall efficiency warrants a multi-input system that accounts for various hardware and software parameters across compute, storage, and network collectively. Arcadia gives Meta\u2019s\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":694,"url":"https:\/\/fde.cat\/index.php\/2023\/03\/23\/big-data-processing-driving-data-migration-for-salesforce-data-cloud\/","url_meta":{"origin":875,"position":5},"title":"Big Data Processing: Driving Data Migration for Salesforce Data Cloud","date":"March 23, 2023","format":false,"excerpt":"The tsunami of data \u2014 set to exceed 180 zettabytes by 2025 \u2014 places significant pressure on companies. Simply having access to customer information is not enough \u2014 companies must also analyze and refine the data to find actionable pieces that power new business. As businesses collect these volumes of\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/875","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=875"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/875\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=875"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=875"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=875"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}