{"id":632,"date":"2022-09-12T16:00:10","date_gmt":"2022-09-12T16:00:10","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/09\/12\/memlab-an-open-source-framework-for-finding-javascript-memory-leaks\/"},"modified":"2022-09-12T16:00:10","modified_gmt":"2022-09-12T16:00:10","slug":"memlab-an-open-source-framework-for-finding-javascript-memory-leaks","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/09\/12\/memlab-an-open-source-framework-for-finding-javascript-memory-leaks\/","title":{"rendered":"MemLab: An open source framework for finding JavaScript memory leaks"},"content":{"rendered":"

We\u2019ve open-sourced<\/span> MemLab<\/span>, a JavaScript memory testing framework that automates memory leak detection.<\/span>
\nF<\/span>inding and addressing the root cause of memory leaks is important for delivering a quality user experience on web applications.<\/span>
\nMemLab has helped engineers and developers at Meta improve user experience and make significant improvements in memory optimization. We hope it will do the same for the larger JavaScript community as well.<\/span><\/p>\n

In 2020, we<\/span> redesigned Facebook.com<\/span><\/a> as a single-page application (SPA) that does most of its rendering and navigation using client-side JavaScript. We used a similar architecture to build most of Meta\u2019s other popular web apps, including Instagram and Workplace. And while this architecture allows us to provide faster user interactions, a better developer experience, and a more app-like feel, maintaining the web app state on the client makes effectively managing client-side memory more complex.<\/span><\/p>\n

People using our web apps will often notice performance and functional correctness issues immediately. A memory leak, however, is a different story. It isn\u2019t immediately perceivable, because it eats up a chunk of memory at a time \u2014 affecting the entire web session and making subsequent interactions slower and less responsive.<\/span><\/p>\n

To help our developers address this, we built<\/span> MemLab<\/span><\/a>, a JavaScript memory testing framework that automates leak detection and makes it easier to root-cause memory leaks. We\u2019ve used MemLab at Meta to successfully contain unsustainable memory increases and identify memory leaks and memory optimization opportunities across our products and infra.<\/span><\/p>\n

We\u2019ve already<\/span> open-sourced MemLab on GitHub<\/span><\/a>, and we\u2019re excited to work with the JavaScript community and have developers start using MemLab today.<\/span><\/p>\n

Why we developed MemLab<\/span><\/h2>\n

Historically, we spent a lot of time<\/span> measuring, optimizing, and controlling page load and interaction time<\/span><\/a>, as well as JavaScript code size. We built automated systems that alerted us when there were regressions in these metrics \u2014 both before and after code landed in production \u2014 so that we could act quickly to fix these issues and prevent the changes from ever landing in production.<\/span><\/p>\n

Comparatively, we hadn\u2019t done much work for managing web in-browser memory. And when we analyzed the new Facebook.com\u2019s memory usage, we found that both memory usage and the number of out-of-memory (OOM) crashes on the client side had been climbing.<\/span><\/p>\n

High memory usage has a statistically significant and negative impact on:<\/span><\/p>\n

Page load and interaction performance (how much time it takes to load a page or perform an interaction)<\/span>
\nUser engagement metrics (active users, time spent on site, number of activities performed)<\/span><\/p>\n

What causes high memory usage in web applications?<\/span><\/h2>\n

Because memory leaks aren\u2019t usually obvious, they seldom get caught in code review, are hard to spot during development, and are often difficult to root-cause in production. But mainstream JavaScript runtimes all have garbage collectors, so how is memory leaking in the first place?<\/span><\/p>\n

JavaScript code can experience memory leaks by keeping hidden references to objects. Hidden references can cause memory leaks in many unexpected ways.<\/span><\/p>\n

For example:<\/span><\/p>\n

var obj = {};
\nconsole.log(obj);
\nobj = null;<\/p>\n

In Chrome, this code leaks <\/span>obj <\/span>even though we set the reference to <\/span>null<\/span>. This happens because Chrome needs to keep an internal reference to the printed object so that it can be inspected in the web console later (even if the web console is not opened).<\/span><\/p>\n

There can also be cases where memory is not technically leaked but grows linearly and unbounded during a user session. The most common causes of this are client-side caches that don\u2019t have any eviction logic built in and infinite scroll lists that don\u2019t have any virtualization to remove earlier items from the list as new content is added.<\/span><\/p>\n

We also didn\u2019t have automated systems and processes in place to control memory, so the only defense against regressions was experts periodically digging into memory leaks via Chrome DevTools, which was not scalable considering we\u2019re shipping a nontrivial number of changes every day.<\/span><\/p>\n

How MemLab works<\/span><\/h2>\n

In a nutshell, MemLab finds memory leaks by running a headless browser through predefined test scenarios and diffing and analyzing the JavaScript heap snapshots.<\/span><\/p>\n

This process happens in six steps:<\/span><\/p>\n

1. Browser interaction<\/span><\/h3>\n

To find leaked objects on a target page (B). MemLab automates a browser using Puppeteer and visits the test pages in the following order:<\/span><\/p>\n

Navigate to a different tab (A) and get heap SA.<\/span>
\nNavigate to the target page (B) and get heap SB.<\/span>
\nCome back to the previous page (A\u2019) and get heap SA\u2019.<\/span><\/p>\n

2. Diffing the heap\u00a0<\/span><\/h3>\n

When we navigate to a page and then navigate away from it, we would expect most of the memory allocated by that page to also be freed \u2014 if not, it is highly suggestive of a memory leak. MemLab finds potential memory leaks by diffing the JavaScript heap and recording the set of objects allocated on page B that weren\u2019t allocated on Page A but are still present when Page A is reloaded. Or, more formally, the superset of objects leaked from the target page can be derived as<\/span> (SB SA) \u2229 SA\u2019 ).<\/span><\/p>\n

3. Refining the list of memory leaks<\/span><\/h3>\n

The leak detector further incorporates framework-specific knowledge to refine the list of leaked objects. For example, Fiber nodes allocated by React (an inner data structure React uses for rendering the virtual DOM) should be released when we clean up after visiting multiple tabs.<\/span><\/p>\n

4. Generating retainer traces<\/span><\/h3>\n

MemLab traverses the heap and generates retainer traces for each leaked object. A retainer trace is an object reference chain from the GC roots (the entry objects in a heap graph from which garbage collectors traverse the heap) to a leaked object. The trace shows why and how a leaked object is kept alive in memory. Breaking the reference chain means the leaked object will no longer be reachable from the GC root and therefore can be garbage collected. By following the retainer trace one step at a time, you can find the reference that should be set to null (but wasn\u2019t, due to a bug).<\/span><\/p>\n

5. Clustering retainer traces<\/span><\/h3>\n

Sometimes certain interactions can trigger thousands of leaked objects. It would be overwhelming to show all the retainer traces in this case. MemLab clusters all retainer traces and shows one trace for each cluster of leaked objects that share similar retainer traces. The trace also includes debug information, such as<\/span> dominator nodes and retained sizes<\/span><\/a>.<\/span><\/p>\n

6. Reporting the leaks<\/span><\/h3>\n

We run MemLab at regular intervals throughout the day to get a continuous signal on memory regressions. Any new regressions are added to an internal dashboard, where clustered retainer traces of all memory leaks detected are gathered and categorized. Developers can then click and view the properties of objects on the retainer trace of each memory leak.<\/span><\/p>\n

(Note: This dashboard is not part of the open source release of MemLab, but something similar could be added to any CI\/CD pipeline.)<\/span><\/p>\n

MemLab\u2019s features<\/span><\/h2>\n

Memory leak detection<\/span><\/h3>\n

For in-browser memory leak detection, the only input MemLab requires from developers is<\/span> a test scenario file<\/span><\/a> that<\/span> defines how to interact with the webpage by overriding three callbacks<\/span><\/a> with the Puppeteer API and CSS selectors. MemLab automatically diffs the JavaScript heap, refines memory leaks, and aggregates results.<\/span><\/p>\n\n

Graph-view API of JavaScript heap<\/span><\/h3>\n

MemLab supports a<\/span> self-defined leak detector as a filter callback<\/span><\/a> that is applied to each leak candidate object allocated by the target interaction but never released afterwards. The leak filter callback can traverse the heap and decide which objects are memory leaks. As an example, our built-in leak detector follows the return chain of a React Fiber node and checks if the Fiber node is detached from the React Fiber tree.<\/span><\/p>\n\n

To allow the context of each candidate leak to be analyzed, MemLab provides a memory-efficient graph view of the JavaScript heap. This enables querying and traversing the JavaScript heap without having any domain knowledge about V8\u2019s heap snapshot file structure.<\/span><\/p>\n

In the graph view, each JavaScript object or native object in the heap is a graph node, and each JavaScript reference in the heap is a graph edge. The heap size of a real application is often large, so the graph view needs to be memory-efficient while providing an intuitive object-oriented heap-traversal API. Therefore, the graph nodes are designed to be virtual and not interconnected through JavaScript references. When the analysis code traverses the heap, the virtual graph view partially constructs the touched section of the graph just-in-time. Any part of the graph can be easily deallocated since those virtual nodes do not have JavaScript references to each other.<\/span><\/p>\n

The heap graph view can be loaded from JavaScript heap snapshots taken from Chromium-based browsers, Node.js, Electron, and<\/span> Hermes<\/span><\/a>. This allows for complex patterns to be analyzed and answers questions such as, \u201cHow many React Fiber nodes are alternate Fiber nodes, which are used in\u00a0 incomplete concurrent renders?\u201d or \u201cWhat is the total retained size of unmounted react components?\u201d.<\/span><\/p>\n

import {getHeapFromFile} from ‘@memlab\/heap-analysis’;
\nconst heapGraph = await getHeapFromFile(heapFile);
\nheapGraph.nodes.forEach(node => {
\n \/\/ heap node traversal
\n node.type
\n node.references
\n);<\/p>\n

Memory assertions<\/span><\/h3>\n

A Node.js program or<\/span> Jest<\/span><\/a> test can also use the graph-view API to get a heap graph view of its own state, to do self-memory checking, and<\/span> write all kinds of memory assertions<\/span><\/a>.<\/span><\/p>\n

import type {IHeapSnapshot} from ‘@memlab\/core’;
\nimport {config, takeNodeMinimalHeap, tagObject} from ‘@memlab\/core’;<\/p>\n

test(‘memory test’, async () => {
\n config.muteConsole = true;
\n const o1 = {};
\n let o2 = {};<\/p>\n

\/\/ tag o1 with marker: “memlab-mark-1”, does not modify o1 in any way
\n tagObject(o1, ‘memlab-mark-1’);
\n \/\/ tag o2 with marker: “memlab-mark-2”, does not modify o2 in any way
\n tagObject(o2, ‘memlab-mark-2’);<\/p>\n

o2 = null;<\/p>\n

const heap: IHeapSnapshot = await takeNodeMinimalHeap();<\/p>\n

\/\/ expect object with marker “memlab-mark-1” exists
\n expect(heap.hasObjectWithTag(‘memlab-mark-1’)).toBe(true);<\/p>\n

\/\/ expect object with marker “memlab-mark-2” can be GCed
\n expect(heap.hasObjectWithTag(‘memlab-mark-2’)).toBe(false);<\/p>\n

}, 30000);<\/p>\n

Memory toolbox<\/span><\/h3>\n

In addition to memory leak detection, MemLab includes a set of built-in<\/span> CLI commands<\/span><\/a> and<\/span> APIs<\/span><\/a> for finding memory optimization opportunities:<\/span><\/p>\n

Break down heap by object shapes<\/span><\/a> (e.g., <\/span>{arguments, postRun, preRun, quite, thisProgram, \u2026}<\/span>) instead of classifying objects based on constructor names (e.g., Object). This is useful to quickly detect and root-cause significant memory used by object literals.<\/span>
\nDetect continuous individual object growth<\/span><\/a> or <\/span>object shape growth<\/span><\/a>. MemLab can take a series of heap snapshots as input and find which object or class of objects keeps growing in size over time.<\/span>
\nFind duplicate string instances<\/span><\/a>. V8 does not always do string interning optimization, which means two JavaScript string primitives with the same value could be represented by two different native objects in V8\u2019s heap.<\/span><\/p>\n

Internal UI for browsing memory leaks. (mockup, for illustration purposes only)
\nInternal UI for diagnosing memory leak traces. (mockup, for illustration purposes only)<\/p>\n

Using MemLab at Meta: Case studies\u00a0<\/span><\/h2>\n

Over the past few years, we\u2019ve used MemLab to detect and diagnose memory leaks and have gathered insights that have helped us optimize memory, significantly improve memory and reliability (reducing OOM crashes), and improve user experience.<\/span><\/p>\n

We reduced OOM crashes on Facebook.com by 50 percent in the first half of 2021.<\/p>\n

React Fiber node cleanup<\/span><\/h3>\n

For rendered components, React builds a Fiber tree \u2014 an inner data structure React uses for rendering the virtual DOM. Although the Fiber tree looks like a tree, it is a bidirectional graph that strongly connects all Fiber nodes, React component instances, and the associated HTML DOM elements. Ideally, React maintains references to the root of the component\u2019s Fiber tree and keeps the Fiber tree from being garbage collected. When a component is unmounted, React breaks the connection between the host root of the component and the rest of the Fiber tree, which can then be garbage collected.<\/span><\/p>\n

The downside of having a strongly connected graph is that if there is any outside reference pointing to any part of the graph, the whole graph cannot be garbage collected. For example, the following export statement caches React components at the module scope level, so the associated Fiber tree and detached DOM elements are never released.<\/span><\/p>\n

export<\/strong> const Component = ((
\n <List> … <\/List>
\n): React.Element<typeof List>);<\/p>\n

It\u2019s also not just the React data structures that would be kept alive. Hooks and their closures can also keep alive all kinds of other objects. This means that a single React component leak could cause the leak of a significant part of a page\u2019s objects, leading to huge memory leaks.<\/p>\n\n

To prevent the cascading effect of memory leaks in the Fiber tree,<\/span> we added a full traversal of a tree that does aggressive cleanup<\/span><\/a> when a component is unmounted in React 18 (thanks to Benoit Girard and the React team). This allows the garbage collector to do a better job at cleaning up an unmounted tree. Any accidental memory leak is bounded to a smaller leak size. This fix reduced average memory usage on Facebook by almost 25 percent, and we saw large improvements across other sites using React when they upgraded. We were worried that this aggressive cleanup could slow down unmounting in React, but surprisingly, we saw a significant performance win because of memory reductions.<\/span><\/p>\n

Relay string interning<\/span><\/h3>\n

By leveraging the heap analysis APIs in MemLab, we found that strings occupied 70 percent of the heap, and half of those strings had at least one duplicated instance. (V8 doesn\u2019t always do string interning, which is an optimization that deduplicates string instances with the same value.)<\/span><\/p>\n

When we further queried duplicated string patterns and clustering retainer traces we found that a significant portion of string memory is consumed by cached key strings in Relay. Relay generates cached keys for fragments by doing duplication, serialization, and concatenation. The string keys are further copied and concatenated for cached keys of other resources (e.g., fragments) in Relay. By collaborating with the Relay and React Apps teams, we optimized Relay cache key strings by interning and shortening overlong string keys on the client side.<\/span><\/p>\n

This optimization enabled Relay to cache more data, allowing the site to show more content to users, especially when there is limited RAM on the client side. We observed a 20 percent reduction in memory p99 as well as OOM crashes, faster page rendering, improved user experience, and revenue wins.<\/span><\/p>\n

Try MemLab today<\/span><\/h2>\n

We\u2019re excited to open-source MemLab and have developers start using it. We are especially interested in seeing which use cases the community finds useful.<\/span><\/p>\n

You can install MemLab through<\/span> npm<\/span><\/a> or build it from the<\/span> GitHub repo<\/span><\/a>:<\/span><\/p>\n

npm i -g memlab<\/span><\/p>\n

We also have a <\/span>quick start guide<\/span><\/a> to help developers get started.<\/span><\/p>\n

If you\u2019ve tried MemLab out on your own project, please <\/span>reach out<\/span><\/a> and let us know how it worked for you!<\/span><\/p>\n

Acknowledgements<\/h2>\n

We\u2019d like to thank Tulga Narmandakh and the rest of the Core Health Experience team for their contributions toward helping open-source MemLab.<\/span><\/p>\n

The post MemLab: An open source framework for finding JavaScript memory leaks<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

We\u2019ve open-sourced MemLab, a JavaScript memory testing framework that automates memory leak detection. Finding and addressing the root cause of memory leaks is important for delivering a quality user experience on web applications. MemLab has helped engineers and developers at Meta improve user experience and make significant improvements in memory optimization. We hope it will… Continue reading MemLab: An open source framework for finding JavaScript memory leaks<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-632","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":506,"url":"https:\/\/fde.cat\/index.php\/2021\/11\/18\/measuring-the-memory-impact-for-hybrid-apps\/","url_meta":{"origin":632,"position":0},"title":"Measuring the Memory Impact for Hybrid Apps","date":"November 18, 2021","format":false,"excerpt":"Memory problems are always challenging to detect and fix for mobile applications, particularly on Android, due to many hardware profiles, OS versions, and OEM skins. With proper memory reporting and analysis, most issues are caught during the development lifecycle. Yet if your application is delivering an entire platform, such as\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":576,"url":"https:\/\/fde.cat\/index.php\/2022\/05\/11\/meta-open-source-is-transferring-jest-to-the-openjs-foundation\/","url_meta":{"origin":632,"position":1},"title":"Meta Open Source is transferring Jest to the OpenJS Foundation","date":"May 11, 2022","format":false,"excerpt":"Meta Open Source is officially transferring Jest, its open source JavaScript testing framework, to the OpenJS Foundation.\u00a0 With over 17 million weekly downloads and over 38,000 GitHub stars, Jest is the most used testing framework in the JavaScript ecosystem and is used by companies of all sizes, including Amazon, Google,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":609,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/20\/using-hermess-quicksort-to-run-doom-a-tale-of-javascript-exploitation\/","url_meta":{"origin":632,"position":2},"title":"Using Hermes\u2019s Quicksort to run Doom: A tale of JavaScript exploitation","date":"July 20, 2022","format":false,"excerpt":"At Meta, our Bug Bounty program is an important element of our \u201cdefense-in-depth\u201d approach to security. Our internal product security teams investigate every bug submission to assess its maximum potential impact so that we can always reward external researchers based on both the bug they found and our further internal\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":637,"url":"https:\/\/fde.cat\/index.php\/2022\/09\/30\/launching-a-new-chromium-based-webview-for-android\/","url_meta":{"origin":632,"position":3},"title":"Launching a new Chromium-based WebView for Android","date":"September 30, 2022","format":false,"excerpt":"Our in-app browser for Facebook on Android has historically relied on an Android System WebView based on Chromium, the open source project that powers many browsers on Android and other operating systems. On other mobile operating systems, the System WebView component cannot be updated without updating the entire operating system.\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":748,"url":"https:\/\/fde.cat\/index.php\/2023\/08\/15\/introducing-immortal-objects-for-python\/","url_meta":{"origin":632,"position":4},"title":"Introducing Immortal Objects for Python","date":"August 15, 2023","format":false,"excerpt":"Instagram has introduced Immortal Objects \u2013 PEP-683 \u2013 to Python. Now, objects can bypass reference count checks and live throughout the entire execution of the runtime, unlocking exciting avenues for true parallelism. At Meta, we use Python (Django) for our frontend server within Instagram. To handle parallelism, we rely on\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":174,"url":"https:\/\/fde.cat\/index.php\/2021\/01\/28\/taming-memory-fragmentation-in-venice-with-jemalloc\/","url_meta":{"origin":632,"position":5},"title":"Taming memory fragmentation in Venice with Jemalloc","date":"January 28, 2021","format":false,"excerpt":"Sometimes, an engineering problem arises that might make us feel like maybe we don't know what we're doing, or at the very least, forces us out of the comfort zone of our area of expertise. That day came for the Venice team at Linkedin when we began to notice that\u2026","rel":"","context":"In "External"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/632","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=632"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/632\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=632"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=632"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=632"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}