{"id":772,"date":"2023-10-17T16:00:07","date_gmt":"2023-10-17T16:00:07","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/10\/17\/automating-product-deprecation\/"},"modified":"2023-10-17T16:00:07","modified_gmt":"2023-10-17T16:00:07","slug":"automating-product-deprecation","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/10\/17\/automating-product-deprecation\/","title":{"rendered":"Automating product deprecation"},"content":{"rendered":"

Systematic Code and Asset Removal Framework (SCARF) is Meta\u2019s unused code and data deletion framework.<\/span>
\nSCARF guides engineers through deprecating a product safely and efficiently via an internal tool.<\/span>
\nSCARF combines this tooling with automation to reduce load on engineers.<\/span><\/p>\n

At Meta, we are constantly innovating and experimenting by building and shipping many different products, and those products comprise thousands of individual features. As part of this healthy technology lifecycle, it is inevitable that certain products or features will be deprecated. For example, in 2015 we launched a<\/span> photo-sharing app called Moments<\/span><\/a>, which was later deprecated in 2019. So, how did we efficiently and safely remove all of the code and data related to Moments without adversely affecting Meta\u2019s other products and services?<\/span><\/p>\n

In this three-part blog series, we will discuss the complexities involved in removing a product from a complex portfolio of products and the framework Meta has built to drive the automation of this process, our Systematic Code and Asset Removal Framework (SCARF). SCARF has had an important impact at Meta. In the last year, it has removed petabytes of unused data across <\/span>12.8M<\/span> different data types stored in 21 different data systems. Over the last five years it has deleted over 100M lines of code.<\/span><\/p>\n

The first post will introduce the complexities faced when systematically deprecating products safely in a large organization and the internal workflow tools we have developed. The second post will explain how SCARF automates the removal of dead code and the infrastructure that powers it. The third post will discuss SCARF\u2019s orchestration for safely identifying and deleting unused data types across various data systems.\u00a0<\/span><\/p>\n

Failure modes<\/span><\/h2>\n

Without established guidance detailing the process for determining when and how to remove a product or feature, a few failure modes might emerge. Consider the example of launching a time-specific feature for a large event that happens once every few years. Does it make sense to keep all the code and data related to it until the next event? Most of the time, maintaining code for a number of years that is unused is less desirable than building a new experience for the next event.<\/span><\/p>\n

Engineers who do attempt the cleanup might find that doing so is a very time-consuming job. Correctly identifying all the pieces of code and data associated with the product and only taking action on those specific pieces of code and data is a laborious process. It\u2019s possible that a table that is still being used, or code that is still required for a shared use case, could be included in the scope of a deletion effort. For example, some tables might be shared between products. Hypothetically, Moments may have started life as an extension to Facebook photos before it became a separate app.<\/span><\/p>\n

Most importantly, it\u2019s crucial that any cleanup efforts only remove things that are actively being deprecated or are already entirely unused. Deleting something that is actively being used in production could cause bad experiences for users. The interconnected nature of features within a large product like Facebook makes this a very real possibility.\u00a0<\/span><\/p>\n

How did we solve this?<\/span><\/h2>\n

Meta has developed playbooks describing how to safely deprecate a product. These playbooks describe how to notify people and give them time to download their data, how to disable the product safely, and when to eventually delete the underlying code and data. They describe how and when to perform a product or feature deprecation, but actually removing the code and data for a product or feature is an engineering problem with an engineering solution. The engineering solution we have built does not replace these guides, but enables engineers to more safely and efficiently complete the product deprecation process.<\/span><\/p>\n

In this post, we\u2019ll describe Meta\u2019s SCARF and how it guides engineers through this cleanup process. In subsequent posts, we will discuss its limitations and how they are mitigated by our automated dead code and data removal platforms. To start, we introduce Meta\u2019s suite of internal tools developed to orchestrate both of these systems in guiding engineers to remove large products with complex dependencies.<\/span><\/p>\n

Introducing workflow management<\/span><\/h2>\n

To simplify the task of removing a product, Meta has built a product deprecation workflow management tool into SCARF to help engineers delete a product\u2019s dead code and unused data safely and efficiently. This tool lets engineers understand and break down the steps they\u2019ll go through during the deprecation and coordinates the actions of SCARF to bring automation to bear with an engineer\u2019s guidance.<\/span><\/p>\n

Engineers can import their product or feature into SCARF, which then determines the constituent pieces of code and data, and identifies internal and external dependencies on these assets. SCARF automatically processes this information to guide engineers on the correct order of operations to delete assets safely and shows their progress relative to the desired end-state.<\/span><\/p>\n

Any assets that are safe to be deleted immediately will be handled by SCARF\u2019s automated code and data cleanup systems (which will be covered in more depth in our subsequent blog posts). Engineers are able to track this automation and accelerate it by using their domain knowledge combined with SCARF\u2019s scoping analysis, which determines the assets that are safe to remove.\u00a0<\/span><\/p>\n

Scoping the deprecation<\/span><\/h2>\n

Understanding when a piece of code or data is used by other components of a product is important, and detecting when an asset is referenced in the codebase of other products is crucial for a safe deprecation (the internal and external dependencies, respectively). For example, we would not want to leave a dead web link between two different products because that could lead to a bad user experience. An engineer deprecating a product must think carefully about each such dependency in the external boundary of their product.<\/span><\/p>\n

To avoid these complications, engineers begin their deprecations by scoping the project, recursively adding any assets that should be deleted, and flagging dependencies that should be severed. For example, if Moments had an integration with the Facebook app\u2019s Sharing feature, this dependency must be broken because the Facebook Sharing feature itself may not be in scope for removal.\u00a0<\/span><\/p>\n\n

However, if the Sharing feature was unique to Moments, that would change the scope of removal as we would want to delete that component as well. Adding new assets into the scope of the project as it progresses requires further analysis to discover the new boundaries of the internal and external dependencies. These dependencies are expected to change over time as an engineer discovers extra components to delete. Alternatively, we may identify code that is actually a shared component and should not be deleted. Attempting to finalize this boundary from the start is very difficult, so we allow developers to redefine the boundary over time as the deprecation progresses to reflect this growing understanding.\u00a0<\/span><\/p>\n

The graph of components, their internal and external dependencies, and related data assets, can grow extremely complex in large products. SCARF simplifies this problem by only requiring engineers to make \u201cFlag Dependency\u201d or \u201cAdd to Project\u201d decisions at the boundaries. It then internally computes the correct deletion order for everything inside the project.<\/span><\/p>\n\n

Creating a deletion roadmap<\/span><\/h2>\n

Once the set of unused assets to be deleted has been determined, SCARF will analyze the internal and external dependencies and create a deletion roadmap that outlines the correct sequence of steps for deleting everything safely. The roadmap is refreshed each day as changes are made, either from targeted assets being deleted or from modifications to the component graph through the Flag and Add actions. This roadmap is one of the workflow management tool\u2019s most crucial features.\u00a0<\/span><\/p>\n

Without guidance, engineers may attempt to delete everything in one fell swoop by removing an entire code directory without accounting for external systems\u2019 dependencies, where changes cannot be committed atomically. Another example includes deleting data before the code that reads and writes it, which may lead to new data being created that must be cleaned again. Deprecations must be staggered to account for these various requirements and this entails that every deprecation be performed in a coordinated, multi-step process.<\/span><\/p>\n

The key implementation detail is the encoding of business logic that allows products and apps to function in Meta\u2019s systems. By encoding how users are able to engage with products and how those products communicate with other Meta services, we can ensure that upstream assets are always deleted before their dependencies. For example, typically a product will comprise code in many different languages across multiple repositories. An engineer needs to delete their mobile code (Java, Objective-C) in order to free up and delete their server-side GraphQL definitions. Deleting those GraphQL definitions makes it possible to delete business logic; deleting business logic makes it possible to delete data schema definitions, which in turn allows unused data to be deleted.\u00a0<\/span><\/p>\n

The following diagram is a hypothetical example of the type of information that is presented to an engineer in their deletion roadmap.\u00a0<\/span><\/p>\n\n

This sequencing isn\u2019t obvious at the get-go! The links between these system boundaries are often weaker than the link between, for example, two classes in the same language; and the link may often only be discovered by an engineer during continuous testing on their code change requests. SCARF\u2019s encoding of these inter-system boundaries enables the deletion roadmap which in turn enables engineers to stagger their removal actions safely.<\/span><\/p>\n

Powering the analysis<\/span><\/h2>\n

SCARF\u2019s workflow management tool is powered by detailed metrics from a combination of both static and dynamic analysis. Information about code is gathered from SCARF\u2019s unified code dependency graph and information about data is gathered from SCARF\u2019s asset usage analysis. These mechanisms will be discussed in more detail in our subsequent blog posts.<\/span><\/p>\n

These metrics give the workflow management tool a very important property: If metrics can be found for an asset, it is in use and should be blocked from automated deletion. Correspondingly, any asset for which no metrics are available is ready for automatic deletion. With these metrics in hand, SCARF can show engineers a precise explanation of exactly what is blocking the automation and what steps they must manually perform in order for the automation to proceed.\u00a0<\/span><\/p>\n

This completeness property allows SCARF to automatically begin removing code and data that is unused, while simultaneously showing engineers which things must be handled manually. At every step of the process, both engineers and automation work together to complete the deprecation.\u00a0<\/span><\/p>\n

Once an engineer has acted upon this information and removed more dead code and unused data in accordance with the deletion roadmap, the continuous indexing and analysis of SCARF will detect these changes and automatically trigger any code\/data cleanup it can. Finally, the workflow management tool will update its deletion roadmap to identify the next set of items requiring manual intervention, and the process is repeated.\u00a0<\/span><\/p>\n\n

Is automation sufficient?<\/span><\/h2>\n

Some of the usage signals SCARF highlights (such as API endpoints that still receive traffic) do not necessarily cause compilation errors if they are ignored and are more subjective as to whether they should stop automated deletion. If the endpoint for a product receives a single request each day, should we delete it right away? We have already identified that the endpoint belongs to a product we are removing so we know it should be removed eventually, but the answer to when it is correct to delete it is ultimately a business decision. Further, we understand that mistakes can happen and that the automation is not expected to be 100 percent perfect. Therefore, SCARF errs on the side of caution as a mistaken deletion could lead to unrecoverable errors or data loss.<\/span><\/p>\n

Correspondingly, SCARF offers the ability to override signals that prevent automation from taking action. In the example above, an engineer might decide we wish to proceed with deleting the API endpoint despite small amounts of traffic. This is extremely valuable when cleaning up internal products, tools, and services. Internal tools are built and deprecated with a much faster cadence than an external product often is and SCARF can help determine each individual internal user of a tool who needs to be asked if they\u2019re agreeable with it being removed.\u00a0<\/span><\/p>\n

This feature creates a feedback loop between engineers and automation. Our automation highlights usage signals; engineers then triage those usage signals (either by modifying code to remove them or, if applicable, marking the signal as not sufficient to block deletion); then our automation can proceed to progress the SCARF project.\u00a0<\/span><\/p>\n

To further accelerate manual changes performed by engineers, SCARF also provides an instant code change feature. SCARF suggests a means for breaking a dependency through a targeted code change and an engineer is able to select one of a predetermined set of possible ways to break it. The change is immediately previewable and a change request can be immediately generated for review by other engineers.<\/span><\/p>\n

After an engineer has progressed through a majority of the deletion roadmap and removed the entry points, client code, server code, and any other in-scope assets, they will be left with unused tables of data that can now be automatically cleaned up by SCARF. This automation will be explained in more detail in our future blog posts.<\/span><\/p>\n

Integration with other tools at Meta<\/span><\/h2>\n

SCARF is useful for more than just product deprecation. We surface deprecation as an option when notifying engineers about routine maintenance or upgrade work on individual assets. For example, when asking developers to migrate away from a legacy internal API, we can include an option for them to deprecate the callsite or affected product and import their code into SCARF. Deprecating something can, in many cases, be more work than performing a simple upgrade, but deprecation means that no future maintenance work is necessary. Once something has been removed, future maintenance costs go to zero.<\/span><\/p>\n

We hope you\u2019ll look forward to our subsequent posts in the series, which will cover SCARF\u2019s automated code and data deletion, respectively.\u00a0<\/span><\/p>\n

The post Automating product deprecation<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

Systematic Code and Asset Removal Framework (SCARF) is Meta\u2019s unused code and data deletion framework. SCARF guides engineers through deprecating a product safely and efficiently via an internal tool. SCARF combines this tooling with automation to reduce load on engineers. At Meta, we are constantly innovating and experimenting by building and shipping many different products,… Continue reading Automating product deprecation<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-772","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":779,"url":"https:\/\/fde.cat\/index.php\/2023\/10\/31\/automating-data-removal\/","url_meta":{"origin":772,"position":0},"title":"Automating data removal","date":"October 31, 2023","format":false,"excerpt":"Meta\u2019s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing unused data types. SCARF scans production data systems to identify tables or assets that are unused and safely removes them. SCARF avoids tedious manual work and ensures that product data is correctly removed when a\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":777,"url":"https:\/\/fde.cat\/index.php\/2023\/10\/24\/automating-dead-code-cleanup\/","url_meta":{"origin":772,"position":1},"title":"Automating dead code cleanup","date":"October 24, 2023","format":false,"excerpt":"Meta\u2019s Systematic Code and Asset Removal Framework (SCARF) has a subsystem for identifying and removing dead code. SCARF combines static and dynamic analysis of programs to detect dead code from both a business and programming language perspective. SCARF automatically creates change requests that delete the dead code identified from the\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":604,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/06\/watch-metas-engineers-discuss-quic-and-tcp-innovations-for-our-network\/","url_meta":{"origin":772,"position":2},"title":"Watch Meta\u2019s engineers discuss QUIC and TCP innovations for our network","date":"July 6, 2022","format":false,"excerpt":"With more than 75 percent of our internet traffic set to use QUIC and HTTP\/3 together, QUIC is slowly moving to become the de facto protocol used for internet communication at Meta. For Meta\u2019s data center network, TCP remains the primary network transport protocol that supports thousands of services on\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":787,"url":"https:\/\/fde.cat\/index.php\/2023\/11\/15\/watch-metas-engineers-on-building-network-infrastructure-for-ai\/","url_meta":{"origin":772,"position":3},"title":"Watch: Meta\u2019s engineers on building network infrastructure for AI","date":"November 15, 2023","format":false,"excerpt":"Meta is building for the future of AI at every level \u2013 from hardware like MTIA v1, Meta\u2019s first-generation AI inference accelerator to publicly released models like Llama 2, Meta\u2019s next-generation large language model, as well as new generative AI (GenAI) tools like Code Llama. Delivering next-generation AI products and\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":756,"url":"https:\/\/fde.cat\/index.php\/2023\/09\/05\/whats-it-like-to-write-code-at-meta\/","url_meta":{"origin":772,"position":4},"title":"What\u2019s it like to write code at Meta?","date":"September 5, 2023","format":false,"excerpt":"Ever wonder what it\u2019s like to write code at Meta\u2019s scale? On the latest episode of the Meta Tech Podcast, Meta engineer Pascal Hartig (@passy) sits down with Dustin Shahidehpour\u00a0and\u00a0Katherine Zak,\u00a0 two software engineers at Meta, about their careers and what it\u2019s really like to ship code at Meta. Why\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":795,"url":"https:\/\/fde.cat\/index.php\/2023\/11\/21\/writing-and-linting-python-at-scale\/","url_meta":{"origin":772,"position":5},"title":"Writing and linting Python at scale","date":"November 21, 2023","format":false,"excerpt":"Python plays a big part at Meta. It powers Instagram\u2019s backend and plays an important role in our configuration systems, as well as much of our AI work. Meta even made contributions to Python 3.12, the latest version of Python. On this episode of the\u00a0Meta Tech Podcast, Meta engineer Pascal\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/772","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=772"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/772\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=772"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=772"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=772"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}