Category: Technology

Root cause analysis (RCA) is an important part of fixing any bug. After all, you can’t solve a problem without getting to the heart of it. But RCA isn’t always simple, especially at a scale like Facebook’s. When billions of people are using an app on a variety of platforms and devices, a single bug… Continue reading Minesweeper automates root cause analysis as a first-line defense against bugs

Authors: Vaishnavi Galgali, Arpeet Kale, Robert Xue Introduction The Salesforce Einstein Vision and Language services are deployed in an AWS Elastic Kubernetes Service (EKS) cluster. One of the primary security and compliance requirements is operating system patching. The cluster nodes that the services are deployed on need to have regular operating system updates. Operating system patching… Continue reading Zero Downtime Node Patching in a Kubernetes Cluster

Native Scrolling in Salesforce Mobile App The Salesforce mobile app is a native app with hybrid functionality available for both iOS and Android platforms. A hybrid app combines the best of both worlds, leveraging native experiences with rich web customizations provided by the Salesforce platform via Flexipages, Lightning Web Components, Aura, and VisualForce. UI Scroller was… Continue reading Native Scrolling in Salesforce mobile app

Every workday, Facebook engineers commit thousands of diffs (which is a change consisting of one or more files) into production. This code velocity allows us to rapidly ship new features, deliver bug fixes and optimizations, and run experiments. However, a natural downside to moving quickly in any industry is the risk of inadvertently causing regressions… Continue reading Faster, more efficient systems for finding and fixing regressions

When I got into the DevOps field, I was exposed to The Five Whys — a popular analytical method used in incident postmortems. The Five Whys is one type of root cause analysis (RCA): “The primary goal of the technique is to determine the root cause of a defect or problem by repeating the question ‘Why?.’ Each… Continue reading How, Not Why: An Alternative to the Five Whys for Post-Mortems

We will be hosting a talk about our work on Scaling a Distributed Priority Queue during our virtual Systems @Scale event at 11 am PT on Wednesday, February 24, followed by a live Q&A session. Please submit any questions to systemsatscale@fb.com before the event. The entire Facebook ecosystem is powered by thousands of distributed systems… Continue reading FOQS: Scaling a distributed priority queue

What the research is:  Silent data corruption, or data errors that go undetected by the larger system, is a widespread problem for large-scale infrastructure systems. This type of corruption can propagate across the stack and manifest as application-level problems. It can also result in data loss and require months to debug and resolve. This work… Continue reading Mitigating the effects of silent data corruption at scale