Month: August 2021

The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix — Part 2 In the first part of this blog, we described a solution for strongly consistent immutable secondary indexes. We extend this solution for mutable indexes here. By doing so, we provide a general solution for strongly consistent global secondary indexes. Mutable Indexes The mutable tables… Continue reading The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix — Part 2

People upload hundreds of millions of videos to Facebook every day. Making sure every video is delivered at the best quality — with the highest resolution and as little buffering as possible — means optimizing not only when and how our video codecs compress and decompress videos for viewing, but also which codecs are used… Continue reading How Facebook encodes your videos

The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix — Part 1 Phoenix is a relational database with a SQL interface that uses HBase as its backing store. This combination allows it to leverage the flexibility and scalability of HBase, which is a distributed key-value store. Phoenix provides additional functionality on top of HBase, including SQL… Continue reading The Design of Strongly Consistent Global Secondary Indexes in Apache Phoenix — Part 1

Training AI models at a large scale isn’t easy. Aside from the need for large amounts of computing power and resources, there is also considerable engineering complexity behind training very large models. At Facebook AI Research (FAIR) Engineering, we have been working on building tools and infrastructure to make training large AI models easier. Our… Continue reading Fully Sharded Data Parallel: faster AI training with fewer GPUs

For years, people have been asking us to create a true multi-device experience that allows people to use WhatsApp on other devices without requiring a smartphone connection. Today, we’re announcing the rollout of a limited public beta test for WhatsApp’s updated multi-device capability.  With this new capability, you can now use WhatsApp on your phone… Continue reading How WhatsApp enables multi-device capability

Our infrastructure supports thousands of services that handle billions of requests per second. We’ve previously discussed how we built our service encryption infrastructure to keep these globally distributed services operating securely and performantly. This post discusses the system we designed to enforce encryption policies within our network and shares some of the lessons we learned… Continue reading Enforcing encryption at scale

What the research is: The Ribbon filter is a new data structure that is more space-efficient than the popular Bloom filters that are widely used for optimizing data retrieval. One of the ways that Bloom, and now Ribbon, filters solve real engineering problems is by providing smooth configurability unmatched by other filters. Bloom filters work… Continue reading Ribbon filter: Practically smaller than Bloom and Xor