Category: Technology

Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Sushant Raikar, Raymond Lam, Ron Hu, Shardul Mahadik, Laura Chen, Khai Tran, Chris Chen, and Nagarathnam Muthusamy Introduction At LinkedIn, our big data compute infrastructure continually grows over time, not only to keep pace with the growth in the number of data applications, or their domains spanning data curation,… Continue reading Coral: A SQL translation, analysis, and rewrite engine for modern data lakehouses

Facebook’s codebase changes each day as engineers develop new features and optimizations for our apps. If not validated, each of these changes could potentially regress the functionality or reliability of our products for billions of people around the world. To mitigate this risk, we maintain an enormous suite of automated regression tests to cover various… Continue reading How do you test your tests?

Facebook’s services rely on fleets of servers in data centers all over the globe — all running applications and delivering the performance our services need. This is why we need to make sure our server hardware is reliable and that we can manage server hardware failures at our scale with as little disruption to our… Continue reading How Facebook keeps its large-scale infrastructure hardware up and running

When I started my journey at LinkedIn ten years ago, the company was just beginning to experience extreme growth in the volume, variety, and velocity of our data. Over the next few years, my colleagues and I in LinkedIn’s data infrastructure team built out foundational technology like Espresso, Databus, and Kafka, among others, to ensure… Continue reading DataHub: Popular metadata architectures explained

As part of our efforts to help expand connectivity around the world, Facebook Connectivity has been prototyping SuperCell, a wide-area coverage solution for increasing mobile connectivity in rural communities. Now, after working with telecom industry partners to conduct several trials and data analyses, we’re ready to share what we’ve learned. Rather than produce this solution… Continue reading SuperCell: Reaching new heights for wider connectivity

Co-authors: Xiang Zhang and Jingyu Zhu Introduction The Lambda architecture has become a popular architectural style that promises both speed and accuracy in data processing by using a hybrid approach of both batch processing and stream processing methods. But it also has some drawbacks, such as complexity and additional development/operational overheads. One of our features… Continue reading From Lambda to Lambda-less: Lessons learned