Author: fde

Pegasus Data Schema (PDSC) is a Pegasus schema definition language that has been used for data modeling with Rest.li services for years. It’s the underlying language that helps define data models, describe the data returned by REST endpoints, and generate derivative schemas for other uses, such as XML schemas and various database schemas. However, writing… Continue reading Pegasus Data Language: Evolving schema definitions for data modeling

Sometimes, an engineering problem arises that might make us feel like maybe we don’t know what we’re doing, or at the very least, forces us out of the comfort zone of our area of expertise. That day came for the Venice team at Linkedin when we began to notice that some Venice processes would consume… Continue reading Taming memory fragmentation in Venice with Jemalloc

Co-authors: Sakshi Jain, Grace Tang, Gaurav Vashist, Yu Wang, John Lu, Ravish Chhabra, Shruti Sharma, Dana Tom, and Ranjeet Ranjan LinkedIn’s vision is to connect every member of the global workforce to economic opportunity. A key driver towards this vision is our world-class hiring marketplace, where we help job seekers find their next dream role… Continue reading Creating a secure and trusted Jobs ecosystem on LinkedIn

Designing a personalized ranking system for more than 2 billion people (all with different interests) and a plethora of content to select from presents significant, complex challenges. This is something we tackle every day with News Feed ranking. Without machine learning (ML), people’s News Feeds could be flooded with content they don’t find as relevant… Continue reading How machine learning powers Facebook’s News Feed ranking algorithm

Co-authors: Jun Jia and Alice Wu Introduction It’s a very common scenario that an AI solution involves composing different jobs, such as data processing and model training or evaluation, into workflows and then submitting them to an orchestration engine for execution. At large companies such as LinkedIn, there may be hundreds of thousands of such… Continue reading Smart Argument Suite: Seamlessly connecting Python jobs

Co-authors: Min Liu, Vangelis Dimopoulos, Elise Georis, Jialiang Mao, Di Luo, and Kang Kang The LinkedIn ecosystem drives member and customer value through a series of marketplaces (e.g., the ads marketplace, the talent marketplace, etc.). We maximize that value by making data-informed product decisions via A/B testing. Traditional A/B tests on our marketplaces, however, are… Continue reading Budget-split testing: A trustworthy and powerful approach to marketplace A/B testing

Over the last year, we have been using real-time feedback to evolve our tooling and provide a more productive experience for LinkedIn’s developers. It’s helped us double our feedback participation, and more importantly, better tailor our recommendations and improvements.  For any engineering organization looking to improve developer experiences, the following questions will provide a good… Continue reading How LinkedIn turned to real-time feedback for developer tooling

Co-authors: Zihan Li, Sudarshan Vasudevan, Lei Sun, and Shirshanka Das Data analytics and AI power many business-critical use cases at LinkedIn. We need to ingest data in a timely and reliable way from a variety of sources, including Kafka, Oracle, and Espresso, bringing it into our Hadoop data lake for subsequent processing by AI and… Continue reading FastIngest: Low-latency Gobblin with Apache Iceberg and ORC format

It goes without saying that 2020 has been a challenging year, to put it lightly. But if anything, the COVID-19 pandemic has shined a light on our need to connect as people. For Facebook, that meant our work has become more important than ever. Whether it was finding new and innovative ways to expand internet… Continue reading 2020 year in review: Connectivity innovations, faster apps, and progress toward net zero