{"id":295,"date":"2021-08-31T14:40:03","date_gmt":"2021-08-31T14:40:03","guid":{"rendered":"https:\/\/fde.cat\/?p=295"},"modified":"2021-08-31T14:40:03","modified_gmt":"2021-08-31T14:40:03","slug":"how-facebook-encodes-your-videos","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/how-facebook-encodes-your-videos\/","title":{"rendered":"How Facebook encodes your videos"},"content":{"rendered":"

People upload hundreds of millions of videos to Facebook every day. Making sure every video is delivered at the best quality \u2014 with the highest resolution and as little buffering as possible \u2014 means optimizing not only when and how our video codecs compress and decompress videos for viewing, but also which codecs are used for which videos. But the sheer volume of video content on Facebook also means finding ways to do this that are efficient and don\u2019t consume a ton of computing power and resources.<\/span><\/p>\n

To help with this, we employ a variety of codecs as well as adaptive bitrate streaming (<\/span>ABR<\/span><\/a>), which improves the viewing experience and reduces buffering by choosing the best quality based on a viewer\u2019s network bandwidth. But while more advanced codecs like VP9 provide better compression performance over older codecs, like H264, they also consume more computing power. From a pure computing perspective, applying the most advanced codecs to every video uploaded to Facebook would be prohibitively inefficient. Which means there needs to be a way to prioritize which videos need to be encoded using more advanced codecs.<\/span><\/p>\n

Today, Facebook deals with its high demand for encoding high-quality video content by combining a benefit-cost model with a machine learning (ML) model that lets us prioritize advanced encoding for highly watched videos. By predicting which videos will be highly watched and encoding them first, we can reduce buffering, improve overall visual quality, and allow people on Facebook who may be limited by their data plans to watch more videos.<\/span><\/p>\n

But this task isn\u2019t as straightforward as allowing content from the most popular uploaders or those with the most friends or followers to jump to the front of the line. There are several factors that have to be taken into consideration so that we can provide the best video experience for people on Facebook while also ensuring that content creators still have their content encoded fairly on the platform.<\/span><\/p>\n

How we used to encode video on Facebook<\/span><\/h2>\n

Traditionally, once a video is uploaded to Facebook, the process to enable ABR kicks in and the original video is quickly re-encoded into multiple resolutions (e.g., 360p, 480p, 720p, 1080p). Once the encodings are made, Facebook\u2019s video encoding system tries to further improve the viewing experience by using more advanced codecs, such as VP9, or more expensive \u201crecipes\u201d (a video industry term for fine-tuning transcoding parameters), such as H264 very slow profile, to compress the video file as much as possible. Different transcoding technologies (using different codec types or codec parameters) have different trade-offs between compression efficiency, visual quality, and how much computing power is needed.<\/span><\/p>\n

The question of how to order jobs in a way that maximizes the overall experience for everyone has already been top of mind. Facebook has a specialized encoding compute pool and dispatcher. It accepts encoding job requests that have a priority value attached to them and puts them into a priority queue where higher-priority encoding tasks are processed first. The video encoding system\u2019s job is then to assign the right priority to each task. It did so by following a list of simple, hard-coded rules. Encoding tasks could be assigned a priority based on a number of factors, including whether a video is a licensed music video, whether the video is for a product, and how many friends or followers the video\u2019s owner has.<\/span><\/p>\n

But there were disadvantages to this approach. As new video codecs became available, it meant expanding the number of rules that needed to be maintained and tweaked. Since different codecs and recipes have different computing requirements, visual quality, and compression performance trade-offs, it is impossible to fully optimize the end user experience by a coarse-grained set of rules.<\/span><\/p>\n

And, perhaps most important, Facebook\u2019s video consumption pattern is extremely skewed, meaning Facebook videos are uploaded by people and pages that have a wide spectrum in terms of their number of friends or followers. Compare the Facebook page of a big company like Disney with that of a vlogger that might have 200 followers. The vlogger can upload their video at the same time, but Disney\u2019s video is likely to get more watch time. However, any video can go viral even if the uploader has a small following. The challenge is to support content creators of all sizes, not just those with the largest audiences, while also acknowledging the reality that having a large audience also likely means more views and longer watch times.<\/span><\/p>\n

Enter the Benefit-Cost model<\/span><\/h2>\n

The new model still uses a set of quick initial H264 ABR encodings to ensure that all uploaded videos are encoded at good quality as soon as possible. What\u2019s changed, however, is how we calculate the priority of encoding jobs after a video is published.<\/span><\/p>\n

The Benefit-Cost model grew out of a few fundamental observations:<\/span><\/p>\n

    \n
  1. A video consumes computing resources only the first time it is encoded. Once it has been encoded, the stored encoding can be delivered as many times as requested without requiring additional compute resources.<\/span><\/li>\n
  2. A relatively small percentage (roughly one-third) of all videos on Facebook generate the majority of overall watch time.<\/span><\/li>\n
  3. Facebook\u2019s data centers have limited amounts of energy to power compute resources.<\/span><\/li>\n
  4. We get the most bang for our buck, so to speak, in terms of maximizing everyone\u2019s video experience within the available power constraints, by applying more compute-intensive \u201crecipes\u201d and advanced codecs to videos that are watched the most.<\/span><\/li>\n<\/ol>\n

    \"\"<\/p>\n

    Based on these observations, we came up with following definitions for benefit, cost, and priority:<\/span><\/p>\n

      \n
    1. Benefit<\/strong> = (relative compression efficiency of the encoding family at fixed quality) * (effective predicted watch time)<\/span><\/li>\n
    2. Cost<\/strong> = normalized compute cost of the missing encodings in the family<\/span><\/li>\n
    3. Priority<\/strong> = Benefit\/Cost<\/span><\/li>\n<\/ol>\n

      Relative compression efficiency of the encoding family at fixed quality:<\/b> We measure benefit in terms of the encoding family\u2019s compression efficiency. \u201cEncoding family\u201d refers to the set of encoding files that can be delivered together. For example, H264 360p, 480p, 720p, and 1080p encoding lanes make up one family, and VP9 360p, 480p, 720p, and 1080p make up another family. One challenge here is comparing compression efficiency between different families at the same visual quality.<\/span><\/p>\n

      To understand this, you first have to understand a metric we\u2019ve developed called Minutes of Video at High Quality per GB datapack (MVHQ). MVHQ links compression efficiency directly to a question people wonder about their internet allowance: Given 1 GB of data, how many <\/span>minutes of high-quality video can we stream?<\/span><\/p>\n

      Mathematically, MVHQ can be understood as:<\/span><\/p>\n

      \"\"<\/p>\n

      For example, let\u2019s say we have a video where the MVHQ using H264 fast preset encoding is 153 minutes, 170 minutes using H264 slow preset encoding, and 200 minutes using VP9. This means delivering the video using VP9 could extend watch time using 1 GB data by 47 minutes (200-153) at a high visual quality threshold compared to H264 fast preset. When calculating the benefit value of this particular video, we use H264 fast as the baseline. We assign 1.0 to H264 fast, 1.1 (170\/153) to H264 slow, and 1.3 (200\/153) to VP9.<\/span><\/p>\n

      The actual MVHQ can be calculated only once an encoding is produced, but we need the value before encodings are available, so we use historical data to estimate the MVHQ for each of the encoding families of a given video.<\/span><\/p>\n

      Effective predicted watch time:<\/b> As described further in the section below, we have a sophisticated ML model that predicts how long a video is going to be watched in the near future across all of its audience. Once we have the predicted watch time at the video level, we estimate how effectively an encoded family can be applied to a video. This is to account for the fact that not all people on Facebook have the latest devices, which can play newer codecs.<\/span><\/p>\n

      For example, about 20 percent of video consumption happens on devices that cannot play videos encoded with VP9. So if the predicted watch time for a video is 100 hours the effective predicted watch time using the widely adopted H264 codec is 100 hours while effective predicted watch time of VP9 encodings is 80 hours.<\/span><\/p>\n

      Normalized compute cost of the missing encodings in the family:<\/b> This is the amount of logical computing cycles we need to make the encoding family deliverable. An encoding family requires a minimum set of resolutions to be made available before we can deliver a video. For example, for a particular video, the VP9 family may require at least four resolutions. But some encodings take longer than others, meaning not all of the resolutions for a video can be made available at the same time.<\/span><\/p>\n

      As an example, let\u2019s say Video A is missing all four lanes in the VP9 family. We can sum up the estimated CPU usage of all four lanes and assign the same normalized cost to all four jobs.\u00a0<\/span><\/p>\n

      If we are only missing two out of four lanes, as shown in Video B, the compute cost is the sum of producing the remaining two encodings. The same cost is applied to both jobs. Since the priority is benefit divided by cost, this has the effect of a task\u2019s priority becoming more urgent as more lanes become available. Encoding lanes do not provide any value until they are deliverable, so it is important to get to a complete lane as quickly as possible. For example, having one video with all of its VP9 lanes adds more value than 10 videos with incomplete (and therefore, undeliverable) VP9 lanes.<\/span><\/p>\n

      \"Video<\/p>\n

      \"\"<\/p>\n

      Predicting watch time with ML<\/span><\/h2>\n

      With a new benefit-cost model in place to tell us how certain videos should be encoded, the next piece of the puzzle is determining which videos should be prioritized for encoding. That\u2019s where we now utilize ML to predict which videos will be watched the most and thus should be prioritized for advanced encodings.<\/span><\/p>\n

      Our model looks at a number of factors to predict how much watch time a video will get within the next hour. It does this by looking at the video uploader\u2019s friend or follower count and the average watch time of their previously uploaded videos, as well as metadata from the video itself including its duration, width, height, privacy status, post type (Live, Stories, Watch, etc.), how old it is, and its past popularity on the platform.<\/span><\/p>\n

      But using all this data to make decisions comes with several built-in challenges:<\/span><\/p>\n

      Watch time has high variance and has a very long-tail skewed nature.<\/b> Even when we focus on predicting the next hour of watch time, a video\u2019s watch time can range anywhere from zero to over 50,000 hours depending on its content, who uploaded it, and the video\u2019s privacy settings. The model must be able to tell not only whether the video will be popular, but also how popular.<\/span><\/p>\n

      The best indicator of next-hour watch time is its previous watch time trajectory.<\/b> Video popularity is generally very volatile by nature. Different videos uploaded by the same content creator can sometimes have vastly different watch times depending on how the community reacts to the content. After experimenting with multiple features, we found that past watch time trajectory is the best predictor of future watch time. This poses two technical challenges in terms of designing the model architecture and balancing the training data:<\/span><\/p>\n