{"id":597,"date":"2022-06-09T16:00:52","date_gmt":"2022-06-09T16:00:52","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/06\/09\/under-the-hood-metas-cloud-gaming-infrastructure\/"},"modified":"2022-06-09T16:00:52","modified_gmt":"2022-06-09T16:00:52","slug":"under-the-hood-metas-cloud-gaming-infrastructure","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/06\/09\/under-the-hood-metas-cloud-gaming-infrastructure\/","title":{"rendered":"Under the hood: Meta\u2019s cloud gaming infrastructure"},"content":{"rendered":"

The promise of cloud gaming is a promise to democratize gaming. Anyone who loves games should be able to enjoy them and share the experience with their friends, no matter where they\u2019re located, and even if they don\u2019t have the latest, most expensive gaming hardware. Facebook <\/span>launched <\/span><\/a>its cloud gaming platform in 2020 to give anyone on Facebook instant access to native Android and Windows games across every screen and web browser.<\/span><\/p>\n

Creating the unprecedented access offered by cloud gaming required engineers at Meta to rise to new challenges and develop a growing hardware infrastructure capable of delivering quality game experiences to people all over the world.<\/span><\/p>\n

But gaming itself is also evolving. From new 3D experiences like AR and VR to what will eventually become the metaverse, people all over the world want to play increasingly immersive games as seamlessly and easily as possible. While it will take a massive effort across the industry to bring the metaverse to fruition, we believe creating the infrastructure and solving the challenges of cloud gaming are pointing us toward solutions for the metaverse as well.<\/span><\/p>\n

Why cloud gaming?<\/span><\/h2>\n

Cloud gaming is about accessibility \u2014 bringing gaming to people regardless of the device they\u2019re using or where they\u2019re located in the world. Placing game apps on the cloud frees people up from having to purchase new hardware and also removes the need for large downloads or waiting for updates. It enables seamless cross-play across desktop and mobile devices as well as flexible app distribution.<\/span><\/p>\n

Cloud gaming also promises an easier developer experience since developers shouldn\u2019t need to concern themselves as much with optimizing games for multiple hardware platforms (e.g., desktop and mobile). In an ideal world, developers will only need to build an app once and it can be distributed to multiple devices without the need for multiple binaries. And for players and developers concerned with security and integrity, cloud gaming means fewer concerns about cheating and piracy.<\/span><\/p>\n

Inside Meta\u2019s cloud gaming infrastructure<\/span><\/h2>\n

Enabling cloud gaming at Meta meant developing a new hardware and software infrastructure to address the challenges inherent in cloud gaming. Cloud-based games need low end-to-end latency to provide a fast and smooth gameplay experience where quality video and audio have to be streamed with as little jitter as possible. The infrastructure also needs to be capable of running multiple games on a single cloud gaming server to be economically efficient. And all this has to be secured against various cyberattacks while also remaining robust and efficient.<\/span><\/p>\n

Edge computing, GPUs, and virtualization<\/span><\/h3>\n

The best way for our cloud gaming infrastructure to provide great latency for players is to bring it as close to them as possible in terms of network distance. Meta\u2019s data centers alone cannot provide the level of ultra-low latency we require for cloud gaming. So we rely on edge computing, where we bring the cloud gaming infrastructure closer to players by deploying in edges that are in metro<\/span>politan<\/span> areas close to large populations.<\/span><\/p>\n

As we increase the number of edge computing sites, we can also improve latency for players.<\/p>\n

Our goal within each edge computing site is to have a unified hosting environment to make sure we can run as many games as possible as smoothly as possible. Today\u2019s games are designed for GPUs, so we partnered with NVIDIA to build a hosting environment on top of NVIDIA Ampere architecture-based GPUs. As games continue to become more graphically intensive and complex, GPUs will provide us with the high fidelity and low latency we need for loading, running, and streaming games.<\/span><\/p>\n

To run games themselves, we use<\/span> Twine<\/span><\/a>, our cluster management system, on top of our <\/span>edge computing operating system. We build orchestration services to manage the streaming signals and use Twine to coordinate the game servers on edge.<\/span><\/p>\n

We built and used container technologies for both Windows and Android games. We have different hosting solutions for Windows and Android games, and the Windows hosting solution comes with the integration with PlayGiga. We\u2019ve built a consolidated orchestration system to manage and run the games for both operating systems. This means we can deliver games and manage capacity with more flexibility across different platforms.<\/span><\/p>\n\n

Video and audio streaming<\/span><\/h3>\n

Ultimately, delivering smooth video and audio is one of the most important parts of the cloud gaming experience. Anyone who has played an online game is familiar with the frustrations that latency can cause.<\/span><\/p>\n

After considering the maturity and compatibility of the technology, we landed on WebRTC with Secure Real-Time Transport Protocol (SRTP) technology as our solution to streaming user inputs and video\/audio frames for games. In doing so, we\u2019ve also been able to significantly improve our video and audio streaming performance over time.<\/span><\/p>\n

Let us start with a basic streaming flow we had for cloud gaming at the beginning.\u00a0<\/span><\/p>\n

Whenever a player clicked to perform an action in a game (e.g., to make their character jump), we captured the click event and sent it to the server, where the game received the event. The game then rendered a frame that contained the result of that action (i.e., the character jumping). We captured that rendered frame, copied it out, and encoded it using a video encoder. The frame was then packetized so it could fit into a User Datagram Protocol (UDP) packet and sent through the network to the player. Finally, on the player\u2019s side, there was a jitter buffer to help smooth play as the packets came in. We decoded the packets into frames, then rendered the frame for the player.<\/span><\/p>\n

All this happened so fast that the player didn\u2019t perceive any time between the click and the action. However, each of these steps takes a tiny bit of time. And when these bits of time add up, it can lead to higher latency and lag for the player.<\/span><\/p>\n

Our current model significantly improves on this and improves latency using GPU encoding. Using GPUs for encoding means that when a game renders a frame, it\u2019s rendered in the GPU and never leaves the GPU\u2019s memory until it\u2019s encoded. This process is much more efficient and doesn\u2019t use a lot of the PCI bus bandwidth between the GPU and main server. The encoded frame also ends up being smaller than the raw frame.<\/span><\/p>\n

Our current model for video and audio streaming significantly improves on latency by using GPU encoding.<\/p>\n

On the networking side, having edge computing sites close to the player also helps reduce video and audio latency \u2014 even more significantly than the savings from moving the entire streaming pipeline to GPUs.<\/span><\/p>\n

Over on the player side, we also now use hardware decoding to reduce the decode time. Video and audio are typically synced together, but we can also send video a bit ahead of audio to improve latency. We can also take advantage of the inherent latency of the player\u2019s computer monitor or phone screen. The screen displays frames one by one at a certain rate (e.g., 30fps or 60fps). We can use those imperceptible intervals between frames to help absorb some of the jitter and smooth out video. For devices with support for higher FPS, the latency can go down further.<\/span><\/p>\n

Keeping cloud gaming secure<\/span><\/h3>\n

Players and developers need to be assured that any cloud gaming experience is secure and safe. Players need to know their data is safe and that games won\u2019t be dominated by cheaters. And developers need to be assured their product will be safe from piracy and other security vulnerabilities.<\/span><\/p>\n

Incorporating edge computing, GPU virtualization, and video\/audio streaming makes the cloud gaming infrastructure very complex. And with this complexity comes unique security challenges. As a Windows- and Android-based system, the system inherently takes on the security challenges of those environments and also needs to be protected against threats like DDOS attacks.<\/span><\/p>\n

To identify and address security issues, we assess security at every stage of development, from design and implementation to testing. This includes threat modeling, security code reviews, fuzz testing, and security testing. We don\u2019t want cloud gaming to become an entry point for attacking other Meta systems, so the cloud gaming infrastructure is completely isolated from Meta\u2019s core data infrastructure. We also have an internal security team working alongside external companies to do regular security assessments on the system.<\/span><\/p>\n

Cloud gaming and the metaverse<\/span><\/h2>\n

The metaverse holds major implications for the future of gaming \u2014 not only in the types of games people will play but also in how those games will be delivered to them. The metaverse will push network connectivity requirements further than ever. The groundwork we\u2019re laying today is going to play an important role in helping the larger industry create the cloud infrastructure that will be needed to handle the complex computing required to create metaverse experiences.<\/span><\/p>\n

Just like AAA games, future metaverse experiences will require the highest fidelity and lowest latency possible. If the experience isn\u2019t frictionless, it won\u2019t work for anyone, which means a metaverse network must have ultra-low latency, high scalability, high throughput, and federated data storage.<\/span><\/p>\n

Whatever this new network architecture looks like, it will require a step-change enhancement to today\u2019s overall network architecture. It will need a reliable, quality of service (QoS)-aware, peer-to-peer communication link and protocol to exchange information among people in the same proximity (e.g., under the same Wi-Fi AP coverage). It needs a unified and ubiquitous network topology to bound latency, jitter, and packet loss performance across the globe. Challenges around improving latency, reliability, and throughput and developing federated databases may force engineers to redesign and redistribute compute and routing resources in the end-to-end communication networks. Then the metaverse will need a ubiquitous end-to-end QoS management strategy at every segment of its networks, as well as at every application\/network protocol layer vertically (i.e., from the application layer at the top of the stack to the OS layer at the bottom of the stack).<\/span><\/p>\n

We\u2019re working on solutions for all this right now. With our cloud gaming infrastructure providing some insights, we\u2019re working with mobile network operators and carriers, hardware chipset makers, and other partners to create solutions that will address the needs of the metaverse.<\/span><\/p>\n

What\u2019s the immediate future of cloud gaming?<\/span><\/h2>\n

A lot of people are going to step into the metaverse for the first time through gaming. As we work towards our long-term vision of the metaverse, we\u2019ll continue to build new 2D experiences to help bridge the gap to the metaverse for people across Meta\u2019s family of apps. The recent <\/span>launch<\/span> <\/a>of Crayta on Facebook Gaming as a cloud-streamed creation platform is a great example of this work.\u00a0<\/span><\/p>\n

As we scale our cloud gaming platform, we\u2019re continuously working to upgrade and improve our cloud gaming infrastructure. Over the next two to three years, we\u2019re working on more international expansion to bring games to more people all over the world. We\u2019re also working with <\/span>mobile network operators and carriers to significantly improve the latency in their access network. On the hardware end, we\u2019re working with chipset makers to improve latency in user devices. We\u2019re also working on new container technologies to provide better streaming efficiency. And, of course, there will always be a continuous push on security as the system grows and improves.<\/span><\/p>\n

Developers can expect significant improvements, including improving the compatibility of the system to reduce developer overhead and providing them with better tools for development, testing, debugging, experimentation, and analytics. <\/span>And players, the most important part of our cloud gaming efforts, can expect new and more immersive gaming experiences coming soon.\u00a0<\/span><\/p>\n

Our goal to let people play great games together \u2014 wherever and whenever they want \u2014 won\u2019t change, but our ongoing efforts with cloud gaming will make sure those experiences only get better.<\/span><\/p>\n

The post Under the hood: Meta\u2019s cloud gaming infrastructure<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

The promise of cloud gaming is a promise to democratize gaming. Anyone who loves games should be able to enjoy them and share the experience with their friends, no matter where they\u2019re located, and even if they don\u2019t have the latest, most expensive gaming hardware. Facebook launched its cloud gaming platform in 2020 to give… Continue reading Under the hood: Meta\u2019s cloud gaming infrastructure<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-597","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":538,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/01\/behind-the-scenes-of-hyperforce-salesforces-infrastructure-for-the-public-cloud\/","url_meta":{"origin":597,"position":0},"title":"Behind the Scenes of Hyperforce: Salesforce\u2019s Infrastructure for the Public Cloud","date":"February 1, 2022","format":false,"excerpt":"Salesforce has been running cloud infrastructure for over two decades, bringing companies and their customers together. When Salesforce first started out in 1999, the world was very different; back then, the only practical way to provide our brand of Software-As-A-Service was to run everything yourself\u200a\u2014\u200anot just the software, but the\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":544,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/22\/the-unified-infrastructure-platform-behind-salesforce-hyperforce\/","url_meta":{"origin":597,"position":1},"title":"The Unified Infrastructure Platform Behind Salesforce Hyperforce","date":"February 22, 2022","format":false,"excerpt":"If you\u2019re paying attention to Salesforce technology at all, you\u2019ve no doubt heard about Hyperforce, our new approach to deploying Salesforce on public cloud providers. As with any big announcement, it can be a little hard to cut through the hyperbolic language and understand what\u2019s going\u00a0on. In this blog series,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":585,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/22\/the-unified-infrastructure-platform-behind-salesforce-hyperforce-2\/","url_meta":{"origin":597,"position":2},"title":"The Unified Infrastructure Platform Behind Salesforce Hyperforce","date":"February 22, 2022","format":false,"excerpt":"If you\u2019re paying attention to Salesforce technology at all, you\u2019ve no doubt heard about\u00a0Hyperforce, our new approach to deploying Salesforce on public cloud providers. As with any big announcement, it can be a little hard to cut through the\u00a0hyperbolic language and understand what\u2019s going on. In this blog series, we\u2019ll\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":896,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/16\/the-unstructured-data-dilemma-how-data-cloud-handles-250-trillion-transactions-weekly\/","url_meta":{"origin":597,"position":3},"title":"The Unstructured Data Dilemma: How Data Cloud Handles 250 Trillion Transactions Weekly","date":"July 16, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we delve into the journeys of engineering leaders who have made notable strides in their areas of expertise. This edition features Adithya Vishwanath, Vice President of Software Engineering at Salesforce. He leads the Data Cloud team, a pivotal platform that integrates diverse data sources,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":644,"url":"https:\/\/fde.cat\/index.php\/2022\/10\/24\/how-salesforce-built-a-cloud-native-task-execution-service\/","url_meta":{"origin":597,"position":4},"title":"How Salesforce Built a Cloud-Native Task Execution Service","date":"October 24, 2022","format":false,"excerpt":"If you\u2019re paying attention to Salesforce technology, you\u2019ve no doubt heard about\u00a0Hyperforce, our new approach to deploying Salesforce on public cloud providers. Start with\u00a0a look at Hyperforce\u2019s architecture. There are many compelling reasons to move to Hyperforce, both for us and our customers. We\u2019re excited to do it in the\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":458,"url":"https:\/\/fde.cat\/index.php\/2021\/09\/20\/cachelib-facebooks-open-source-caching-engine-for-web-scale-services\/","url_meta":{"origin":597,"position":5},"title":"CacheLib, Facebook\u2019s open source caching engine for web-scale services","date":"September 20, 2021","format":false,"excerpt":"Caching plays an important role in helping people access their information efficiently. For example, when an email app loads, it temporarily caches some messages, so the user can refresh the page without the app retrieving the same messages. However, large-scale caching has long been a complex engineering challenge. Companies must\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/597","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=597"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/597\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=597"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=597"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=597"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}