{"id":310,"date":"2021-08-31T14:40:03","date_gmt":"2021-08-31T14:40:03","guid":{"rendered":"https:\/\/fde.cat\/?p=310"},"modified":"2021-08-31T14:40:03","modified_gmt":"2021-08-31T14:40:03","slug":"peering-automation-at-facebook","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/peering-automation-at-facebook\/","title":{"rendered":"Peering automation at Facebook"},"content":{"rendered":"

Traffic on the internet travels across many different kinds of links. A fast and reliable way to exchange traffic between different networks and service providers is through peering. Initially, we managed peering via a time-intensive manual process. Reliable peering is essential for Facebook and for everyone\u2019s internet use. But there is no industry standard for how to set up a scalable, automatic peering management system. So we\u2019ve developed a new automated method, which allows for faster self-service peering configuration. We\u2019re sharing a few best practices we have learned in automating our public peering in a hope of wider adoption of our approach in the internet community.<\/span><\/p>\n

How does this work? Take Facebook, for example. Your friend has just posted a video of an extremely cute cat, and you are about to watch it. Let us follow the path of the cat video before it reaches your device:<\/span><\/p>\n<\/p>\n

\u00a0<\/p>\n

Option A: Often the slower, less reliable, higher-latency route:<\/span><\/p>\n

You see your friend\u2019s post with a cute cat video, and you click on it and can\u2019t wait to watch it! Before the video reaches your device, Facebook needs to send it to your ISP using the best-performing, shortest route available. There might be many other networks (commonly referred to as transit networks) between Facebook and your ISP. They might be interconnected in suboptimal locations with potential capacity constraints, causing the awesome cat video to reach you slowly. Nobody wants to watch a buffering cat video!<\/span><\/p>\n

Option B: Often the faster, more reliable, more direct route:<\/span><\/p>\n

You clicked the cat video to watch the cute cat! Before the video even reaches your device, Facebook\u2019s traffic controller realizes there is a fast, direct way to your ISP, without other networks in the middle. The cat video travels through a peering exchange, a common meeting point where lots of different types of networks interconnect by establishing <\/span>Border Gateway Protocol (BGP)<\/span><\/a> sessions between their routers. These <\/span>peering<\/span><\/a> sessions allow them to directly exchange bits, including cat videos, which helps improve the quality, performance, latency, and reliability of the user experience.\u00a0 <\/span><\/p>\n

Why we automated public peering<\/span><\/h2>\n

Across the industry, configuring peering manually is known to be a painfully slow, inefficient, error-prone process. This challenge gets bigger as networks connect to new internet exchanges (IX) and connect multiple routers to each IX.<\/span><\/p>\n

Before developing our automated system, we suffered the same struggle. Peers would email us to request to establish peering sessions. Next, one of our Edge engineers would verify the email and check our mutual traffic levels. To confirm the traffic levels were appropriate, that team member had to check numerous internal dashboards, reports, and rulebooks, as well as external resources, such as the potential peer\u2019s <\/span>PeeringDB<\/span><\/a> record. The team member then would use a few internal tools to configure BGP sessions, reply back to the peer, and wait for the peer to configure their side of the network.\u00a0\u00a0<\/span><\/p>\n

This approach had several problems. First, there was no centralized place to see the incoming peering requests or the existing peering status. Requests could arrive over email, or several other out-of-band systems. Edge engineers had to track, parse, and hand-verify every request. Next, for each request, the team member had to manually launch and monitor an in-house tool for each peer, and then, once finished, type a response to each peering request. At last count, we estimate that this process took more than nine hours per week \u2014 wasting a whole day of each workweek on a needlessly manual process.<\/span><\/p>\n

Our solution<\/span><\/h2>\n

We are thrilled to announce that peers can now request their own public peering sessions through our facebook.com\/peering<\/a> page.\u00a0<\/span><\/p>\n

PeeringDB Oauth<\/span><\/h3>\n

PeeringDB is an open source, not-for-profit, volunteer-run database of networks and their peering network information, verified and vetted by PeeringDB administrators. We believe there is value in the PeeringDB database, and, along with <\/span>many others<\/span><\/a> in the industry, we support it through sponsorship. Since most peering networks already maintain their PeeringDB records as a source of truth for other networks to consume, we see PeeringDB\u2019s OAuth service as an opportunity to standardize a peering management authentication method.\u00a0 <\/span><\/p>\n

To ensure that peering requests made on our peering page are from an authorized person, we require the requester to authenticate using their PeeringDB login and leverage <\/span>PeeringDB\u2019s OAuth<\/span><\/a> service on behalf of their network\u2019s organization. The peer does not need to provide any other authentication \u2014 no Facebook account is required. Once authenticated, the peer will see a list of all their network\u2019s existing public peering sessions with Facebook and can submit new requests.<\/span><\/p>\n<\/p>\n

After requesting sessions, our internal process takes over. All the peer has to do is await our automated emails and configure their side of the network.\u00a0 <\/span><\/p>\n

We have also set up a monitoring system to sort our peering@ mailbox. If the system detects a peering request, it automatically replies with instructions to direct the peer to our peering page. Of course, we still monitor the inbox to respond to any nonpeering inquiries or support requests. But this new engine has significantly reduced time spent combing through email and verifying requests.<\/span><\/p>\n

Once received, the request goes to an auditing queue. If the request is approved, another service launches a workflow to set up peering. First, it queries PeeringDB and our internal tables to validate and collect the mutual peering information, such as IP address and max-prefix settings. Next, it configures the sessions on Facebook\u2019s routers. After that, it emails the peer to confirm that BGP sessions are ready on Facebook\u2019s side and waiting for the peer to turn up their side. The workflow then checks daily to see whether the sessions are established. On the second, third, seventh, and 13th days, it sends an email to the peer to remind them to configure the sessions. As soon our workflow detects that all sessions have been established, our workflow sends a final confirmation email. At that point, our peer should be able to see the new sessions as active in the table on facebook.com\/peering.\u00a0<\/span><\/p>\n

Creating an industry standard<\/span><\/h2>\n

Since launch, we have received more than 170 peering requests and approved 149 of them. As a result, we have automatically pushed more than 1,400 public peering sessions \u2014 for a time savings of more than eight hours per week.<\/span><\/p>\n<\/p>\n

With PeeringDB OAuth<\/a>, we are able to check the validity of peering request submissions and automate more steps in the peering turn-up process. Based on our experience using this system, we recommend leveraging PeeringDB OAuth as the industry standard in other public peering automation applications and implementations.<\/span><\/p>\n

Building on our public peering automation success, we are investigating ways to automate our private network interconnects (PNIs). Private peering is the larger-volume counterpart to public peering, and we hope to offer a self-service option later this year. We are also exploring the possibility of using PeeringDB OAuth as an alternative login service <\/span>for other services we offer our network partners.\u00a0 <\/span><\/p>\n

The post Peering automation at Facebook<\/a> appeared first on Facebook Engineering<\/a>.<\/p>\n

Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Traffic on the internet travels across many different kinds of links. A fast and reliable way to exchange traffic between different networks and service providers is through peering. Initially, we managed peering via a time-intensive manual process. Reliable peering is essential for Facebook and for everyone\u2019s internet use. But there is no industry standard for… Continue reading Peering automation at Facebook<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-310","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":325,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/driving-towards-an-open-internet-ecosystem-to-help-tackle-the-digital-divide\/","url_meta":{"origin":310,"position":0},"title":"Driving towards an open internet ecosystem to help tackle the digital divide","date":"August 31, 2021","format":false,"excerpt":"Connectivity is an integral part of Facebook\u2019s mission to bring people closer together, and the COVID-19 pandemic has only heightened the demand for critical internet access. According to the latest edition of our Inclusive Internet Index, nearly 70 percent of people around the world believe that increased internet usage in\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":307,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/running-border-gateway-protocol-in-large-scale-data-centers\/","url_meta":{"origin":310,"position":1},"title":"Running Border Gateway Protocol in large-scale data centers","date":"August 31, 2021","format":false,"excerpt":"What the research is: A first-of-its-kind study that details the scalable design, software implementation, and operations of Facebook\u2019s data center routing design, based on Border Gateway Protocol (BGP). BGP was originally designed to interconnect autonomous internet service providers (ISPs) on the global internet. Highly scalable and widely acknowledged as an\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":484,"url":"https:\/\/fde.cat\/index.php\/2021\/10\/05\/more-details-about-the-october-4-outage\/","url_meta":{"origin":310,"position":2},"title":"More details about the October 4 outage","date":"October 5, 2021","format":false,"excerpt":"Now that our platforms are up and running as usual after yesterday\u2019s outage, I thought it would be worth sharing a little more detail on what happened and why \u2014 and most importantly, how we\u2019re learning from it.\u00a0 This outage was triggered by the system that manages our global backbone\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":478,"url":"https:\/\/fde.cat\/index.php\/2021\/09\/28\/2africa-pearls-subsea-cable-connects-africa-europe-and-asia-to-bring-affordable-high-speed-internet-to-3-billion-people\/","url_meta":{"origin":310,"position":3},"title":"2Africa Pearls subsea cable connects Africa, Europe, and Asia to bring affordable, high-speed internet to 3 billion people","date":"September 28, 2021","format":false,"excerpt":"Facebook invests in subsea cables as part of our continued efforts to build the infrastructure that carries internet traffic and helps bring more people online to a faster internet. Today, we are thrilled to announce, along with regional and global partners, a new segment of subsea cable called 2Africa Pearls,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":286,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/sre-weekly-issue-262\/","url_meta":{"origin":310,"position":4},"title":"SRE Weekly Issue #262","date":"August 31, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, StackHawk: Join the Secure Coding Summit to hear from industry-leading AppSec and DevSecOps practitioners, analysts, and visionaries as they share their best pro tips to level up your code security. http:\/\/sthwk.com\/secure-code-summit Articles The Prerequisites for Chaos Engineering Chaos Engineering isn\u2019t adding chaos\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":503,"url":"https:\/\/fde.cat\/index.php\/2021\/11\/11\/malbec-subsea-cable-connects-argentina-and-brazil-with-the-rest-of-the-world\/","url_meta":{"origin":310,"position":5},"title":"Malbec subsea cable connects Argentina and Brazil with the rest of the world","date":"November 11, 2021","format":false,"excerpt":"Meta, in partnership with GlobeNet, has launched Malbec, a 2,500 km subsea cable that connects the Brazilian cities of Rio de Janeiro and S\u00e3o Paulo. In the future, it will also connect Porto Alegre, Brazil to Buenos Aires. The cable landing station based in Las Toninas, Province of Buenos Aires,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/310","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=310"}],"version-history":[{"count":1,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/310\/revisions"}],"predecessor-version":[{"id":400,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/310\/revisions\/400"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=310"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=310"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=310"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}