{"id":704,"date":"2023-04-17T16:15:48","date_gmt":"2023-04-17T16:15:48","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/04\/17\/a-fine-grained-network-traffic-analysis-with-millisampler\/"},"modified":"2023-04-17T16:15:48","modified_gmt":"2023-04-17T16:15:48","slug":"a-fine-grained-network-traffic-analysis-with-millisampler","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/04\/17\/a-fine-grained-network-traffic-analysis-with-millisampler\/","title":{"rendered":"A fine-grained network traffic analysis with Millisampler"},"content":{"rendered":"

What the research is:\u00a0<\/span><\/h2>\n

Millisampler<\/a> is one of Meta\u2019s latest characterization tools and allows us to observe, characterize, and debug network performance<\/a> at high-granularity timescales efficiently. This lightweight network traffic characterization tool for continual monitoring operates at fine, configurable timescales. It collects time series of ingress and egress traffic volumes, number of active flows, incoming ECN marks, and ingress and egress retransmissions. Additionally, Millisampler is also able to identify in-region traffic and cross-region traffic (longer RTT). Millisampler runs on our server fleet collecting short, periodic snapshots of this data at 100us, 1ms, and 10ms time granularities, stores it in local disk, and makes it available for several days for on-demand analysis. Since the data is only aggregated flow-level header information, it does not contain any personally identifiable information (PII). Even with the minimal amount of information it collects, Millisampler data has proven very useful in practice,\u00a0 particularly when combined with existing coarser-grained data \u2014 we are able to see clearly how switch buffers or host NICs, for example, might be unable to handle the ingress traffic pattern<\/a>.<\/span><\/p>\n

<\/div>\n

\u00a0<\/p>\n

How it works:\u00a0<\/span><\/h2>\n

Millisampler comprises userspace<\/a> code to schedule runs, store data, and serve data, and an eBPF-based tc filter that runs in the kernel to collect fine-timescale data. The user code attaches the tc filter and enables data collection. A tc filter is among the first programmable steps on the receipt of a packet and near the last step on transmission. On ingress, this means that the eBPF code executes on the CPU core that is processing the soft irq (bottom half) as the packet is directed toward the owning socket. Because processing happens on many CPU cores, to avoid locks, we use per-CPU variables, which increase the memory requirement to eliminate risk of contention. To minimize overhead, we sample periodically and for short periods of time. Userspace therefore configures two parameters in Millisampler: the sampling interval <\/a> and the number of samples. We schedule runs with three sampling intervals: 10ms, 1ms, and 100\u03bcs, with a fixed number of samples to 2,000 for all sampling intervals. This means that our observation periods range from 200ms (100\u03bcs sampling rate) to 20s (10ms sampling rate), allowing us to observe events at sub-RTT to cross-region RTT time scales, and, at the same time, fix the memory footprint of each run to 2,000 64-bit counters per CPU core for each value we measure.<\/span><\/p>\n

Millisampler collects a variety of metrics. It computes ingress and egress total bytes and ingress ECN-marked bytes from the lengths and CE bits of the packets. Millisampler also sounds<\/span>TTLd marked retransmits<\/span><\/a>. Millisampler uses a 128-bit sketch to estimate the number of active (incoming and outgoing) connections. Using the sketch results in an approximation of the connection count that is precise up to a dozen connections and saturates at around 500 connections per sampling interval. Although there is space for additional precision, in practice, more than the actual number of connections, the qualitative variation between a few connections to dozens or hundreds of connections has been helpful toward identifying patterns of traffic with more connections (heavy incast) as opposed to more traffic with fewer connections. <\/span><\/p>\n

Why it matters:<\/span><\/h2>\n

Millisampler is a powerful tool for troubleshooting and performance analysis. Two contrasting network performance faults that we solved at Meta in the last few years relied on our needing a fine-grained view of traffic. The first problem featured synchronized traffic bursts at fine time scales, and seeing this motivated us to build and deploy Millisampler to catch it quickly if it happened again. The second, which an early Millisampler prototype helped root-cause, featured a NIC driver bug that caused it to stop delivering packets for milliseconds at a time, thereby proving the value of Millisampler in complex investigations. While Millisampler (or Millisampler-like data) played an important role in these investigations, it was only as part of our rich ecosystem of data collection tools that track a dizzying array of metrics across hosts and a network.<\/span><\/p>\n

Beyond such incidents, Millisampler data has also proven useful in characterizing and analyzing traffic characteristics of services, allowing us to design and deploy a range of solutions to help improve their performance. For example, we have been able to characterize the nature of bursts across a number of services in order to understand the intensity of incast and tune transport performance accordingly. We have also been able to look at complex interactions between short-RTT and long-RTT flows and understand how bursts of either affect fairness for the other. In a following post, we will look at an extension of Millisampler \u2014 Syncmillisampler \u2014 where we run Millisampler synchronously across all hosts in a rack and use that data to identify buffer contention in the top-of-rack ASICs.<\/span><\/p>\n

Read the full paper:<\/span><\/h2>\n
\n

A microscopic view of bursts, buffer contention, and loss in data centers<\/a><\/p>\n<\/div>\n

Acknowledgements:<\/span><\/h2>\n

Ehab Ghabashneh, Cristian Lumezanu, Raghu Nallamothu, and Rob Sherwood also contributed to the design and implementation of Millisampler.<\/span><\/p>\n

The post A fine-grained network traffic analysis with Millisampler<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

What the research is:\u00a0 Millisampler is one of Meta\u2019s latest characterization tools and allows us to observe, characterize, and debug network performance at high-granularity timescales efficiently. This lightweight network traffic characterization tool for continual monitoring operates at fine, configurable timescales. It collects time series of ingress and egress traffic volumes, number of active flows, incoming… Continue reading A fine-grained network traffic analysis with Millisampler<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-704","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":319,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/network-hose-managing-uncertain-network-demand-with-model-simplicity\/","url_meta":{"origin":704,"position":0},"title":"Network hose: Managing uncertain network demand with model simplicity","date":"August 31, 2021","format":false,"excerpt":"Our production backbone network connects our data centers and delivers content to our users. The network supports a vast number of different services, distributed across a multitude of data centers. Traffic patterns shift over time from one data center to another due to the introduction of new services, service architecture\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":630,"url":"https:\/\/fde.cat\/index.php\/2022\/09\/07\/network-entitlement-a-contract-based-network-sharing-solution\/","url_meta":{"origin":704,"position":1},"title":"Network Entitlement: A contract-based network sharing solution","date":"September 7, 2022","format":false,"excerpt":"Meta\u2019s overall network usage and traffic volume has increased as we\u2019ve continued to add new services. Due to the scarcity of fiber resources, we\u2019re developing an explicit resource reservation framework to effectively plan, manage, and operate the shared consumption of network bandwidth, which will help us keep up with demand\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":670,"url":"https:\/\/fde.cat\/index.php\/2023\/01\/27\/watch-metas-engineers-discuss-optimizing-large-scale-networks\/","url_meta":{"origin":704,"position":2},"title":"Watch Meta\u2019s engineers discuss optimizing large-scale networks","date":"January 27, 2023","format":false,"excerpt":"Managing network solutions amidst a growing scale inherently brings challenges around performance, deployment, and operational complexities.\u00a0 At Meta, we\u2019ve found that these challenges broadly fall into three themes: 1.) \u00a0 Data center networking: Over the past decade, on the physical front, we have seen a rise in vendor-specific hardware that\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":344,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/risk-driven-backbone-management-during-covid-19-and-beyond\/","url_meta":{"origin":704,"position":3},"title":"Risk-driven backbone management during COVID-19 and beyond","date":"August 31, 2021","format":false,"excerpt":"What the research is:\u00a0 A first-of-its-kind study detailing our backbone management strategy to ensure high service performance throughout the COVID-19 pandemic. The pandemic moved most social interactions online and caused an unprecedented stress test on our global network infrastructure with tens of data center regions. At this scale, failures such\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":842,"url":"https:\/\/fde.cat\/index.php\/2024\/03\/20\/optimizing-rtc-bandwidth-estimation-with-machine-learning\/","url_meta":{"origin":704,"position":4},"title":"Optimizing RTC bandwidth estimation with machine learning","date":"March 20, 2024","format":false,"excerpt":"Bandwidth estimation (BWE) and congestion control play an important role in delivering high-quality real-time communication (RTC) across Meta\u2019s family of apps. We\u2019ve adopted a machine learning (ML)-based approach that allows us to solve networking problems holistically across cross-layers such as BWE, network resiliency, and transport. We\u2019re sharing our experiment results\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":604,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/06\/watch-metas-engineers-discuss-quic-and-tcp-innovations-for-our-network\/","url_meta":{"origin":704,"position":5},"title":"Watch Meta\u2019s engineers discuss QUIC and TCP innovations for our network","date":"July 6, 2022","format":false,"excerpt":"With more than 75 percent of our internet traffic set to use QUIC and HTTP\/3 together, QUIC is slowly moving to become the de facto protocol used for internet communication at Meta. For Meta\u2019s data center network, TCP remains the primary network transport protocol that supports thousands of services on\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/704","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=704"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/704\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=704"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=704"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=704"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}