{"id":174,"date":"2021-01-28T19:59:00","date_gmt":"2021-01-28T19:59:00","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2021\/01\/28\/taming-memory-fragmentation-in-venice-with-jemalloc\/"},"modified":"2021-02-02T13:41:44","modified_gmt":"2021-02-02T13:41:44","slug":"taming-memory-fragmentation-in-venice-with-jemalloc","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/01\/28\/taming-memory-fragmentation-in-venice-with-jemalloc\/","title":{"rendered":"Taming memory fragmentation in Venice with Jemalloc"},"content":{"rendered":"
\n
\n <\/a>\n <\/div>\n
\n
\n

Sometimes, an engineering problem arises that might make us feel like maybe we don’t know what we’re doing, or at the very least, forces us out of the comfort zone of our area of expertise. That day came for the Venice team at Linkedin when we began to notice that some Venice<\/a> processes would consume all available memory and crash if given enough time to run. This ended up being due to memory fragmentation. In this blog post, we’ll explain the different ways we tried to diagnose the problem, and how we solved it.<\/p>\n

What is Venice?<\/h2>\n

Venice<\/a> is LinkedIn’s platform for serving derived data. Venice was designed for very low latency lookups while also supporting high throughput ingestion of data. Reads are served directly from Venice, while writes enter the system asynchronously, either pushed as full versioned data sets, or as incremental updates coming from either Hadoop or stream sources like Samza<\/a>. At its core, Venice is a sharded multi-tenant clustering software that leverages Kafka<\/a> and RocksDB<\/a> to power high throughput lookups for LinkedIn features (like People You May Know).<\/p>\n

You can learn more about Venice here<\/a>.<\/p>\n

The symptom of the problem<\/h2>\n

As mentioned earlier, we noticed that if we left a server node alone for long enough, it would end up crashing. We generally try to release Venice to production on a weekly cadence to align with our development cycle, but we began to notice that if we slipped and left the software running for longer (a few weeks), things started to get scary. Nodes would go offline across clusters in batches. OS system metrics and logs seemed to point to a growing lack of available memory. Moreover, we had error messages in backtraces from generated core dumps and hs_err_pid<\/span> logs that all seemed to indicate failures on allocations, as well as sometimes having lines from dmesg<\/a> telling us that OOM killer had stepped in. With some added metrics, we could see that, over time, resident memory (RSS) would trend steadily upward along with virtual memory (VSS).<\/p>\n

A memory leak!
<\/b>A lot of software engineers have played this game before. There are plenty of tools and strategies out there for puzzling memory leaks, and many an article offering succor for such maladies. I am a Java developer, and I’m used to working with tools like visualvm<\/a> and OQL<\/a>. This influenced our initial track of investigation. We grabbed some heap dumps and got to digging.<\/p>\n

visualvm has a nice feature where you can compare two heap dumps and see what the difference is in memory footprint and object counts. If hooking up a profiling tool to a production service is problematic, a handy alternative can be to start the service, grab a heap dump, let it run for a while until you see that the process has hit its \u201cdegraded\u201d state, and grab another heap dump at that point. You can then plug both dumps into visualvm and get a report on the difference of objects.\u00a0\u00a0<\/p>\n

But we didn\u2019t find anything. Unfortunately though, this process didn’t tell us very much. We were not seeing an accumulation of objects in the heap that would indicate that we were leaving around an increasing volume of objects. Moreover, the heap sizes weren’t actually getting bigger over time.<\/p>\n

Could it be something off-heap?
<\/b>jcmd<\/a> is another great tool that comes bundled with Java. It has utilities for tracking native memory allocations (i.e., off-heap memory allocations made by the JVM). You can execute it on your process with the VM.native_memory<\/span> command, and get a report on the memory usage on your process. You can also add the baseline<\/span> argument and you can set a starting point on what the memory usage is at time of execution, so you can come back later and see what the difference is. You’ll get a nice report that looks like the following:<\/p>\n

jcmd <pid> VM.native_memory<\/span><\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n

\n
\n <\/a>\n <\/div>\n