{"id":513,"date":"2021-12-09T16:02:39","date_gmt":"2021-12-09T16:02:39","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2021\/12\/09\/using-redis-hash-instead-of-set-to-reduce-cache-size-and-operating-costs\/"},"modified":"2021-12-09T16:02:39","modified_gmt":"2021-12-09T16:02:39","slug":"using-redis-hash-instead-of-set-to-reduce-cache-size-and-operating-costs","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/12\/09\/using-redis-hash-instead-of-set-to-reduce-cache-size-and-operating-costs\/","title":{"rendered":"Using Redis HASH instead of SET to reduce cache size and operating costs"},"content":{"rendered":"

What if we told you that there was a way to dramatically reduce the cost to operate on cloud providers? That\u2019s what we found when we dug into the different data structures offered in Redis<\/a>. Before we committed to one, we did some research into the difference in memory usage between using HASH versus using SET. We found that HASHes can be used to potentially reduce the cost to operate by at least 50%<\/em><\/strong> over SET. Read on to see how we determined this and how you can put these savings into practice for yourself!<\/p>\n

Introduction<\/h3>\n

Redis is an in-memory data structure store often used as a distributed cache enabling rapid operations. It offers simple key-value stores but also more complicated data structures such as lists, hashes (map\/dict), sorted sets, etc. This blog post assumes some understanding of Redis. Read more about Redis\u00a0here<\/a>.<\/p>\n

Salesforce Activity Platform apps and customers have a configurable data retention period in which we will TTL (time-to-live)\/delete any data older than that set time. Our data can be either be at the Organization (parent) level or at the User (child) level. A parent level deletion job can only be recognized as complete if and only if all child level jobs are also complete. Our deletion pipeline used a Mongo set data structure to keep track of completed child jobs from fanout for parallel execution. However, this fanout introduced a massive load on Mongo as each child job will be required to update its respective parent set. Thus, the number of database calls scaled linearly with the average number of users per org. We decided to move this set to Redis in order to reduce load on\u00a0Mongo.<\/p>\n

Since this deletion job runs daily, we can can encounter a fluctuation of how much data we\u2019re deleting and how many jobs we\u2019re executing. Thus, to provision the correct and proper size of the cache, we performed memory usage calculations related to our use case. The testing done in this post was performed on more generalized test data, but the findings and optimizations we decided to take are still applicable.<\/p>\n

Obviously use built-in Redis SET,\u00a0right?<\/h3>\n

Redis offers a SET data structure that is very intuitive and simple to use. However, it\u2019s healthy to be cautious about datastore implementations of data structures. Our initial concerns revolved around the extra memory needed to maintain the SET data structure. Could there be a trade-off in using the data structure that is simpler to interact with? The Redis docs note memory optimizations for small HASHes, but we wanted to see if it would benefit our use\u00a0case.<\/p>\n

Luckily, simulating a SET by using a HASH data structure requires minimal additional work. To do so, we paired each key (what we would want to store into the SET) with a small value such as \u201c1.\u201d In reality, this value could be anything. However, since the value is not important to our operation, it is ideal to keep it something small. Effectively, the \u201ckey-set\u201d of our HASH is our \u201cset\u201d data structure.<\/p>\n

Our strategy results in a view of the HASH like\u00a0this:<\/p>\n

“redisHashKey”:
{
“valueToStore1”: 1,
“valueToStore2”: 1,
“valueToStore3”: 1,
“valueToStore4”: 1
}<\/p>\n

> HSET hashKey valueToStore 1 # insert
> HDEL hashKey valueToDelete # delete
> HKEYS hashKey # retrieve “set”
> HLEN hashKey # retrieve size<\/p>\n

as opposed\u00a0to\u2026<\/p>\n

“redisSetKey”: {“valueToStore1″,”valueToStore2″,”valueToStore3″,”valueToStore4”}> SADD setKey valueToStore # insert
> SREM setKey valueToDelete # delete
> SMEMBERS setKey # retrieve set
> SCARD setKey # retrieve size<\/p>\n

From testing, we see that HASH can actually be better than SET for tracking unique elements. We\u2019ve been able to cut down our memory usage by more than half employing this simple\u00a0trick.<\/p>\n

\u201chash-max-ziplist-entries\u201d config for\u00a0HASH<\/h4>\n

If using HASH, there is a mysterious config to be aware of that isn\u2019t heavily documented. You can read more about it in the official Redis docs<\/a> as well as a great blog post from Peter Bengtsson<\/a>.<\/p>\n

Effectively, this config can be thought of as \u201cRedis will employ tricks to optimize memory usage for HASHes where the number of keys within the HASH is less than the configured value.\u201d Otherwise, it effectively has the same memory usage as storing plain key-value pairs.<\/p>\n

Maybe not! We tested it\u00a0out.<\/h3>\n

We performed our tests with a simple script\u00a0that:<\/p>\n

Flushed the\u00a0cacheAssign the config value of \u201chash-max-ziplist-entries\u201dInserted multiple ranges of values to be stored (24 len random\u00a0string)Measured the memory usage of the data structure<\/p>\n

The plots below will show the memory usage of the data structure as the number of entries grow. It will also show the first derivative of the original \u201cnum entries vs memory usage\u201d curve which equates to the \u201ccost to add new entry\u201d as the data structure grows.<\/p>\n

Here is a code snippet of the script used to generate these\u00a0graphs:<\/p>\n

import redis, string, random, csv
import numpy as np
import matplotlib.pyplot as pltr = redis.Redis(host=’localhost’, port=6379, db=0)
config_name = ‘hash-max-ziplist-entries’
hashKey = ‘hash1’
setKey = ‘set1’def get_random_entry():
return ”.join(random.choice(string.ascii_lowercase) for i in range(24))def set_hash_max_ziplist(value):
r.config_set(config_name, value)
print(‘Setting config %s’ % (r.config_get(config_name)))def run_sadd_10k(): #adding to SET
print(‘Flushing cache: %s’ % r.flushall())
x_num_entries = np.arange(10000)
y_mem_usage = np.zeros(len(x_num_entries))
for i in x_num_entries:
r.sadd(setKey, get_random_entry())
mem_usage = r.memory_usage(setKey)
y_mem_usage[i] = mem_usage
diff_y = np.diff(y_mem_usage)
return x_num_entries, y_mem_usage, diff_y<\/p>\n

def run_hset_10k(): #adding to HASH
print(‘Flushing cache: %s’ % r.flushall())
x_num_entries = np.arange(10000)
y_mem_usage = np.zeros(len(x_num_entries))
for i in x_num_entries:
r.hset(hashKey, get_random_entry(), 1)
mem_usage = r.memory_usage(hashKey)
y_mem_usage[i] = mem_usage
diff_y = np.diff(y_mem_usage)
return x_num_entries, y_mem_usage, diff_y<\/p>\n

set_hash_max_ziplist(512) #set hash-max-ziplist-entries to 512
x,y,dy = run_hset_10k() #run_hset_10k or run_sadd_10k
csv_matrix = np.transpose(np.array([x,y,np.insert(dy, 0, 0, axis=0)]))
np.savetxt(“run_hset_10k_512.csv”, csv_matrix, delimiter=’,’, header=”Num Entry,Mem Usage, Diff”, comments=””)
plt.title(“HASH Num entries VS memory usage – 1k entries\/512 config”)
plt.xlabel(“num entries”)
plt.ylabel(“memory usage (in bytes)”)
plt.plot(x, y, color =”red”)
plt.show()<\/p>\n

Adding to SET\u00a0: hash-max-ziplist-entries=512\u00a0: 1k\u00a0entries<\/h4>\n

Both arrows point to constant slope sections of\u00a0graph<\/p>\n

Adding to SET\u00a0: hash-max-ziplist-entries=512\u00a0: 10k\u00a0entries<\/h4>\n

Adding to HASH\u00a0: hash-max-ziplist-entries=512\u00a0: 1k\u00a0entries<\/h4>\n

Cost per entry increases after initial spike from 28 to 72\u00a0bytes<\/p>\n

Adding to HASH\u00a0: hash-max-ziplist-entries=512\u00a0: 10k\u00a0entries<\/h4>\n

There are some things we can see by just looking at the graphs with 1k entries and \u2018hash-max-ziplist-entries\u2019=512.<\/p>\n

When (num entries < hash-max-ziplist-entries), HASH uses less memory than SET, and it costs less to add a new\u00a0entry.SET cost per child is consistent throughout (ignoring optimizations).HASH will increase after exceeding the config, but then remains overall constant.Before exceeding the config, HASH is 100% linear and more predictable than\u00a0SET.Given the same config value, less memory is used by SET for a large number of\u00a0entries.<\/p>\n

How does changing \u2018hash-max-ziplist-entries\u2019 affect memory\u00a0usage?<\/h3>\n

Below, we will change the config from 512 to 32768 on 50k entries as a dramatic example. SET does not seem to be affected by this change when comparing 50k entry runs at both different config\u00a0values.<\/em><\/p>\n

However, we see a gradual and constant increase in memory usage from 0\u201332767 entries and a spike at x=32768 (instead of x=512) <\/strong>on the HASH plots. This confirms the effects of changing this\u00a0config.<\/p>\n

Does size increase of one HASH affect\u00a0others?<\/h3>\n

An important question to ask is, if one HASH exceeds the config size that Redis optimizes for, does it nullify HASH optimizations for the entire cache? What if one of our \u201csets\u201d is an outlier with more entries than expected?<\/p>\n

For this, we\u2019ll set the config value to 1028<\/strong>. We will pre-populate HASH2 and HASH3 with 900 and 700 entries respectively. Then we will record the memory usage of all three sets as HASH1 is\u00a0growing.<\/p>\n

As you can see, HASH2 and HASH3 are unaffected by the loss of optimization from HASH1 exceeding the\u00a0limit.<\/p>\n

When do these spikes\u00a0happen?<\/h3>\n

If we look at the \u201ccost per new entry\u201d graph for HASH or SET, we can see that the increase spikes happen in powers of two. Currently, these fluctuations in cost to add new entry are unexplained outliers.<\/p>\n

*Note*<\/strong> for the plot above, since it is based on the first derivative, the x axis is off by one. The y value (for the plot below and previous plots) represents the cost to add the (x+1)th\u00a0entry.Here is a snippet of the CSV for the run\u00a0above<\/p>\n

We can try to infer that the increases are from extending the size of the backing collection. The decreases may be from some sort of metadata compression or aggregation algorithm. However, the initial loss of optimization is greatest in percentage relative to current memory usage. (511\u2192512 is 250% increase, 4095\u21924096 is\u00a020%)<\/p>\n

Conclusion<\/h3>\n

The data shows that, when possible, it is more space effective to represent a \u201cset\u201d data structure as a map\/dict in Redis (SET vs HASH). In our case, the majority of our SET sizes will be below the default value for hash_max_ziplist_entries which is 512. And, in the scenario where the SET size exceeds this configured value (which we do run into sometimes), we can be confident that our existing and new HASHes will not be affected.<\/p>\n

There is still more work to be done when looking at the performance trade-offs for increasing the config value. Given that Redis is in-memory, it is already faster than we need. It is ideal to keep the size of all your HASHes below the optimization limit. Although more of a separate topic, there are other techniques that can be employed to better ensure even distribution across multiple HASH objects, such as generating a hashcode from your object key to use\u00a0instead.<\/p>\n

Cost savings<\/h4>\n

At the time of writing this, AWS ElasticCache Pricing<\/a> shows that cache size and pricing changes are directly linked. Double in cache size, double the cost to operate per hour. From the plots showing the memory usage of 1k entries with hash_max_ziplist_entries configured to 512, there can be a memory saving of ~56% using HASH instead of SET (14403 bytes vs 32920\u00a0bytes).<\/p>\n

Employing this simple trick can help scale down the instance type potentially saving 50% on cost to\u00a0operate.<\/em><\/strong><\/p>\n

Redis already recommends provisioning an instance that is double your peak<\/strong> memory usage. Regardless, you should run these calculations on your use cases to see how much can be\u00a0saved.<\/p>\n

Learn how you can join our team to tackle problems like this one by <\/em>joining our Talent\u00a0Network<\/em><\/a>!<\/em><\/p>\n

Using Redis HASH instead of SET to reduce cache size and operating costs<\/a> was originally published in Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

What if we told you that there was a way to dramatically reduce the cost to operate on cloud providers? That\u2019s what we found when we dug into the different data structures offered in Redis. Before we committed to one, we did some research into the difference in memory usage between using HASH versus using… Continue reading Using Redis HASH instead of SET to reduce cache size and operating costs<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-513","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":483,"url":"https:\/\/fde.cat\/index.php\/2021\/10\/05\/lessons-learned-using-spring-data-redis\/","url_meta":{"origin":513,"position":0},"title":"Lessons Learned using Spring Data Redis","date":"October 5, 2021","format":false,"excerpt":"Context Our Commerce Cloud team that is in charge of the Omnichannel Inventory service uses Redis as a remote cache to store data that lends itself for caching. The remote cache allows our multiple processes to get a synchronized and single view of the cached data. (See our previous blog\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":299,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/caching-with-the-salesforce-commerce-sdk\/","url_meta":{"origin":513,"position":1},"title":"Caching with the Salesforce Commerce SDK","date":"August 31, 2021","format":false,"excerpt":"Co-written by Brian\u00a0RedmondEvery e-commerce application is going to need caching. For some of our customers, millions of shoppers may look at the same product information and, if you have to request that information from the service every time, your application will not scale. This is why we built the Commerce\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":337,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/coordinated-rate-limiting-in-microservices\/","url_meta":{"origin":513,"position":2},"title":"Coordinated Rate Limiting in Microservices","date":"August 31, 2021","format":false,"excerpt":"The ProblemAny multitenant service with public REST APIs needs to be able to protect itself from excessive usage by one or more tenants. Additionally, as the number of instances that support these services is dynamic and varies based on load, the need arrises to perform coordinated rate limiting on a\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":466,"url":"https:\/\/fde.cat\/index.php\/2021\/09\/16\/autonomous-monitoring-and-healing-networks\/","url_meta":{"origin":513,"position":3},"title":"Autonomous Monitoring and Healing Networks","date":"September 16, 2021","format":false,"excerpt":"Autonomous Monitoring and Self-Healing Networks Occasional failure is inevitable in any network system. The need of the hour is a robust, self-reliant automated monitoring tool that provides great insight and a lesser degree of manual intervention. We need autonomous interventions that save us time and enhance system availability. What Salesforce\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":864,"url":"https:\/\/fde.cat\/index.php\/2024\/05\/13\/sre-weekly-issue-424\/","url_meta":{"origin":513,"position":4},"title":"SRE Weekly Issue #424","date":"May 13, 2024","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, FireHydrant: FireHydrant is now AI-powered for faster, smarter incidents! Power up your incidents with auto-generated real-time summaries, retrospectives, and status page updates. https:\/\/firehydrant.com\/blog\/ai-for-incident-management-is-here\/ My Availability Investment Playbook Here\u2019s an ultra-practical guide to pushing for reliability investments at your company, formatted as a\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":659,"url":"https:\/\/fde.cat\/index.php\/2022\/12\/05\/sre-weekly-issue-350\/","url_meta":{"origin":513,"position":5},"title":"SRE Weekly Issue #350","date":"December 5, 2022","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, Rootly: Manage incidents directly from Slack with Rootly\u00a0\ud83d\ude92. Rootly automates manual tasks like creating an incident channel, Jira ticket and Zoom rooms, inviting responders, creating statuspage updates, postmortem timelines and more. Want to see why companies like Canva and Grammarly love us?:\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/513","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=513"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/513\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=513"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=513"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=513"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}