Our Commerce Cloud team that is in charge of the Omnichannel Inventory service uses Redis<\/a> as a remote cache to store data that lends itself for caching. The remote cache allows our multiple processes to get a synchronized and single view of the cached data. (See our previous blog post, Coordinated Rate Limiting in Microservices <\/a>for an example of how we used Redis for Rate Limiting access to our system by tenants.) The usage pattern is entries that were shorter lived, high cache-hit and shared among instances<\/strong>. To interact with Redis, we utilized Spring Data Redis <\/a>(with Lettuce<\/a>), which has been helping us to share our data entries among our instances and provide a low-code solution to interact with\u00a0Redis.<\/p>\n A subsequent deployment of our applications showed an oddity, where the memory consumption on Redis was continuously increasing, with no indication of reducing.<\/p>\n The memory growth on the Redis Server is reflected in the following graph:<\/p>\n The memory consumption showed an almost linear increase over time, with increased throughput on the system and no significant reclamation over time. The scenario reached such extremes that it warranted a manual flush<\/em> of the Redis database as the memory increased and neared 100%. The above seemed to point toward a memory leak happening with the Redis\u00a0entries.<\/p>\n The first suspect was that the Redis entries were either configured without a Time to Live (TTL) or with a TTL value that was beyond the one intended. This was indicative that the Redis Repository entity class we had for Rate Limiting was devoid of any TTL configuration:<\/p>\n @RedisHash(“rate”) \/\/ CRUD repository. To validate this on Data that the TTL was indeed not set, a connection was made to the Redis Server instance and the Redis command TTL <name entry key> was used to check the value for TTL on some of the entries that were\u00a0listed.<\/p>\n TTL “rate:block_00001” As shown above, there were entries that had a TTL of -1 indicating non-expiring. While this clearly appeared a suspect for the issue at hand, and fixing it to explicitly set TTL values to practice good software hygiene seemed the direction forward, there was some skepticism that this were the real cause of the problem due to the relatively small amount of entries and memory used being reported.<\/p>\n With the TTL added, the entry code is show\u00a0below.<\/p>\n @RedisHash(“rate”) @TimeToLive<\/strong> When doing this exercise, we also noticed that on the entities that was, there was an extra entry<\/strong> created using the hash value that we declared, and this was happening in all of our interfaces that extended from CRUDRepository.<\/p>\n An example was the following hash declared for rate<\/em>\u00a0entries.<\/p>\n @RedisHash(“rate”)<\/p>\n In order to check this, we used the following Redis commands:<\/p>\n KEYS * As you can see, there are two<\/strong> entries. One is an entry with key name \u201crate:block_00001\u201d and an extra entry with key \u201crate\u201d. The \u201crate:block_00001\u201dwas expected, but the other entry was surprising to\u00a0find.<\/p>\n Monitoring the system over time also showed that the memory associated with the \u201crate\u201d key was steadily increasing.<\/p>\n >MEMORY USAGE “rate” In addition the to increased memory growth, the TTL on the \u201crate\u201d entry was -1 as shown by the\u00a0below:<\/p>\n >TTL “rate” It clearly pointed to the most plausible suspect where its growth showed no sign of reducing over\u00a0time.<\/p>\n So, what was this entry and why was it\u00a0growing?<\/p>\n Spring Data Redis creates a SET data type in Redis for every @RedisHash<\/em>. The entries of the SET act as an index for many of the Spring Data Redis operations used by the CRUD repository.<\/p>\n The SET entries, for example, look like the\u00a0below:<\/p>\n >SMEMBERS “rate” We decided to post our situation in Stack Overflow<\/a> and on Spring Data Redis\u2019s GitHub page<\/a> requesting some assistance from the community on the issue on how best to address this issue\u200a\u2014\u200aeither to prevent the growth of this SET, or how to prevent its creation, as we really did not need any other indexing\u00a0feature.<\/p>\n While awaiting a response from the community, we discovered that enabling a property<\/a> of the Spring Data Redis annotation EnableRedisRepositories<\/em> will actually make Spring Data Redis listen for KEY<\/strong> Events and clean up the Set over time as it receives KEY expired\u00a0events<\/a>.<\/p>\n @EnableRedisRepositories(enableKeyspaceEvents With this setting enabled, Spring Data Redis will ensure that the memory of the Set does not keep increasing and that expired entries are purged out (See this Stack Overflow Question<\/a> on details).<\/p>\n “rate” The Phantom Keys are created so that Spring Data Redis can propagate a RedisKeyExpiredEvent<\/em><\/a> with relevant data to Spring Framework\u2019s ApplicationEvent<\/a> subscribers. The Phantom (or Shadow) entry is longer lived than the entry it is shadowing, so when the primary entry expired event is received by Spring Data Redis, it will obtain values from the Shadow entry to propagate the RedisKeyExpiredEvent, <\/em>which will house a copy of the expired domain object in addition to the\u00a0key.<\/p>\n The following code in Spring Data Redis receives the phantom entry expiration and purges the item from the\u00a0index:<\/p>\n static class MappingExpirationListener extends KeyExpirationEventMessageListener {<\/p>\n private final RedisOperations<?, ?> ops; ops.execute((RedisCallback<Void>) connection -> { The primary concern with this approach is the additional processing overhead that Spring Data Redis incurs for having to consume the expired event stream and perform the clean up. It should also be noted that, since the Redis Pub\/Sub messages are not persistent, if entries expire while the application is down, then expired events are not processed, and those entries will not get cleaned up from the\u00a0SET.<\/p>\n Using CRUDRepository effectively means more shadow\/support entries are created for each entry, causing more consumption of memory from the Redis server\u2019s database. If one does not need the details of the expiration in the Spring Boot Application when an entry expired, you can disable the generation of the Phantom entries with the following change to the EnableRedisRespositories<\/em> annotation.<\/p>\n @EnableRedisRepositories(.. shadowCopy = ShadowCopy.OFF<\/strong>)<\/p>\n The net effect of the above is that Spring Data Redis will no longer create the Shadow copy but will still subscribe for the Keyspace events and purge the SET of the entry. Spring Boot Application Events propagated will only contain the KEY and not the full Domain\u00a0object.<\/p>\n With all the above discovered around performance and additional memory storage, we decided that it felt that for the use cases we were dealing with, this additional overhead added by Redis CRUDRepository and KEY Space events was not something that was appealing to us. For this reason, we decided to explore a more leaner approach.<\/p>\n We made a proof-of-concept application to test the differences in response time between using CrudRepository or working directly with the RedisTemplate class that exposes the Redis server operations. Through testing we observed RedisTemplate to be more favorable.<\/p>\n This graph is from the metrics in our system using CRUDRepository<\/em>, which includes the overhead of handling the\u00a0indexes.This graph represents metrics from the proof-of-concept application that doesn\u2019t use CRUDRepository<\/em> and works directly with RedisTemplate<\/em>.<\/p>\n The comparison was made by executing GET operations non-stop for five minutes and taking an average of the time it took to complete the operation. What we saw is that the almost all GET operations using CRUDRepository were in the milliseconds range, while the proof-of-concept without CRUDRepository was mostly on the nano-seconds realm. Another thing we noticed was that CRUDRepository also has a tendency to have more upticks when performing the operations, increasing the latency of executing its operations.<\/p>\n Based on the research, we were down to the following paths\u00a0forward:<\/p>\n Spring Data Redis CrudRepository: <\/strong>Enable Redis Repository\u2019s key space events and enable Spring Data Redis to purge the Set type of expired entries. The upside of this approach is that it is a low-code solution achieved by setting a value on an annotation, letting Spring Data Redis subscribe for KEY expired events and clean up behind the scenes. The downside is, for our case, the additional usage of memory for something we never utilize, i.e., the SET index and the processing overhead occurred by Spring Data Redis subscribing for Keyspace events and performing the\u00a0cleanup.Custom Repository using RedisTemplate:<\/strong> Handle the Redis I\/O Operation without using the CRUD Repository, use the RedisTemplate,<\/em> and build the basic needed operations. The upside is that it results in creating only what data we need in Redis, i.e., the hash entries, and not other artifacts like the SET index. We avoid the processing overhead of Spring Data Redis subscribing and processing Keyspace events for clean up. The downside is that we stop availing the low-code magic of Spring Data Redis\u2019s CRUD repository and the work it does behind the scenes and instead do all the work with\u00a0code.<\/p>\n After considering all of our findings, especially the metrics around the proof-of-concept application and our system, and the needs that we have on the team, which are more about quick response times and low memory usage, the direction we adopted was not to use the CrudRepository<\/em> but to use the RedisTemplate<\/a> to interact with the Redis server. It presents a solution that includes far less unknown behavior, due to the code being more transparent and the functionality more straight\u00a0forward.<\/p>\n Our code ended up looking like\u00a0this:<\/p>\n public class RateRedisEntry implements Serializable { template.setConnectionFactory(getLettuceConnectionFactory());<\/p>\n return template; private final RedisTemplate<String, RateRedisEntry> redisTemplate;<\/p>\n public RedisCachedRateRepositoryImpl(RedisTemplate<String, RateRedisEntry> redisTemplate) { public Optional<RateRedisEntry> find(String key, Tags tags) { public void put(final @NonNull RateRedisEntry rateEntry, Tags tags) { private String composeHeader(String key) { By using it this way, we worked directly with the entries, so there are no risks of unwanted indexes or structures being\u00a0stored.<\/p>\n Once our solution got deployed, our memory usage totally dropped and remained stable, with any peaks going down after the TTL of the entries reached\u00a00.<\/p>\n Another benefit that we saw of using the RedisTemplate direct approach was an improvement in our response times of the operations we were performing, as shown on the comparison metrics above from running the GET operation for some time on our proof-of-concept and our system. Before the change, we saw values that averaged in the milliseconds<\/strong>; after deploying the change, we started seeing that most operations were being completed in nanoseconds<\/strong>.<\/p>\n The magic of Spring Data Redis Crud Operations is achieved by the creation of additional data structures like the SET for Indexes. These additional data structures are not cleaned up when items expire without enabling<\/strong> Spring Data Redis to listen for KEY space events. For caching patterns where entries are very long-lived or where the set of entries is tractable and finite, Spring Data Redis with CrudRepositories provides a low-code solution for CRUD operations for\u00a0Redis.<\/p>\n However, for caching patterns where the data is cached and shared by multiple processes and where the entries have a smaller window where they can be cached, avoiding listening for KEY events and using the RedisTemplate<\/em> to perform Redis operations for the CRUD operations needed seems\u00a0optimal.<\/p>\n Share your thoughts in the comments below; we\u2019d love to hear of your experiences with Spring Data Redis or Redis in\u00a0general.<\/p>\n Appreciation Note: Thanks to Sanjay Acharya<\/a> and Balachandar Mariappan for all the help given with the reviewing and refining. You guys are awesome\u2026\u00a0\ud83d\ude42<\/p>\n Lessons Learned using Spring Data Redis<\/a> was originally published in Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"Investigation<\/h3>\n
public class RateRedisEntry implements Serializable {
@Id
private String tenantEndpointByBlock; \/\/ A HTTP end point
…
}<\/p>\n
@Repository
public interface RateRepository extends CrudRepository<RateRedisEntry, String> {}<\/p>\n
-1<\/p>\n
public class RateRedisEntry implements Serializable {
@Id
private String tenantEndpointByBlock;<\/p>\n
<\/strong>private Integer expirationSeconds;
…
}<\/p>\n
1) “rate”
2) “block_00001”<\/p>\n
(integer) 153034<\/strong>
.
.
.
> MEMORY USAGE “rate”
(integer) 153876<\/strong>
.
.
> <\/strong>MEMORY USAGE <\/strong>“rate”
(integer) 163492<\/strong><\/p>\n
-1
>TYPE “rate”
set<\/p>\n
1) “block_00001”
2) “block_00002”
3) “block_00003”
…<\/p>\n
= EnableKeyspaceEvents.ON_STARTUP)<\/strong><\/p>\n
“rate:block_00001”
“rate:block_00001:phantom”<\/strong> <– Phantom entry in addition to base<\/strong>
…<\/p>\n
…
@Override
public void onMessage(Message message, @Nullable byte[] pattern) {
…
RedisKeyExpiredEvent event = new RedisKeyExpiredEvent(channel, key, value);<\/p>\n
\/\/ Removes entry from the Set
connection.sRem(converter.getConversionService()
.convert(event.getKeyspace(), byte[].class), event.getId());
…
});
}
..
}<\/p>\nSolution<\/h3>\n
private String tenantEndpointByBlock;
private Integer expirationSeconds;
…
}
@Bean
public RedisTemplate<String, RateRedisEntry> redisTemplate() {
RedisTemplate<String, RateRedisEntry> template = new RedisTemplate<>();<\/p>\n
}
public class RedisCachedRateRepositoryImpl implements RedisCachedRateRepository {<\/p>\n
this.redisTemplate = redisTemplate;
}<\/p>\n
return Optional.ofNullable(this.redisTemplate.opsForValue()
.get(composeHeader(key)));
}<\/p>\n
this.redisTemplate.opsForValue().set(composeHeader(rateEntry.getTenantEndpointByBlock()),
rateEntry, Duration.ofSeconds(rateEntry.getRedisTTLInSeconds()));
}<\/p>\n
return String.format(“rate:%s”, key);
}
}<\/p>\nConclusion<\/h3>\n