ATF<\/strong>\u200a\u2014\u200aAvailable to Fulfill inventoryOn-Hand\u200a<\/strong>\u2014\u200aPhysical amount of Inventory availableSKU<\/strong>\u200a\u2014\u200aStock Keeping Unit, which represents a distinct type of item for\u00a0sale.Location<\/strong>\u200a\u2014\u200aRepresentation of a physical location like a store or warehouse where SKU\u2019s are\u00a0presentLocation Group<\/strong>\u200a\u2014\u200aA Logical aggregation of typically one or more Locations.Reservation or Inventory Reservation<\/strong>\u200a\u2014\u200aReserving a quantity of a SKU. For example: Reservation of 5 iPod-Nanos-RedInventory Records<\/strong>\u200a\u2014\u200aA quantity for a SKU at a particular location.<\/p>\n Salesforce Commerce Cloud<\/a> is a product that provides e-commerce solutions for both business-to-business (B2B) and <\/em>business-to-consumer (B2C) companies. To continue enhancing what we are able to offer our customers, we needed to build a multi-tenant, omnichannel inventory availability solution that could be shared by multiple Salesforce products, including B2C Enterprise, Order Management, and B2B. This solution needed to be highly available, highly performant, and highly accurate in its bookkeeping, while unlocking new features like Location Level Inventory Aggregation.<\/em><\/p>\n Location Level Inventory Aggregation is a feature that enables the aggregation of inventory from one or more locations into a Location Group, thereby allowing inventory reservations to be performed at either Location or Location Group level for BOPIS<\/em> (Buy On-line and Pickup in store) among other omnichannel scenarios.<\/p>\n The solution which was to be built had to deal with high contention reservation and row lock concerns associated with scenarios like flash sales over a few highly sought after items in a limited time\u00a0window.<\/p>\n Put yourselves in the shoes of a major retailer with the holiday shopping season fast approaching and you\u2019ll be primed to understand the need for this solution.<\/p>\n Accuracy<\/strong>: Inventory reservations must be accurate The solution was expected to support thousands of tenants globally with the maximum number of inventory records per tenant to extend into the billions. The solution also needed to be able to process hundreds of reservation requests per second with low latency while doing the same returning availability of SKUs<\/em> for each tenant. There was also a requirement to maintain an audit log of changes to the inventory for diagnosability and forensics.<\/p>\n Thinking back to our imagined role as a major retailer, we can put these requirements even more simply: if someone reserves a pair of, say, sneakers, that reservation needs to be based on inventory that actually exists, and the person shouldn\u2019t be able to reserve the same pair of shoes twice. As the retailer, you need everything behind the scenes of your website to happen seamlessly.<\/p>\n So how did we satisfy all of these requirements in way that could scale to the levels needed by our many customers?<\/p>\n Event Sourcing<\/a> pattern is an approach to operations around data where changes are recorded to an append-only<\/strong> datastore in a sequential manner as discrete immutable events. The store behaves as the system of record maintaining the history of changes, and domain objects can be materialized by replaying the individual events and folding<\/strong> them into the aggregate.<\/p>\n Event Sourcing<\/p>\n Or as Greg Young stated it rather simplistically \u2014<\/p>\n \u200a\u2014\u200a@gregyoung<\/a><\/p>\n Reconstituting the aggregate by folding the events up during every query is not tractable. For this reason SNAPSHOTs<\/em> are implemented and only the events since the last snapshot are folded-in to provide the current state of the aggregate. We should also note that the Event Log is an ordered sequence of facts<\/strong> of things that have happened and is not series of intentions or commands.<\/p>\n Reconstituting the aggregate by folding the events up during every query is not tractable. For this reason, SNAPSHOTs<\/em> are implemented and only the events since the last snapshot are folded-in<\/em> to provide the current state of the aggregate. We should also note that the Event Log is an ordered sequence of facts<\/strong> of things that have happened and is not series of intentions or commands.<\/p>\n Inventory availability mutations typically occur to change on-hand<\/em> values or create reservations. These operations can positively or negatively affect the Available to Fulfill (ATF) of a stock-keeping unit (SKU, basically a barcode, but used here to mean more generally \u201can\u00a0item\u201d).<\/p>\n For the sake of illustration in this blog, let us consider a fictional merchant, Acme. <\/em>Acme has a website, acme.com<\/a>, and has a warehouse in Salt Lake City, UT, from where they ship their merchandise. In addition, Acme also has physical stores in Las Vegas, NV and Seattle, WA. These two stores also support shipping from the store of products purchased on acme.com.<\/p>\n Acme Location Graph\u00a0Setup<\/p>\n In the example, we have a SKU, acme-hat-blue<\/em> which is present at the warehouse, and at the Seattle and Las Vegas stores. The warehouse has 220 on-hand and there are 20 reservations made against it, effectively leaving 200 items that can be fulfilled by the warehouse\u200a\u2014\u200aare available to be sold. The Location Inventory State is shown\u00a0below:<\/p>\n Location Inventory Aggregate<\/p>\n In the Event Sourcing model, a bunch of immutable<\/strong> events occur and these are then applied in order to build the state of the aggregate.<\/p>\n The following shows a sample of how the Location Inventory Aggregate<\/em> is built over time with different events. The yellow rows on the table show the impacted data points at each\u00a0state:<\/p>\n Location Inventory Aggregate Updates over\u00a0time<\/p>\n In the example shown above, the initial state of the aggregate is NULL<\/strong>, i.e., nothing in it. When the first import event comes in, it gets applied, and the aggregate reflects the same. Subsequent reservation events affect the aggregate similarly, thereby altering the ATF\u00a0number.<\/p>\n The important thing to note here is that the Event Log is a sequence of persistent facts that have occurred, and, at any time, the aggregate can be reconstituted from the Event Log by replaying all events since inception. However, this is not likely to be tractable as the number of events grow. To avoid this, events are folded and snapshotted<\/strong> at a regular interval and stored in an entity called Location Inventory Aggregate<\/em> as\u00a0shown.<\/p>\n As we needed to surface the Location Group Inventory by aggregating inventory across different locations, we do the same using an entity that is realized called GroupInventoryView<\/strong>. This Group Inventory View has the aggregated inventory counts in\u00a0it.<\/p>\n Group Inventory View<\/p>\n GroupInventoryView<\/em> entries are computed by rolling\/folding<\/strong> up the inventory availability of the different locations part of the group into<\/strong> the group. One of our initial approaches involved computing this aggregation on the fly when an API\u2019s invocation was performed to obtain the Location Group\u2019s inventory where the different location level inventory would be combined to serve the Group Level Numbers. This, however, proved to be a performance concern due to the amount of I\/O, slower response time, and additional compute added to the invocation.<\/p>\n To address this problem, we availed one of the core benefits of Event Sourcing, which are subscriptions<\/strong> where additional actions can be taken by subscribing to the Event Sourced Log. This is similar to Change Data Capture (CDC) in databases. We utilized this CDC ability to roll up the Location level inventory availability into the GroupInventoryView<\/em>by subscribing to the event stream and updating the Location Group numbers and persisting them to the GroupInventoryView<\/em>. This approach worked for us due to the acceptable staleness<\/strong> of the records allowed for the GroupInventoryView<\/em>.<\/p>\n Group Inventory View Update using Event Sourcing Subscription Feature<\/p>\n High Contention Reservation<\/p>\n One of our primary objectives is to avoid overselling, as outlined in the first functional requirement above, while also providing a performant and responsive experience during inventory reservation. When inventory approaches low levels, if more than a single caller is competing to grab the item, the regular approach using a transactional datastore is to lock the row and update\/decrement the inventory would result in large latency due to high contention. In other words, a website, store terminal or other consumer attempting to do inventory reservations would be slow, and the request might even error out without completing, so the user experience of the person trying to reserve the item would\u00a0suffer.<\/p>\n To address this problem, we decided to use an optimistic concurrency <\/em><\/strong>approach <\/em><\/strong>to avoid row locking. We utilize an Event Sequence number that <\/strong>is monotonically increasing associated with a SKU-Location<\/strong> record.<\/p>\n The following shows a sample Location Inventory Events for Reservation and Inventory Import<\/em>:<\/p>\n Location Inventory Event for Reservation \u2014<\/p>\n { “name” : “LocationInventoryEvent”, Location Inventory Event for Inventory Import\u00a0\u2014<\/p>\n { The algorithm to apply a reservation<\/strong> event looks like\u00a0this:<\/p>\n function reserveInventory<\/strong> [Event reservationEvent<\/strong>, 2. if eventLog<\/strong>.get(reservationEvent<\/strong>.eventId) is not<\/strong> NULL<\/strong> 7. reservationEvent<\/strong>.event_sequence_number 10. return success<\/strong><\/p>\n The event log is queried to determine the current sequence number of the last inserted event. The Reservation Event\u2019s sequence number is set to one above the last known sequence number. An append of the Event is then attempted on the event log. A constraint exists on the event log that ensures uniqueness of [Location-id, SKU, event_sequence_number], <\/strong>so if another thread had attempted to insert an event for the SKU\/Location and had gotten through, this insert would have failed. A retry is attempted for a few times before giving\u00a0up.<\/p>\n Reads performed against the inventory solution by consumers are not promised to be strongly consistent.<\/p>\n Weakly Vs. Strongly Consistent Read<\/strong><\/p>\n Queries performed against the aggregate represent data that is not guaranteed to be strongly consistent and reflective of the latest state of the event log. This is acceptable for almost all read use cases where stale data on the READ side is fine. If a more consistent read were required, it would entail reading in the Location Inventory Aggregate and applying all events in the stream to it in real time, which could result in higher\u00a0latency.<\/p>\n We considered other options to solve this problem, such as directly mutating rows and then having some form of Change Data Capture (CDC) to capture the changes for auditability and forensics. During the evaluations, it became more apparent that, rather than maintaining a separate Event Stream of changes, using Event Sourcing Pattern with its Event Log gave us the historical changes baked into the solution without having to deal with the complexities of a CDC and a separate\u00a0log.<\/p>\n CDC<\/p>\n We wanted our solution to also avoid<\/strong> dealing with distributed transactions, for example, where we created an entry in a database and then fired a message into a message broker due to the inherent unreliability of such operations from an atomic perspective.<\/p>\n To implement the solution, we had to pick the appropriate data store and eventing system. The following were some of the needs of our Event log and Aggregates:<\/p>\n Event log<\/strong><\/p>\n Event log is totally ordered, immutable (attempts to update or delete events are rejected) and is append-onlyEvent log contains each event exactly once (attempts to append a duplicate event are rejected)Event log append may enforce additional user-defined constraints (e.g., attempt to append a reservation event is rejected if insufficient inventory available)High performance append log predictability from a time perspective on\u00a0inserts.<\/p>\n From an aggregate perspective, we needed to make sure the solution supports SNAPSHOTs<\/em>.<\/p>\n We considered several popular log-like data-stores like Kafka, Cassandra, and EventStore to build the above but had certain reservations at that time around each of\u00a0them:<\/p>\n Kafka<\/strong>\u200a\u2014\u200aDid not have the ability to support unique events (not to be confused with Kafka Producer idempotence). Kakfa also presented a challenge to be able to support consistent query\/read all events for a particular SKU+Location without an explosion of\u00a0Topics.EventStore<\/strong>\u2014 Seemed to have challenges with scaling of high throughput writes, though reads looked more promising.Cassandra<\/strong>\u200a\u2014\u200aCassandra had the ability to enforce event uniqueness with LWT (Light Weight Transactions) albeit with higher<\/strong>\u00a0latency.<\/p>\n Out solution ended up using an in-house data service called Zero Object Service<\/em> (ZOS), which is an elastically scalable microservice designed to handle data storage and retrieval for multi-tenant, meta-data-driven, high-volume, and low-latency use cases. It does so by providing a set of template patterns like Event Sourcing and Reservation Sets that are built on top of key-value datastores that support per-row atomic writes with strong consistency. The actual implementation of the ZOS Event Sourcing template is beyond the scope of this blog post, but, with ZOS, the Inventory service was able to achieve its desired features for event sourcing and meet the desired scale. The solution was deployed on public cloud across multiple global\u00a0regions.<\/p>\n Our implementation and adoption of the pattern left us with some learnings that we\u2019d like to\u00a0share:<\/p>\n One of the foremost things that stands out is our Product and Engineering teams embraced the concept of eventual consistency and that it is acceptable that inventory numbers have different degrees of staleness at different points of the architecture. Inventory on a web page might show available, but, when you try to buy it, it could be sold out. Product Searches and other indices do not need to reflect 100% consistent inventory results and neither do Product detail or Product listing pages. But, when attempting to Reserve\/Check-out, it is extremely important to be accurate to not promise something that could potentially not be available. For that reason, the writes<\/em> for Inventory reservations (as we refer to them) were strongly consistent operations.<\/p>\n Event Sourcing as a pattern can be a bit difficult to wrap one\u2019s head around. It often is easily confused with a simple event stream\/topic, where the messages themselves are no longer relevant once the stream is processed. Our team had to go through a learning curve around Event Sourcing, how it differs from CQRS<\/a> and the benefits it brings over straight up manipulation of data and CRUD. Embracing the need to maintain two data points, namely an event log and the entity\/aggregate\/view, instead of interacting with just the entity\/aggregate\/view took some time to digest. The fact that the Event Sourcing pattern promotes the separation of Write\/Read paths, thereby enabling performance optimizations on either end, was a big benefit that we observed in our performance tests. The ability to build aggregates from the stream while coalescing multiple event records were performance optimizations that gave us the benefit of cost saving as well as reducing the number of I\/O operations. Avoiding lock contention by multiple threads by serializing the updates to entities over single threads for mutation seemed to gain us good performance during High Contention<\/strong> scenarios like flash\u00a0sales.<\/p>\n Another benefit of Event Sourcing is subscriptions<\/strong> and providing our consumers the ability to subscribe for changes, which meant that they could cache the availability state closer to their application and reach back to the inventory solution for deltas (changes) while always using the inventory solution for performing strongly consistent reservations. This effectively led to improved performance for consumers as their data was closer to them for the READ path, as well as reducing the cost to serve, as the Inventory Solution did not have to be accessed for every GET availability operation.<\/p>\n The benefit of ensuring we have a historical log of changes with the Event Sourcing system without the need for a supplemental audit log helped us many times during forensics to replay and re-construct states. The audit is not something that is an accessory to the primary model but is instead what drives the\u00a0model.<\/p>\n Below are a couple of spots where we found the event log to be very helpful for supporting live customers<\/p>\n The event log provides footprints to trace back the path to arrive at the current inventory state for a SKU. By providing the Event log to the consumers as a self service tool, it helped them trouble shoot the actions leading to the current\u00a0state.The release version of the service is stamped on every event which is helpful for finding the affected events in case of bug introduced in the release, and we are able to isolate the affected events and replay those events to get it to the correct\u00a0state.<\/p>\n The performance of the Architecture of the Inventory Availability Service\u2192 Zero Object Service\u2192 Key\/Value Store <\/em>provided the desired results we needed from the solution for throughput and responsiveness of inventory reservations and availability retrieval.<\/p>\n Since launching in February 2021, the Inventory solution has garnered the interest and adoption of many brands globally. The solution offers a headless API as part of the overall suite of Commerce API offerings. (Read more about that at the <\/em>Commerce Cloud Developer Center<\/em><\/a>.)<\/em> It integrates with Salesforce B2C Commerce, which many merchants have loved using over the years, and also with Salesforce Order Management.<\/p>\n We would like to thank and acknowledge Robert Libby and Benjamin Busjeager for their contributions toward this solution and also for helping review this blog. We also would like to acknowledge and appreciate all the teamwork and effort from members of Product and Technology at Salesforce who were involved in the making of this offering.<\/p>\n Interested in working on problems like these? Join our Talent Network<\/a> or reach out to Sanjay on his blog<\/a>, Twitter<\/a>, or LinkedIn<\/a> and Balachandar on his LinkedIn<\/a>.<\/p>\n Event Sourcing for an Inventory Availability Solution<\/a> was originally published in Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"The Opportunity<\/h3>\n
Requirements<\/h3>\n
Functional Requirements<\/h4>\n
\u200a\u2014\u200aNo duplicate<\/strong> reservations should be allowed
\u200a\u2014\u200aSolution must prevent overselling<\/strong>, i.e., prevent an inventory reservation from happening if there is no quantity available.Consistency<\/strong>: Availability reads can display relatively stale inventory levels and do not need be immediately consistent to what is written.
\u200a\u2014\u200aNo need for inventory updates to be reflected in a call to obtain availability immediately after an inventory update. Acceptable to have up to 60<\/strong> seconds of staleness between the time the write is reflected in the\u00a0read.<\/p>\nNon-Functional Requirements<\/h4>\n
Direction for Inventory Solution<\/h3>\n
Event Sourcing<\/h4>\n
Using Subscriptions to create the Group Inventory View<\/h4>\n
High Contention on Select SKUs during Reservations<\/h4>\n
“event_type” : “LocationReservation<\/strong>“,
“sku” : “acme-hat-blue”,
“location_id” : “warehouse”,
“event_sequence_number” : 1122,
“quantity”: 5<\/strong>
}<\/p>\n
“name” : “LocationInventoryEvent”,
“event_type” : “Import<\/strong>“,
“sku” : “acme-hat-blue”,
“location_id” : “warehouse”,
“event_sequence_number” : 1122,
“on_hand” : 423<\/strong>,
…
}<\/p>\n
EventLog eventLog<\/strong>,
LocationInventoryAggregateTable aggregateTable<\/strong>]
1. current_sequence_number
= eventLog<\/strong>.query(reservationEvent<\/strong>.location_id, reservationEvent<\/strong>.sku) <\/p>\n
return error<\/strong> \/\/ Idempotence check 3. aggregate =
aggregateTable<\/strong>.readAggregate(reservationEvent<\/strong>.location_id,
reservationEvent<\/strong>.sku)\/\/ Get all Events not applied to aggregate including event with the current_sequence_number
4. pending_event_list =
eventLog<\/strong>.getPendingEvents(aggregate.sequence_number<\/strong>,
current_sequence_number<\/strong>)\/\/ Re-Constitute the Aggregate State
5. updatedAggregateState = foldEventsIntoAggregate<\/strong>(aggregate<\/strong>, pending_event_list<\/strong>)6. if reservationEvent<\/strong>.quantity > updatedAggregateState<\/strong>.ATF
return error<\/strong> \/\/ Insufficient quantity constraint violation <\/p>\n
= current_sequence_number + 1<\/strong>\/\/ Constraint on sequence_number<\/strong> [location_id, sku, sequence_number<\/em>]
8. result = insert reservationEvent<\/strong> into eventLog<\/strong> 9. if result is false<\/strong>
restart from Step #1<\/strong> until a timeout or finite retries<\/p>\nStaleness of Read\u00a0Data<\/h4>\n
Other Options Considered for the\u00a0Solution<\/h3>\n
Resulting Solution<\/h3>\n
Learnings from our Implementation<\/h3>\n
Consistency and Staleness<\/h4>\n
Event Sourcing\u00a0Pattern<\/h4>\n
Audit and Forensics<\/h4>\n
Performance<\/h4>\n
Conclusion<\/h3>\n
Credits and Acknowledgements<\/h3>\n