{"id":225,"date":"2021-02-02T20:01:16","date_gmt":"2021-02-02T20:01:16","guid":{"rendered":"https:\/\/fde.cat\/?p=225"},"modified":"2021-02-02T20:01:17","modified_gmt":"2021-02-02T20:01:17","slug":"flow-scheduling-for-the-einstein-ml-platform","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/flow-scheduling-for-the-einstein-ml-platform\/","title":{"rendered":"Flow Scheduling for the Einstein ML Platform"},"content":{"rendered":"

At Salesforce, we have thousands of customers using a variety of products. Some of our products are enhanced with machine learning (ML) capabilities. With just a few clicks, customers can get insights about their data. Behind the scenes, it\u2019s the Einstein Platform that builds hundreds of thousands of models, unique for each customer and product, and serves those predictions back to the customer.<\/p>\n

\"\"<\/figure>\n

Each customer comes with their unique data structure and distribution. The sheer scale of the number of customers and their unique data configurations sets new requirements for our processes. For example, it\u2019s simply not practical to have a data scientist look at every model that gets trained and verify the model quality. The established process in the ML industry is to first start by manually \u201cconstructing\u201d a model, tweaking the parameters by hand, and then training and deploying the model. In contrast, we needed a process to automate the \u201cconstruction\u201d phase in addition to model training, validation, and deployment. We wanted to automate the end-to-end process of creating a model. To solve this particular problem, we use our own (Auto)ML library on top of the open source TransmogrifAI<\/a> library.<\/p>\n

While many ML solutions deal with processing a handful of very large datasets (a big data problem), our main use case is a small data problem. Namely, to process many thousands of small-to medium sized datasets and generate models for each one of them. <\/p>\n

The various steps involved in producing a model need to be codified in a flow. The flow automates the entire process: pulling data from our sources, performing data cleaning and preparation, running data transformations, and finally training and validating the model. There are already well-established orchestration systems to solve authoring ML and big data flows. However, they\u2019re designed with the assumption that there are only a few (up to a few hundred) flow instances that need to run at any point in time. That\u2019s not the case for Salesforce. We have hundreds of thousands of flow instances\u200a\u2014\u200aseveral for each of our customers\u200a\u2014\u200athat must run reliably and in isolation. <\/p>\n

So we built, from the ground up, a workflow orchestration system designed to deal with our scale requirements. A key requirement of a scalable orchestration solution is to have a scalable scheduler\u200a\u2014\u200anamely, the ability to trigger flow executions reliably at a particular configured section in time. In fact, the extensive tests we performed on other existing orchestration solutions revealed key problems on their scheduling approach that wouldn\u2019t work for our use cases. For example, we ran into difficulties running more than 1,000 flows concurrently. <\/p>\n

The most straightforward solution is a cron, a reliable method for scheduling processes in Unix-like systems. However, this solution isn\u2019t scalable, and if the node fails, all of our schedules are gone. Similar issues occur when using single-process schedulers. <\/p>\n

What about a distributed cron? This solution is fault-tolerant and scalable, and node failures won\u2019t cause the schedules to be lost. This is the scheduling solution that other enterprise companies use. However, this approach puts a significant load on our system. Consider this graph of request counts for 100 flows that need to be executed every\u00a0minute:<\/p>\n

\"\"<\/figure>\n

This means that, at scale, our server will be hit with a lot of requests on the minute, but will be idle the rest of the time. We want to smooth out the execution of flows to something like\u00a0this:<\/p>\n

\"\"<\/figure>\n

This way we get a smaller but more continuous load on our servers for the same performance!<\/p>\n

So back to the drawing board we\u00a0go.<\/p>\n

A Timely\u00a0Solution<\/h3>\n

This graphic shows our scheduler solution.<\/p>\n

\"\"<\/figure>\n

Many of these components are common to most services. We have API nodes that respond to HTTP requests. We have a database cluster that handles persistence and keeps track of schedules. The key components of this scheduler are the two lambdas connected with a\u00a0queue.<\/p>\n

What\u2019s in a Schedule?<\/h3>\n

Before we can begin building a schedule architecture, we need to first define what a \u201cschedule\u201d is in more detail. In our system, a schedule is an action (HTTP call) that must performed on a period within a margin of\u00a0error.<\/strong><\/p>\n

Let\u2019s break it down one point at a\u00a0time:<\/p>\n