{"id":228,"date":"2021-02-02T20:02:39","date_gmt":"2021-02-02T20:02:39","guid":{"rendered":"https:\/\/fde.cat\/?p=228"},"modified":"2021-02-02T20:02:39","modified_gmt":"2021-02-02T20:02:39","slug":"realtime-predictions-in-a-multitenant-environment","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/realtime-predictions-in-a-multitenant-environment\/","title":{"rendered":"Realtime Predictions in a Multitenant Environment"},"content":{"rendered":"

Real-time Predictions in a Multitenant Environment<\/h3>\n

Introduction<\/h3>\n

The Einstein Vision and Language Platform Team at Salesforce enables data management<\/a>, training<\/a>, and prediction for deep learning-based Vision and Language<\/a> use cases. Consumers of the platform can use our API gateway<\/a> to upload datasets, train those datasets, and ultimately use the models generated out of training to run predictions against a given input payload (for example, input text during a live chat). The platform is multitenant and supports all the above operations for different customers simultaneously.<\/p>\n

\"\"
Prediction Service in Action\u200a\u2014\u200alive chat text input is sent to Prediction Service to understand customer intention.<\/figcaption><\/figure>\n

Prediction Service is one of the Einstein Vision and Language Platform services that runs behind the scenes. As the name suggests, its job is to serve predictions. The nature of the predictions is real-time. For example, a customer may train a custom model to learn different intents from a set of chat messages. Then, during a live chat session, the utterances are sent to the Prediction Service where the service uses the customer-trained model to identify and return the intent of those utterances. Based on the intent returned by the model, the chatbot takes the appropriate decision (for example, checking the status of an order). Failing to return prediction result within the Service-Level Agreement (SLA) window may result in poor end-user experience.<\/p>\n

\"\"
Figure 1.0: Einstein Vision and Language Platform Prediction Functional Overview<\/em><\/figcaption><\/figure>\n

In addition to being real-time, predictions are multitenant: multiple customers train one or more models and use them to get predictions. Together, real-time & multitenancy aspects of predictions make it an interesting challenge to solve. The real-time aspect of predictions implies that the time taken to serve a prediction request should be low\u200a\u2014\u200afor us it depends on use-case; typically the SLA\u2019s for latencies range from 150 ms to 500 ms. And the multitenancy aspect means managing models for thousands of tenants in a way that model access from memory to disk, and file store (AWS S3) to disk is minimized.<\/p>\n

\"\"
Figure 1.1: Multiple Tenants Run Predictions Simultaneously<\/em><\/figcaption><\/figure>\n

Typical Prediction Request\/Response<\/h3>\n

Before we dive deeper, let\u2019s take a look at a typical prediction request. This is what our service looks like to the consumer.<\/p>\n

Request<\/h4>\n

The following example shows the request parameters for a language model (intent classification<\/a>) prediction request. The modelId is the unique ID that\u2019s returned after you train a dataset and create a model. The document parameter contains the input payload for prediction.<\/p>\n

{
  \"modelId\": \"2ELVAO5BNVGZLBHMYTV7CGD5AY\",
  \"document\": \"my password stopped working\"
}<\/pre>\n
Response<\/h4>\n
After you send the request via an API call, you get a response that looks like the following JSON. The response consists of the predicted label\u2014alternatively called category\u2014for the input payload. Along with label, the model returns a probability for each label, which indicates the confidence of the prediction.<\/p>\n
In the response below, the model is 99.04% confident that the input, my password stopped working, belongs to the label Password Help. Similarly, the rest of the probability scores indicate that the input doesn\u2019t belong to other\u00a0labels.<\/p>\n
{
  \"probabilities\": [
    {
      \"label\": \"Password Help\",
      \"probability\": 0.99040705
    },
    {
      \"label\": \"Order Change\",
      \"probability\": 0.003532466
    },
    {
      \"label\": \"Shipping Info\",
      \"probability\": 0.003473858
    },
    {
      \"label\": \"Billing\",
      \"probability\": 0.0024010758
    },
    {
      \"label\": \"Sales Opportunity\",
      \"probability\": 0.00018560764
    }
  ],
  \"object\": \"predictresponse\"
}<\/pre>\n
Sending a prediction request is just a simple cURL\u00a0call.<\/p>\n
curl -X POST 
-H \"Authorization: Bearer <TOKEN>\" 
-H \"Cache-Control: no-cache\" 
-H \"Content-Type: application\/json\" 
-d \"{\"modelId\":\"2ELVAO5BNVGZLBHMYTV7CGD5AY\",\"document\":\"my password stopped working\"}\" 
https:\/\/api.einstein.ai\/v2\/language\/intent<\/a><\/pre>\n
You can quickly sign up for free<\/a> and build your own model using your own data or one of the sample datasets in our API reference<\/a>. Or try one of our pre-built<\/a> models. Feel free to drop a note in comments and let us know about your experience using our\u00a0API.<\/p>\n
Lifecycle of a Prediction Request<\/h3>\n
To understand how predictions are handled in a multitenant environment, let\u2019s look at the lifecycle of a single prediction request.<\/p>\n
\"\"
Figure 2.0: Lifecycle of a Prediction Request<\/em><\/figcaption><\/figure>\n
After you train a dataset, the model artifact is stored on an AWS S3 file store. The model metadata (for example, model ID, S3 location) is stored in the database via the Data Management Service (DM)<\/a>. To get the location of the model, Prediction Service queries DM service using the customer\u2019s tenant ID and model ID. DM returns a pre-signed S3 URL with a temporary access\u00a0token.<\/p>\n
Prediction Service then downloads the model tar from AWS S3 and stores it on the Elastic File System (EFS) disk (if the model does not exist on the disk already). Then the model tar is extracted and the model is loaded into memory. Finally, the pre-processed input is run through the model (neural network) and the output is returned\u00a0back.<\/p>\n
An example of pre-processing is tokenization & embedding lookup. Typically, the prediction input (eg. live chat) for language models (eg. intent classification or Named Entity Recognition) is first tokenized where the input is split into words. Further, the words are then used to do a lookup to get their vector representations. This lookup table of words & their vector representations is called an embedding\u200a\u2014\u200aglove<\/a> is one type of embedding. This is what vector representation of word \u201csalesforce\u201d looks like\u200a\u2014\u200aonly first five dimensions shown:<\/p>\n
salesforce 0.23245 -0.31159 0.28237 0.10496 0.3087\u00a0….<\/p>\n
The vector representation is then passed to the model for prediction. The model returns a set of labels and their probability scores as shown in example response\u00a0above.<\/p>\n
Note:<\/strong> Embeddings are intentionally kept separate from model artifact to reduce memory & disk footprint, which factors into overall performance.<\/p>\n
Prediction Service\u00a0Overview<\/h3>\n
The Einstein Vision and Language Platform runs on a Kubernetes cluster. All services are deployed using Kubernetes Deployment to launch multiple replicas of each service as Pods which are exposed via cluster-scoped internal Kubernetes Service endpoints.<\/p>\n
The diagram below shows language and vision predictors running on the Kubernetes cluster. Predictor is a group of pods responsible for carrying out prediction\/inference and they are fronted by a Kubernetes service. The API Gateway acts as a proxy to the Prediction Service and routes requests across different predictors.<\/p>\n
\"\"
Figure 3.0: Prediction Service\u00a0Overview<\/em><\/figcaption><\/figure>\n
Predictor Pod<\/h4>\n
Let\u2019s dive deeper into the Predictor. The Predictor is collection of pods fronted by a Kubernetes service that acts as a load balancer. Each pod in the Predictor consists of two containers: model management sidecar<\/strong> and inference<\/strong>.<\/p>\n
\"\"
Figure 3.1: Predictor Pod<\/em><\/figcaption><\/figure>\n
sidecar container<\/strong><\/p>\n