![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*CnF2LsaP2afwG0eMowXHlg.png?w=750&ssl=1\")
<\/figure>\nLet\u2019s take a deep dive and validate our hypothesis.<\/p>\n
We decided to start with a univariate approach on a single metric which represents the average page load time for our customers.<\/p>\n
How many models do we need? Answering this question isn\u2019t simple.<\/p>\n
1 time-series metric =<\/strong> 1 model<\/p>\n Too easy? Unfortunately the real world is a difficult one.<\/p>\n The inbound traffic pattern and seasonality is completely different for various regions (Asia Pacific\/North America\/Europe etc). We have to factor in this variable to generate good predictions.<\/p>\n 10 Regions x<\/strong> 1 time-series metric =<\/strong> 10 models<\/p>\n Are we sure that each datacenter serving a region has the same type of hardware specification?<\/p>\n Not at all. Based on the procurement period, we keep updating the hardware specification. So, it\u2019s good to have host level models.<\/p>\n 10 Regions x<\/strong> ~10k hosts x<\/strong> 1 time-series metric =<\/strong> 100k models<\/p>\n Are we done yet? Here comes the interesting part.<\/p>\n Let\u2019s say we collect the metric in a minute interval and use that for training the model. Using the lowest granularity available is very important to find minor service degradations, but the downside of this approach is noisy alerts. Alerts caused by minor glitches are normally not actionable and do not cause any customer impact. One way to generate quality alerts from time-series data is to create an ensemble of models trained on data aggregated at different time intervals.<\/p>\n We generated models for the metric with data aggregated over 1 minute, 15 minutes, and 1 hour. The higher the interval, the higher the weightage we give to the models. A spike on data averaged at a minute level may not be an incident, but the same spike on data averaged at an hourly level can cause a serious incident.<\/p>\n Let\u2019s count the number of models again:<\/p>\n 10 Regions x 10k hosts x 1 time-series metric x Ensemble of 3 models = 300k models<\/p>\n Now you can see that, starting from 1 model, we are about to produce 300k models. But its not 1 million yet, right? This is rough math for a univariate approach. Designing a proactive incident detection system uses a lot of system and application metrics. Once we add multiple metrics to the system, the final number would be the number of metrics times 300k.<\/p>\n Now that we have time-series models with very high accuracy, the next challenge is model serving. It is of no use if we can\u2019t serve these models fast for real-time data. We have a couple of options here: use one of the open source platforms, or build one ourselves. As a first step, we evaluated MLflow<\/a> and Kubeflow<\/a>. (There are many other open source products, but we decided to evaluate the most popular ones first.) Both of these platforms offer solid model serving options with a lot of features. But these projects were in very early stages with a lot of changes going in when we started this project in early 2019, so we decided to park these options and started to work on building an easy way to serve models. Simple and fast is what we needed, not a lot of features.<\/p>\n The first thing was to come up with an abstraction of prediction for models generated by various libraries.<\/p>\n We implemented predict<\/em> for all distinct kinds of models that we use. Models are generated with various open source libraries and custom algorithms developed internally.<\/p>\n The last step is to invoke the predict method from a simple web app. We used Flask<\/a>. You can see the sample code below.<\/p>\n This simple code snippet can serve models. But we need to optimise the response latency to enable smooth predictions for real-time data. Let\u2019s see how we do that.<\/p>\n Compressing the models and adding a caching layer was important. Fetching models from storage was slow, as some of the models are large in size. Compressing models helped to reduce the storage size and network throughput more than 6x. We were able to serve ~1 million+ models generated out of 10\u201315 time-series metrics smoothly with this simple system.<\/p>\n This blog content is based on our experience and learnings from building an anomaly detection system. There are multiple ways of achieving this. You can factor in each variable as features and train a single model to serve this purpose, but the quality of alerts produced by this approach worked much better for us.<\/em><\/strong><\/p>\n Journey to a million models<\/a> was originally published in Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n","protected":false},"excerpt":{"rendered":"![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1024\/1*tICDjrV_S1Ns0t27a9ZQFA.png?w=750&ssl=1\")
\nServing models (fast)<\/h3>\n
class Model(abc.ABC):\u2029\u2029 def predict(self, model_id, data_points):\u2029 \"\"\"\u2029 Predicts anomaly scores for time series data \u2029 \"\"\"<\/pre>\n
import pandas as pd\u2029\u2029@app.route('\/predict', methods=['POST'])\u2029def predict():\u2029 \"\"\"\u2029 Predicts anomaly score for time-series data\u2029 \"\"\"\u2029 request_df = pd.DataFrame(requests.json[\"datapoints\"])\u2029 model = Model(requests.json.get(\"model_id\", default_model))\u2029 prediction_df = model.predict(request_df)\u2029 return jsonify({'anomaly_scores: prediction_df.to_dict('list')})<\/pre>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/517\/1*bIbHAeoVR5p1CIH2_apdSQ.png?w=750&ssl=1\")
<\/figure>\n<\/p>\n
\n