![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1000\/1*LViwMW_VeUi__lC5LFMT8g.png?w=750&ssl=1\")
<\/figure>\n\u201cIf it could be demonstrated that any complex organ existed, which could not possibly have been formed by numerous, successive, slight modifications, my theory would absolutely break down. But I can find no such case.\u201d \u2015 Charles\u00a0Darwin<\/em><\/p><\/blockquote>\n
What do you mean constrained by the value of the solution? Shouldn\u2019t you always use the most robust solution available? Sometimes, but often, no. I like the example from Bruce Schneier<\/a>: The most secure way you can encrypt a document is a truly random one-time pad\u2026 but you\u2019d be rather silly to use that to secure your 13yo daughter\u2019s diary. So when you decide on how to solve a problem, you should of course think about how well it\u2019ll solve it and how robust it\u2019ll be in the future, but you also need to question if it\u2019s necessary and if there even will be a future? Much of the work we do on my team is experimental, and a lot will simply be abandoned once we prove it\u2019s not useful or effective. So each time you see one of these solutions and think \u201cWell that\u2019s suboptimal, you should have XYZ,\u201d stop and ask, \u201cWould XYZ have been a good use of resources at the\u00a0time?\u201d<\/p>\n
\u201cThe difference between a programmer and an engineer is the consideration of development costs, both in the present and\u00a0future!\u201d<\/em><\/p><\/blockquote>\n
Anyways, enough caveats, onto the evolution! I\u2019ve included a scale of how much pain the problem causes versus how much pain implementing the solution would be using \ud83d\udd25.<\/p>\n
Problem<\/strong>: We need to run a single machine learning (ML) algorithm on data for all of our clients. The data is big and the ML is complicated, being difficult to run efficiently. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25
Solution<\/strong>: We created EC2 instance clusters for each client (around half a dozen at this point) and a script that starts the clusters, starts to run the ML, uploads the results, and stops the instances. Each client had a different number of instances in their cluster based on the client data size. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25<\/p>\nProblem<\/strong>: This is expensive! At the time, AWS charged per hour rounded up. So even if the ML took only 5 minutes, we\u2019d pay for an entire hour on machines! Also, the size of available machines had grown, so we no longer needed clusters to run a single client, they could be ran on a single powerful instance. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25
Solution<\/strong>: Create a system that is able to run multiple clients sequentially on a single box, cleaning up after each. This system feeds off a queue of which clients need to run and autoscales the number of boxes based on the amount in the queue. It writes data to a central DB about each instance and job so we can learn how to binpack better in the future and save money. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25<\/p>\nProblem<\/strong>: Using machine learning to improve things is awesome, let\u2019s create more models\/algorithms! Wait.. the current system was created to only run one type. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25
Solution<\/strong>: Abstract the class that runs ML and use subclasses for each new model. The class needs to know how to prepare the environment, run the job, and cleanup when all is done (either successfully or in error). Pain: \ud83d\udd25<\/p>\nProblem<\/strong>: Creating a new subclass each time the data science team comes up with something new is a bit of a pain. It\u2019s not a good use of resources to ask the data scientists to create the subclasses since they\u2019d need to learn to build\/test\/follow standards of the project. Pain: \ud83d\udd25\ud83d\udd25
Solution<\/strong>: Remove the subclasses and create a runtime configured class that reads from YAML files. Pain: \ud83d\udd25\ud83d\udd25<\/p>\nProblem<\/strong>: As more and more algorithms are added and each needing slightly different configuration and the system itself needs more configuration for each job, the YAML files are becoming complicated and prone to human error. Pain: \ud83d\udd25\ud83d\udd25
Solution<\/strong>: Create a UI to configure jobs! This also gives a chance to create a reporting system about the outcomes of each job. So a double win! Pain: \ud83d\udd25\ud83d\udd25<\/p>\nProblem<\/strong>: The ML jobs are getting more divergent. They require different versions of libraries to run, and no one has time to update the old jobs that are going just fine. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25
Solution<\/strong>: Instead of running the jobs directly on the box, we can encapsulate the environment into Docker containers. We provide a skeleton of a few different types to the data science team and let them customize to their hearts\u2019 content. This allows different dependencies, and makes clean up after each job far easier! Pain: \ud83d\udd25\ud83d\udd25<\/p>\nProblem<\/strong>: While all the ML jobs can run on the beefy machines originally set up, it\u2019s a bit wasteful. Some require GPU, some only a good CPU, some high memory but very little CPU, and some require very little at all to run. Keeping everything on the large boxes is wasteful. Pain: \ud83d\udd25
Solution<\/strong>: Create multiple queues for different types of boxes available and separate Auto Scaling Groups for each queue. Pain: \ud83d\udd25<\/p>\nProblem<\/strong>: As we grow in the number of clients and ML jobs, the queues are getting backed up a bit. Some jobs are dependent on others and some jobs needed to run more often than others. Unfortunately, they all run in a more or less FIFO nature. Pain: \ud83d\udd25
Considered Solution<\/strong>: Create a priority queue and have jobs assigned an importance score. This was not chosen since we are currently using the basic AWS queueing service and writing our own was not worthwhile. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25
Solution<\/strong>: Move from a queue\/Auto Scaling Group per resource to one per job type. Pain: \ud83d\udd25<\/p>\nProblem<\/strong>: Creating new queue\/Auto Scaling Groups each time a new ML algorithm comes up is a bit of a hassle. Pain: \ud83d\udd25
Partial Solution<\/strong>: Use a terraform module to reduce the amount of work to simply creating a few lines of code and running a command. Pain: \ud83d\udd25<\/p>\nProblem<\/strong>: We went from a half dozen clients with a single ML job to thousands of clients with dozens of ML jobs. The single DB is getting hammered and that\u2019s causing job failures. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25
Temporary Solution<\/strong>: Remove stats being written about each instance; they aren\u2019t that useful since AWS no longer charges for whole hours. Add read replicas, this is a fairly common solution to reduce DB load. This reduces the load, but still doesn\u2019t improve scaling. Pain:\u00a0\ud83d\udd25<\/p>\nProblem<\/strong>: More DB issues. We now have hundreds of ML boxes running at a time during peak. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25
Considered Solutions<\/strong>: Move from standard SQL DB to cassandra. This was deemed to be more work than useful since it\u2019d require rewriting the entire DB layer of the job runner and the reports. Pain: \ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25\ud83d\udd25
Solution<\/strong>: None of the writes are needed immediately in the DB. They are primarily just there for reports. So, let\u2019s create a queue of DB writes, we can then reuse the whole system\u2019s \u201cread from queue and perform job\u201d to perform the DB writes. This requires a relatively small amount of code to be written, and reuses the throttling system already in place. Pain: \ud83d\udd25\ud83d\udd25<\/p>\nPossible Future Problem<\/strong>: Creating the new queues\/Auto Scaling Groups still requiring engineering time, which isn\u2019t ideal. Pain: \ud83d\udd25
Possible Future Solution<\/strong>: Moving this creating into the UI. Of course, having a UI that creates infrastructure is generally a dangerous thing. So perhaps we\u2019ll revisit making a new priority queue service. Pain: \ud83d\udd25\ud83d\udd25<\/p>\nPossible Future Problem<\/strong>: This bespoke binpacking isn\u2019t really the greatest and as the number of clients and jobs grow, the small difference will start to add up to real costs. Pain: \ud83d\udd25\ud83d\udd25
Possible Future Solution<\/strong>: Move this system into Kubernetes, which we already have in place for other services and which already knows how to efficiently pack things together. Pain:\u00a0\ud83d\udd25\ud83d\udd25\ud83d\udd25<\/p>\n\u201cJust as smart financial debt can help you reach major life goals faster, not all technical debt is bad, and managing it well can yield tremendous benefits for your company.\u201d\u200a\u2014\u200a<\/em>hackernoon.com<\/em><\/a><\/p><\/blockquote>\n
As mentioned above, this article is in an experimental format. One thing that became clear while<\/em> I was writing is the pain scale. When we faced a challenge and brainstormed solutions, we\u2019d analyze how much trouble implementation would be, but we never boiled it down to a simple scale like in this article. Looking back, it is obvious, to paraphrase Dr. Henry Cloud, that changes happen when the pain of the system remaining the same exceeds or equals the pain of changing. Each decision made is an attempt to minimize pain\/cost or, more positively, maximize expenditure. We make decisions with trade-offs and knowledge that nothing is final; the system shall continue to\u00a0evolve.<\/p>\n
<\/p>\n
\nThe Origin of MLMon<\/a> was originally published in Salesforce Engineering<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n
Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"