{"id":733,"date":"2023-07-11T23:35:37","date_gmt":"2023-07-11T23:35:37","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/07\/11\/how-is-salesforce-einstein-optimizing-ai-classification-model-accuracy\/"},"modified":"2023-07-11T23:35:37","modified_gmt":"2023-07-11T23:35:37","slug":"how-is-salesforce-einstein-optimizing-ai-classification-model-accuracy","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/07\/11\/how-is-salesforce-einstein-optimizing-ai-classification-model-accuracy\/","title":{"rendered":"How is Salesforce Einstein Optimizing AI Classification Model Accuracy?"},"content":{"rendered":"

In our \u201cEngineering Energizers\u201d Q&A series, we examine the professional journeys that have shaped Salesforce Engineering leaders. Meet Matan Rabi, Senior Software Engineer on Salesforce Einstein\u2019s<\/a> Machine Learning Observability Platform (MLOP) team. Matan and his team strive to optimize the accuracy of Einstein\u2019s AI classification models, empowering customers across industries to enhance efficiency and provide world-class customer support.<\/p>\n

Read on to learn how Matan and his team tackle engineering challenges with the latest technologies to advance the state-of-the-art in AI.<\/p>\n

What is your team\u2019s mission?<\/strong><\/p>\n

Composed primarily of backend engineers, my team primarily works behind the scenes, harnessing observability features to measure and increase the accuracy of custom AI classification models \u2014 significantly improving the end-user experience.<\/p>\n

To achieve this goal, we thoroughly analyze the predictions generated by Einstein AI classification models for our external customers and compare that with the ground truth of what happened in the real world. This apples-to-apples comparison enables us to identify areas for improvement and how we can take modeling accuracy to the next level for our customers.<\/p>\n

MLOP uses mathematical formulas based on a confusion matrix to determine a model\u2019s overall performance.<\/em><\/p>\n

MLOP also validates models by subjecting them to tests using historical data provided by our customers. These tests instill high confidence in the models\u2019 value even before deployment.<\/p>\n

In addition to accuracy, we analyze the expenses for developing and maintaining the models. This is achieved by tagging different models\u2019 usage data to determine the various cost to serve for each tenant within an application, which helps to reduce the overall cost of the models. These savings are ultimately passed on to our external customers.<\/p>\n

Matan explains why he\u2019s proud to be an engineer at Salesforce.<\/em><\/p>\n

What are Einstein AI classification models?<\/strong><\/p>\n

Einstein AI classification models ingest and analyze customer data to formulate predictions of a particular customer outcome from a known subset of outcomes, enabling Salesforce customers with actionable intelligence to better serve their customers. The models play a vital role across numerous industries.<\/p>\n

For example, when a customer connects with their cable service provider to discuss a service issue, Einstein\u2019s AI model case classification feature monitors the discussion between the customer and the customer service representative (CSR) and rapidly determines the severity of the problem. It then ranks the customer\u2019s needs as high, medium, or low, enabling the CSR to effectively provide a recommended solution.<\/p>\n

Likewise, in the medical field, when a patient contacts their doctor\u2019s office to schedule an appointment, the receptionist may ask some questions to understand the patient\u2019s symptoms in order to determine the urgency of care. Einstein\u2019s case classification feature monitors the conversation to predict the priority of the patient\u2019s case, informing the receptionist that a patient with a potentially serious health condition should be treated as soon as possible.<\/p>\n

At a high level, how does MLOP measure AI classification model accuracy?<\/strong><\/p>\n

First, my team gathers data from the customer and contrasts it against the data on which the models were trained. We then segment the data to examine different model elements. Back to my CSR use case example, this could include analyzing the person\u2019s location, age, and other variables.<\/p>\n

MLOP calculates a model\u2019s performance for each relevant segment group to detect specific issues.<\/em><\/p>\n

For example, the model may have been trained on people from New York and proved successful there but it was also used by CSRs in Oklahoma, where people\u2019s values and attitudes may be dissimilar. In that instance, the model did not have the same level of success as it did in New York. Thus, we would recommend the model be tuned for Oklahoma CSRs to improve its inclusivity.<\/p>\n

What\u2019s a cutting-edge tool that your team uses to analyze AI models?<\/strong><\/p>\n

Reviewing AI modeling often requires MLOP to process terabytes of data to arrive at conclusions, which could take days or weeks to complete without the right resources.<\/p>\n

To overcome this challenge, our team leverages Apache Spark\u2019s distributed computation feature, empowering us to perform computations in parallel \u2014 examining huge amounts of data at once. Paralleling on one machine is fast \u2014 especially when the machine has multiple cores. Seeking to go even faster, our team learned how to spread the computational load across many machines.<\/p>\n

What\u2019s the endgame? We created an \u201cinfinite\u201d parallelism factor by harnessing a vast network of commodity hardware. This enables the team to analyze tens of millions of data points in the blink of an eye to better predict and understand model accuracy.<\/p>\n

How does your team deal with risks when analyzing AI models?<\/strong><\/p>\n

To measure and visualize a model\u2019s performance, we must first obtain approval from internal legal and security departments. This lengthy process remains a risk for us because it could stall our team\u2019s development time.<\/p>\n

During the approval process, on a case-by-case basis, we must articulate our plan for handling customer data. If approved, we must encrypt the model\u2019s customer data to keep their personal data private from our team. Ultimately, this process upholds Salesforce\u2019s number one value: Trust.<\/p>\n

Placeholder for video:
https:\/\/salesforce.vidyard.com\/watch\/uiXvVi3sqoqbqphxHRG4Jb<\/a>
Caption: Matan shares a story on Salesforce hackathons, one of his favorite engineering experiences.<\/em><\/p>\n

What would someone find surprising about your team?<\/strong><\/p>\n

MLOP\u2019s primary focus is delivering the best product and remaining relevant. To accomplish that, we constantly experiment with new tools to increase our agility, effectiveness, and efficiency \u2014 not just on our current project, but also the next one. In fact, we are always thinking one or two years down the line. Consequently, our team always pushes themselves to improve our development processes. We are never complacent with the status quo.<\/p>\n

Learn more<\/strong><\/h4>\n

Hungry for more AI stories? Check out this blog<\/a> to explore the latest AI projects at Salesforce.<\/p>\n

Stay connected \u2014 join our Talent Community<\/a>!<\/p>\n

Check out our <\/a>Technology and Product teams<\/a> to learn how you can get involved.<\/p>\n

The post How is Salesforce Einstein Optimizing AI Classification Model Accuracy?<\/a> appeared first on Salesforce Engineering Blog<\/a>.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

In our \u201cEngineering Energizers\u201d Q&A series, we examine the professional journeys that have shaped Salesforce Engineering leaders. Meet Matan Rabi, Senior Software Engineer on Salesforce Einstein\u2019s Machine Learning Observability Platform (MLOP) team. Matan and his team strive to optimize the accuracy of Einstein\u2019s AI classification models, empowering customers across industries to enhance efficiency and provide… Continue reading How is Salesforce Einstein Optimizing AI Classification Model Accuracy?<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-733","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":887,"url":"https:\/\/fde.cat\/index.php\/2024\/06\/25\/the-future-of-ai-testing-salesforces-next-gen-framework-for-ai-model-performance\/","url_meta":{"origin":733,"position":0},"title":"The Future of AI Testing: Salesforce\u2019s Next Gen Framework for AI Model Performance","date":"June 25, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we explore the innovative minds shaping the future of Salesforce engineering. Today, we meet Erwin Karbasi, who leads the development of the Salesforce Central Evaluation Framework (SF Eval), a revolutionary internal tool used by Salesforce engineers to assess the performance of generative AI models.\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":317,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/a-deep-dive-on-text-classification-at-salesforce\/","url_meta":{"origin":733,"position":1},"title":"A Deep Dive on Text Classification at Salesforce","date":"August 31, 2021","format":false,"excerpt":"published on Towards Data\u00a0SciencePutting from a Sand Trap (Image by\u00a0Author)We\u2019re excited to announce that Noah Burbank, a Principal Data Scientist in Sales Cloud, has recently published a deep dive into text classification at Salesforce on Towards Data Science. The article, How to choose the right model for text classification in\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":736,"url":"https:\/\/fde.cat\/index.php\/2023\/07\/18\/how-can-apache-spark-windowing-supercharge-your-performance-and-simplify-coding\/","url_meta":{"origin":733,"position":2},"title":"How Can Apache Spark Windowing Supercharge Your Performance and Simplify Coding?","date":"July 18, 2023","format":false,"excerpt":"By Matan Rabi and Scott Nyberg. Salesforce Einstein\u2019s data scales generate terabytes of data each day to train, test, and observe AI models. By only utilizing one machine, this time-consuming process could take days or weeks to complete. In response, Einstein\u2019s Machine Learning Observability Platform (MLOP) team uses Apache Spark,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":840,"url":"https:\/\/fde.cat\/index.php\/2024\/03\/20\/aiops-engineering-secrets-revealed-how-ai-and-automation-slash-thousands-of-manual-hours-annually\/","url_meta":{"origin":733,"position":3},"title":"AIOps Engineering Secrets Revealed: How AI and Automation Slash Thousands of Manual Hours Annually","date":"March 20, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we explore the remarkable journeys of engineering leaders who have made significant contributions in their respective fields. Today, we meet Sravanthi Konduru, a Lead Member of the Technical Staff for Salesforce Engineering, who helps drive the development of the Warden AIOps platform. Explore how\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":848,"url":"https:\/\/fde.cat\/index.php\/2024\/04\/01\/unveiling-the-cutting-edge-features-of-ml-console-for-ai-model-lifecycle-management\/","url_meta":{"origin":733,"position":4},"title":"Unveiling the Cutting-Edge Features of ML Console for AI Model Lifecycle Management","date":"April 1, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we explore the journeys of engineering leaders who have made remarkable contributions in their fields. Today, we meet Venkat Krishnamani, a Lead Member of the Technical Staff for Salesforce Engineering and the lead engineer for Salesforce Einstein\u2019s Machine Learning (ML) Console. This vital tool\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":866,"url":"https:\/\/fde.cat\/index.php\/2024\/05\/15\/revealing-einsteins-blueprint-for-creating-the-new-unified-ai-platform-from-siloed-legacy-stacks\/","url_meta":{"origin":733,"position":5},"title":"Revealing Einstein\u2019s Blueprint for Creating the New, Unified AI Platform from Siloed Legacy Stacks","date":"May 15, 2024","format":false,"excerpt":"In our insightful \u201cEngineering Energizers\u201d Q&A series, we delve into the inspiring journeys of engineering leaders who have achieved remarkable success in their specific domains. Today, we meet Indira Iyer, Senior Vice President of Salesforce Engineering, leading Salesforce Einstein development. Her team\u2019s mission is to build Salesforce\u2019s next-gen AI Platform,\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/733","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=733"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/733\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=733"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=733"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=733"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}