{"id":884,"date":"2024-06-21T21:31:57","date_gmt":"2024-06-21T21:31:57","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2024\/06\/21\/how-einstein-copilot-sharpens-large-language-model-outputs-and-redefines-ai-data-testing\/"},"modified":"2024-06-21T21:31:57","modified_gmt":"2024-06-21T21:31:57","slug":"how-einstein-copilot-sharpens-large-language-model-outputs-and-redefines-ai-data-testing","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2024\/06\/21\/how-einstein-copilot-sharpens-large-language-model-outputs-and-redefines-ai-data-testing\/","title":{"rendered":"How Einstein Copilot Sharpens Large Language Model Outputs and Redefines AI Data Testing"},"content":{"rendered":"

In our \u201cEngineering Energizers\u201d Q&A series, we explore the paths of engineering leaders who have attained significant accomplishments in their respective fields. Today, we spotlight Armita Peymandoust, Senior Vice President of Software Engineering at Salesforce, who spearheads the development of Einstein Copilot<\/strong><\/a>, a conversational AI assistant for CRM that integrates data, metadata, prompts, and workflows to perform intelligent analysis and task execution, thereby optimizing efficiency and improving task completion rates.<\/p>\n

Explore how Armita\u2019s team addresses large language model (LLM) complexities by implementing guardrails to ensure accurate and reliable AI outputs, ensures customer data privacy by using synthetic and public datasets to test features, and much more!<\/p>\n

What are the primary technical challenges in developing Einstein Copilot?<\/strong><\/h5>\n

The development of Einstein Copilot at Salesforce faces two primary technical challenges that are crucial to its success and functionality.<\/p>\n

The first challenge involves LLM management and utilization<\/strong>. Einstein Copilot uses LLM to perform complex tasks by generating accurate and contextually appropriate responses. However, the technology is relatively new and the team continuously navigates learning how best to implement and control it. A significant aspect of this challenge is preventing the model from \u201challucinating\u201d\u2014producing plausible but incorrect or irrelevant outputs. To address this, the team focuses on creating strict guardrails and providing specific, contextual grounding to the model, which helps in narrowing down the response scope and maintaining output reliability.<\/p>\n

The second major challenge is ensuring the availability of appropriate and realistic datasets for testing the AI features of Einstein Copilot<\/strong>. Since Salesforce policy prohibits the use of customer data for development purposes, the team must use alternative methods to test and refine their features. They primarily rely on synthetic data, which they generate to mimic real-world scenarios that Einstein Copilot might encounter. Additionally, they utilize public data sources available for research to further validate and compare the performance of different model configurations. In some cases, they also engage in pilot agreements with customers, allowing them to access real data in a controlled and ethical manner. This not only helps in fine-tuning Einstein Copilot\u2019s features but also ensures that the data used respects customer privacy and data ownership.<\/p>\n

Armita discusses Salesforce\u2019s engineering culture.<\/em><\/p>\n

What are the challenges and typical adoption process for integrating Einstein Copilot into existing customer workflows?<\/strong><\/h5>\n

Integrating Einstein Copilot into existing customer workflows poses significant challenges, requiring careful change management. This often prolongs the feedback loop, as customers need time to test and adapt to new features before fully integrating them.<\/p>\n

For instance, when Salesforce releases a new feature of Einstein Copilot, initial use is typically limited to admins and power users. This gradual adoption process ensures the AI solutions are trusted and seamlessly integrate into established systems without disruption.<\/p>\n

This methodical approach to innovation underscores Salesforce\u2019s commitment to delivering dependable AI solutions that meet customer needs and fit into their operational realities.<\/p>\n

A diagram depicting the testing of generative copilots and agents at scale.<\/em><\/p>\n

How does the collaboration between your engineering and product teams contribute to the development of effective and relevant Einstein Copilot features?<\/strong><\/h5>\n

It\u2019s really about the synergy between our engineering and product teams. Our engineering team is deeply involved in exploring what\u2019s technically feasible with the latest AI advancements. They essentially set the stage by showcasing the art of the possible.<\/p>\n

On the other side, our product team steps in with a strong understanding of the challenges and needs our customers face. This combination allows us to pinpoint which business problems can be effectively solved using new technologies.<\/strong> It\u2019s a dynamic interplay where both teams bring something vital to the table, ensuring that the solutions we develop are cutting-edge, directly relevant, and highly beneficial to our users.<\/p>\n

We also take a very measured approach to deciding which features to push forward. This involves a careful assessment of the costs associated with developing and maintaining these features versus the value they deliver to our customers and to Salesforce as a business. It\u2019s all about making strategic choices that maximize impact while optimizing resource use.<\/strong><\/p>\n

This collaborative process keeps us agile and responsive, enabling us to adapt quickly in a fast-changing market environment. It ensures that the AI features we develop are innovative and aligned perfectly with what our customers need to succeed.<\/p>\n

Armita discusses a new project in development, in collaboration with Salesforce\u2019s AI Research team<\/em><\/p>\n

What is the iterative process your team follows when developing new features for Einstein Copilot?<\/strong><\/h5>\n

The development of new features for Einstein Copilot at Salesforce follows a meticulous iterative process. Initially, the team defines the problem they aim to solve and constructs a specific prompt for it. This prompt engineering is crucial as it shapes the subsequent development and testing phases.<\/p>\n

Once a feature is built, it undergoes rigorous testing to evaluate the quality and accuracy of the generated outputs. This testing involves using labeling tools, human labelers, and state-of-the-art evaluation metrics. After the release of these features, the team heavily relies on customer feedback, which is gathered both qualitatively and quantitatively within the product. This feedback is integral to the iterative cycle, enabling the team to refine and enhance the features continuously.<\/p>\n

This structured approach ensures that each feature not only meets the initial design specifications but also evolves based on direct user input, aligning closely with customer needs and expectations.<\/p>\n

Armita explains why engineers should join Salesforce.<\/em><\/p>\n

Diving deeper, how does your team handle customer feedback and integrate it into Einstein Copilot\u2019s development process?<\/strong><\/h5>\n

The team takes a meticulous approach to monitor and evaluate every feature developed, especially given the innovative nature of generative AI technology. Due to the novelty of these applications, it can be challenging to anticipate customer preferences and needs accurately. To address this, the team implements comprehensive instrumentation within the features, allowing them to capture detailed data on customer interactions. This includes tracking actions such as generation, editing, and acceptance of outputs, which provides a rich source of both quantitative and qualitative feedback.<\/p>\n

This feedback is invaluable, as it not only helps in assessing the effectiveness of each feature but also informs the necessary adjustments to enhance functionality and user satisfaction. By continuously analyzing how customers interact with the features, the team can make data-driven decisions to refine and improve the AI solutions<\/strong>, ensuring they align more closely with user expectations and improve their overall experience. This dynamic feedback loop is integral to the iterative development process, enabling the team to adapt swiftly and effectively to user feedback and evolving market needs.<\/p>\n

\n
Learn More<\/h5>\n

Hungry for more AI stories? Learn how the new Einstein Copilot for Tableau is building the future of AI-driven analytics in this blog<\/a>.<\/p>\n

Stay connected \u2014 join our Talent Community<\/a>!<\/p>\n

Check out our Technology and Product<\/a> teams to learn how you can get involved.<\/p>\n<\/div>\n

The post How Einstein Copilot Sharpens Large Language Model Outputs and Redefines AI Data Testing<\/a> appeared first on Salesforce Engineering Blog<\/a>.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

In our \u201cEngineering Energizers\u201d Q&A series, we explore the paths of engineering leaders who have attained significant accomplishments in their respective fields. Today, we spotlight Armita Peymandoust, Senior Vice President of Software Engineering at Salesforce, who spearheads the development of Einstein Copilot, a conversational AI assistant for CRM that integrates data, metadata, prompts, and workflows… Continue reading How Einstein Copilot Sharpens Large Language Model Outputs and Redefines AI Data Testing<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-884","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":859,"url":"https:\/\/fde.cat\/index.php\/2024\/04\/23\/einstein-copilot-for-tableau-building-the-next-generation-of-ai-driven-analytics\/","url_meta":{"origin":884,"position":0},"title":"Einstein Copilot for Tableau: Building the Next Generation of AI-Driven Analytics","date":"April 23, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we explore the extraordinary journeys of engineering leaders who have achieved success in their specific domains. Today, we meet John He, Vice President of Software Engineering, who leads the development of Einstein Copilot for Tableau \u2014 an innovative tool that redefines how users interact\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":887,"url":"https:\/\/fde.cat\/index.php\/2024\/06\/25\/the-future-of-ai-testing-salesforces-next-gen-framework-for-ai-model-performance\/","url_meta":{"origin":884,"position":1},"title":"The Future of AI Testing: Salesforce\u2019s Next Gen Framework for AI Model Performance","date":"June 25, 2024","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we explore the innovative minds shaping the future of Salesforce engineering. Today, we meet Erwin Karbasi, who leads the development of the Salesforce Central Evaluation Framework (SF Eval), a revolutionary internal tool used by Salesforce engineers to assess the performance of generative AI models.\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":283,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/ai-research-to-production-with-einstein-reply-recommendations\/","url_meta":{"origin":884,"position":2},"title":"AI Research to Production with Einstein Reply Recommendations","date":"August 31, 2021","format":false,"excerpt":"We all know that AI is here and it\u2019s quickly changing our lives. However, the impacts of AI are unevenly distributed and it favors those with \u201cmore data,\u201d leaving those with \u201cfew data\u201d behind. This runs counter to our Salesforce core values of Customer Success and Equality, so we set\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":785,"url":"https:\/\/fde.cat\/index.php\/2023\/11\/07\/einstein-for-flow-bringing-ai-innovation-to-the-next-generation-of-automation\/","url_meta":{"origin":884,"position":3},"title":"Einstein for Flow: Bringing AI Innovation to the Next Generation of Automation","date":"November 7, 2023","format":false,"excerpt":"By Vera Vetter, Zeyuan Chen, Ran Xu, and Scott Nyberg In our \u201cEngineering Energizers\u201d Q&A series, we examine the professional journeys that have shaped Salesforce Engineering leaders. Meet Vera Vetter, Product Management Director for Salesforce AI Research and a co-Product Manager for Einstein for Flow, a game-changing AI product that\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":751,"url":"https:\/\/fde.cat\/index.php\/2023\/08\/22\/how-is-einstein-gpt-shaping-the-future-of-salesforce-development-and-unleashing-developer-productivity\/","url_meta":{"origin":884,"position":4},"title":"How is Einstein GPT Shaping the Future of Salesforce Development and Unleashing Developer Productivity?","date":"August 22, 2023","format":false,"excerpt":"By Yingbo Zhou and Scott Nyberg In our \u201cEngineering Energizers\u201d Q&A series, we examine the professional life experiences that have shaped Salesforce Engineering leaders. Meet Yingbo Zhou, a Senior Director of Research for Salesforce AI Research, where he leads the team to develop the model for Einstein GPT for Developers\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":791,"url":"https:\/\/fde.cat\/index.php\/2023\/08\/22\/how-is-einstein-shaping-the-future-of-salesforce-development-and-unleashing-developer-productivity\/","url_meta":{"origin":884,"position":5},"title":"How is Einstein Shaping the Future of Salesforce Development and Unleashing Developer Productivity?","date":"August 22, 2023","format":false,"excerpt":"By Yingbo Zhou and Scott Nyberg In our \u201cEngineering Energizers\u201d Q&A series, we examine the professional life experiences that have shaped Salesforce Engineering leaders. Meet Yingbo Zhou, a Senior Director of Research for Salesforce AI Research, where he leads the team to develop the model for Einstein for Developers, a\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/884","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=884"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/884\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=884"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=884"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=884"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}