{"id":709,"date":"2023-05-03T01:48:54","date_gmt":"2023-05-03T01:48:54","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2023\/05\/03\/inside-marketing-clouds-new-automation-systems-updating-500m-marketing-leads-daily\/"},"modified":"2023-05-03T01:48:54","modified_gmt":"2023-05-03T01:48:54","slug":"inside-marketing-clouds-new-automation-systems-updating-500m-marketing-leads-daily","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2023\/05\/03\/inside-marketing-clouds-new-automation-systems-updating-500m-marketing-leads-daily\/","title":{"rendered":"Inside Marketing Cloud\u2019s New Automation Systems: Updating 500M+ Marketing Leads Daily"},"content":{"rendered":"

To remain competitive in today\u2019s marketplace, companies\u2019 marketing leads must convert into sales; however; 40% of business leaders believe their existing marketing efforts are outdated<\/a>. Pivoting from archaic marketing tools to automated software like Salesforce\u2019s Marketing Cloud A<\/a>ccount Engagement<\/a> (MCAE) significantly enhances companies\u2019 lead generation process and helps boost revenue. Companies that use mature lead generation processes produce 133% more revenue<\/a> than average.<\/p>\n

MCAE helps Salesforce customers simplify and scale how they market to their customers, removing the need for manual, hands-on parsing to qualify leads. MCAE simplifies marketers\u2019 work via automated lead segmentation and auto<\/a>mated, customized customer experiences<\/a> \u2014 including email campaigns \u2014 which rapidly qualify leads for their company\u2019s sales team to convert.<\/p>\n

Marketing Cloud\u2019s production engineering scale team is the engine behind MCAE\u2019s engineering, supporting the above features while updating customers\u2019 constantly changing lead data in near real-time.<\/p>\n

Among its many activities, the scale team updates existing lead segmentation and inserts new segmentations.<\/em><\/p>\n

Ensuring lead integrity through systems maintenance formerly consumed much of the team\u2019s day-to-day, requiring them to perform hands-on management of their message queuing service and database querying system. However, the team recently adopted powerful new tools to solve these challenges \u2014 further improving the quality of customer leads.<\/p>\n

How does the automation system power scaling and reliability for message-based communications?<\/strong><\/p>\n

To drive the effectiveness of customized customer experiences \u2014 known as En<\/a>gagement Studio<\/a> campaigns \u2014 MCAE\u2019s automation system continually ingests customer change events and determines if any customer automation should be updated. This ensures the right emails are sent to the right leads at the right time.<\/p>\n

For example, a dynamic data change event in the system \u2014 such as a customer lead relocating from the West Coast to the East Coast \u2014 could trigger the automation system to delete the lead from a Western list and add them to an Eastern list. Lists that are controlled by the automation system are called dynamic lists, and they are a core tenet of MCAE.<\/p>\n

This single event could also spark a butterfly effect, causing cascading changes throughout the system, requiring 15 or more dynamic lists \u2014 which are controlled by the automation system and a core tenet within MCAE \u2014 to be updated.<\/p>\n

Customers expect our systems to be highly available and near real-time. Operating at scale is a requirement in today\u2019s age, and our system handles hundreds of millions of events each day while being as performant as possible. This high change volume overwhelmed the scale team\u2019s legacy automation system \u2014 consisting of a fleet of self-provisioned instances \u2014 causing the system to crash periodically, creating delays and stability issues for customers.<\/p>\n

The system seamlessly scales in support of thousands of requests per second.<\/em><\/p>\n

Consequently, the team devoted considerable hands-on time to managing the system, providing security updates and maintenance to keep the service running smoothly.<\/p>\n

To improve message queuing scale and reliability, the team recently migrated to Queue Platform (QP), an internally built system that democratizes message queueing for all internal teams for powering their services. QP leverages a new underlying infrastructure to provide a near-infinite scale for message throughput. QP also intelligently detects the amount of pending work that needs to be performed and automatically scales the number of message consumers appropriately. Migrating existing mission-critical systems onto QP required large amounts of refactoring, which posed a great challenge for the team.<\/p>\n

To support the migration, the team re-wrote mission-critical parts of their software. The risk during this migration was high because marketers rely on this data to determine which leads to email. If the team had incorrectly computed their data, it could have resulted in MCAE customers sending emails to the wrong leads. Consequently, this could have damaged Salesforce\u2019s reputation and violated compliance laws. Ultimately, the team seamlessly executed the migration without incurring any bugs or disrupting customer service while continuing to operate at the scale required \u2014 a major achievement for a refactoring of this size.<\/p>\n

Migrating from a bespoke system tailored to one use case to a versatile platform that powers many features has delivered major improvements. Since the migration, QP continues to serve as a force multiplier for the organization, providing internal teams with an out-of-the-box, reliable, and scalable queuing solution. QP handles mundane and time-consuming tasks such as infrastructure maintenance and allows developers to spend more time focused on customer success.<\/p>\n

How does the automation system circumvent lead data processing bottlenecks?<\/strong><\/p>\n

Keeping list-based memberships updated creates many technical challenges on the backend, including how to most efficiently query (retrieve) and process lead data from databases in the wake of countless daily events that can create bottlenecks due to the high scale of circulated data.<\/p>\n

Querying lead data occurs when MCAE\u2019s automation system detects change events in real-time and determines which data must be pulled from the database. Querying typically triggers MCAE\u2019s decision engine to take the appropriate action, including updating a list. Horizontally scaling our databases has been a recipe for success. For example, by using several read replicas \u2014 multiple instances of their main database \u2014 query processing spread across multiple server databases to handle increased workloads. This horizontal scaling enables the system to more efficiently query databases, reducing the strain on the main database and decreasing cycle times sometimes by an order of magnitude.<\/p>\n

Additionally, the automation system must dynamically scale to meet the various changes in system demand throughout the day. The system implements custom scaling logic, powered by Kubernetes<\/a>, that intelligently detects increases in lead data processing demand and responds accordingly by scaling system capacity.As system demand returns to its baseline, the extra resources are deallocated, which subsequently reduces the system\u2019s cost-to-serve.<\/p>\n

Learn More<\/strong><\/h3>\n

Stay connected \u2014 join our Talent Community<\/a>!<\/p>\n

Check out our Technology and Product teams<\/a> to learn how you can get involved.<\/p>\n

For a closer look at the MCAE scale team, check out this blog<\/a>.<\/p>\n

The post Inside Marketing Cloud\u2019s New Automation Systems: Updating 500M+ Marketing Leads Daily<\/a> appeared first on Salesforce Engineering Blog<\/a>.<\/p>\n

Read More<\/a><\/p>","protected":false},"excerpt":{"rendered":"

To remain competitive in today\u2019s marketplace, companies\u2019 marketing leads must convert into sales; however; 40% of business leaders believe their existing marketing efforts are outdated. Pivoting from archaic marketing tools to automated software like Salesforce\u2019s Marketing Cloud Account Engagement (MCAE) significantly enhances companies\u2019 lead generation process and helps boost revenue. Companies that use mature lead… Continue reading Inside Marketing Cloud\u2019s New Automation Systems: Updating 500M+ Marketing Leads Daily<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-709","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":707,"url":"https:\/\/fde.cat\/index.php\/2023\/04\/25\/solving-complex-and-demanding-marketing-automation-challenges-at-scale\/","url_meta":{"origin":709,"position":0},"title":"Solving Complex and Demanding Marketing Automation Challenges at Scale","date":"April 25, 2023","format":false,"excerpt":"In our \u201cEngineering Energizers\u201d Q&A series, we examine the life experiences and career paths that have shaped Salesforce engineering leaders. Meet Charlie Curtis, a lead member of the technical staff for Marketing Cloud\u2019s production engineering team. Charlie and his Atlanta-based team develop cutting-edge automation software that helps users simplify their\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":845,"url":"https:\/\/fde.cat\/index.php\/2024\/03\/25\/mission-impossible-inside-the-unprecedented-integration-of-100-products-within-salesforces-new-marketing-cloud-growth-edition\/","url_meta":{"origin":709,"position":1},"title":"Mission Impossible: Inside the Unprecedented Integration of 100 Products Within Salesforce\u2019s New Marketing Cloud Growth Edition","date":"March 25, 2024","format":false,"excerpt":"In this edition of our \u201cEngineering Energizers\u201d Q&A series, we spotlight Jeanine Walters, Principal Architect and lead architect behind Marketing Cloud Growth Edition at Salesforce. With over 20 years of architecting innovative solutions at Salesforce, Jeanine has played a pivotal role in creating a game-changing marketing application that empowers small\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":538,"url":"https:\/\/fde.cat\/index.php\/2022\/02\/01\/behind-the-scenes-of-hyperforce-salesforces-infrastructure-for-the-public-cloud\/","url_meta":{"origin":709,"position":2},"title":"Behind the Scenes of Hyperforce: Salesforce\u2019s Infrastructure for the Public Cloud","date":"February 1, 2022","format":false,"excerpt":"Salesforce has been running cloud infrastructure for over two decades, bringing companies and their customers together. When Salesforce first started out in 1999, the world was very different; back then, the only practical way to provide our brand of Software-As-A-Service was to run everything yourself\u200a\u2014\u200anot just the software, but the\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":566,"url":"https:\/\/fde.cat\/index.php\/2022\/04\/26\/sql-notebooks-combining-the-power-of-jupyter-and-sql-editors-for-data-analytics\/","url_meta":{"origin":709,"position":3},"title":"SQL Notebooks: Combining the power of Jupyter and SQL editors for data analytics","date":"April 26, 2022","format":false,"excerpt":"At Meta, our internal data tools are the main channel from our data scientists to our production engineers. As such, it\u2019s important for us to empower our scientists and engineers not only to use data to make decisions, but also to do so in a secure and compliant way. We\u2019ve\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":182,"url":"https:\/\/fde.cat\/index.php\/2020\/12\/07\/datahub-popular-metadata-architectures-explained\/","url_meta":{"origin":709,"position":4},"title":"DataHub: Popular metadata architectures explained","date":"December 7, 2020","format":false,"excerpt":"When I started my journey at LinkedIn ten years ago, the company was just beginning to experience extreme growth in the volume, variety, and velocity of our data. Over the next few years, my colleagues and I in LinkedIn\u2019s data infrastructure team built out foundational technology like Espresso, Databus, and\u2026","rel":"","context":"In "External"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":705,"url":"https:\/\/fde.cat\/index.php\/2023\/04\/18\/ai-based-identity-resolution-the-key-for-linking-diverse-customer-data\/","url_meta":{"origin":709,"position":5},"title":"AI-based Identity Resolution: The Key for Linking Diverse Customer Data","date":"April 18, 2023","format":false,"excerpt":"Companies want a comprehensive view of their customers, enabling them to solve business and marketing challenges, such as personalization, segmentation, and targeting \u2014 but they face an uphill battle as they are drowning in data. For example, many companies cannot match the identity of a customer who visits their website\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/709","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=709"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/709\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=709"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=709"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=709"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}