{"id":229,"date":"2021-02-02T20:02:02","date_gmt":"2021-02-02T20:02:02","guid":{"rendered":"https:\/\/fde.cat\/?p=229"},"modified":"2021-02-02T20:02:03","modified_gmt":"2021-02-02T20:02:03","slug":"ml-lake-building-salesforces-data-platform-for-machine-learning","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/02\/02\/ml-lake-building-salesforces-data-platform-for-machine-learning\/","title":{"rendered":"ML Lake: Building Salesforce\u2019s Data Platform for Machine Learning"},"content":{"rendered":"

Salesforce uses machine learning to improve every aspect of its product suite. With the help of Salesforce Einstein<\/a>, companies are improving productivity and accelerating key decision-making.<\/p>\n

\"\"<\/figure>\n

Data is a critical component of all machine learning applications and Salesforce is no exception. In this post I will share some unique challenges Salesforce has in the realm of data management and how ML Lake addresses these challenges to enable internal teams to build predictive capabilities into all Salesforce products, making every feature in Salesforce smarter and easier to use.<\/p>\n

ML Lake enables Salesforce application developers and data scientists to easily build machine learning capabilities on customer and non-customer data. It is a shared service<\/strong> that provides the right data<\/strong>, optimizes the right access patterns<\/strong>, and alleviates the machine learning application developer from having to manage data pipelines, storage, security <\/strong>and compliance<\/strong>.<\/p>\n

Multitenancy, Integration, Metadata<\/h3>\n

Before diving into ML Lake\u2019s internals and architecture, it is important to introduce the functional and non-functional requirements that inspired us to build it. Salesforce is a cloud enterprise company that offers vertical solutions in areas such as Sales<\/a>, Service<\/a> and Marketing<\/a>, as well as general-purpose low-code\/no-code platform<\/a> capabilities. There are thousands of different features of Salesforce that our customers leverage and customize<\/a> to the needs of their unique business challenges.<\/p>\n

Multitenancy has been described as the \u201ckeystone of the Salesforce architecture<\/a>.\u201d Data at Salesforce is owned by our customers and is segregated to ensure that data from one customer does not mix with another. As in most of Salesforce systems, machine learning applications requires granular access controls to ensure tenant-level data isolation. Additionally, Salesforce has grown via both acquisitions and organically, resulting in an architecture comprised on varied stacks and databases.<\/p>\n

A key reason for building ML Lake is to make it easy for applications to get the data they need, while centralizing the security controls required for maintaining trust<\/a>. Teams building applications typically underestimate the level of effort required for ETL and integration. The total cost of copying and storing data includes many hidden costs in the area of compliance, synchronization and security.<\/p>\n

Salesforce customers have embraced the extensibility of the platform and have tailored their Salesforce systems to suit their unique business requirements. Every customer\u2019s object model is different, and understanding this uniqueness by machine learning applications is critical for Einstein<\/a>. In order to leverage this metadata and use it in conjunction with highly structured and customized data to build high-quality ML models tailored to each customer, Salesforce has developed and open-sourced TransmogrifAI<\/a>, an AutoML library specifically tailored to the needs of the enterprise.<\/p>\n

ML Lake must not only scale in total data size, but also in the number and variety of datasets it houses. A common industry practice is to carefully maintain and curate a number of key datasets for ML or analytics use cases. At Salesforce\u2019s scale of data and customization this is impossible. Everything has to be tracked and automated, enabled by maintaining extensive metadata in ML Lake. This metadata is vital for model training and serving, as well as compliance operations.<\/p>\n

Architecture<\/h3>\n\"\"\n

ML Lake is deployed in multiple AWS regions as a shared service for use by internal Salesforce teams and applications running in a variety of stacks in both public cloud providers and Salesforce\u2019s own data centers. It exposes a set of OpenAPI-based interfaces running in a Spring Boot<\/a>-based Java microservice. It uses Postgres to store application state and metadata. Data for machine learning is stored in S3 in buckets managed and secured by ML Lake.<\/p>\n

Data Lake<\/h3>\n

AWS S3 was chosen as the backing store for ML Lake\u2019s data lake for its resiliency, cost-effectiveness, and ease of integration with data processing engines. It houses datasets containing customer data from different parts of Salesforce, non-customer data, such as public datasets containing word embeddings, as well as data generated and uploaded by internal machine learning applications.<\/p>\n

A typical flow for a machine learning application interacting with ML Lake is to request metadata for a particular dataset, which contains a pointer to an S3 path housing the data, request a granularly-scoped data access token, and interact with the actual data using S3 API or S3\u2019s integration into common data tooling like Apache Spark<\/a> or Pandas<\/a>.<\/p>\n

The majority of Salesforce data is highly structured and with schemas often customized by clients. It is important for ML Lake to support structured datasets well, allow partitioning and filtering of large datasets, and support consistent schema changes and data updates. The current de-facto standard for table formats is the Hive Metastore<\/a> but it addresses only a subset of ML Lake\u2019s needs. There are a number of exciting open source projects in this space that the ML Lake team has evaluated, eventually choosing Apache Iceberg<\/a>. Iceberg is the table format for all ML Lake\u2019s structured datasets.<\/p>\n

Pipelines<\/h3>\n

ML Lake uses the term \u201cpipelines\u201d for jobs bringing data in and out of it. The pipeline capabilities provided by ML Lake are controlled via a set of APIs exposed to internal applications. The APIs control bi-directional data movement of raw feature data into ML Lake, as well predictions and related data back to customer-facing systems in Salesforce, while hiding the complexity behind a simple facade.<\/p>\n

The pipelines service centralizes the management of data movement jobs and handles common concerns like retries, error handling and reporting. Pipeline jobs are implemented in Scala<\/a> and Spark<\/a> running on EMR clusters and utilize custom connectors to various parts of Salesforce, coupled with an intra-Salesforce integration auth mechanism that complies with the strict rules of granular and explicit authorization mandated by the Product Security organization.<\/p>\n

ML Lake automatically provides GDPR compliance for data stored inside it. Compliance-related pipeline jobs continuously ingest GDPR signals, such as record deletions and do-not-profile flags<\/a>, and periodically remove that data from ML Lake. Jobs that ingest these signals, as well as jobs that process data deletions, are also implemented in Scala\/Spark.<\/p>\n

Data Catalog<\/h3>\n

Data lake is a common industry approach to storing large volumes of disparate data, and allowing varied types of access to it without over-optimizing for specific access patterns up front.<\/p>\n

Read about our engagement activities data lake<\/a> built with Delta Lake.<\/em><\/p>\n<\/blockquote>\n

While data lakes allow flexibility and freedom, an ungoverned data lake often becomes a \u201cdata graveyard<\/a>.\u201d For ML Lake, it is important to track and catalog the data stored inside for the following reasons:<\/p>\n