{"id":646,"date":"2022-10-31T16:30:38","date_gmt":"2022-10-31T16:30:38","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2022\/10\/31\/improving-instagram-notification-management-with-machine-learning-and-causal-inference\/"},"modified":"2022-10-31T16:30:38","modified_gmt":"2022-10-31T16:30:38","slug":"improving-instagram-notification-management-with-machine-learning-and-causal-inference","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2022\/10\/31\/improving-instagram-notification-management-with-machine-learning-and-causal-inference\/","title":{"rendered":"Improving Instagram notification management with machine learning and causal inference"},"content":{"rendered":"

We\u2019re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram \u2013 particularly on daily digest push notifications.<\/span>
\nBy using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications sent while also improving overall user experience.<\/span><\/p>\n

On Instagram, notifications play an important role in providing efficient communication channels between Instagram and our users. As the types of notifications<\/a> have increased, a need has arisen to provide people with personalized notification experiences to help avoid them receiving excess notifications or ones they may not find to be important.<\/span><\/p>\n

At Meta, we have been applying statistics and machine learning<\/a> (ML) for notification personalization and management on Instagram. Today, we would like to share an example of how we used causal inference and ML to control sending for daily digest push notifications.<\/span><\/p>\n

Moving beyond click-through rate models<\/span><\/h2>\n

A daily digest push notification about stories is a type of notification that lists a digest of stories that are shared and ready for someone to view. When such a notification is delivered to someone\u2019s device, they may click on the notification to view the content. Traditionally, an ML model called a click-through rate (CTR) model is used to predict how likely someone is to click on a notification. CTR models have been working well in many applications across the industry. The predicted click probability is used as a proxy to indicate the notification\u2019s quality to the user. If the predicted click probability is too low, the notification will be dropped in the middle of its sending flow and the user won\u2019t receive the notification because it has been deemed low quality.<\/span><\/p>\n

CTR model-based filtering worked well for daily digest notification in the sense that the actual average click rate when using the CTR model was significantly higher than without the model. However, we also noticed that using the CTR model meant a large portion of the daily digest notifications were sent to users who are relatively active in terms of using Instagram. For many highly active Instagram users, even without sending these daily digest notifications, they would be able to view the corresponding stories in an organic manner. <\/span>This actually opens up an opportunity to provide a better user experience by sending fewer notifications to active users who are likely to view the stories listed in the notifications organically.<\/span><\/p>\n

The challenging aspect is how to identify these users. If we reduce the notifications sent to users who are active based on them receiving such notifications, it\u2019s possible those users will become less active. In other words, if the right users aren\u2019t properly selected for sending reductions we risk creating a decline in user engagement.<\/span><\/p>\n

Causal inference and ML<\/span><\/h2>\n

Essentially, it\u2019s a user selection problem. We would like to maximize the efficiency of sent notifications by selecting proper user cohorts. The solution we adopted to tackle this problem is the combination of causal inference and ML.<\/span><\/p>\n

For problem formulation, let\u2019s assume there is a fixed computational cost to send each daily digest notification, and also there is a total budget for these notifications to spend. Now it becomes a budget allocation problem. The key to solving this problem is figuring out the incremental value of sending a daily digest notification compared to not sending. For example, the incremental value for user <\/span>i<\/span><\/em> can be defined in terms of user activeness, i.e., ui=Pri(active|do(send notification)) \u2013 Pri(active|do(drop notification))<\/em>. <\/span>For some user cohorts, they would be active without receiving the daily digest notifications and thus the incremental values would be small; selecting these cohorts to send the digest notifications is inefficient and may even spam these users. For better product experience and efficiency, we can sort the notifications by the incremental values in descending order and select the top notifications with high incremental values to send, to maximize the overall incremental value with limited budget (sending volume).<\/span><\/p>\n

The next question is how to estimate the incremental value before we make the send or drop decision. It is a challenging question because for the same notification, we can either send it or drop it; there is no way to observe both scenarios. Essentially, this is a causal inference problem and <\/span>uplift modeling techniques<\/span><\/a> can be used. To <\/span>apply uplift models<\/span><\/a>, we designed a randomized experiment in which each notification was randomly sent or dropped, as illustrated in Figure 1.<\/span><\/p>\n

Figure 1: A randomized experiment for uplift modeling, where a notification eligible to send will be dropped randomly with 50 percent probability.<\/p>\n

Based on the data collected from this randomized experiment, we developed a neural network-based uplift model to predict the incremental impact between not sending and sending the daily digest notifications about stories at user level. Given the estimates of incremental impact for all notifications, the solution of the above budget allocation problem is trivial. However, in practice the notifications are generated and scored online and thus we cannot have incremental impact estimates ready for all candidate notifications in advance.\u00a0<\/span><\/p>\n

As a consequence, we need an online approach to determine which notifications to send or drop. One simple but effective solution is to compare the online generated score with a fixed threshold \u2013 if the score is higher than the threshold we can send it. By doing so, we intend to maintain a fixed notification sending rate <\/span>r<\/span><\/em> where <\/span> 0 < r < 1<\/span><\/em>.\u00a0<\/span><\/p>\n

When we applied this approach in online testing we observed sending rate fluctuations because the uplift (incremental impact) estimates generated from ML models may shift from time to time due to various reasons. To stabilize the sending rate, we utilize an online quantile computation service to transform the raw uplift estimates towards a standard uniform distribution while preserving the orders. To maintain the sending rate to <\/span>r<\/span>, we simply compare the transformed uplift estimate with <\/span>r<\/span><\/em> to make the sending decision, since the transformed uplift estimate Z~U(0,1),<\/em>\u00a0 Pr( Z >= 1 \u2013 r ) = r<\/em><\/span>. This process is illustrated in Figure 2.<\/span><\/p>\n

Figure 2: Order-preserving score transformation for sending rate control<\/p>\n

Better notifications with causal inference and ML<\/span><\/h2>\n

By applying this model and targeting the users + notifications with high incremental impact, we reduced the sending volume substantially compared to using the CTR model and also saw no decline in user engagement. The benefit of this work is twofold: improved user experience and reduced resource usage.<\/span><\/p>\n

In the Instagram Notifications Systems team, ML and statistics have been applied in different areas to improve user notification experience. If you want to learn more about this work or are interested in joining one of our engineering teams, please visit our <\/span>careers page<\/span><\/a>, and follow us<\/span> on Facebook<\/span><\/a>.<\/span><\/p>\n

The post Improving Instagram notification management with machine learning and causal inference<\/a> appeared first on Engineering at Meta<\/a>.<\/p>\n

Engineering at Meta<\/p>","protected":false},"excerpt":{"rendered":"

We\u2019re sharing how Meta is applying statistics and machine learning (ML) to improve notification personalization and management on Instagram \u2013 particularly on daily digest push notifications. By using causal inference and ML to identify highly active users who are likely to see more content organically, we have been able to reduce the number of notifications… Continue reading Improving Instagram notification management with machine learning and causal inference<\/span><\/a><\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[7],"tags":[],"class_list":["post-646","post","type-post","status-publish","format-standard","hentry","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":599,"url":"https:\/\/fde.cat\/index.php\/2022\/06\/14\/applying-federated-learning-to-protect-data-on-mobile-devices\/","url_meta":{"origin":646,"position":0},"title":"Applying federated learning to protect data on mobile devices","date":"June 14, 2022","format":false,"excerpt":"What the research is: Federated learning with differential privacy (FL-DP) is one of the latest privacy-enhancing technologies being evaluated at Meta as we constantly work to enhance user privacy and further safeguard users\u2019 data in the products we design, build, and maintain. FL-DP enhances privacy in two important ways: It\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":893,"url":"https:\/\/fde.cat\/index.php\/2024\/07\/10\/metas-approach-to-machine-learning-prediction-robustness\/","url_meta":{"origin":646,"position":1},"title":"Meta\u2019s approach to machine learning prediction robustness","date":"July 10, 2024","format":false,"excerpt":"Meta\u2019s advertising business leverages large-scale machine learning (ML) recommendation models that power millions of ads recommendations per second across Meta\u2019s family of apps. Maintaining reliability of these ML systems helps ensure the highest level of service and uninterrupted benefit delivery to our users and advertisers. To minimize disruptions and ensure\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":744,"url":"https:\/\/fde.cat\/index.php\/2023\/08\/09\/scaling-the-instagram-explore-recommendations-system\/","url_meta":{"origin":646,"position":2},"title":"Scaling the Instagram Explore recommendations system","date":"August 9, 2023","format":false,"excerpt":"Explore is one of the largest recommendation systems on Instagram. We leverage machine learning to make sure people are always seeing content that is the most interesting and relevant to them. Using more advanced machine learning models, like Two Towers neural networks, we\u2019ve been able to make the Explore recommendation\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":343,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/sre-weekly-issue-282\/","url_meta":{"origin":646,"position":3},"title":"SRE Weekly Issue #282","date":"August 31, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, StackHawk: ICYMI ZAP Creator and Project Lead Simon Bennetts recently unveiled ZAP\u2019s new automation framework. Watch the session and see how it works: https:\/\/sthwk.com\/Automation-Framework Articles A thorough introduction to bpftrace I really need to learn bpftrace, and this article is a great\u2026","rel":"","context":"In "SRE"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":613,"url":"https:\/\/fde.cat\/index.php\/2022\/07\/26\/launching-instagram-messaging-on-desktop\/","url_meta":{"origin":646,"position":4},"title":"Launching Instagram Messaging on desktop","date":"July 26, 2022","format":false,"excerpt":"In 2020 we launched Instagram Messaging (referred to in this post simply as \u201cMessaging\u201d) for personal desktop computers. We believe that this feature will improve everyday experiences and enable new use cases for all of our desktop web users. In this post, we go through some of our overall learnings\u2026","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":851,"url":"https:\/\/fde.cat\/index.php\/2024\/04\/10\/introducing-the-next-gen-meta-training-and-inference-accelerator\/","url_meta":{"origin":646,"position":5},"title":"Introducing the next-gen Meta Training and Inference Accelerator","date":"April 10, 2024","format":false,"excerpt":"The post Introducing the next-gen Meta Training and Inference Accelerator appeared first on Engineering at Meta. Engineering at Meta","rel":"","context":"In "Technology"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/646","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=646"}],"version-history":[{"count":0,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/646\/revisions"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=646"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=646"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=646"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}