{"id":178,"date":"2021-01-12T17:00:00","date_gmt":"2021-01-12T17:00:00","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2021\/01\/12\/how-linkedin-turned-to-real-time-feedback-for-developer-tooling\/"},"modified":"2021-02-02T13:47:07","modified_gmt":"2021-02-02T13:47:07","slug":"how-linkedin-turned-to-real-time-feedback-for-developer-tooling","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/01\/12\/how-linkedin-turned-to-real-time-feedback-for-developer-tooling\/","title":{"rendered":"How LinkedIn turned to real-time feedback for developer tooling"},"content":{"rendered":"<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p>Over the last year, we have been using real-time feedback to evolve our tooling and provide a more productive experience for LinkedIn&#8217;s developers. It\u2019s helped us double our feedback participation, and more importantly, better tailor our recommendations and improvements.\u00a0<\/p>\n<p>For any engineering organization looking to improve developer experiences, the following questions will provide a good starting point:<\/p>\n<ul>\n<li>How can we make our developers more productive and happier?\u00a0<\/li>\n<li>What investments should I make, as an owner of an internal-facing developer-focused tool, to best help my users?\u00a0<\/li>\n<li>What if there was a way to get immediate feedback from developers right at the time they experience a problem or have a great experience?\u00a0<\/li>\n<\/ul>\n<p>By sharing our learnings from our implementation journey of real-time feedback, we hope it will help other organizations improve their own processes to create a natural feedback loop between their developers and tool owners.\u00a0<\/p>\n<h2>The challenges of traditional surveys<\/h2>\n<p>For the past several years, we had relied on quarterly developer surveys as a way to reach engineers and solicit comprehensive feedback on the tools they used. However, we found ourselves running into a number of challenges with this approach:\u00a0<\/p>\n<ul>\n<li><b>Developers sometimes provided opinions on tools that they hadn\u2019t used in a long time. <\/b>In some cases, we found that over 90% of the people that rated a specific tool had not used the tool in the 6 months before the survey was released.<\/li>\n<li><b>Tool owners had very little context about the provided feedback. <\/b>What was the developer providing a rating about? What specific tool or action were they referring to? This made large portions of the feedback unactionable.<\/li>\n<li><b>It was difficult to measure what was going on between periodic surveys. <\/b>When features and bug fixes were released between surveys, we couldn\u2019t tell if the user rating was applicable to the \u201cbefore\u201d state or the \u201cafter\u201d state. In addition, we couldn\u2019t get a pulse on developers\u2019 thoughts in between surveys.<\/li>\n<\/ul>\n<p>At a high-level, the results helped us understand the general sentiment about the entire tooling ecosystem and allowed us to make significant progress on the tooling <a href=\"https:\/\/measuringu.com\/microsoft-nsat\/\" target=\"_blank\" rel=\"noopener\">NSAT<\/a>. However, we lacked a solid understanding of the specifics of the pain-points and sought a more accurate pulse on how internal users were feeling about the tools at their disposal.<\/p>\n<h2>The solution: Real-time Feedback<\/h2>\n<p>In order to overcome these challenges, we developed a mechanism called Real-Time Feedback. Real-time Feedback is a system that first collects information about actions that developers take across our tooling ecosystem and then, based on this contextual information, decides if, when, and how to solicit feedback from the developer.\u00a0<\/p>\n<p>For example, we might notice that a developer has just completed a deployment, and hasn\u2019t been asked for feedback on any tooling activity in the last two weeks. This presents a case in which we can send an email asking for feedback on how the tooling experience was for that specific deployment. The seamless integration of feedback solicitation into the day-to-day workflow means that developers don\u2019t need to sit down once a quarter and rely on their memory, but instead can just weigh in a little bit at a time, when they are automatically prompted for feedback.<\/p>\n<h2>Capturing context<\/h2>\n<p>When we deliver feedback to the developers of our tools, we are able to include context about what specific activity the developer was engaged in when they provided that feedback.<\/p>\n<p>As noted above, one of the challenges with periodic surveys was extracting actionable information from the feedback. Comments that mentioned \u201csometimes\u201d were especially difficult to understand. By knowing exactly what happened at the specific time, and asking in the moment (Real-time), the feedback we pass along is much more precise with additional detail on the specific session and use.<\/p>\n<p>To capture this contextual information, we created a system that logs developers\u2019 actions via multiple channels: internal web UIs (by utilizing <a href=\"https:\/\/en.wikipedia.org\/wiki\/Matomo_(software)\" target=\"_blank\" rel=\"noopener\">Matomo<\/a>), command-line interfaces (CLIs), and internal APIs (by tapping into our internal logging and auditing mechanisms that publish information to <a href=\"https:\/\/engineering.linkedin.com\/blog\/2019\/apache-kafka-trillion-messages\" target=\"_blank\" rel=\"noopener\">Apache Kafka<\/a>).\u00a0<\/p>\n<p>Context capturing is a key element of our strategy, and is used for more than just feedback. We plan on using it as the foundation of a new system that will help developers get help with tools. The theory is that if we have the full context of what the developer was doing when they asked for help, it\u2019s going to be much easier to provide that support.<\/p>\n<h2>Participation rates<\/h2>\n<p>For any survey process, a major factor in its efficacy relies in the willingness of the user populations to respond. In fact, as we transitioned into the Real-time Feedback methodology, one major concern we had was consistent participation rates.<\/p>\n<p>It turns out that using a real-time solicitation approach was much more effective, in terms of population coverage, than our traditional surveys. We managed to double our survey-participation population (from roughly 15% in the periodic surveys, to over 30% with the real-time approach).\u00a0<\/p>\n<p>Thanks to this increased participation, we\u2019re now able to implement better <a href=\"https:\/\/en.wikipedia.org\/wiki\/Market_segmentation\" target=\"_blank\" rel=\"noopener\">market segmentation<\/a> on the developer population. At LinkedIn, there are many kinds of developers\u2014UX developers, backend developers, <a href=\"https:\/\/engineering.linkedin.com\/blog\/2017\/05\/building-the-sre-culture-at-linkedin\" target=\"_blank\" rel=\"noopener\">site reliability engineers<\/a>, and <a href=\"https:\/\/engineering.linkedin.com\/blog\/2019\/01\/scaling-machine-learning-productivity-at-linkedin\" target=\"_blank\" rel=\"noopener\">machine-learning experts<\/a>, to name a few. These different developer types\u2014groupings\/cohorts of developers\u2014have different needs, usage patterns, and productivity issues. More precise targeting and segmentation will lead to a better and more personalized tooling that cater to the specific needs of each developer type.\u00a0<\/p>\n<p>We\u2019re also cognizant not to overwhelm developers with too many requests for feedback\u2014otherwise, participation could drop dramatically. We, therefore, implemented smart throttling mechanisms to ensure that we\u2019re only asking for feedback when a developer is done with their intent (that is, they are at the end of a flow). The solicitation itself is centered around their experience, rather than around a specific tool or layer of the tech stack. For example, we might ask them how their whole deployment went, as opposed to how their experience was with just one tool.<\/p>\n<p>We also make sure to honor preferences with respect to feedback solicitation. Developers can choose how often they want to be asked for feedback, and what channels (such as email, web, Slack, etc.) they want to be asked on. We believe that allowing for this flexibility so that delivery of feedback is incorporated into each developer\u2019s workflow has also played a role in increasing participation.<\/p>\n<p>Developers can choose how often they want to be asked for feedback, and what channels they want to be asked on. These channels include:\u00a0<\/p>\n<ul>\n<li><b>Email: <\/b>This relies solely on the email client to capture the feedback (single-click, based on <a href=\"https:\/\/en.wikipedia.org\/wiki\/Mailto\" target=\"_blank\" rel=\"noopener\">mailto links<\/a>).\u00a0<\/li>\n<li><b>Pluggable UI widget: <\/b>We developed an in-product pluggable UI widget that can serve all kinds of solicitation mechanisms, both passive and active (e.g., in-line, <a href=\"https:\/\/en.wiktionary.org\/wiki\/toast_notification\" target=\"_blank\" rel=\"noopener\">toast notifications<\/a>, and pop-ups).\u00a0<\/li>\n<li><b>Slack: <\/b>By integrating with instant messaging, we also developed a way to collect feedback over Slack.\u00a0<\/li>\n<li><b>Web portal: <\/b>We developed a web portal that allows developers to provide feedback as a stand-alone experience, as well as interact (i.e., voting, commenting) with what other users may have reported.\u00a0<\/li>\n<\/ul>\n<h2>Taking action on feedback<\/h2>\n<p>Listening to feedback is a great starting point, but what comes after when the feedback is implemented is what motivated us to create Real-time Feedback. Once we gather feedback from the various listening channels, we synthesize them into shareable reports (documents and other offerings, such as dashboards). We make sure that the relevant teams are acting on the feedback, and when they choose not to, we make sure that the reasoning is shared back with the developer that provided the feedback. Making sure that this loop is completed is important to sustaining a healthy culture of feedback so that the providers of feedback know they\u2019re being listened to and continue providing valuable input.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<div class=\"resourceImageBlock section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceimageblock\"><\/a>\n <\/div>\n<ul class=\"resource-image-block single\">\n<li class=\"resource-image\"> <img decoding=\"async\" src=\"https:\/\/content.linkedin.com\/content\/dam\/engineering\/site-assets\/images\/blog\/posts\/2021\/01\/rtf-1.svg\" alt=\"diagram-showing-the-act-listen-share-framework-that-real-time-feedback-enables\"> <\/li>\n<\/ul>\n<\/div>\n<div class=\"resourceParagraph section\">\n<div class=\"component-anchor-container\">\n  <a class=\"component-anchor\" name=\"post_par_resourceparagraph_811747156\"><\/a>\n <\/div>\n<div class=\"resource-text-section\">\n<div class=\"resource-paragraph rich-text\">\n<p><i>Our \u201cListen, Act, and Share\u201d framework<\/i><\/p>\n<p>Fostering those two relationship types\u2014inwards (towards tool developers) and outwards (towards the engineers across the company)\u2014allows us to have a symbiotic relationship. We have our ears to the ground gaining insights about what will make developers more productive and happy, and are also communicating with tool developers to suggest improvements that will better help the audiences they serve.\u00a0<\/p>\n<h2>Insights gained from feedback<\/h2>\n<p>We\u2019d like to share two examples of major insights that we gained from the subjective feedback we\u2019ve collected.<\/p>\n<ul>\n<li>Our in-house CI system provides estimates for the expected pipeline runtime. It turns out that these estimates, which sometimes were not very accurate, contributed to doubts among developers on the perceived reliability issue of the tooling in question.<\/li>\n<li>Our <a href=\"https:\/\/engineering.linkedin.com\/blog\/2015\/11\/monitoring-the-pulse-of-linkedin\" target=\"_blank\" rel=\"noopener\">canary-deployment monitoring system (EKG)<\/a> added a protection layer when <a href=\"https:\/\/engineering.linkedin.com\/blog\/2017\/05\/trafficshift--load-testing-at-scale\" target=\"_blank\" rel=\"noopener\">traffic shifts occur<\/a> to fail the analysis, in order to make sure developers are aware that the analysis is unreliable, as the control might have changed their behavior. Unfortunately, developers ended up thinking that EKG itself was unreliable, and ignored the analysis. By getting feedback about this phenomenon, we\u2019ve improved our systems and are now handling this case in a better way.<\/li>\n<\/ul>\n<p>To summarize, realizing the blindspots of our previous approach with traditional surveys helped us rethink how we collect feedback. By creating Real-Time Feedback, an omnichannel contextual way of collecting feedback from developers, we\u2019ve been able to increase both the quantity and quality of feedback received, leading us to drive more actionable insights in how we can better support our developers.<\/p>\n<h2>Acknowledgments<\/h2>\n<p>We\u2019d like to thank <a href=\"https:\/\/www.linkedin.com\/in\/junejavineet\/\" target=\"_blank\" rel=\"noopener\">Vineet Juneja<\/a> who\u2019s helped guide us in this area based on his <a href=\"https:\/\/www.infoworld.com\/article\/3444277\/the-key-to-developer-productivity-and-happiness-measure-and-improve.html\" target=\"_blank\" rel=\"noopener\">previous experience<\/a>; <a href=\"https:\/\/www.linkedin.com\/in\/benlainw\/\" target=\"_blank\" rel=\"noopener\">Ben Lai<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/awaist\/\" target=\"_blank\" rel=\"noopener\">Awais Tariq<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/nnagampalli\/\" target=\"_blank\" rel=\"noopener\">Narsi Nagampalli<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/jgaldes\/\" target=\"_blank\" rel=\"noopener\">Jeff Galdes<\/a>, and <a href=\"https:\/\/www.linkedin.com\/in\/jared-green-0182086\/\" target=\"_blank\" rel=\"noopener\">Jared Green<\/a> from the management team; <a href=\"https:\/\/www.linkedin.com\/in\/namanjain94\/\" target=\"_blank\" rel=\"noopener\">Naman Jain<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/tholsapp\/\" target=\"_blank\" rel=\"noopener\">Troy Holsapple<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/sahilpatwardhan\/\" target=\"_blank\" rel=\"noopener\">Sahil Patwardhan<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/yuting-sun-a92a8227\/\" target=\"_blank\" rel=\"noopener\">Yuting Sun<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/aaron-dai-62278094\/\" target=\"_blank\" rel=\"noopener\">Aaron Dai<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/barry-warsaw\/\" target=\"_blank\" rel=\"noopener\">Barry Warsaw<\/a>, and <a href=\"https:\/\/www.linkedin.com\/in\/mkanat\/\" target=\"_blank\" rel=\"noopener\">Max Kanat-Alexander<\/a> from the engineering team (a special shoutout to <a href=\"https:\/\/www.linkedin.com\/in\/mkanat\/\" target=\"_blank\" rel=\"noopener\">Max<\/a> for his help with this article!); our UX partners, <a href=\"https:\/\/www.linkedin.com\/in\/arunram92\/\" target=\"_blank\" rel=\"noopener\">Arun Yegappan<\/a>, <a href=\"https:\/\/www.linkedin.com\/in\/kuan-ying-chen-8349b4a0\/\" target=\"_blank\" rel=\"noopener\">Kuan-Ying Chen<\/a>, and <a href=\"https:\/\/www.linkedin.com\/in\/kylemarksmith\/\" target=\"_blank\" rel=\"noopener\">Kyle Smith<\/a>; and finally, our Data Science partner, <a href=\"https:\/\/www.linkedin.com\/in\/yue-wu-b2466b184\/\" target=\"_blank\" rel=\"noopener\">Yue Wu<\/a>. Many teams across LinkedIn collaborated with us to roll out Real-Time Feedback\u2014we are thankful to them all for their support.<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n<p><a href=\"https:\/\/engineering.linkedin.com\/blog\/2021\/real-time-feedback-for-developer-tooling\" target=\"_blank\" rel=\"noopener\">Read More<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Over the last year, we have been using real-time feedback to evolve our tooling and provide a more productive experience for LinkedIn&#8217;s developers. It\u2019s helped us double our feedback participation, and more importantly, better tailor our recommendations and improvements.\u00a0 For any engineering organization looking to improve developer experiences, the following questions will provide a good&hellip; <a class=\"more-link\" href=\"https:\/\/fde.cat\/index.php\/2021\/01\/12\/how-linkedin-turned-to-real-time-feedback-for-developer-tooling\/\">Continue reading <span class=\"screen-reader-text\">How LinkedIn turned to real-time feedback for developer tooling<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[1,7],"tags":[],"class_list":["post-178","post","type-post","status-publish","format-standard","hentry","category-external","category-technology","entry"],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":683,"url":"https:\/\/fde.cat\/index.php\/2023\/02\/21\/what-is-the-secret-behind-increasing-salesforces-developer-velocity\/","url_meta":{"origin":178,"position":0},"title":"What is the Secret Behind Increasing Salesforce\u2019s Developer Velocity?","date":"February 21, 2023","format":false,"excerpt":"From retail to healthcare to IT and beyond, countless industries rely on software development to enhance business performance. However, to optimize software innovation and performance, companies must create enhanced environments that remove productivity blockers and deliver great experiences for developers. By empowering engineers to focus more on building new features\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":302,"url":"https:\/\/fde.cat\/index.php\/2021\/08\/31\/a-brief-history-of-rust-at-facebook\/","url_meta":{"origin":178,"position":1},"title":"A brief history of Rust at Facebook","date":"August 31, 2021","format":false,"excerpt":"Facebook is embracing Rust, one of the most loved and fastest-growing programming languages available today. In addition to bringing new talent to its Rust team, Facebook has announced that it is officially joining the nonprofit Rust Foundation. Alongside fellow members including Mozilla (the creators of Rust), AWS, Microsoft, and Google,\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":799,"url":"https:\/\/fde.cat\/index.php\/2023\/12\/05\/explaining-salesforces-large-scale-migration-to-git-how-we-enhanced-developer-productivity\/","url_meta":{"origin":178,"position":2},"title":"Explaining Salesforce\u2019s Large-Scale Migration to Git: How We Enhanced Developer Productivity","date":"December 5, 2023","format":false,"excerpt":"By Patrick Calahan and Scott Nyberg As new developer productivity technologies emerge, small and nimble enterprises with newer codebases swiftly embrace innovation. Conversely, larger organizations, rooted in larger and aging codebases, face obstacles replacing legacy technologies. Salesforce faced such a challenge with its primary Source Code Management (SCM) system. For\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":863,"url":"https:\/\/fde.cat\/index.php\/2024\/05\/07\/the-new-einstein-for-developers-streamlines-your-workflow-say-goodbye-to-time-consuming-tasks\/","url_meta":{"origin":178,"position":3},"title":"The New Einstein for Developers Streamlines Your Workflow: Say Goodbye to Time-Consuming Tasks","date":"May 7, 2024","format":false,"excerpt":"In our latest article for our \u201cEngineering Energizers\u201d Q&A series, we meet Christopher Williams, Vice President of Software Engineering. With over 17 years at Salesforce under his belt, Christopher today leads the development of Einstein for Developers. From eliminating time-consuming tasks to providing intelligent code completions, this groundbreaking tool empowers\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":467,"url":"https:\/\/fde.cat\/index.php\/2021\/09\/20\/sre-weekly-issue-288\/","url_meta":{"origin":178,"position":4},"title":"SRE Weekly Issue #288","date":"September 20, 2021","format":false,"excerpt":"View on sreweekly.com A message from our sponsor, StackHawk: Want to see what\u2019s new with automated security tooling? Tune in on September 30 to see how StackHawk and Semgrep are making it possible to embed security testing in CI\/CD. https:\/\/sthwk.com\/whats-new-webinar Articles Tammy Bryant Butow on SRE Apprentices Faced with a\u2026","rel":"","context":"In &quot;SRE&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":679,"url":"https:\/\/fde.cat\/index.php\/2023\/02\/14\/3-ways-salesforce-boosts-developer-productivity-on-hyperforce\/","url_meta":{"origin":178,"position":5},"title":"3 Ways Salesforce Boosts Developer Productivity on Hyperforce","date":"February 14, 2023","format":false,"excerpt":"In 2018, Salesforce began development of Hyperforce \u2014 a next-gen infrastructure platform that leverages public cloud to securely and swiftly deliver Salesforce software to customers worldwide. The platform development\u2019s team priorities were focused: build Hyperforce, get it sentient, and provide cloud-native tools that drive internal product developers\u2019 innovations, empowering them\u2026","rel":"","context":"In &quot;Technology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/178","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/comments?post=178"}],"version-history":[{"count":1,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/178\/revisions"}],"predecessor-version":[{"id":206,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/posts\/178\/revisions\/206"}],"wp:attachment":[{"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/media?parent=178"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/categories?post=178"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fde.cat\/index.php\/wp-json\/wp\/v2\/tags?post=178"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}