{"id":177,"date":"2021-01-21T20:01:00","date_gmt":"2021-01-21T20:01:00","guid":{"rendered":"https:\/\/fde.cat\/index.php\/2021\/01\/21\/budget-split-testing-a-trustworthy-and-powerful-approach-to-marketplace-a-b-testing\/"},"modified":"2021-02-02T13:46:58","modified_gmt":"2021-02-02T13:46:58","slug":"budget-split-testing-a-trustworthy-and-powerful-approach-to-marketplace-a-b-testing","status":"publish","type":"post","link":"https:\/\/fde.cat\/index.php\/2021\/01\/21\/budget-split-testing-a-trustworthy-and-powerful-approach-to-marketplace-a-b-testing\/","title":{"rendered":"Budget-split testing: A trustworthy and powerful approach to marketplace A\/B testing"},"content":{"rendered":"
\n
\n <\/a>\n <\/div>\n
\n
\n

Co-authors: Min Liu<\/a>, Vangelis Dimopoulos<\/a>, Elise Georis<\/a>, Jialiang Mao<\/a>, Di Luo<\/a>, and Kang Kang<\/a><\/i><\/p>\n

The LinkedIn ecosystem drives member and customer value through a series of marketplaces (e.g., the ads marketplace<\/a>, the talent marketplace<\/a>, etc.). We maximize that value by making data-informed product decisions via A\/B testing<\/a>. Traditional A\/B tests on our marketplaces, however, are often statistically biased and under-powered. To mitigate this, we developed \u201cbudget-split\u201d testing, which provides more trustworthy and powerful marketplace A\/B testing. Read on to learn about the problem, solution, and successful results, using the ads marketplace<\/a> as a running example. For more technical details, please refer to the paper \u201cTrustworthy Online Marketplace Experimentation with Budget-split Design<\/a>.\u201d<\/p>\n

Problems with marketplace A\/B testing<\/h2>\n

To add some important context, modern online ad marketplaces use auction-based models<\/a> for ad assignment. Advertisers set an objective, an audience, a campaign budget, and a bidding strategy to each ad campaign. Each \u201cresult\u201d (member click, view, etc., depending on the objective) utilizes a portion of the overall campaign budget, for a set duration, until the campaign ends or there is no more budget available. The maximum revenue generated by a campaign cannot exceed its set budget.<\/p>\n

When running A\/B tests on the ads marketplace, we noticed two types of problems:<\/p>\n

    \n
  1. \n

    When testing a new ad feature, we\u2019d often see a strong metric impact in our experiment, but wouldn\u2019t observe the same level of impact when launched to the entire marketplace.<\/p>\n<\/li>\n

  2. \n

    Many tests required an unacceptably long time to achieve statistically significant results.\u00a0<\/p>\n<\/li>\n<\/ol>\n

    The first problem exemplified cannibalization bias, while the second stemmed from insufficient statistical power.<\/p>\n

    Cannibalization bias
    <\/b>We can illustrate cannibalization bias with a hypothetical example (note: real world manifestations of this bias are less extreme forms of this hypothetical). Suppose that we want to test how a new ad feature (e.g., improving the match between ads and members) impacts ad impressions and revenue. Prior to our experiment, let\u2019s say all ad campaigns were spending 100% of their budgets (i.e., no new feature can increase ads revenue further). If we test our new feature in a traditional A\/B test and observe increases in the number of ad impressions, the test would also show a corresponding increase in revenue for the treatment group. Once we launch the feature to the entire marketplace, however, we won\u2019t see that same increase in revenue because (remember) all campaigns were already spending 100% of their budgets. So why did our A\/B test lead us to the wrong conclusion?<\/p>\n<\/p><\/div>\n<\/p><\/div>\n<\/div>\n

    \n
    \n <\/a>\n <\/div>\n