LinkedIn details how it keeps new features from promoting inequality

In a newly published paper, LinkedIn describes the internal approach it’s taken to pairing product testing with economic metrics to lower users’ barriers to networking opportunities. The company claims that across thousands of A/B tests it analyzed, the approach reshaped research and design practices across teams, increasing understanding of the underlying causes of inequality.

As the coauthors of the paper point out, even products that appear to have been designed in a “responsible” or “fair” way, based on assumptions of parity, can drive a wedge between users. For instance, an app update that improves overall engagement but runs slowly on older devices might affect members across demographic categories in a manner that doesn’t appear in a typical A/B test, because traditional A/B testing looks at averages focused on an idealized “average user.”

LinkedIn’s solution taps experimentation platforms to analyze product changes, AI model revisions, and internal business decisions, with the goal of measuring the effect on real-world users. It complements the hundreds of A/B tests LinkedIn runs each day, which track thousands of variables from visual changes in apps to improvements in recommendation algorithms.

Starting last year, LinkedIn says it began tracking the inequality impact of the experiments on core business and member value metrics. It also created a special multidisciplinary team that discusses the impact of notable experiments, and that invites the owners of the products to a working session to discuss the impact.

Each experiment started with the questions:

• If feature A were to be rolled out, what would be the share of contributions from the top 1% of members, in terms of engagement and contributions?
• Would inequality impact go up or down between LinkedIn’s most and least engaged members?

From the year’s worth of data, LinkedIn found that seemingly metric-neutral interventions aren’t often neutral for everyone; while metrics might not be affected by, say, a back-end infrastructure change, some members are. It also found that notifications have a strong impact on inequality of engagement, and that strategically batching notifications for highly engaged members results in a qualitatively better user experience for that group of users.

LinkedIn also reports that a rich onboarding experience for new members has a positive impact on both average engagement and quality of engagement, since it helps members at the highest risk of dropping off. And speed, availability, and low-bandwidth optimized apps turned out to matter a great deal to inclusiveness, because members who only have access to slower devices and connections may experience other inequalities.

Going forward, LinkedIn says it hopes to collaborate with experts across domains to look for new ideas outside of the typical best practices in the technology industry.

“Combining measures of inequality and A/B testing provides us two distinct advantages,” wrote LinkedIn in a blog post. “First, instead of only measuring inequality impact, we can also trace it back to its causes: a specific set of features and product decisions … Second, unlike classical algorithmic fairness approaches, it helps us identify features that increase inequality impact without having to rely only on explicitly protected categories … We hope that increased understanding of the underlying causes of inequality can lead to similar approaches to ethical product design across several different industries.”