MIT Study: Considering Three Options Reveals Hidden Preference Patterns

Random utility models (RUMs) have been a cornerstone of predicting human preferences for nearly a century. But a new MIT study suggests we've been missing crucial information by relying too heavily on pairwise comparisons.

The power of three: A key to unlocking preference correlations

The research, presented at the International Conference on Learning Representations, demonstrates that considering three options at a time can reveal important correlations in preferences that two-way comparisons miss entirely.

Constantinos Daskalakis, an MIT professor involved in the study, explains the limitation of pairwise comparisons: 'With this way of assessing people's preferences, looking at just two things at a time, it is impossible to find correlations between the numerous choices.'

These correlations can be critical. For example:

• A voter favoring gun control might also support government-sponsored child care
• A fan of independent films might prefer foreign movies over Hollywood blockbusters

Failing to capture these relationships can lead to inaccurate predictions and potentially dissatisfied users.

Why this matters: From streaming to AI alignment

The implications of this research extend far beyond academic interest:

1. Digital Platforms: Streaming services and recommendation systems could improve their ability to suggest relevant content, potentially increasing user satisfaction and retention.

2. AI Alignment: As Daskalakis notes, 'RUMs play a central role in the commercial viability and usefulness of large language models.' Improved preference modeling could lead to better-aligned AI systems.

3. Public Policy: RUMs are used in predicting responses to infrastructure changes or budget allocations. More accurate models could lead to better-informed decision-making.

The method: Combining three-way rankings

The researchers proved that while pairwise comparisons alone cannot reveal correlations, asking people to rank three alternatives can unlock this information. Sobhan Mohammadpour, an MIT PhD student involved in the study, explains:

'You would get a bunch of people to rank three items. You could then utilize the method we developed for merging those individual results into one big model that can provide us with the big picture.'

Importantly, the team demonstrated that efficient algorithms exist for this purpose, and the number of required experiments doesn't grow exponentially with the number of items being evaluated.

Limitations and future work

While this research represents a significant advance, several questions remain:

1. Real-world implementation: How easily can existing systems transition to incorporate three-way comparisons?
2. User experience: Will users find ranking three options significantly more burdensome than pairwise comparisons?
3. Data quality: As with any preference data, there's a risk of noise or inconsistency in user responses.

The researchers emphasize that building and refining utility models will remain an active area of study. As Daskalakis puts it, 'You have to keep improving and updating your model in an iterative process until, hopefully, you can make good predictions.'

The bigger picture: A fundamental shift in preference modeling

This study challenges a nearly century-old assumption in the field of psychometrics and random utility modeling. By demonstrating the limitations of pairwise comparisons and offering a practical alternative, the MIT team has opened up new possibilities for more accurate and nuanced preference prediction across a wide range of applications.

As digital platforms, AI systems, and policymakers increasingly rely on understanding and predicting human preferences, this research provides a valuable new tool for capturing the complex, correlated nature of human decision-making.