Abstract
Popularity bias in recommendation lists refers to over-representation of popular content and is a challenge for many recommendation algorithms. Previous research has suggested several offline metrics to quantify popularity bias, which commonly relate the popularity of items in users' recommendation lists to the popularity of items in their interaction history. Discrepancies between these two factors are referred to as popularity miscalibration. While popularity metrics provide a straightforward and well-defined means to measure popularity bias, it is unknown whether they actually reflect users' perception of popularity bias. To address this research gap, we conduct a crowd-sourced user study on Prolific, involving 56 participants, to (1) investigate whether the level of perceived popularity miscalibration differs between common recommendation algorithms, (2) assess the correlation between perceived popularity miscalibration and its corresponding quantification according to a common offline metric. We conduct our study in a well-defined and important domain, namely music recommendation using the standardized LFM-2b dataset, and quantify popularity miscalibration of five recommendation algorithms by utilizing Jensen-Shannon distance (JSD). Challenging the findings of previous studies, we observe that users generally do perceive significant differences in terms of popularity bias between algorithms if this bias is framed as popularity miscalibration. In addition, JSD correlates moderately with users' perception of popularity, but not with their perception of unpopularity.
Original language | English |
---|---|
Title of host publication | SIGIR 2023 - Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval |
Publisher | Association of Computing Machinery |
Pages | 1889-1893 |
Number of pages | 5 |
ISBN (Electronic) | 9781450394086 |
DOIs | |
Publication status | Published - 19 Jul 2023 |
Event | 46th International ACM SIGIR Conference on Research and Development in Information Retrieval: SIGIR 2023 - Taipei, Taiwan, Province of China Duration: 23 Jul 2023 → 27 Jul 2023 |
Conference
Conference | 46th International ACM SIGIR Conference on Research and Development in Information Retrieval |
---|---|
Abbreviated title | SIGIR 2023 |
Country/Territory | Taiwan, Province of China |
City | Taipei |
Period | 23/07/23 → 27/07/23 |
Keywords
- ecological validity
- metrics
- miscalibration
- music recommendation
- popularity bias
- popularity calibration
- recommender systems
ASJC Scopus subject areas
- Computer Graphics and Computer-Aided Design
- Information Systems
- Software