A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations

Dominik Kowald; Gregor Mayr; Markus Schedl; Elisabeth Lex

doi:10.1007/978-3-031-37249-0_1

A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations

Dominik Kowald^*, Gregor Mayr, Markus Schedl, Elisabeth Lex

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Abstract

Recent research has suggested different metrics to measure the inconsistency of recommendation performance, including the accuracy difference between user groups, miscalibration, and popularity lift. However, a study that relates miscalibration and popularity lift to recommendation accuracy across different user groups is still missing. Additionally, it is unclear if particular genres contribute to the emergence of inconsistency in recommendation performance across user groups. In this paper, we present an analysis of these three aspects of five well-known recommendation algorithms for user groups that differ in their preference for popular content. Additionally, we study how different genres affect the inconsistency of recommendation performance, and how this is aligned with the popularity of the genres. Using data from Last.fm, MovieLens, and MyAnimeList, we present two key findings. First, we find that users with little interest in popular content receive the worst recommendation accuracy, and that this is aligned with miscalibration and popularity lift. Second, our experiments show that particular genres contribute to a different extent to the inconsistency of recommendation performance, especially in terms of miscalibration in the case of the MyAnimeList dataset.

Original language	English
Title of host publication	Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers
Editors	Ludovico Boratto, Mirko Marras, Stefano Faralli, Giovanni Stilo
Publisher	Springer Science and Business Media Deutschland GmbH
Number of pages	16
ISBN (Print)	9783031372483
DOIs	https://doi.org/10.1007/978-3-031-37249-0_1
Publication status	Published - 2023
Event	4th International Workshop on Algorithmic Bias in Search and Recommendation, part of the 45th European Conference on Information Retrieval: BIAS 2023 - Dublin, Ireland Duration: 2 Apr 2023 → 2 Apr 2023

Publication series

Name	Communications in Computer and Information Science
Volume	1840 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	4th International Workshop on Algorithmic Bias in Search and Recommendation, part of the 45th European Conference on Information Retrieval
Abbreviated title	BIAS 2023/ECIR 2023
Country/Territory	Ireland
City	Dublin
Period	2/04/23 → 2/04/23

Keywords

Accuracy
Miscalibration
Popularity bias
Popularity lift
Recommendation inconsistency
Recommender systems

ASJC Scopus subject areas

General Computer Science
General Mathematics

Access to Document

10.1007/978-3-031-37249-0_1

Cite this

Kowald, D., Mayr, G., Schedl, M., & Lex, E. (2023). A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations. In L. Boratto, M. Marras, S. Faralli, & G. Stilo (Eds.), Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers (Communications in Computer and Information Science; Vol. 1840 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-37249-0_1

A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations. / Kowald, Dominik; Mayr, Gregor; Schedl, Markus et al.
Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers. ed. / Ludovico Boratto; Mirko Marras; Stefano Faralli; Giovanni Stilo. Springer Science and Business Media Deutschland GmbH, 2023. (Communications in Computer and Information Science; Vol. 1840 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Kowald, D, Mayr, G, Schedl, M & Lex, E 2023, A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations. in L Boratto, M Marras, S Faralli & G Stilo (eds), Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers. Communications in Computer and Information Science, vol. 1840 CCIS, Springer Science and Business Media Deutschland GmbH, 4th International Workshop on Algorithmic Bias in Search and Recommendation, part of the 45th European Conference on Information Retrieval, Dublin, Ireland, 2/04/23. https://doi.org/10.1007/978-3-031-37249-0_1

Kowald D, Mayr G, Schedl M, Lex E. A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations. In Boratto L, Marras M, Faralli S, Stilo G, editors, Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2023. (Communications in Computer and Information Science). doi: 10.1007/978-3-031-37249-0_1

Kowald, Dominik ; Mayr, Gregor ; Schedl, Markus et al. / A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations. Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers. editor / Ludovico Boratto ; Mirko Marras ; Stefano Faralli ; Giovanni Stilo. Springer Science and Business Media Deutschland GmbH, 2023. (Communications in Computer and Information Science).

@inproceedings{f192f02adb894a0a9b85456987fc517d,

title = "A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations",

abstract = "Recent research has suggested different metrics to measure the inconsistency of recommendation performance, including the accuracy difference between user groups, miscalibration, and popularity lift. However, a study that relates miscalibration and popularity lift to recommendation accuracy across different user groups is still missing. Additionally, it is unclear if particular genres contribute to the emergence of inconsistency in recommendation performance across user groups. In this paper, we present an analysis of these three aspects of five well-known recommendation algorithms for user groups that differ in their preference for popular content. Additionally, we study how different genres affect the inconsistency of recommendation performance, and how this is aligned with the popularity of the genres. Using data from Last.fm, MovieLens, and MyAnimeList, we present two key findings. First, we find that users with little interest in popular content receive the worst recommendation accuracy, and that this is aligned with miscalibration and popularity lift. Second, our experiments show that particular genres contribute to a different extent to the inconsistency of recommendation performance, especially in terms of miscalibration in the case of the MyAnimeList dataset.",

keywords = "Accuracy, Miscalibration, Popularity bias, Popularity lift, Recommendation inconsistency, Recommender systems",

author = "Dominik Kowald and Gregor Mayr and Markus Schedl and Elisabeth Lex",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 4th International Workshop on Algorithmic Bias in Search and Recommendation, part of the 45th European Conference on Information Retrieval : BIAS 2023, BIAS 2023/ECIR 2023 ; Conference date: 02-04-2023 Through 02-04-2023",

year = "2023",

doi = "10.1007/978-3-031-37249-0_1",

language = "English",

isbn = "9783031372483",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

editor = "Ludovico Boratto and Mirko Marras and Stefano Faralli and Giovanni Stilo",

booktitle = "Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers",

address = "Germany",

}

TY - GEN

T1 - A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations

AU - Kowald, Dominik

AU - Mayr, Gregor

AU - Schedl, Markus

AU - Lex, Elisabeth

PY - 2023

Y1 - 2023

N2 - Recent research has suggested different metrics to measure the inconsistency of recommendation performance, including the accuracy difference between user groups, miscalibration, and popularity lift. However, a study that relates miscalibration and popularity lift to recommendation accuracy across different user groups is still missing. Additionally, it is unclear if particular genres contribute to the emergence of inconsistency in recommendation performance across user groups. In this paper, we present an analysis of these three aspects of five well-known recommendation algorithms for user groups that differ in their preference for popular content. Additionally, we study how different genres affect the inconsistency of recommendation performance, and how this is aligned with the popularity of the genres. Using data from Last.fm, MovieLens, and MyAnimeList, we present two key findings. First, we find that users with little interest in popular content receive the worst recommendation accuracy, and that this is aligned with miscalibration and popularity lift. Second, our experiments show that particular genres contribute to a different extent to the inconsistency of recommendation performance, especially in terms of miscalibration in the case of the MyAnimeList dataset.

AB - Recent research has suggested different metrics to measure the inconsistency of recommendation performance, including the accuracy difference between user groups, miscalibration, and popularity lift. However, a study that relates miscalibration and popularity lift to recommendation accuracy across different user groups is still missing. Additionally, it is unclear if particular genres contribute to the emergence of inconsistency in recommendation performance across user groups. In this paper, we present an analysis of these three aspects of five well-known recommendation algorithms for user groups that differ in their preference for popular content. Additionally, we study how different genres affect the inconsistency of recommendation performance, and how this is aligned with the popularity of the genres. Using data from Last.fm, MovieLens, and MyAnimeList, we present two key findings. First, we find that users with little interest in popular content receive the worst recommendation accuracy, and that this is aligned with miscalibration and popularity lift. Second, our experiments show that particular genres contribute to a different extent to the inconsistency of recommendation performance, especially in terms of miscalibration in the case of the MyAnimeList dataset.

KW - Accuracy

KW - Miscalibration

KW - Popularity bias

KW - Popularity lift

KW - Recommendation inconsistency

KW - Recommender systems

UR - http://www.scopus.com/inward/record.url?scp=85169072189&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-37249-0_1

DO - 10.1007/978-3-031-37249-0_1

M3 - Conference paper

AN - SCOPUS:85169072189

SN - 9783031372483

T3 - Communications in Computer and Information Science

BT - Advances in Bias and Fairness in Information Retrieval - 4th International Workshop, BIAS 2023, Revised Selected Papers

A2 - Boratto, Ludovico

A2 - Marras, Mirko

A2 - Faralli, Stefano

A2 - Stilo, Giovanni

PB - Springer Science and Business Media Deutschland GmbH

T2 - 4th International Workshop on Algorithmic Bias in Search and Recommendation, part of the 45th European Conference on Information Retrieval

Y2 - 2 April 2023 through 2 April 2023

ER -