What’s in a Review: Discrepancies Between Expert and Amateur Reviews of Video Games on Metacritic

Tiago Filipe Teixeira dos Santos, Florian Lemmerich, Markus Strohmaier, Denis Helic

Research output: Contribution to journalConference articlepeer-review


As video game press (“experts”) and casual gamers (“amateurs”) have different motivations when writing video game reviews, discrepancies in their reviews may arise. To study such potential discrepancies, we conduct a large-scale investigation of more than 1 million reviews on the Metacritic review platform. In particular, we assess the existence and nature of discrepancies in video game appraisal by experts and amateurs, and how they manifest in ratings, over time, and in review language. Leveraging these insights, we explore the predictive power of early expert vs. amateur reviews in forecasting video game reputation in the short- and long-term. We find that amateurs, in contrast to experts, give more polarized ratings of video games, rate games surprisingly long after game release, and are positively biased towards older games. On a textual level, we observe that experts write rather complex, less readable texts than amateurs, whose reviews are more emotionally charged. While in the short-term amateur reviews are remarkably predictive of game reputation among other amateurs (achieving 91% ROC AUC in a binary classification), both expert and amateur reviews are equally well suited for long-term predictions. Overall, our work is the first large-scale comparative study of video game reviewing behavior, with practical implications for amateurs when deciding which games to play, and for game developers when planning which games to design, develop, or continuously support. More broadly, our work contributes to the discussion of wisdom of the few vs. wisdom of the crowds, as we uncover the limits of experts in capturing the views of amateurs in the particular context of video game reviews. © 2019 Copyright held by the owner/author(s)
Original languageEnglish
Article number140
Number of pages22
JournalProceedings of the ACM on Human-Computer Interaction
Issue numberCSCW
Publication statusPublished - Nov 2019
Event22nd ACM Conference on Computer-Supported Cooperative Work and Social Computing: CSCW 2019 - Austin, United States
Duration: 9 Nov 201913 Nov 2019


Dive into the research topics of 'What’s in a Review: Discrepancies Between Expert and Amateur Reviews of Video Games on Metacritic'. Together they form a unique fingerprint.

Cite this