Computing recommendations from free-form text

While searching for consumer goods, users frequently ask for suggestions from their peers by writing short free-form textual requests. For example, when searching for movies users may ask for “Drama movies with a mind-bending story and a surprise ending, such as Fight Club” in one of the many online discussion boards. Despite the recent developments in large language models (LLMs) and natural language processing (NLP), modern recommender systems still struggle to process such requests. Therefore, in this paper we evaluate several approaches for annotating structured information from such short, free-form natural language user texts to calculate recommendations. We set up this evaluation as a two phase processes including (a) identification of the best NLP approach to identify key elements of users’ requests, and (b) assessment of the quality of recommendations computed with such elements. For our evaluation, we use a gold-standard reddit movie recommendation dataset consisting of annotations, manually created by crowdworkers who extracted keywords, actor names and movie titles. Using this dataset we evaluate a collection of more than 30 NLP and five recommender approaches. In addition, we perform an ablation study to assess relative annotation importance for movie recommendations. We find that domain-specific deep learning models, trained on a subset of data as well as embedding-based recommendation approaches are able to match the recommendation performance of recommendations computed from manual annotations. These promising results warrant further investigation in automatic processing of short free-form texts for computation of recommendations. Specifically, we provide insights into which NLP models and configurations work best for automatically annotating free text to compute (movie) recommendations, hence substantially reducing the search space for combinations of NLP and recommendation algorithms in the movie and potentially other domains.