Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction Article Swipe

MAKE IMPACT

Wang-Cheng Kang , Jianmo Ni , Nikhil Mehta , Maheswaran Sathiamoorthy , Lichan Hong , Ed H. , Derek Zhiyuan Cheng · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2305.06474

Large Language Models (LLMs) have demonstrated exceptional capabilities in generalizing to new tasks in a zero-shot or few-shot manner. However, the extent to which LLMs can comprehend user preferences based on their previous behavior remains an emerging and still unclear research question. Traditionally, Collaborative Filtering (CF) has been the most effective method for these tasks, predominantly relying on the extensive volume of rating data. In contrast, LLMs typically demand considerably less data while maintaining an exhaustive world knowledge about each item, such as movies or products. In this paper, we conduct a thorough examination of both CF and LLMs within the classic task of user rating prediction, which involves predicting a user's rating for a candidate item based on their past ratings. We investigate various LLMs in different sizes, ranging from 250M to 540B parameters and evaluate their performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct comprehensive analysis to compare between LLMs and strong CF methods, and find that zero-shot LLMs lag behind traditional recommender models that have the access to user interaction data, indicating the importance of user interaction data. However, through fine-tuning, LLMs achieve comparable or even better performance with only a small fraction of the training data, demonstrating their potential through data efficiency.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2305.06474

PDF

https://arxiv.org/pdf/2305.06474

OA Status

green

Cited By

24

Related Works

10

OpenAlex ID

https://openalex.org/W4376311940

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4376311940

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2305.06474

Digital Object Identifier

Title

Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-05-10

Full publication date if available

Authors

Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed H., Derek Zhiyuan Cheng

List of authors in order

Landing page

https://arxiv.org/abs/2305.06474

Publisher landing page

PDF URL

https://arxiv.org/pdf/2305.06474

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2305.06474

Direct OA link when available

Concepts

Task (project management), Shot (pellet), Computer science, Economics, Management, Organic chemistry, Chemistry

Top concepts (fields/topics) attached by OpenAlex

Cited by

24

Total citation count in OpenAlex

Citations by year (recent)

2025: 5, 2024: 10, 2023: 9

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload