Evaluating the use of semantics in collaborative recommender systems: A user study

In this paper we report on a pilot user study aimed at evaluating two aspects of recommender systems that have not been the aim of previous user studies in the field. Firstly, item semantics may be incorporated into a collaborative recommender system and we wish to measure the effect on user satisfaction. Secondly, we would like to evaluate different approaches to collecting ratings from users: the ratings that are used to seed their profile with a collaborative filtering system. Key indications from the study are: users do prefer recommendations generated by semantically enhanced recommender systems; the user's satisfaction with a recommendation set is different from the sum of their satisfaction with the individual items with the set and the approach to collecting item ratings from the user should be tailored to the algorithm being used. Finally, recommender systems within the movie domain seem to be more useful for "movie buffs" rather than the "average movie watcher" for whom recommending simply the most popular movies seems to be most appropriate.