Research Area: Machine Learning
Conversational Recommender Systems (CRS) have received increased interest in recent years due to advances in natural language processing and the wider use of voice controlled smart assistants. One technical approach to build such systems is to learn, in an end-to-end way, from recorded dialogs between humans. Recent proposals rely on neural architectures for learning such models. These models are often evaluated both with the help of computational metrics and with the help of human annotators. In the latter case, the task of human judges may consist of assessing the utterances generated by the models, e.g., in terms of their consistency with previous dialog utterances. However, such assessments may tell us not enough about the true usefulness of the resulting recommendation model, in particular when the judges only assess how good one model is compared to another. In this work, we therefore analyze the utterances generated by two recent end-to-end learning approaches for CRS on an absolute scale. Our initial analyses reveals that for each system about one third of the system utterances are not meaningful in the given context and would probably lead to a broken conversation. Furthermore, about less than two third of the recommendations were considered to be meaningful. Interestingly, none of the two systems “generated” utterances, as almost all system responses were already present in the training data. Overall, our works shows that (i) current approaches that are published at high-quality research outlets may have severe limitations regarding their usability in practice and (ii) our academic evaluation approaches for CRS should be reconsidered.
Keywords:
Conversational Recommender Systems
Evaluation
Author(s) Name: Dietmar Jannach, Ahtsham Manzoor
Journal name: IntRS Workshop at ACM RecSys
Conferrence name:
Publisher name: IntRS@RecSys
DOI:
Volume Information: Volume 2682
