Research Area: Machine Learning
Conversational recommender systems (CRS) are software agents that support users in their decision-making process in an interactive way. While such systems were traditionally mostly manually engineered, recent works increasingly rely on machine learning models that are trained on larger corpora of recorded recommendation dialogues between humans. One promise of such end-to-end learning approaches therefore is that they avoid the knowledge-engineering bottlenecks of traditional systems. Recent empirical evaluations of such learning-based systems sometimes demonstrate continuous progress relative to previous systems. Therefore, it may not be entirely clear how useable these systems are on an absolute scale. To address this research question, we evaluated two recent end-to-end learning approaches presented at top-tier scientific conferences with the help of human judges. A first study showed that in both investigated systems about one third of the system responses were not considered meaningful in the given dialogue context, which questions the applicability of these systems in practice. In a second study, we benchmarked the two systems against a trivial rule-based approach, again with human judges. In this second study, the participants considered the quality of the responses of the rule-based approach significantly better on average than those of the learning-based systems. Overall, besides pointing to open challenges of state-of-the-art learning-based approaches, our studies indicate that we must improve our evaluation methodology for CRS to ensure progress in this field.
Keywords:
Conversational recommender systems
Evaluation
End-to-end learning
one-shot recommendations
Author(s) Name: Ahtsham Manzoor, Dietmar Jannach
Journal name: Computers in Human Behavior Reports
Conferrence name:
Publisher name: Elsevier
DOI: 10.1016/j.chbr.2021.100139
Volume Information: Volume 4
