Research Area:  Machine Learning
Conversational recommender systems aim to interactively support online users in their information search and decision-making processes in an intuitive way. With the latest advances in voice-controlled devices, natural language processing, and AI in general, such systems received increased attention in recent years. Technically, conversational recommenders are usually complex multi-component applications and often consist of multiple machine learning models and a natural language user interface. Evaluating such a complex system in a holistic way can therefore be challenging, as it requires (i) the assessment of the quality of the different learning components, and (ii) the quality perception of the system as a whole by users. Thus, a mixed methods approach is often required, which may combine objective (computational) and subjective (perception-oriented) evaluation techniques. In this paper, we review common evaluation approaches for conversational recommender systems, identify possible limitations, and outline future directions towards more holistic evaluation practices.
Keywords:  
Conversational recommender systems
Dialogue systems
Interactive systems
Evaluation
Author(s) Name:  Dietmar Jannach
Journal name:  Artificial Intelligence Review
Conferrence name:  
Publisher name:  Springer
DOI:  10.1007/s10462-022-10229-x
Volume Information:   56
Paper Link:   https://link.springer.com/article/10.1007/s10462-022-10229-x