Research Area:  Machine Learning

Abstract:

In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost- and time-intensive. Thus, much work has been put into finding methods which allow a reduction in involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented, conversational, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then present the evaluation methods regarding that class.

Keywords:  

Author(s) Name:   methods for dialogue systems Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre & Mark Cieliebak

Publisher name:  Springer

DOI:  10.1007/s10462-020-09866-x

Volume Information:  volume 54, pages755–810 (2021)

Paper Link:   https://link.springer.com/article/10.1007/s10462-020-09866-x