Rebo Junior: Analysis of Dialogue Structure Quality for a Reflection Guidance Chatbot

Conversational user interfaces open up new opportunities for reflection guidance. This paper presents a computer-mediated dialogue structure for reflecting on learning tasks, Rebo Junior, and its evaluation in the context of apprenticeship training. We answer three research questions. Firstly, how apprentices react to Rebo Junior; secondly, whether Rebo Junior’s dialogue structure is
apt to lead apprentices in reflective conversations; and thirdly, how user engagement with Rebo Junior develops over time. During three months, 17 apprentices led 153 reflective conversations with Rebo Junior in the context of a training workshop, 117 in phase one and 36 in phase two of the study (five to thirteen
interactions per apprentice). We coded interactions manually for coherence, level of reflectivity, and user engagement. Our results show that apprentices react well to the intervention and that the dialogue structure is successful in leading apprentices through different levels of reflection (114 out of 153 showed observable reflection on the learning experience; 133 out of 153 expressed learning or planned behaviour change for future tasks). Furthermore, the interactions between the apprentices and Rebo Junior result in coherent conversations (149 out of 153 were coherent). Contrary to expectations, engagement did not decrease
over time in either phase. With the present paper, we therefore publish a dialogue structure for reflecting on learning tasks that has worked extremely well despite no adaptivity in the conversational interface. Overall, we interpret the results of
our work as underscoring the importance of dialogue structure quality in conversational agents.