A multimodal corpus of rapid dialogue games

Abstract

This paper presents a multimodal corpus of spoken human-human dialogues collected as participants played a series of Rapid DialogueGames (RDGs). The corpus consists of a collection of about 11 hours of spoken audio, video, and Microsoft Kinect data taken from 384game interactions (dialogues). The games used for collecting the corpus required participants to give verbal descriptions of linguisticexpressions or visual images and were specifically designed to engage players in a fast-paced conversation under time pressure. As aresult, the corpus contains many examples of participants attempting to communicate quickly in specific game situations, and it alsoincludes a variety of spontaneous conversational phenomena such as hesitations, filled pauses, overlapping speech, and low-latencyresponses. The corpus has been created to facilitate research in incremental speech processing for spoken dialogue systems. Potentially,the corpus could be used in several areas of speech and language research, including speech recognition, natural language understanding,natural language generation, and dialogue management.