Leveraging Multi-Modality in Understanding and Summarizing Task-Oriented Dialogue Systems

Context & Background

Conversational agents are increasingly used in customer support and mental health triage. However, task-oriented dialogue systems face severe issues with 'topic drift' (getting off track) and cannot understand user emotions, leading to poor summaries and failed interactions.

Problems to be Addressed

Most dialog summary datasets are text-only, ignoring visual and speech features. Traditional summarization models fail to identify crucial dialog turns, leading to incorrect or incomplete case summaries.

Aims and Objectives

1. Formulate emotion-aware and intent-aware joint dialog classifiers.
2. Integrate text, image, and voice inputs into dialogue models.
3. Design topic-aware abstractive summarization models.

Methodology

The project builds multi-modal attention networks that process textual utterances, voice inflection, and visual facial inputs. Contrastive learning aligns these modalities. Summarization models use attention to highlight key dialogue segments, generating concise, drift-free conversation summaries.

Expected Outcomes

A multimodal dialogue summarization framework, a benchmark dataset of emotional dialogues, and publications in top NLP conferences (ACL, EMNLP).