Multimodal Knowledge Graphs for Legal Question Answering and Reasoning

Context & Background

Knowledge Graphs (KGs) represent semantic entities and their relationships. However, most legal and law enforcement KGs are built entirely from text data, neglecting visual evidence (CCTV footage, crime scene images) and audio records. A multimodal knowledge graph is essential to enable intelligent reasoning and complete case comprehension in the legal domain.

Problems to be Addressed

1. Extracting structured RDF triples (subject, predicate, object) from non-textual modalities is highly complex.
2. Resolving contradictions or corroborations between textual records (like First Information Reports) and scene graphs extracted from video/images is currently not handled by existing systems.

Aims and Objectives

1. Build an extraction pipeline to create scene graphs from images and video footage.
2. Integrate these visual scene graphs into text-based legal KGs using Semantic Web standards (RDF, OWL).
3. Design a reasoning engine that identifies complementary, corroborative, or contradictory links to resolve cases.

Methodology

Using Deep Learning for computer vision, object detection and spatial relationship models extract structured triples from visual crime data. An ontology aligning text-based legal terms with visual scene graphs is developed. Ontological reasoners apply logical rules to cross-check text narratives with visual evidence, highlighting discrepancies.

Expected Outcomes

A reasoning framework for law enforcement agencies, a multimodal database connecting crime scenes with FIR text, and a query dashboard enabling investigators to search and match visual evidence.