AI-Based Interactive and Multimodal 2D Illustrations in Mixed Reality

Context & Background

Traditional textbooks rely on static 2D illustrations, which can make physical and mechanical systems hard for students to visualize. Immersive Mixed Reality (MR) technologies offer an opportunity to convert static drawings into interactive, animated simulations, promoting engagement and learning.

Problems to be Addressed

Creating 3D interactive content currently requires specialized designers and programmers, making it expensive and slow. There are no tools that allow teachers or content authors to easily generate MR animations directly from 2D textbook pages.

Aims and Objectives

1. Build an algorithmic framework for interactive educational illustrations in MR.
2. Develop AI tools to automate content authoring workflows.
3. Deploy open-source software libraries for interactive textbook design.

Methodology

The project uses computational geometry and AI to infer 3D models and context from 2D drawings. Initial prototypes are developed using WebXR (HTML5) for accessibility. The tool integrates with Microsoft Hololens MR headsets, enabling students to interact with animated physics and biology models in real-space.

Expected Outcomes

An open-source MR authoring software library, interactive NCERT textbook illustrations, and user testing reviews in schools.