Gesture-Driven Storytelling: Designing interactive experiences for cultural heritage interpretation in blended spaces

Storytelling has long been key to cultural heritage interpretation by encouraging participation and engagement. However, traditional methods often lack immersive qualities. The convergence of extended reality, interaction design, and storytelling offers transformative potential for cultural heritage interpretation in blended spaces. Most literature concerned with gesture-based interactions focuses on pre-defined gestures and user experiences. This study explores how gesture-driven storytelling supports interaction design for cultural heritage interpretation in blended spaces. A design workflow was developed to guide this process, comprising three stages: storyboarding, gesture coding, and prototype development. Here, storyboarding visualizes interaction concepts, gesture coding creates custom gestures, and prototype development integrates them in a creative coding environment. A pilot workshop was conducted where participants followed this workflow to create gesture-driven storytelling prototypes. Using prewritten code snippets, MediaPipe’s pre-trained machine learning models for hand tracking, and a pre-coded Seljuk architectural ornament, participants developed interactive prototypes. Following the outcomes of this workshop, this study discusses the potential and challenges of gesture-driven storytelling for cultural heritage interpretation. This study contributes to human-computer interaction design and cultural heritage by demonstrating the potential of XR to transform cultural heritage interpretation and providing a reusable workflow that enables customized storytelling experiences.