1. Overview
  2. 1The experience
  3. 2Where it started
  4. 3How it works
  5. 4What I learned and what’s next

Plot Twist

Step into a story as the main character, and decide how it unfolds by acting out your choices.

Type
Immersive · TouchDesigner
Role
Concept, story engine, gesture interaction, music system and the TouchDesigner build
Duration
1 week · CIID course Designing with AI
Teachers
Kay van den Aker, Lesley Lock
Team
With Naman Jain (particle-shadow visuals)

Overview

Plot Twist is an immersive, choose-your-own-adventure story for the walls of CIID’s thousand-year-old home in Bergamo. It’s told in the second person, and your body enacts the main character’s actions to unfold the story.

1The experience

Slink into the shadows or raise a sword to fight?

A pulsing orb is projected on the wall. As you walk toward it, medieval music grows louder. Touch it, the music swells, and the story begins, with you as the main character: you’re standing in a banquet hall…

A shadow-like projection of you appears on the ancient stone. It shifts color to match the mood of the story, and the music keeps changing underneath to support the tone.

The orb
Shadow shifts color upon action

When the story reaches a branching point, it asks you to choose. Do you raise your sword arm to prepare for battle, or crouch into the shadows? You just do it. There’s nothing to explain, and people were really enthralled by it. A lot of them came away saying some version of “oh, I can’t believe I did that,” and the surprising turns in the story caused real delight.

2Where it started

What if these ancient walls could talk?

Our brief was to design an AI experience off the screen: no text or speech, just an embodied experience that connects people to the space around them. Our process was very site-specific. We explored CIID’s building, intrigued by odd corners and liminal spaces like the stairwell, and bodystormed scenarios throughout the space.

Bodystorming in the stairwell
Bodystorming at the stone wall

Ultimately, we were captivated by the fact that CIID is housed in a thousand-year-old building, connected to the neighboring monastery. Those walls have witnessed countless stories over the centuries. What if these walls could talk? How might we let someone today step into a medieval past these walls could have seen? That led to the idea of an immersive choose-your-own-adventure, where you act out a story from the past with your body.

The room at CIID in Bergamo, with a stone wall, wooden beams and two people testing at the wall
The 1000-year-old walls of CIID

3How it works

Built in TouchDesigner, in a week

I built it in TouchDesigner, which I’d never touched before that week. MediaPipe tracks the body, Gemini writes the story, Gemini’s text-to-speech voices it, and Google’s Lyria makes the music.

Reading the body

Gestures are relative. Exact positions in the frame were hard to calibrate, so I measured movement instead. Raising your right or left arm is judged against your shoulders, and ducking or stepping aside is about where your head was and how it moved. That way it works for anyone, wherever they stand.

MediaPipe body tracking
The TouchDesigner build

Generating a spoken story, in real time

The story tree

The story stays one step ahead. Every person gets a brand-new story, so the tales really do evolve with each playthrough. But generating at every choice would have kept people waiting, so some clever pre-generation was necessary.

The walls remember. Each person’s choices shape the next person’s story. If someone chose to fight, the next story leaned more action-oriented. If they slunk into the shadows and met people, it got more social.

Music that follows the mood

I gave Lyria, Google’s real-time music generator, a medieval, Renaissance-y base and a set of dials: suspense, romance, action, melancholy and wonder. Each step of the story sends new values for those dials, so the music really shifts mid-story as things get tense or triumphant.

The Lyria RealTime Controller web app: connection status, settings and a live audio waveform
Lyria music controller
Mood dials

4What I learned and what’s next

Imagination can enrich simple gestures

Stepping sideways or raising an arm is a pretty mundane movement. But inside the story, it becomes escaping into the shadows or drawing a sword. I didn’t have to build in recognition for complex gestures; there was plenty possible with simple ones, once you added an imaginative lens to the movement.

A player raises his arm at the stone wall; an imagined sword is drawn in his hand, and his particle shadow glows red
A raised arm becomes raising a sword

What I’d try next

I’d love to try it with different visuals, or none at all. I wasn’t even sure it needed them, though they did give people something to look at while the story played. Would people stay with it without them? Would it feel like they’re performing for an audience?

We also had the idea of the story noticing when another person walks into the frame, so a group could play together and each newcomer could become an unplanned twist. There just wasn’t time for that in a one-week project.

Six people at the stone wall, each raising an arm beside their glowing particle shadow
People playing at the wall

Next project

Auracle →

← Back to all work