Overview
Plot Twist is an immersive, choose-your-own-adventure story for the walls of CIID’s thousand-year-old home in Bergamo. It’s told in the second person, and your body enacts the main character’s actions to unfold the story.
1The experience
Slink into the shadows or raise a sword to fight?
A pulsing orb is projected on the wall. As you walk toward it, medieval music grows louder. Touch it, the music swells, and the story begins, with you as the main character: you’re standing in a banquet hall…
A shadow-like projection of you appears on the ancient stone. It shifts color to match the mood of the story, and the music keeps changing underneath to support the tone.
When the story reaches a branching point, it asks you to choose. Do you raise your sword arm to prepare for battle, or crouch into the shadows? You just do it. There’s nothing to explain, and people were really enthralled by it. A lot of them came away saying some version of “oh, I can’t believe I did that,” and the surprising turns in the story caused real delight.
2Where it started
What if these ancient walls could talk?
Our brief was to design an AI experience off the screen: no text or speech, just an embodied experience that connects people to the space around them. Our process was very site-specific. We explored CIID’s building, intrigued by odd corners and liminal spaces like the stairwell, and bodystormed scenarios throughout the space.
Ultimately, we were captivated by the fact that CIID is housed in a thousand-year-old building, connected to the neighboring monastery. Those walls have witnessed countless stories over the centuries. What if these walls could talk? How might we let someone today step into a medieval past these walls could have seen? That led to the idea of an immersive choose-your-own-adventure, where you act out a story from the past with your body.

3How it works
Built in TouchDesigner, in a week
I built it in TouchDesigner, which I’d never touched before that week. MediaPipe tracks the body, Gemini writes the story, Gemini’s text-to-speech voices it, and Google’s Lyria makes the music.
Reading the body
Gestures are relative. Exact positions in the frame were hard to calibrate, so I measured movement instead. Raising your right or left arm is judged against your shoulders, and ducking or stepping aside is about where your head was and how it moved. That way it works for anyone, wherever they stand.
Generating a spoken story, in real time
The story stays one step ahead. Every person gets a brand-new story, so the tales really do evolve with each playthrough. But generating at every choice would have kept people waiting, so some clever pre-generation was necessary.
The walls remember. Each person’s choices shape the next person’s story. If someone chose to fight, the next story leaned more action-oriented. If they slunk into the shadows and met people, it got more social.
Music that follows the mood
I gave Lyria, Google’s real-time music generator, a medieval, Renaissance-y base and a set of dials: suspense, romance, action, melancholy and wonder. Each step of the story sends new values for those dials, so the music really shifts mid-story as things get tense or triumphant.

4What I learned and what’s next
Imagination can enrich simple gestures
Stepping sideways or raising an arm is a pretty mundane movement. But inside the story, it becomes escaping into the shadows or drawing a sword. I didn’t have to build in recognition for complex gestures; there was plenty possible with simple ones, once you added an imaginative lens to the movement.

What I’d try next
I’d love to try it with different visuals, or none at all. I wasn’t even sure it needed them, though they did give people something to look at while the story played. Would people stay with it without them? Would it feel like they’re performing for an audience?
We also had the idea of the story noticing when another person walks into the frame, so a group could play together and each newcomer could become an unplanned twist. There just wasn’t time for that in a one-week project.
