← projects

Interactive Hologram

Team (contributor) · 2024

PythonOpenCVMediaPipePyAutoGUIPyAudioFigmaAfter Effects

GitHub

Overview

A glass pyramid sits on top of a screen. The screen plays black-backgrounded footage, the four faces reflect it, and your eye assembles the reflections into an object floating in mid-air. The trick is old, it is called Pepper’s ghost, and it has been fooling theatre audiences since 1862.

What makes this one interesting is that you can reach out and change what it shows. A webcam watches your hand. There is no controller, no touchscreen, no remote.

The glass pyramid on its stand, a blue candle floating inside it
The pyramid at the De Vinci Innovation Center. What looks like it is floating is a reflection of the screen underneath.

The sleight of hand: Figma is the engine

There is no application. The thing you are interacting with is a Figma prototype, running full screen with the interface chrome switched off.

That is the decision the whole project turns on, and it is a good one. A Figma prototype already gives you states, transitions and clickable hotspots for free, and it can be redesigned in an afternoon by someone who does not write code. So instead of building an app, the team built a prototype and then wrote a small Python program whose only job is to move the real mouse and click on it.

The consequence is that Python and Figma agree on nothing except screen coordinates. When an audio peak fires, the code does pyautogui.moveTo(90, 990) and clicks — and a button has been placed at exactly that spot in the Figma file. Move the button, and you have to edit the Python. It is brittle in the way a stage trick is brittle, and like a stage trick it works.

What the screen actually plays: labelled elements on pure black, so that only the bright parts reflect and the background disappears.

Your hand becomes the cursor

InteractiveHolo.py opens the webcam with OpenCV, runs each frame through MediaPipe Hands, and gets back 21 landmarks per hand. Three of them do all the work:

  • The little-finger tip drives the cursor. Its normalised position is scaled to the screen and handed to pyautogui.moveTo(...). Using the pinky rather than the index is a deliberate choice: the index is busy doing something else.
  • The thumb tip and the index tip are the click. The code measures the distance between them and fires a click when it drops under a threshold. Pinch to select.
  • The wrist anchors the debug overlay, drawn back onto the video feed so you can see what the machine sees while you tune it.

Nothing here is trained or learned. It is geometry, a threshold, and one honest constant to adjust per machine — the README says so plainly, which is the right way to ship a rig that has to work in a room you have never been in.

A laptop showing the Figma control view next to the pyramid, a blue figure floating inside
Two screens, one system: the control monitor runs the prototype, the pyramid shows the result.
Three luminous blue forms reflected in the faces of the pyramid
Four reflections, one object. Move your head and the parallax holds.

Blowing on it counts as a click

The part I like most. A second thread opens the webcam’s microphone with PyAudio, computes the RMS of every buffer, converts it to decibels, and if it crosses 78 dB it clicks a fixed point on screen.

In practice that means you blow on the hologram and it reacts. It was used to blow out a candle. A whole input modality obtained from one line of arithmetic on a signal that was already there, because the webcam had a microphone nobody was using.

The pyramid showing a product with a label floating beside it
The second use: put a real object at the centre of the pyramid and float its information around it.

The hardware

Three pieces, and the README is exact about which is which: a webcam for the hand and the sound, a display screen built into the hologram, and a control monitor where the Python and the Figma actually run. The audience only ever sees the first two.

The object holding all of it up — the wooden frame and the plastic pyramid that does the reflecting — was built by Gaëtan Carrade. It is the part everyone touches and nobody credits: the angles of those four faces are what decides whether the illusion holds.

Annotated photo of the rig identifying the webcam, the displaying screen and the control monitor
The rig, annotated. The control monitor stays out of frame during a demo.
The hologram unit on a workbench next to a laptop
On the bench. The pyramid is a separate object you place over the screen.

My part in it

The project is Thomas Bodénan’s. The concept, the electronics and the rig are his, and he is the one who worked out how to make a webcam drive an interface.

What I brought was the other half of the illusion: the motion design and the entire Figma side. Every element you see floating was animated in After Effects, exported on a black background so it would reflect cleanly, and assembled into the prototype with its states and its hotspots. When the Python clicks a coordinate, it is clicking on something I placed.

He also taught me the camera interaction — how to get from a webcam frame to landmarks to a cursor. That was the first time I made a camera drive an interface, and it is a thread I kept pulling: the motion capture in Body Shape Control and the muscle-driven controller in 2D-EMG-GAME both start from the same idea, that a sensor already in the room can become an input if you are willing to do the arithmetic.

Team project, 2024, at the De Vinci Innovation Center. Concept, electronics and code by Thomas Bodénan (repository linked above); wooden structure and pyramid by Gaëtan Carrade; motion design and Figma by me. Technical details on this page are drawn from the README and the source.