MIDI Reality – Mixed Reality with Kinect and MIDI Keyboard

Tech & music are my favorites. I’d love to do them together.
So I had this idea to try covering a song with a MIDI keyboard, and a Kinect v2 sensor for mixed-reality floating caption effects.
That was when the song (Life Is Worth Living by Justin Bieber) was still new. However, the recording process’s been so procrastinated.
Since it had taken for a very long time, I decided to not making a better cover, but this somewhat bad, weird and incomplete one. And, explaining how that was done more focusedly.

You can consider it as how most mixed reality / augmented reality effects are possibly done behind nowadays. Literally overlaying parts of(or full) virtuality on a reality layer.

Furtherly, using regular AR marker technology to track position of MIDI keyboard, Kinect can capture around for a movable camera view(Requires a stored 3D background data also). Or just use HoloLens!