AR Piano – A New Learning Phenomenon

Definitely worth seeing: A tweet from Fayez Salka – physician, software company and 3D artist – shows how he uses AR (augmented reality) technology to bring a virtual pianist to life on a real piano. Back to the future, it gets more and more wacky. But take a look for yourself:

️?

Augment Reality for musicians – really impressive

In the video, the resourceful Salka strolls through a piano shop. He points his mobile phone camera at various pianos. Immediately an AR pianist appears on the screen and starts playing various songs by award-winning pianists. In real time, the notes scroll over the bottom of the screen.

Precise key recognition even without 3D depth camera

The app recognises the piano including the notes of its keyboard. It determines the three-dimensional position and orientation of the piano in a 3D space. This is done with a precision of up to 1 centimetre. This is carried out merely by looking at each individual frame, 3D depth cameras are not needed.

This precise positioning is important so that the virtual pianist can be aligned and visually superimposed at the exact place on the piano. The interesting thing about this is that, due to the self-learning algorithm, this works with any piano model of any brand. Even the lighting conditions are irrelevant. It is not necessary to set special image points or image markers. Also, the piano does not have to be scanned beforehand. Unbelievable!

Now published on App Store

The app is now available on the iOS app store. Fayez Salka and his business partner Anas Wattar of Montréal-based Massive Technologies Inc. have eliminated the final bugs of the virtual wonder and made it available for others online. There is most probably some creative brainstorming to be done, for which purposes the app could be used. The future is exciting!

Re-experience piano music on smartphones & mobile devices

In its current state, the creative lateral thinker sees the app as an excellent tool for experiencing piano music on the smartphone in a completely new way. This is truly something special, a completely new approach with previously unimaginable scenarios for learning to play an instrument. In further development, the app could cover many more areas in music and a variety of instruments.

A new development step in music’s AI sector

To put it bluntly: A new step in AI (artificial intelligence) development is being implemented here. Whereas music not made by hand was previously limited to the use of playback, sequences and the like, the former concept is now being completely turned upside down: You simply point your smartphone at a piano and the whole process takes place.

Learning from “virtual people”

You will be played what you would like to learn in an absolutely direct and visually comprehensible way. However, it is not the “artificial” piano that plays, but a virtual person. So you can establish a personal relationship with your smartphone. Finally, an AR app that has massive potential for ambitious young musicians.

What could this technology possibly lead to?

Let’s not fool ourselves, after all, despite all its comprehensive functionality, this app is also just the beginning. It is conceivable that this is not just a fascinating optical gimmick, but rather a didactic step into the future world of music for almost all instruments. It’s a funny thought that you could have your music teacher permanently in your pocket in the near future. A small request by simple keystroke or voice command; s/he will play the next lesson for you. As many times as you want, and with consistent perfection, until you have understood everything.

Lots of ideas still in the oven

For Salka the project started as a hobby. Meanwhile the cooperation with Massive Technologies has become his main job. And it is already clear that there is still a lot to come. Fayez reports that some ideas are still cooking in the oven, we are curious and will follow the developments with great pleasure. To illustrate the potential: An older app from Massive Technologies is called Piano 3D. A product with over 7 million downloads, and counting…

Below are links to all of Massive Technologies‘ apps so far: