Controlling GNOME 3 with your hands

Last week Igalia released an important piece of software called Skeltrack which, to put it simple, allows to retrieve the human skeleton joints from depth images. It had a good coverage from many important news websites and blogs and I received good feedback with kind words and even use cases I hadn’t thought of.

Still, one thing is to have the simple demo of drawing a sort of stick man from one’s joints like shown in the Skeltrack’s example video, another is to actually do something more useful with it. This is the way to check how reliable the library is, so Igalia has built a prototype that consists of controlling the GNOME 3 desktop or even playing some racing games using gestures that are interpreted from the positions of the joints that Skeltrack gives.

It uses one or both hands to control the mouse pointer, perform clicks, drag things around and it even simulates a pinch gesture which adjusts the zoom level. Hands can be also interpreted as if holding a steering wheel, making racing games so much fun.

The results are shown in the following video:

(direct link to video in Vimeo)

Just like pretty much everything we do at Igalia, this demo is also Free Software, so you can get its code from GitHub (check the README to see what the gestures are and what they do), tweak it to your needs or, if you need specialized help, you can always hire us.