Controlling GNOME 3 with your hands

Last week Igalia released an important piece of software called Skeltrack which, to put it simple, allows to retrieve the human skeleton joints from depth images. It had a good coverage from many important news websites and blogs and I received good feedback with kind words and even use cases I hadn’t thought of.

Still, one thing is to have the simple demo of drawing a sort of stick man from one’s joints like shown in the Skeltrack’s example video, another is to actually do something more useful with it. This is the way to check how reliable the library is, so Igalia has built a prototype that consists of controlling the GNOME 3 desktop or even playing some racing games using gestures that are interpreted from the positions of the joints that Skeltrack gives.

It uses one or both hands to control the mouse pointer, perform clicks, drag things around and it even simulates a pinch gesture which adjusts the zoom level. Hands can be also interpreted as if holding a steering wheel, making racing games so much fun.

The results are shown in the following video:

(direct link to video in Vimeo)

Just like pretty much everything we do at Igalia, this demo is also Free Software, so you can get its code from GitHub (check the README to see what the gestures are and what they do), tweak it to your needs or, if you need specialized help, you can always hire us.

Announcing Skeltrack

After being able to control the basic features of a Kinect device (or any time-of-flight camera), the next thing many users look for is skeleton tracking. Skeleton tracking means to easily retrieve different joints of the human skeleton from depth images.

The most famous solutions (or the only ones) to do so are the Microsoft Kinect SDK or the OpenNI framework. If you are looking for a Free software solution though, you are out of luck. Microsoft’s SDK, apart from obviously being close, does not even allow a commercial use of it which is something the OpenNI framework does but this is as far as the meaning of the word “open” in OpenNI goes… You cannot adapt/improve their code nor learn from it. To solve this problem, Igalia has just created Skeltrack.

Free Software Skeleton Tracking

Skeltrack is a Free and Open Source Software library whose goal is to provide easy to use human skeleton tracking.
Skeltrack’s implementation was based on a paper by Andreas Baak but, apart from other internal differences, it doesn’t use a pose database. This means that the skeleton joints extraction is based on mathematics and heuristics, no calibration pose nor pose database is needed.

It provides an asynchronous API written in GLib, supports single user tracking (one skeleton only) and tracks up to 7 joints currently: head, shoulders, elbows and hands.

Take a look at the video below to get an idea of what it can do:

(watch it in Vimeo)

How to use it

We took a more modular approach than the two projects mentioned above. This means that Skeltrack expects to be given a depth buffer (i.e. depth image from the Kinect) with nothing but the user on it as opposed to connecting directly to a Kinect device.
Still, three easy steps should be all that is needed for Skeltrack to be ready to use:
1) Use the GFreenect library we released a couple of months ago, connect to the depth stream signal and get the depth buffer;
2) If there are other objects (chairs, tables, walls, etc.) apart from the user, then these should be removed by performing background subtraction or by excluding everything that is out of a threshold;
2) Skeltrack performs some calculations that might be heavy depending on the machine and the buffer size so users should reduce the size of the buffer that is given to Skeltrack by using its dimension-reduction property as the reduction factor.

Take a look at the test-kinect example (shown in the video) shipped with Skeltrack where the steps above are implemented.
The documentation for the project is available here.

Skeltrack is in its early beginnings and we want to detect more skeleton joints and work on stabilizing the results so, its features and API might change in the future.

Feel free to get the code, file bugs and send patches at its GitHub repository.
Hopefully more people will contribute to it until we finally have a rock solid, easy to use, Free Software skeleton tracking library.

I need to thank Søren Hauberg who was kind enough to point me in the right direction when I needed.