Category Archives: augmented reality

Salutations Interactive Installation: Making of

Posted on by
1

Since the post about the Salutations Interactive Installation in the Museum für Kommunikation in Berlin, many people asked me for videos of the installation so we, the Igalia Interactivity team, came up with a better idea — a making of.

The video shows the an early test application that was created from the one shipped with Skeltrack, then some testing with colleagues at Igalia’s office and finally the installation in Berlin and the final result:


(link to original video in YouTube)

You can get the source code for the application from Igalia’s GitHub.

Skeltrack 0.1.10 is out

Posted on by
3

That’s right, a new version of the world’s first Free Software skeleton tracking library is out.
In every version we try to make Skeltrack more robust and this one is no exception.

Head&Shoulders

We have changed the way the shoulders are inferred. This heuristic now uses a circumference around the user’s head and an arc with which it searches for the shoulders.
Since we like to keep giving developers the ability to tweak the algorithm’s parameters, we had to change the properties related to the shoulders. We should probably improve the documentation with a visual explanation of how those properties work but meanwhile you can check the properties’ documentation.

Centering Joints

Another issue we had was that the extremas we initially calculate result in e.g. the point at tip of the a finger (for a hand joint) or the top of the head. This was not an issue specifically but it might result in more unstable joints. For example, the Kinect device in particular might give blind spots in very bushy hair which would result in the head joint jittering more than usual.
To fix this, we calculate the average of points around an extrema and assign it with that value. The radius of the sphere surrounding an extrema that is used to calculate this average can be controlled by using the extrema-sphere-radius property. Thus, if this behavior is not desired, this feature can be turned off just by simply assigning a 0 to this property.

Here is a couple of pictures describing this issue:

Picture of Skeltrack's test without averaged extremas

Without the averaged extremas (extrema-sphere-radius set to 0)

Picture of Skeltrack's test with averaged extremas

With the averaged extremas (extrema-sphere-radius set to 300)

Vertical Kinect

Due to a project that Igalia Interactivity has been working on, we had to use the Kinect in a vertical stance. By doing this we discovered a small bug that prevented Skeltrack to be used with a vertical depth image. This is corrected in this 0.1.10 version and while fixing it, we found out that it seems the other skeleton tracking alternatives also do not support the Kinect in a vertical stance; this might mean that if you want to use skeleton tracking with the Kinect vertically, your only choice is either to use Skeltrack or to convince Microsoft or PrimeSense to fix their solutions for you :)

Picture of Skeltrack's test example using a Kinect in a vertical stance

Skeltrack using a Kinect in a vertical stance

Last but not least, the function skeltrack_skeleton_new was returning a GObject instance by mistake. We have corrected that and it now returns a pointer to SkeltrackSkeleton as expected.

Special thanks to Iago, our intern at the Igalia Interactivity team, for coding most of these nifty features.

Be sure to clone Skeltrack at GitHub and read the docs, you are welcome to participate in its development.

Controlling GNOME 3 with your hands

Posted on by
22

Last week Igalia released an important piece of software called Skeltrack which, to put it simple, allows to retrieve the human skeleton joints from depth images. It had a good coverage from many important news websites and blogs and I received good feedback with kind words and even use cases I hadn’t thought of.

Still, one thing is to have the simple demo of drawing a sort of stick man from one’s joints like shown in the Skeltrack’s example video, another is to actually do something more useful with it. This is the way to check how reliable the library is, so Igalia has built a prototype that consists of controlling the GNOME 3 desktop or even playing some racing games using gestures that are interpreted from the positions of the joints that Skeltrack gives.

It uses one or both hands to control the mouse pointer, perform clicks, drag things around and it even simulates a pinch gesture which adjusts the zoom level. Hands can be also interpreted as if holding a steering wheel, making racing games so much fun.

The results are shown in the following video:

(direct link to video in Vimeo)

Just like pretty much everything we do at Igalia, this demo is also Free Software, so you can get its code from GitHub (check the README to see what the gestures are and what they do), tweak it to your needs or, if you need specialized help, you can always hire us.

Announcing GFreenect

Posted on by
9

As mentioned in my last post, Edu the mighty Cuban and I have been playing with the Kinect and developed an interactive installation for Igalia‘s 10th anniversary party using OpenFrameworks. (By the way, some people asked me for that application’s code so yesterday I cleaned it and it’s available on Gitorious)
OpenFrameworks offers a number of functionalities either from its core libraries or by means of add-ons and indeed there is an add-on that wraps libfreenect, the Free Software library that allows to control the Kinect.

Using OpenFrameworks was easy, it makes it fast to start developing with it but in many aspects it’s completely different from the way we’re used to work on GNOME. We are used to have single, independent libraries that do one thing and do it well and are used as needed by application developers, for example, do not include a sound library if my application is never going to use it.
Having modules such as GTK+, Clutter, Cairo, GStreamer, etc. already gives us flexible ways to develop certain parts of applications similar to the demo mentioned before: we just had to draw the fish using Clutter/Cairo, implement their behavior and show the Clutter stage. Of course we also would need a way to control the Kinect and it would be really nice if it could offer us an easy to use API for those familiar with GLib…

Ladies and gents, we give you… GFreenect

GFreenect is a wrapper for the Freenect library written using Glib in order to control a Kinect device and make it easy to use with GNOME technologies.
It doesn’t simply wrap the Freenect library but also offers ways of using it that are familiar to you if you have developed something using other GNOME libraries.
One example of this enhanced functionality is that we focused on offering an asynchronous API (although there are some synchronous alternative methods as well). Another example is that when setting the device’s tilt angle, a signal will be emitted when it has finished setting the angle, since it might be useful for some applications.

One of the purposes of having it written with GLib is the GObject Introspection capability. This allowed us to include an example application that controls the various features of the Kinect and was written in Python effortlessly. A screenshot of this app is shown below:

GFreenectView Screenshot

And that’s it! You can find the code for GFreenect in Gitorious (including documentation for this 0.1.2 version). Bear with us if you find some bugs, it is fresh out of the oven.

We hope you find GFreenect useful for your projects and please give us feedback if you find some issues or have any good suggestions.

Augmented Reality at Igalia’s 10 Years Anniversary

Posted on by
9

This is a long pending post. It should have been written about 3 months ago but only now we found the time to do it.

Some of us in Igalia have been interested in Augmented Reality (AR) and Computer Vision (CV) for a while and we wanted to do something within these fields for the company’s 10th anniversary party which took place in last November.
Since our time was limited (due to our work in a client’s project) we came up with a simple AR installation to show at the party — a fish pond, or aquarium.

So a week before the party Edu and I bought a Kinect device and implemented this fish pond using OpenFrameworks, an Open Source framework for writing interactive applications using C++.
As you might notice from the pictures, we are more programming artists than graphical artists so the fish may look a bit funky but they’re drawn in only 10 lines of code using graphics functions (and I kinda like their looks). Due to limitations of the place where we installed the application, we had to place the projector in a lower angle instead of above the “action area” and this of course resulted in bigger shadows but people could still play with it without many issues.

The fish’s behavior was implemented using Reynolds’s algorithms and the idea is that when a person “touches” the pond, the fish will go there as if there was food being dropped. When the person stops touching the pond, the fish will hang for a little while and then will disperse randomly to other points of the pond.

As you see, the idea is pretty simple but it was a nice entertainment for kids and grown ups around the party.

We will likely be doing more AR and CV for the coming months so stay tuned!