Last week Igalia released an important piece of software called Skeltrack which, to put it simple, allows to retrieve the human skeleton joints from depth images. It had a good coverage from many important news websites and blogs and I received good feedback with kind words and even use cases I hadn’t thought of.
Still, one thing is to have the simple demo of drawing a sort of stick man from one’s joints like shown in the Skeltrack’s example video, another is to actually do something more useful with it. This is the way to check how reliable the library is, so Igalia has built a prototype that consists of controlling the GNOME 3 desktop or even playing some racing games using gestures that are interpreted from the positions of the joints that Skeltrack gives.
It uses one or both hands to control the mouse pointer, perform clicks, drag things around and it even simulates a pinch gesture which adjusts the zoom level. Hands can be also interpreted as if holding a steering wheel, making racing games so much fun.
The results are shown in the following video:
(direct link to video in Vimeo)
Just like pretty much everything we do at Igalia, this demo is also Free Software, so you can get its code from GitHub (check the README to see what the gestures are and what they do), tweak it to your needs or, if you need specialized help, you can always hire us.
Holy carp.
Allow me to pilot a F16 with flightgear with this and you have world domination.
Hi,
Skeltrack works with depth cameras, like the Kinect so you would need a device like this to be able to use it.
Can you please post your video as HTML5/WebM or Theora?
Hi Jack,
Could you tell me why should I post the video in those formats? (I just did the embed straight out of Vimeo)
Hi Joaquim Rocha.
First of all let me thank you for sharing your code.
I’m developing a mouse control with kinect in Qt, one of the parts of my thesis, I am having some problems about jitter in the joints from skeletal tracking.
I was checking your code, I was wondering what type of filtering are you using in the smooth_point function of Skeltrack Desktop Control’s main.cpp.
Is this an implementation of double exponential buffer?
I really appreciate your help and your time.
static Point *
smooth_point (guint16 *buffer, guint width, guint height, SkeltrackJoint *joint)
{
if (joint == NULL)
return NULL;
Point *closest;
gint i, j, x, y, radius, min, count;
radius = 16;
x = joint->screen_x;
y = joint->screen_y;
if (x >= width || y >= height)
return NULL;
closest = g_slice_new0 (Point);
closest->x = x;
closest->y = y;
closest->z = joint->z;
min = closest->z – 50;
count = 1;
for (i = x – radius; i < x + radius; i+=2)
{
if (i = width)
continue;
for (j = y – radius; j < y + radius; j+=2)
{
guint16 current;
if (j = height || (j == y && i == x))
continue;
current = buffer[j * width + i];
if (current z && current >= min)
{
closest->x += x;
closest->y += y;
count++;
}
}
}
closest->x /= count;
closest->y /= count;
return closest;
}
Hi Herman,
About the function you pasted here, I implemented that because Skeltrack gives the points with intervals originated from the dimension reduction factor. Using this function I get values in between those intervals.
As for the mouse jittering, the way I’m moving the mouse is to increment its current position by values that are directly proportional to the distance of the destination point and the current point but those values are only a fraction of that distance. This way the mouse will take a little longer to move to its destination but it’ll move smoother.
Hope this explanation helped.
Hi Joaquim,
Thank you very much, now I think I understand, with “dimension reduction factor” you meant the projection from 3D hand point (in “world” coordinates) to 2D hand point (in depth video frame coordinates) ?
Please let me know if I got it: The hand coordinates that you are using in “set_mouse_pointer” are between 640 and 480 (even with smoothing), you calculate the distance between the mouse cursor and the end point (the new hand point), this distance is reduced to a new near end point is where you move the mouse with XTest
Hey Herman,
About the dimension reduction: No. It is just the factor by which I reduce the number of points considered in the depth image; this is so the computation is easier when calculating the body extremas.
About the way to move the mouse: I do set the mouse pointer between 640×480 (the resolution of the Kinect’s depth camera) and translate this to the real screen’s resolution; the rest (if I understood you well), is how you say.
Thanks for your help Joaquim, I got it now, I really appreciate your help and your time.
No prob, Herman.
Hello,
thanks for posting this code; i have intalled successfuly on gnome 3 the skeltrak desktop control but i can’t understand how i can configure it because the signal is noisy.
In your video demo is very precise but not the same in my system.
Could you help me please?
thanks a lot Leonardo
Hi,
Try to place the Kinect in front of you (if you haven’t tried it yet) and to activate Skeltrack’s smoothing with a low value (increased smoothing).
Please note that that application is not a polished one and that it was intended as a demo to what can be done with Skeltrack.