When I explained how the Leap Motion device could be used on Fedora 19, I mentioned how I had one of those early prototypes. Well, Leap Motion was extremely kind and sent me an actual device as a thank you for starting the thread asking for Linux support. Now that GUADEC is over and I am spending my vacation in Portugal, I had a little time to play with my fancy new device and wrote a relatively small script to control GNOME with it. I call it the über original name of: Leap GNOME Controller!
For those who don’t care about technical details, here’s the video showing what can be done with Leap, GNOME and this script. Technical details follow below the video:
The two videos that compose the one above were recorded with an HD camera and GNOME Shell’s screencast recorder. I tried to sync them the best I could but a certain delay can be noticed, especially at the end of the video.
Leap Motion provides a “close source” shared library and a high-level API with the respective documentation for the many bindings it has. To code it quickly, I used the Python bindings and Xlib to fake the input events.
Leap Motion’s APIs make it really easy for one to simulate a touch-screen. It even offers a “screen tap” gesture that should be the obvious choice when mapping a finger touch/tap to a mouse click. However, this didn’t work very well. The problem is that if we are tracking one finger to control the mouse movement, when performing the “screen tap” gesture, the finger (and mouse) will of course move. Making it as frustrating as seen on ArsTechnica hands-on video.
I came up with a solution for this by dropping the “screen tap” gesture and using the “key tap” instead. The “key tap” is a simple, quick down-and-up finger movement, like pressing a key. This is much more precise and easier for a user to do than the “screen tap”. Of course that when the finger moves for performing the gesture, the mouse pointer would move as well, so I came up with a little trick to work around this: when the mouse pointer doesn’t move more than a couple of pixels for half a second, it will stop and only move again if the user makes it move for more than 150 pixels. This allows for the user to stop the pointer with precision where it needs to be and perform the gesture without making the pointer move.
The Leap device offers a lot of possibilities for adding many gestures. Ideally they should be implemented per application but being able to control the shell is already pretty useful, so it would be wonderful to fine-tune the current gestures and add new ones. I also wish the library’s source code were open because I ran into small issues and I wish I could take a look at the source code, instead of trying to fix it based on the theories of what might be wrong.
I haven’t explored the AirSpace appstore yet so I don’t know if it is worth adding (or possible to add) this script there but I will check it out.
Have fun with Leap and GNOME!
9 thoughts on “Controlling GNOME with Leap”
I can’t make it work, I would need a little help please…
here’s what I get:
Traceback (most recent call last):
File “./leap-gnome-controller.py”, line 21, in
ImportError: No module named leap
Thanks for helping me out and for your work.
your experience was just as fun as mine, using simple dbus commands, i created all sorts of gestures to control my KDE desktop, pretty cool stuff.
Will this work with Ubuntu 12.04 Unity?
Hi, why you didn’t use the gnome 3 JS environment?
You need to make sure leap is accessible to your Python path. Try this before running: export PYTHONPATH=/path/to/leap.y
Hope it helps!
First of all, thank you for putting this together, it looks really promising.
I am having similar problems as Bountam. I keeps giving me import errors and file not found errors. I managed point it toward the lib folder provided my Leap, like you suggested, but I am stuck at
ImportError: No module named Xlib
Any help would be appreciated.
You should install the Python bindings for Xlib. That’s a package called python-xlib on Fedora.
Im getting the same problem as Bountam.
Where is the path to the leap.y (im guessing its actually .py) I cant seem to find it anywhere? im running the 1.0.9 sdk.
Thanks in advance
It is .py. You should have that module installed by one of the Leap Motion packages and available in your PYTHONPATH.
Comments are closed.