Controlling GNOME with Leap

When I explained how the Leap Motion device could be used on Fedora 19, I mentioned how I had one of those early prototypes. Well, Leap Motion was extremely kind and sent me an actual device as a thank you for starting the thread asking for Linux support. Now that GUADEC is over and I am spending my vacation in Portugal, I had a little time to play with my fancy new device and wrote a relatively small script to control GNOME with it. I call it the über original name of: Leap GNOME Controller!

For those who don’t care about technical details, here’s the video showing what can be done with Leap, GNOME and this script. Technical details follow below the video:

The two videos that compose the one above were recorded with an HD camera and GNOME Shell’s screencast recorder. I tried to sync them the best I could but a certain delay can be noticed, especially at the end of the video.

The code

Leap Motion provides a “close source” shared library and a high-level API with the respective documentation for the many bindings it has. To code it quickly, I used the Python bindings and Xlib to fake the input events.

Leap Motion’s APIs make it really easy for one to simulate a touch-screen. It even offers a “screen tap” gesture that should be the obvious choice when mapping a finger touch/tap to a mouse click. However, this didn’t work very well. The problem is that if we are tracking one finger to control the mouse movement, when performing the “screen tap” gesture, the finger (and mouse) will of course move. Making it as frustrating as seen on ArsTechnica hands-on video.

I came up with a solution for this by dropping the “screen tap” gesture and using the “key tap” instead. The “key tap” is a simple, quick down-and-up finger movement, like pressing a key. This is much more precise and easier for a user to do than the “screen tap”. Of course that when the finger moves for performing the gesture, the mouse pointer would move as well, so I came up with a little trick to work around this: when the mouse pointer doesn’t move more than a couple of pixels for half a second, it will stop and only move again if the user makes it move for more than 150 pixels. This allows for the user to stop the pointer with precision where it needs to be and perform the gesture without making the pointer move.

Future

The Leap device offers a lot of possibilities for adding many gestures. Ideally they should be implemented per application but being able to control the shell is already pretty useful, so it would be wonderful to fine-tune the current gestures and add new ones. I also wish the library’s source code were open because I ran into small issues and I wish I could take a look at the source code, instead of trying to fix it based on the theories of what might be wrong.

I haven’t explored the AirSpace appstore yet so I don’t know if it is worth adding (or possible to add) this script there but I will check it out.

Have fun with Leap and GNOME!

9 thoughts on “Controlling GNOME with Leap

  1. I can’t make it work, I would need a little help please…
    here’s what I get:
    ~/Téléchargements/Leap-GNOME-Controller-master$ ./leap-gnome-controller.py
    Traceback (most recent call last):
    File “./leap-gnome-controller.py”, line 21, in
    import leap
    ImportError: No module named leap

    Thanks for helping me out and for your work.

  2. your experience was just as fun as mine, using simple dbus commands, i created all sorts of gestures to control my KDE desktop, pretty cool stuff.

  3. Hi,
    First of all, thank you for putting this together, it looks really promising.

    I am having similar problems as Bountam. I keeps giving me import errors and file not found errors. I managed point it toward the lib folder provided my Leap, like you suggested, but I am stuck at

    ImportError: No module named Xlib

    Any help would be appreciated.

  4. Im getting the same problem as Bountam.

    Where is the path to the leap.y (im guessing its actually .py) I cant seem to find it anywhere? im running the 1.0.9 sdk.

    Thanks in advance

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>