Interactive diabolo

As part of my last year of study in the Art, Science and Technology Masters at Grenoble INP, I had to realize a project that mixed art, science and technology. I worked with Thomas Moncond'huy on a project of interactive juggling with a diabolo. We had about three months to develop our project and zero budget.

We are both juggling amateurs and we liked the idea of visuals and sounds generated by juggling tricks, thus making the juggling experience interactive. We divided our project in three parts: tracking of the diabolo and the hands of the juggler, sound generation and image generation.

Tracking program

The tracking program was in charge of recognizing the trick made by the juggler and sending it to the sound and image generation programs.

Diabolo and hands tracking

The juggler makes a diabolo trick with a combination of movements of the sticks causing the diabolo to have a particular movement. However, a diabolo trick is not only a particular movement of the diabolo, you have also to consider the movements of the sticks and sometimes the juggler.

To achieve this trick recognition, we needed a way of tracking the movements of the diabolo and the sticks. This is probably the part that took us the most time.

First, inspired by Johnny Chung Lee's Wiimote tracking system we attached infrared LEDs to the diabolo and to the end of the sticks and tracked them with a Wiimote. The tracking was very nice and fluid, but unfortunately there was spatial limitations that were too important: the LEDs emission cone and the Wiimote's reception cone were too narrow. It means that the sticks and the diabolo had to point to the Wiimote to be tracked, which is impossible if you want to do interesting tricks with the diabolo. Moreover, the width and height of the space where the tracking was correct was too small comparing to the space needed for doing diabolo tricks.

We realized it would be difficult to track the sticks with the time and budget we had, so we chose to track the hands of the juggler instead. Although the movement of the sticks is characteristic of a diabolo trick, we found that tracking the movements of the hands was relevant enough for the recognition of basic tricks.

The tracking of the hands was made using a Kinect lent by one of our professors and OpenNI. As the Kinect includes a RGB camera, we also used it to track the diabolo with a simple color tracking.

Juggling trick recognition

Thanks to the Kinect, we could track the position of the diabolo and the hands of the juggler in real-time. The trick recognition consisted in determining the trick made by the juggler only knowing the positions of its hands and the diabolo, and it was a bit, well, tricky.

We selected a limited number of tricks that we would want to recognize: vertical acceleration, horizontal acceleration (side whips), leg satellite arm satellite and elevator. We started with simple filters to recognize these tricks. While it worked fine for some tricks (like accelerations), it has proved to be very difficult for others more complex, like satellites.

For these tricks we used dynamic time warping with the Gesture Recognition Toolkit which is an open-source, C++ machine learning library designed for real-time gesture recognition. We made datasets of the tricks we wanted to recognize and fed them to GRT who was then able to tell us what trick we were doing in real-time.

Although our trick recognition worked, we had interferences between tricks that would compromise an elegant sound and image generation downstream. That's why we used a sequencer. Thereby, a trick could only be detected if it was programmed in the sequencer.

The sequencer waits for the currently programmed trick to be recognized and when it is, it fires an event that goes to the sound and image generation programs via OSC. Then the sequencer jumps to the next programmed trick and waits for it.

Below is a diagram of the achitecture of the tracking program:

Sound and image generation

Now that we had a good tracking of the diabolo and the hands, we could generate sound and images using the coordinates and events sent by the tracking program.

Sound generation

The sound generation was done using SuperCollider. We had several instruments that were triggered and modulated according to the events sent via OSC from the tracking program. For instance:

• a ratchet noise was played as the diabolo was rotating around the leg of the juggler
• a drone effect was played during the elevator trick and the pitch of the drone was mapped to the elevation of the diabolo
• a percussive sound was played at each stroke during acceleration phases

Image generation

The idea of the visualization was to materialize the interaction that the juggler could feel with the diabolo. We rapidly chose to use a particle system because it was easy to develop and fun to play with.

The particles were rendered with an additive blending as it is a good way of visualizing the particle density (higher density, brighter color). Two post processing effects were added: a trail to the particles using an FBO and a glowing effect using the ofxPostProcessing addon.

The visualization program receives the positions of the hands and the diabolo sent from the tracking program via OSC. The positions of the hands are materialized with two blue discs, and the diabolo with a red disc. The particles are able to move between the diabolo disc and the two hands discs. The images shown below are screenshots of the actual image generation:

Conclusion

We finished the project in time and everything worked well. However, a lot of improvements could be done. An important part would be removing the sequencer in the tracking program, this means a better trick recognition and a better tracking. A way of improving the tracking would be to track the sticks directly maybe using IMU sensors. A better tracking of the diabolo would be necessary as well, although the color tracking works fine with the good lighting conditions.

As we spent a lot of time getting the tracking right, we had little time to do the sound and image generation although it was the funniest part of the project. The particle system is very basic and it would be fun to add events to it for instance, an explosion when a particular trick is achieved.

If I'd have to redo it, it would probably be quite different as I know openFrameworks a lot better and I improved my programming skills since then. Anyway, I still put the code of the project on github AST_diabolo_tracking and AST_diabolo_particles for archive purposes. Although it should still work, the code needs a good cleaning and refactoring but I can't do it a the moment as I don't have the equipement to make it work anymore.