It’s been 20 years since Tom Cruise wowed us with gesture control in ‘Minority Report’. So where’s the technology? Nick Ross swipes through the history of this relatively untapped technology.

Twenty years ago, Tom Cruise vehicle, ‘Minority Report’ appeared. While the ‘pre-crime’ plot is forgettable, what stuck with many technologists was the gesture-based UI.

Multiple, images would appear in front of him and, wearing special finger gloves, he’d rapidly cycle, zoom and navigate through them to perform an entire crime scene investigation just by spending a few minutes waving his hands around.

With so much sci-fi tech having already been realised, why aren’t we waving our hands around like Tom Cruise? The pandemic has reinvigorated the desire for touch-free interactions and a plethora of IoT devices are begging for gesture-based controls.

So, what’s held it back, where is the technology going and what opportunities does it currently present?

A quick history

Few entities have pushed gesture control more than the video games industry.

Starting with 1989’s infamous, Nintendo Power Glove, it collapsed out of the blocks thanks to its hapless, analogue, ultrasonic location-detection technology and its fibre-optic finger switches struggling to actuate accurately. The appalling UX frustrated everyone, and Nintendo wouldn’t provide a fix until 2006’s Wii console which utilised digital accelerometers (and Bluetooth transmission) for axis control plus infra-red positional detection in battery-powered, hardware wands.

Also notable was Microsoft’s Xbox 360 ‘Kinect’ which used cameras while flooding a room with infra-red dots to map depth of objects. However, gamers quickly discovered that waving their arms around to perform basic gaming operations was exhausting, inaccurate and less effective than pressing buttons on a controller.

While sports games and dance games saw success, it was discontinued in 2017 with the primary proponents being robot enthusiasts who regarded the Kinect’s depth-perception, cameras and useful SDK as an inexpensive, programmable eyeball.

What does gesture control need?

While touchscreen-gestures are inherently different, the principles behind Apple’s 2018 iPhone X development are universal to the field. The removal of the ‘Home’ Button required a total transition to gesture control for a broad, consumer audience.

The company boiled the challenges down to three principles: 1) Remove all lag from the operating system. 2) Accurately detect lightweight movements (including the acceleration of a finger) to infer trajectory and predict intent. 3) allow overlapping gestures in order to remove the frustration born from waiting for one event to finish before activating another.

Embracing muscle memory was also deemed critical: generations of iPhone users were used to the ‘Home’ button and so its old location became the starting point for gestures. The addition of motion blur, the feeling of weight (light swiping for photos but more pressure being required for moving meatier apps) plus inertial scrolling (where webpages grind to a halt after an initial flick) all meant that making gestures on an iPhone felt like an extension of oneself… something that’s sorely missing from many rivals.

Losing the screens

Development of free-space gestures hasn’t been nearly so successful but nor has it been idle.

In 2013, the USB-connected Leap Motion infrared hand scanner and Microsoft’s Augmented Reality Hololens system embraced three-axis, hand-and-finger detection for interacting on-screen and in mixed reality (respectively).

Industrial Design firm, Teague, delved deeper into these movements by examining the principles of proprioception (our body’s innate ability to sense its own movements) to establish which gestures were most effective. It subsequently designed a ‘Seamful’ UI (as opposed to seamless) using three, broad categories.

A ‘Seam’ was an intrinsically known range of movement, like moving one’s thumb away from the index finger. This accurate movement was perfect for zero-to-100-per-cent, incremental adjustments. Weaker fingers were best used for tapping against a thumb (a principle called ‘Hierarchy’) and suited stepped-adjustments; for example, moving from 0-25-50-100%. However, ‘Directionality’ – navigation through space – proved more challenging. The wrist and fingers proved ineffective for vector control compared to simply moving your palm and steering like Superman.

Other modern gesture recognition technology involves cameras and shape recognition. A great example involves IBM AI Specialist, Nicholas Renotte demonstrating how, in just 30 minutes, machine learning could be used to create a real-time sign-language translator. Using Tensorflow’s Object Detection API he collects images, trains the Tensorflow API to detect them and then achieves live translation using a laptop’s webcam. While his results are primitive, the speed and simplicity of development are eye-opening.

While there’s still significant development in these areas, the results are still often experimental, basic and gimmicky. It’s little wonder that VR’s superior spatial recognition and controllers have instead been embraced by many developers.

We’ve seen incredibly immersive games (like Half-Life: Alyx), virtual, engineering sessions in Nvidia’s Holodeck and the likes of Hollywood concept artist, Colie Wertz, using the sensitive Oculus Touch controllers to create to-scale, 3D spaceships in minutes simply using a laptop and Gravity Sketch VR software. However, while the barriers to entry are dropping, you still need bulky, expensive headsets and powerful computers to make everything tick.

The current state of gesture control…

… is not good. Modern BMWs and Subarus currently use gestures to adjust climate control and volume but the experience is poor. This and applications which use phone screens for AR controls make people want buttons.

There are some high hopes for devices like Haptx’s force-feedback gloves which improve edge perception in virtual words (and on robot arms). Facebook also has a prototype to improve interactivity in its Metaverse.

Meanwhile, PixArt’s Multi-Object Gesture Recognition kit includes an SoC that enables basic, camera-based recognition but it’s a core technology rather than ready-made solution.

Arguably, the technology with most potential is radar. Texas Instruments’ mmWave radar sensors offer gesture recognition and monitoring without the privacy intrusion of cameras: making it of particular interest to the aged care space. The technology is also at the core Google’s IoT-friendly Project Soli ecosystem which uses radar and EdgeAI to decode what it sees and machine learning to identify intention. It can detect sub-millimetre motion and understand where someone’s attention is focused before learning what they want.

Unfortunately, words like ‘potential’ and ‘research’ still plague the gesture-control market. But powerful hardware and software development kits are widely available, waiting to be leveraged. So, while commercial, on-sellable applications aren’t abundant, the potential to create a killer application is high.