Tangram Game

Game tailored for gestural interaction with fogscreens

Overview

The project aimed to explore and evaluate the potential of using mid-air gestural interactions with a fogscreens as an engaging gaming medium for children diagnosed with attention-deficit/hyperactivity disorder (ADHD).

The collaborative, interactive Tangram puzzle game was disigned to support self-regulation, collboration, cognitive and communication skills, such as, listening and maintaining focus.

Goal

The goal was to evaluate and optimize usability for mid-air gestural interaction, ensuring the interface was intuitive, effective, and motivating for young users. Key focus areas included:

Designing seamless interaction patterns for object selection and drag-and-drop in a fog-based interface;
Understanding children's engagement, challenges, and prefernces in touchless digital interaction;
Refining the UX based on behavioral insights, interaction performance, and qualitative feedback.

This project contributes to innovative UX solutions in immersive gaming and therapeutic applications, bringing the gap between human-technology interaction and cognitive training for neurodiverse users.

Role

Interaction designer, Developer, UX researcher

Tools

Development: C++, Microsoft Kinect SDK

Data analysis: SPSS, Python

Team

Two UX researchers, one developer

Timeline

Aug - Oct 2018 (3 monthes)

Emphathy & Define

During the ideation phase, our team closely colaborated with neuropsychologists and observed therapy sessions to clarify and define essential game requirements tailored specifically for children with ADHD.

These expert insights allowed us to identify two critical needs for the design:

Avoid touch-based and wearable devices. Children with attention defficits may become easily frustrated, leading to potential damage or breaking of physical devices. Thefore, the chosen interaction should remain fully touchless and free from wearable technology.
Balance collaboration and competition. To sustain children's motivation and engagement throughout therapeutic sessions, the game design should throughfully integrate collaboartive teamwork alongside competitive elements, fostering both cooperation and friendly competition among players.

These foundational insights informed our approach to developing an engaging, safe, and effective therapeutic gaming experience.

Ideate

Brainstormed multiple interaction scenarious and game concepts tailored to children with ADHD;
Selected Tangram puzzles for their strong cognetive benefits and potential to foster collaboration and problemsolving;
Explored mid-air gestures as an intuitive, contact-free input methods, ideal for maintaing focus and reducing frustration in therapeutic enviroments;
Chose to use a fogscreen as the interaction surface due to its non-breakable, touchless nature, making it safe and durable for clinical use

Prototype

We developed the game prototype in two parallel phases: the graphical game interface and the gesture-based interaction system.

Designed the core gameplay around Tangram puzzle mechanics, knwn to enchance spatial reasoning and self-regulation;
Created a visual interface featureing a control panel and play area, optimized for projection on a fogscreen (black backgroung, contrast clear element colors);
Built a gesture interaction layer using Microsoft Kinect SDC to support natural mid-air gestures;
Implemented key interaction scenarious such as puzzle piece selection and drag-and-drop to support intuitive, touchless manipulation.
Defined paired gameplay roles: the instructor, who guides, and the implementer, who performs gestures - promoting collaboration and shared attention.

Graphical game interface specifically designed for fogscreen interaction - featuring high-contrast visuals and clear spatial layout to supprt mid-air gesture control.

Graphical interface with different visual feedbacks

Green color for curently active element

A dimond piece was selected from the caonrol panel (turned green) and is curently moving to the play zone

Yellow color for selected element

After arriving at its designated place, the diamond piece turns yellow, signifying its active state (available for manipulations in the play area)

Red color for informing about collision

The triangle turns red when hovers over the blue pieces, signifying a non-permitted collision of the shapes

Interactions in game

Usabilllity Testing

Usability study was conducted with 18 children with attention deficits (aged 8-12) across 6 game sessions, allowing us to observe the effects of prolonged fogscreen-based mid-air interaction.

Quantitative Results:

Game Success Rate: 81% average, indicating good overall task completion.
Movement Time: Average 5.34 seconds, showing efficiency in piece manipulation.
Success Rate for drag-and-drop: High, though certain movements required adjustments due to interaction challenges.

Qualitative Insights:

Most participants rated the game as easy or moderately difficult and expressed a strong willingness to play again.
Positive feedback: Children enjoyed assembling the puzzle and engaging with the fog-based interface, often expressing excitement and creativity in their movements.
Negative feedback: Some reported fatigue, visibility issues with the fog, and occasional lagging in system response.
Emotional responses such as joy and engagement were frequently observed, alongside some frustration during technical difficulties.

Observational Findings:

Users demonstrated spontaneous playful behaviors (e.g., drawing in the fog or interacting with it outside of game tasks).
The intuitive nature of the interaction allowed most children to quickly understand how to play without additional instructions.
Interaction challenges primarily arose from tracking limitations and physical fatigue rather than design flaws.

Usability issues that we identified in usability tests:

Issue 1. Missed or unclear visual feedback

When participants selected puzzle pieces from the placeholder on the control panel, visual feedback was displayed on the puzzle piece in the play zone. However, due to the large size of fogscreen and the limited visibility caused by fog turbulence, the participants often overlooked feedback that was not aligned with their hand position or line of sight. Succes rate for selection was 45%, as users often did repetitive selections of already selected pieces or selections of wrong pieces

Solution: Visual feedback should be displayed where the hand is currently interacted with fogscreen.

Issue 2. Premature loss of control

Children instinctively pulled the hand toward their body; exiting the fog unintentionally dropped the piece and required readjustment.

Solution: Definition of the exit point from the drag-and-drop as closer to a user (relative to the entry point along the depth (Z) axis of the interaction space).

Outcomes

The Tangram game delivered an engaging and intuitive user experience, successfully combining mid-air gestures with fogscreen interaction.
This innovative setup effectively supported both task success and user enjoyment, with measurable improvements in usability observed over time.
The gesture-based interaction system — incorporating selection and drag-and-drop actions — proved to be robust and adaptable. This system can be repurposed for other projects without significant modifications, offering a versatile solution for touchless interactive applications.

Publications

Remizova, V., Sand, A., MacKenzie, I. S., Špakov, O., Nyyssönen, K., Rakkolainen, I., Kylliäinen, A., Surakka, V., & Gizatdinova, Y. (2023). Mid-Air Gestural Interaction with a Large Fogscreen. Multimodal Technologies and Interaction, 7(7), 63. https://www.mdpi.com/2414-4088/7/7/63

Remizova, V., Sand, A., Špakov, O., Lylykangas, J., Qin, M., Helminen, T., Takio, F., Rantanen, K., Kylliäinen, A., Surakka, V., & Gizatdinova, Y. (2025). Foggy Fun: An Exploration of Mid-Air Gestural Interaction with a Fogscreen by Children with Attention Deficits. Behaviour & Information Technology, pp. 1–24. doi.org/10.1080/0144929X.2025.2469661

Page updated

Google Sites

Report abuse