Ninja fruit with STM32N6!

This project demonstrates the power of STM32N6 with a ninja fruit or pop the bubbles! Game. It uses a pose estimation model to detect certain gestures and their positions to interact with the game.

This is an entry for the STM32 Edge AI Contest

Do you want to do some exercise as you play with the latest hardware accelerated models? I have created a game that demonstrates how we can interact in real time with the fast-inference times of STM32N6 and the camera-display integration. It can be useful in trade shows to showcase the capabilities of edge AI models in an interactive way to any person visiting the booth.

I was inspired by games that my daughter loves to play, she loves to pop bubbles! This kind of games help build better coordination in a playful way.

This is an interactive demo implemented on the STM32N6570 Discovery Kit, combining real-time computer vision, gesture recognition, and graphical rendering. The game leverages the MoveNet embedded pose-detection model running on-device to capture the player’s body movements (arm swipes, wrist positions, and nose position). These gestures are mapped into game interactions, allowing the player to slice virtual fruits (or pop virtual bubbles) displayed on the LCD.

The game tracks a score, and runs until you miss 5 fruits/bubbles. There are incorporated game levels, with increasing difficulty.

Game Modes

There are two primary gameplay modes in the game, selectable by pressing the USER1 button before starting or after a game over:

Slice Mode

Player slices using swiping gestures (left/right arm swipes, sword-like strikes).
A swipe trajectory is calculated from tracked wrist positions, and intersection with fruit objects triggers slicing.
Fruits are split graphically into halves with an animation.

Pop Mode

Fruits are popped by touching them with either wrist.
In Nose Mode (special effect), the player must pop fruits using their nose instead.
This creates an alternate, more playful interaction style.

Occasionally, special fruits appear with unique effects: They activate a nose mode. For a limited time, fruits must be popped using the player’s nose keypoint instead of wrists.

Gameplay Flow

Start Screen

The LCD shows the title, instructions, and current mode. The game starts when the first valid gesture is detected.

Fruit Spawning

Fruits are randomly generated at the top of the screen with varying horizontal drift and fall speed.

Difficulty increases over time: the spawn rate accelerates every 30 seconds (new level).

Interaction

In Slice Mode: swiping gestures produce virtual blade trajectories that cut fruits.

In Pop Mode: collision between wrist/nose positions and fruit centers pops fruits.

Each fruit type (apple, orange, banana, strawberry) has a different score value.

Game Over

If the player misses more than 5 fruits, the game ends.

A “GAME OVER” screen shows the final score and level, and any gesture will restart the game.

Fruits are drawn as simple colored circles, each fruit has a different score associated.

Software components

The project stems from the Getting-started Pose Estimation demo project from STM32. A simplified block diagram that shows the workflow to incorporate the game logic is shown:

Pose estimation model

The project uses movenet, with the provided model checkpoint in STM’s model Zoo.

The keypoints for that model are the following, and are used to detect the gestures. More gestures can be easily coded if needed, using the desired keypoints.

#define KEYPOINT_NOSE 0

#define KEYPOINT_LEFT_SHOULDER 1

#define KEYPOINT_RIGHT_SHOULDER 2

#define KEYPOINT_LEFT_ELBOW 3

#define KEYPOINT_RIGHT_ELBOW 4

#define KEYPOINT_LEFT_WRIST 5

#define KEYPOINT_RIGHT_WRIST 6

#define KEYPOINT_LEFT_HIP 7

#define KEYPOINT_RIGHT_HIP 8

#define KEYPOINT_LEFT_KNEE 9

#define KEYPOINT_RIGHT_KNEE 10

#define KEYPOINT_LEFT_ANKLE 11

#define KEYPOINT_RIGHT_ANKLE 12

Configuration parameters, found in gesture_detection.h

The names and values of the constants are shown:

MIN_CONFIDENCE 0.5f // Minimum confidence to accept a keypoint as valid
SWIPE_MIN_DISTANCE 0.3f // Minimum distance for swipe (normalized)
SWIPE_MIN_SPEED 0.05f // Minimum speed for swipe
SWIPE_MAX_FRAMES 8 // Maximum frames for swipe gestureGESTURE_DISPLAY_TIME 2000 // How long to display gesture (ms)

Game configuration parameters (can be found in ninja_fruit_game.h)

The names and values of the constants are shown:

MAX_FRUITS 8

FRUIT_SPAWN_INTERVAL 2000 // milliseconds

FRUIT_FALL_SPEED 0.15f // normalized screen coordinates per second

FRUIT_SIZE 40 // pixels

SLICE_TOLERANCE 40 // pixels for slice detection

MAX_MISSED_FRUITS 5 // game over condition

SLICE_ANIMATION_TIME 500 // milliseconds

SPECIAL_MODE_DURATION 20000 // milliseconds special modes last

MODE_MESSAGE_DURATION 2000 // milliseconds mode change messages display

SPECIAL_FRUIT_CHANCE 20 // Calculated as random % SPECIAL_FRUIT_CHANCE, so chance is 1/20, ~5% chance

Gesture logic for the following movements is implemented:

GESTURE_RIGHT_ARM_SWIPE_LEFT
GESTURE_RIGHT_ARM_SWIPE_RIGHT

GESTURE_LEFT_ARM_SWIPE_LEFT

GESTURE_LEFT_ARM_SWIPE_RIGHT

GESTURE_SWORD_OVERHEAD_STRIKE

GESTURE_SWORD_SIDE_SLASH

This gives further flexibility in case we want to change game logic so that only certain angles are valid, or to extend the application of pose gestures to other sword-like games that need to know attack or parry movements.

This is a proof of concept, some parameters can be adjusted to increase the sensitivity to the gestures of interest and make the game more robust. You only need to modify the values you see fit in the places mentioned above.

I hope you enjoy this project!