Skip to main content

Hand Gesture

In this tutorial, you’ll learn how to use Hand Tracker to create customized hand tracking effects. You’ll also learn how to show different objects with the event and rotate objects using the Visual Scripting panel.

This video uses Speech Synthesis powered by Tiktok-SAMI

Prepare

Start by downloading the following package and locating the Unfinished_HandGesture project:

hand_gesture.zip

You will see all the required assets have been imported into the Assets panel.

hand gesture materials in the assets panel

Open the Preview menu in the Preview panel, choose + Add media, and select the hand.mp4 file.

add media in the preview panel  hand in the preview panel

Index Finger Gesture Controller

First, try to create an effect of a thunder sign attached to the index finger. It should only appear when it detects the index finger gesture.

Create an Index Finger Hand Tracker

Start by adding a Hand Tracker object by clicking the Add button [+] > AR Tracking > Hand Tracker in the Hierarchy panel. Create an Empty object and nest the Hand Tracker under it. Create another Empty object and nest it under the Hand Tracker.

hand tracker and empty object in the hierarchy panel

Rename these object like below or other names that are reasonable for your hand gesture controller (you don’t need to add the [MODIFY ME] suffix):

rename objects

Select the IndexF object, or the object with Hand Tracker, and find the Hand Tracker component in the Inspector panel. Change the Anchor from Hand Center to Index Finger Tip. Then select the Rotate AxisZ checkbox so that the object will have the same Z rotation as the finger.

set the anchor of indexf  disable the rotate axisz property of indexf

Adjust Model, Material, and Transform

Select the Lightning object in the Hierarchy panel.

select the lightning object in the hierarchy panel

Add a Model component by clicking + Add component > 3D > Model in the Inspector panel.

add a model component in the inspector panel

In the Model component, change Mesh to lightingBolt, and Material to be lightning_mat.

set the mesh and material properties of the model component in the inspector panel

In the Transform component, change the Scale values, X, Y, and Z, to 0.1, and the Y Position to 2. This will create some distance between the fingertip and the lightning object.

set the transform of the model in the inspector panel

You will see the preview like this:

preview of the lightning and fingertip

Control 3D Lightning Rotation With Visual Scripting

Next you need to control the rotation of the light object and keep it rotating all the time. Select the Lightning object to open the Inspector panel.

Find the Rotation parameter under Transform and click + to select Set Rotation. You can see the node was created in the Visual Scripting panel.

rotation parameter of the transform component
lightning node in visual scripting

Next you need to control the rotate speed. To do this, you can go to the Visual Scripting panel, create a variable by clicking Variables > Add variable. Then change the variable name to RotateSpeed, Type to Number, and Value to 240 (240 degrees per second).

create a variable in visual scripting

Next click the Add button [+] next the variable name and select Get Variable. This will create a variable node in the Visual Scripting panel.

get variable in visual scripting
rotatespeed node in visual scripting

Next you need to add the Get time node to control the rotation by clicking Add node > Time > Get time. In the same way, you can add the Mod and Multiply nodes.

get time node in visual scripting

You also need a Combine node to convert these variables to Vector 3f. Add the Combine node and then click the data type on the Combine node and set the output data to Vector3f.

add a combine node to convert variables to vector 3f

Now, let’s link the nodes together. Link trigger from the Update node to Set of Lightning so it will be updated every frame. Link the following together: Get time, Value1 of Multiply, RotateSpeed, Value2 of Multiply. Then link the value of Multiply to Value1 of Mod, to Rotation of Lightning. Change Value2 of Mod to 360, so that the Y angle will change between 0 and 360 (so it is smooth).

link nodes in visual scripting
tip

Feel free to change the RotateSpeed variable to control the speed of the rotation.

Control the Controller's Visibility

Next you need to hide and show the controller object using the Hand gesture Event. You can turn off the visibility of the Index Finger Gesture Controller to hide them at the beginning.

control the visibility of the index finger gesture controller in the hierarchy panel

Let’s add the Gesture (Index Finger up) node by right-clicking the panel and selecting it from the Hand category and add the Set Visibility node from the Utility category. Duplicate the Set visibility node and check the Value. You’ll then need to drag the Index Finger Gesture Controller to the Visual Scripting panel and connect them together like below.

add the gesture node in the visual scripting panel

Hand Open Gesture Controller

For this session, we will try to implement the effect of having five accessories assigned to each fingers and they won’t show up until it detects the hand open gesture.

Create Five Finger Trackers

Just like you did in the Creating an Index Finger Hand Tracker section, you need to create another five finger tracker, but instead of adding the Lightning object, you need to add 3D Plane objects under each finger tracker.

You can start by adding a Hand Tracker entity by clicking the Add button [+] > AR Tracking > Hand Tracker in the Hierarchy panel. Then create an Empty object and drag the Hand Tracker under it.

add a hand tracker object in the hierarchy panel

In the same way, you can add a 3D plane by clicking the Add button [+] > 3D > Plane in the Hierarchy panel. Nest it under the Hand Tracker. Then you can change the name of each object to make it more sense (you don’t need to add the “MODIFY ME” suffix).

add a plane object in the hierarchy panel

Go to the Hierarchy panel and select IndexF. Find the Hand Tracker component in the Inspector panel. Change the Anchor from the default Hand Center to Index Finger Tip. Select the Rotate AxisZ checkbox.

hand tracker component in the inspector panel

You will now see a default 3D plane following your index finger in the Preview panel.

finger with plane in the preview panel

Adjust Material and Transform

Now select the Plane object in the Hierarchy panel. Change the material from default to watermelon_matin the Inspector panel under the Model component.

change the material of the model component in the inspector panel

In the Transform component, change the Scale, the X, Y, and Z values, to 0.258, and the Y value of Position to 2. This will create some distance between fingertip and the plane.

set the transform of the plane in the inspector panel

Add Other Hand Trackers

You can simply duplicate the index finger hand tracker 4 times, rename them and change the Anchor of each Hand Tracker component to Thumb Tip/Middle Finger Tip/Ring Finger Tip/Pinkie Tip.

Then change the material to cactus_mat/doughnut_mat/ribbon_mat/heart_mat. After these steps you will see the following in the Preview panel:

different hand trackers in the hierarchy panel  preview of hand with different materials assigned to each finger

Control Controller's Visibility

Next you need to hide and show the controller object using the Hand Gesture Event. You can turn off the visibility of the Index Finger Gesture Controller to hide them at the beginning.

finger and hand gesture controllers in the hierarchy panel

Let’s add the Gesture (Hand Open) node by right-clicking the panel and selecting it from the Hand category and add the Set Visibility node from the Utility category. Duplicate the Set Visibility node and select the Value checkbox. You’ll then need to drag Hand Open Gesture Controller to the Visual Scripting panel and connect them together like below.

add gesture hand open node and set visibility node in the visual scripting panel

Refresh the preview and now we can see your very own effect in the Scene and Preview!

The logic will be: when you show index finger up gesture, it will the 3D lighting object. When you show the hand open gesture, it will show the other 2D objects and hide the lighting object.