A-Frame Hand-tracking

A-Frame native hand-tracking (https://aframe.io/docs/1.4.0/components/hand-tracking-controls.html) does not support customized interaction in the scene, although giving you a good mesh representation of the hand. However, you can use Handy-Work library https://github.com/vincentianxing/handy-work,

The Handy-Work does not have a good documentation. Here's a basic tutorial.

Set Up Your Project

Create a new HTML file and include the A-Frame and Handy-Work libraries by adding the following script tags in the head of your HTML file:

<!DOCTYPE html>

<html>

<head>

</head>

Create the A-Frame Scene

Let's start by adding a basic A-Frame scene in the body of your HTML:

<body>

<a-scene>

</a-scene>

</body>

</html>

You can add any 3D objects you want to interact with. For this tutorial, we'll add a simple sphere:

<a-sphere position="0 1.5 -3" radius="1" color="#EF2D5E"></a-sphere>

Add Handy-Work Component

Now, let's add a new A-Frame component that will use the Handy-Work library to track the user's hands. Add this script inside your head tag:

AFRAME.registerComponent('hand-tracking', {

init: function () {

const sceneEl = this.el;

handyWork.start();

handyWork.addEventListener('frame', (e) => {

// access hand data with e.detail

const hands = e.detail;

// to do: handle hand tracking data

});

}

});

</script>

This initializes the hand tracking when the scene is loaded and listens for new hand tracking frames. Every time a new frame is received, the listener function will be called with an event object containing the tracked hand data.

Use Hand Data

Now you'll want to use the hand data to interact with your 3D objects. Let's make the sphere move when the user pinches their fingers. Update your script like so:

AFRAME.registerComponent('hand-tracking', {

init: function () {

const sceneEl = this.el;

const sphereEl = sceneEl.querySelector('a-sphere');

handyWork.start();

handyWork.addEventListener('frame', (e) => {

const hands = e.detail;

hands.forEach(hand => {

if (hand.pinchStrength > 0.7) {

// move the sphere when the user pinches their fingers

const {x, y, z} = hand.pointerPose.position;

sphereEl.setAttribute('position', {x, y, z});

}

});

}

});

</script>

This will move the sphere to the position of the hand when the user pinches their fingers.

Add the Hand-Tracking Component to the Scene

Finally, add your new hand-tracking component to the A-Frame scene:

<a-scene hand-tracking>

<a-sphere position="0 1.5 -3" radius="1" color="#EF2D5E"></a-sphere>

</a-scene>

You should now be able to move the sphere in your A-Frame scene using hand tracking.