Hand Annotation Workflow

Written by: Ellie Na (2026)

OVerview

This page covers the design and implementation of a 3D annotation workflow in VR like how a user draws in space, saves a snapshot, and retrieves it later to explore whether hand annotation could function as a practical tool.

Requirement

Hand Gesture Tracking set up (see Unity Hand Gesture Tracking page)
XR Poke Interaction set up (see XR Poke Interaction page)

Why Not TouchScreenKeyboard?

TouchScreenKeyboard.Open() is Unity's built-in API for triggering a system keyboard. In VR it either doesn't appear, appears flat on the screen (not in world space), or can't be interacted with via hand tracking. It also gives you no control over layout, style, or behavior.

The custom approach here instead:

Lives in world space alongside your other UI
Is pressed with the same finger poke system as every other button
Directly manipulates TMP_InputField.text. no event routing needed
Can be generated automatically with one click in the Editor

Required Scripts

There are three scripts involved

VRKeyboard.cs that manages the target input field and processes key input

using UnityEngine;using TMPro;
public class VRKeyboard : MonoBehaviour{    private TMP_InputField targetInputField;
    /// <summary>    /// Call this to point the keyboard at a specific input field.    /// </summary>    public void SetTargetInputField(TMP_InputField inputField)    {        targetInputField = inputField;    }
    public void OnKeyPressed(string key, KeyType keyType)    {        if (targetInputField == null) return;
        switch (keyType)        {            case KeyType.Character:                targetInputField.text += key;                break;
            case KeyType.Backspace:                if (targetInputField.text.Length > 0)                    targetInputField.text = targetInputField.text                        .Substring(0, targetInputField.text.Length - 1);                break;
            case KeyType.Space:                targetInputField.text += " ";                break;
            case KeyType.Done:                gameObject.SetActive(false); // hide keyboard                break;        }    }}
public enum KeyType { Character, Backspace, Space, Done }

VRKeyButton.cs that sits on each key, knows its key type and value, calls the keyboard

using UnityEngine;
public class VRKeyButton : MonoBehaviour{ [SerializeField] private VRKeyboard keyboard; [SerializeField] private string keyValue; [SerializeField] private KeyType keyType = KeyType.Character;
// Called by SimpleTouchButton's onButtonPressed UnityEvent public void OnKeyPoked() { keyboard.OnKeyPressed(keyValue, keyType); }}

VRKeyboardGenerator.cs that auto-generates the full keyboard layout as editor tool

using UnityEngine;using UnityEngine.UI;using TMPro;
public class VRKeyboardGenerator : MonoBehaviour{    [SerializeField] private VRKeyboard keyboard;    [SerializeField] private float keySize = 60f;    [SerializeField] private float keySpacing = 8f;
    private string[] rows = new string[]    {        "1234567890",        "QWERTYUIOP",        "ASDFGHJKL",        "ZXCVBNM"    };
    [ContextMenu("Generate Keyboard")]    public void GenerateKeyboard()    {        ClearKeys();
        float startY = 0f;        int rowIndex = 0;
        foreach (string row in rows)        {            GenerateRow(row, startY - (rowIndex * (keySize + keySpacing)), row.Length);            rowIndex++;        }
        // Special keys        float specialY = startY - (rowIndex * (keySize + keySpacing));        CreateKey("DEL", -200f, specialY, keySize * 1.5f, keySize, KeyType.Backspace);        CreateKey(" ", 0f, specialY, keySize * 3f, keySize, KeyType.Space);        CreateKey("DONE", 200f, specialY, keySize * 1.5f, keySize, KeyType.Done);    }
    [ContextMenu("Clear Keys")]    public void ClearKeys()    {        // Remove all child GameObjects        for (int i = transform.childCount - 1; i >= 0; i--)            DestroyImmediate(transform.GetChild(i).gameObject);    }
    private void GenerateRow(string keys, float yPos, int count)    {        float totalWidth = count * keySize + (count - 1) * keySpacing;        float startX = -totalWidth / 2f + keySize / 2f;
        for (int i = 0; i < keys.Length; i++)        {            float xPos = startX + i * (keySize + keySpacing);            CreateKey(keys[i].ToString(), xPos, yPos, keySize, keySize, KeyType.Character);        }    }
    private void CreateKey(string label, float x, float y,                           float w, float h, KeyType type)    {        // Create GameObject        GameObject keyObj = new GameObject("Key_" + label);        keyObj.transform.SetParent(transform, false);
        // RectTransform        RectTransform rt = keyObj.AddComponent<RectTransform>();        rt.sizeDelta = new Vector2(w, h);        rt.anchoredPosition = new Vector2(x, y);
        // Image (rounded background)        Image img = keyObj.AddComponent<Image>();        img.color = new Color(0.9f, 0.9f, 0.9f, 1f);
        // Button        keyObj.AddComponent<Button>();
        // Box Collider (trigger, for poke detection)        BoxCollider col = keyObj.AddComponent<BoxCollider>();        col.size = new Vector3(w, h, 20f);        col.isTrigger = true;
        // VRKeyButton        VRKeyButton keyButton = keyObj.AddComponent<VRKeyButton>();        // (set keyboard reference and keyValue via serialized fields)
        // SimpleTouchButton        SimpleTouchButton touch = keyObj.AddComponent<SimpleTouchButton>();        // wire onButtonPressed → keyButton.OnKeyPoked() in Inspector
        // Label text        GameObject textObj = new GameObject("Text");        textObj.transform.SetParent(keyObj.transform, false);        TextMeshProUGUI tmp = textObj.AddComponent<TextMeshProUGUI>();        tmp.text = label;        tmp.alignment = TextAlignmentOptions.Center;        tmp.fontSize = 24;        tmp.color = Color.black;        RectTransform textRt = textObj.GetComponent<RectTransform>();        textRt.anchorMin = Vector2.zero;        textRt.anchorMax = Vector2.one;        textRt.sizeDelta = Vector2.zero;    }}

How to use the generator?

Create an empty GameObject in the Hierarchy, name it VRKeyboard - Add the VRKeyboardGenerator component - Assign the VRKeyboard reference in the Inspector - Right-click the component header 'Generate Keyboard'

The full layout is created instantly as child GameObjects, all saved in the scene, not generated at runtime.

Run in your headset!

Common Issues

These are the issues I encountered while building this keyboard and how I troubleshot them.

#1 Keys don't respond to pokes?

Confirm BoxCollider on each key has Is Trigger = true
Check HandColliderSetup is active on both hands
Verify SimpleTouchButton.onButtonPressed is wired to VRKeyButton.OnKeyPoked()

#2 Text appears in wrong input field?

SetTargetInputField() must be called each time you open the keyboard for a new field — the target is not set automatically

#3 Keyboard not visible in headset?

Confirm the keyboard is a child of (or positioned within) a World Space Canvas
Check it's included in Build Settings scenes

# 4Generator doesn't appear in right-click menu?

The [ContextMenu] attribute only works in the Unity Editor and make sure you're right-clicking the component header in the Inspector, not in the Hierarchy