Imitating a grip on an object

Question

Adamc93 OP

Created 2w

Replies 1

Boosts 0

Participants 2

I'm playing about with the hand tracking systems in reality kit / Vision Pro

I thought it would be interesting if I could attach a virtual object to a hand when the hand is gripping (thought it would be fun to attach a basic cylinder to mimic a wand from Harry Potter)

I'm able to detect when the user is gripping but having trouble placing an object as though it's within the hand.

The simplest version of this is using an AnchorEntity pointing to the user's palm which kind of works, but quickly breaks the illusion when you rotate the wrist or hand.

It seems as though I will have to roll my own anchor entity using the various points of the user's hand and I thought calculating some median point between the thumb and little finger tips would be a good start but it's proven a little difficult as we need both rotation and position.

I'm already out of my depth with reality kit and matrices (and thanks to ChatGPT) I have some code, but as soon as I apply the position manually (as opposed to a hand anchor entity) it fails to render on the user's hand.

It feels like this should already have been something someone has looked in to, any ideas on what might be the issue here?

Note: HandTrackingSystem.handTracking is a HandTrackingProvider()

guard let anchors = HandTrackingSystem.handTracking.latestAnchors.leftHand else {
    return
}

if
    let thumb = anchors.handSkeleton?.joint(.thumbTip),
    let little = anchors.handSkeleton?.joint(.littleFingerTip)
{
    let thumbPos = simd_make_float3(thumb.anchorFromJointTransform.columns.3)
    let littlePos = simd_make_float3(little.anchorFromJointTransform.columns.3)

    let midPos = (thumbPos + littlePos) / 2
    let direction = normalize(littlePos - thumbPos)
    let rotation = simd_quatf(from: [0, 1, 0], to: direction)

    wandEntity.transform.translation = midPos
    wandEntity.transform.rotation = rotation

    content.add(wandEntity)
}

Boost

Answer 1

Vision Pro Engineer OP

Apple

2w

Hi @Adamc93

Could you try replacing your thumbPos and littlePos calculations with the following code?

let thumbPos = simd_make_float3((anchors.originFromAnchorTransform * thumb.anchorFromJointTransform).columns.3)
let littlePos = simd_make_float3((anchors.originFromAnchorTransform * little.anchorFromJointTransform).columns.3)

The key difference here being that by multiplying by the originFromAnchorTransform matrix the positions you've calculated will then be in world-space (instead of anchor-space), which is the same coordinate space that it appears your wandEntity is in as well.

Just to be safe you can also update the code that sets the translation and rotation of your wandEntity to the following to ensure that you are updating the world-space position and orientation of your entity (as opposed to the local-space translation and rotation):

wandEntity.setPosition(midPos, relativeTo: nil)
wandEntity.setOrientation(rotation, relativeTo: nil)

Let me know if you have any other questions!

P.S. Consider taking a look at this forum thread about picking up and throwing objects using hand tracking! Some of the code shared in that discussion may be relevant to what you're trying to achieve here.

1