ARKit camera transform orientation vector doesn't match physical device heading (despite `.gravityAndHeading`)

Question

himanshujnaidu OP

Created 3w

Replies 1

Boosts 0

Participants 2

Hi all,

I'm working on an ARKit-based iOS app where I need to accurately determine the direction the device is facing to localize objects in the real world. I'm using:

let config = ARWorldTrackingConfiguration()
config.worldAlignment = .gravityAndHeading

Thus, I would expect the world alignment to behave as given in the gravityAndHeading page.

The AR session is started after verifying that CLLocationManager.headingAccuracy <= 20, and the compass appears to be calibrated.

However, I'm seeing a major inconsistency: When the rear camera is physically pointed toward true North, I would expect:

cameraTransform.columns.2.z ≈ -1  // (i.e. ARKit's -Z pointing North)

But instead, I'm consistently seeing:

cameraTransform.columns.2.z ≈ +0.97  // Implies camera is facing South

Meanwhile, the translation vector behaves as expected: As I physically move North, cameraTransform.columns.3.z becomes more negative, matching the world’s +Z = South assumption.

For example, let's say I have the device in landscapeRight (or landscapeLeft for UIDeviceOrientation). Let's say the device rear camera is pointing towards True North, and I start moving towards True North. I get something like this:

Camera Transform = simd_float4x4(
[
[0.98446155, -0.030119859, 0.172998, 0.0], 
[0.023979114, 0.9990097, 0.037477385, 0.0], 
[-0.17395553, -0.032746706, 0.98420894, 0.0], 
[0.024039675, -0.037087332, -0.22780673, 0.99999994]
])

As you can see, the cameraTransform.columns.2.z is positive despite the rear camera pointing towards True North, while cameraTransform.columns.3.z is correctly positive as the device is moving towards True North.

So here is my question:

Why is cameraTransform.columns.2.z positive when the rear camera is physically facing North?

Any clarity would be deeply appreciated. I've read the documentation and tested with different heading accuracies and AR session resets, but I keep running into this orientation mismatch.

Thanks in advance!

Boost

Answer 1

Vision Pro Engineer OP

Apple

3w

Hello @himanshujnaidu,

I can't tell for sure but it sounds like the matrix you show above is different than how you are interpreting it.

Camera Transform = simd_float4x4(
[
[0.98446155, -0.030119859, 0.172998, 0.0],  // column 0
[0.023979114, 0.9990097, 0.037477385, 0.0],  // column 1
[-0.17395553, -0.032746706, 0.98420894, 0.0], // column 2
[0.024039675, -0.037087332, -0.22780673, 0.99999994] // column 3
])

First up, keep in mind that float4x4's init method is column major. So each 'row' in that code is a column in the matrix.

So in this case the position of the camera relative to the origin is 0.024039675, -0.037087332, -0.22780673

Second, the upper-left part of the float4x4 matrix is a float3x3 and represents the orientation. The eulerAngles property on the frame.camera re-interprets this 3x3 matrix as the familiar Euler angles.

You can find the orientation of the camera relative to the frame via let rotation = frame.camera.eulerAngles

rotation.y will be zero when facing north, 90° when facing east.

The individual elements in the matrix will not tell you the orientation typically. So I would not expect any value in that matrix alone to correspond to the orientation of the camera relative to the origin.

0