Projecting a Cube with a Number in ARKit

Question

Tomato OP

Created 3w

Replies 4

Boosts 0

Participants 2

I'm a novice in RealityKit and ARKit. I'm using ARKit in SwiftUI to show a cube with a number as shown below.

import SwiftUI
import RealityKit
import ARKit

struct ContentView : View {
    var body: some View {
        return ARViewContainer()
    }
}

#Preview {
    ContentView()
}

struct ARViewContainer: UIViewRepresentable {
    typealias UIViewType = ARView
    
    func makeUIView(context: UIViewRepresentableContext<ARViewContainer>) -> ARView {
        let arView = ARView(frame: .zero, cameraMode: .ar, automaticallyConfigureSession: true)
        arView.enableTapGesture()
        return arView
    }

    func updateUIView(_ uiView: ARView, context: UIViewRepresentableContext<ARViewContainer>) {
    }
}

extension ARView {
    func enableTapGesture() {
        let tapGestureRecognizer = UITapGestureRecognizer(target: self, action: #selector(handleTap(recognizer:)))
        self.addGestureRecognizer(tapGestureRecognizer)
    }
    
    @objc func handleTap(recognizer: UITapGestureRecognizer) {
        let tapLocation = recognizer.location(in: self) // print("Tap location: \(tapLocation)")
        guard let rayResult = self.ray(through: tapLocation) else { return }
        
        let results = self.raycast(from: tapLocation, allowing: .estimatedPlane, alignment: .any)
        if let firstResult = results.first {
			let position = simd_make_float3(firstResult.worldTransform.columns.3)
			placeObject(at: position)
		}
    }
    
    func placeObject(at position: SIMD3<Float>) {
        let mesh = MeshResource.generateBox(size: 0.3)
        let material = SimpleMaterial(color: UIColor.systemRed, roughness: 0.3, isMetallic: true)
        let modelEntity = ModelEntity(mesh: mesh, materials: [material])
        var unlitMaterial = UnlitMaterial()
        if let textureResource = generateTextResource(text: "1", textColor: UIColor.white) {
            unlitMaterial.color = .init(tint: .white, texture: .init(textureResource))
            modelEntity.model?.materials = [unlitMaterial]
            let id = UUID().uuidString
            modelEntity.name = id
            modelEntity.transform.scale = [0.3, 0.1, 0.3]
            modelEntity.generateCollisionShapes(recursive: true)
            let anchorEntity = AnchorEntity(world: position)
            anchorEntity.addChild(modelEntity)
            self.scene.addAnchor(anchorEntity)
        }
    }
    
    func generateTextResource(text: String, textColor: UIColor) -> TextureResource? {
        if let image = text.image(withAttributes: [NSAttributedString.Key.foregroundColor: textColor], size: CGSize(width: 18, height: 18)), let cgImage = image.cgImage {
            let textureResource = try? TextureResource(image: cgImage, options: TextureResource.CreateOptions.init(semantic: nil))
            return textureResource
        }
        return nil
    }
}

I tap the floor and get a cube with '1' as shown below.

The background color of the cube is black, I guess. Where does this color come from and how can I change it into, say, red? Thanks.

Boost

Answer 1

Vision Pro Engineer OP

Apple

3w

Hi @Tomato,

I believe the issue lies here:

            unlitMaterial.color = .init(tint: .white, texture: .init(textureResource))
            modelEntity.model?.materials = [unlitMaterial]

The call to set the materials on the entity have replaced the red color material you set earlier.

Instead do modelEntity.model?.materials = [material, unlitMaterial]

Also, the texture that you make for the string ("1" in this case) needs to have a transparent background rather than black.

I think you can do that here:

if let image = text.image(withAttributes: [NSAttributedString.Key.foregroundColor: textColor], size: CGSize(width: 18, height: 18)), let cgImage = image.cgImage {

by adding the backgroundColor and setting it to clear. That should work but I haven't run that code to make sure.

Let me know if you have further questions.

1

Answer 2

Tomato OP

3w

Thank you for your kind assistance, Vision Pro Engineer.

No. 1, it seems that NSAttributedString.Key.backgroundColor has nothing to do with my issue. No. 2, I think you only suggest that I change the modelEntity.model?.materials line as follows.

modelEntity.model?.materials = [unlitMaterial]

into

modelEntity.model?.materials = [material, unlitMaterial]

Yes, I do get a material color other than black with or without NSAttributedString.Key.backgroundColor. I then lose the text material. So the text number is MIA.

func placeObject(at position: SIMD3<Float>) {
    let mesh = MeshResource.generateBox(size: 0.3)
    let material = SimpleMaterial(color: UIColor.systemRed, roughness: 0.3, isMetallic: true)
    let modelEntity = ModelEntity(mesh: mesh, materials: [material])
    var unlitMaterial = UnlitMaterial()
    if let textureResource = generateTextResource(text: "1", textColor: UIColor.white) {
        unlitMaterial.color = .init(tint: .white, texture: .init(textureResource))
        modelEntity.model?.materials = [material, unlitMaterial]
        let id = UUID().uuidString
        modelEntity.name = id
        modelEntity.transform.scale = [0.3, 0.1, 0.3]
        modelEntity.generateCollisionShapes(recursive: true)
        let anchorEntity = AnchorEntity(world: position)
        anchorEntity.addChild(modelEntity)
        self.scene.addAnchor(anchorEntity)
    }
}

func generateTextResource(text: String, textColor: UIColor) -> TextureResource? {
    if let image = text.image(withAttributes: [NSAttributedString.Key.foregroundColor: textColor, NSAttributedString.Key.backgroundColor: UIColor.clear], size: CGSize(width: 18, height: 18)), let cgImage = image.cgImage {
        let textureResource = try? TextureResource(image: cgImage, options: TextureResource.CreateOptions.init(semantic: nil))
        return textureResource
    }
    return nil
}

0

Answer 3

Vision Pro Engineer OP

Apple

2w

Hi @Tomato,

Thanks for getting back to me with more clarification.

I think your goal is to have a box with some rendered text on it that has a specific background color? If that is correct I'd change your texture generation code like this:

    if let image = text.image(withAttributes: [NSAttributedString.Key.foregroundColor: textColor, NSAttributedString.Key.backgroundColor: UIColor.red], size: CGSize(width: 18, height: 18)), let cgImage = image.cgImage {
        let textureResource = try? TextureResource(image: cgImage, options: TextureResource.CreateOptions.init(semantic: nil))
        return textureResource
    }
    return nil
}

That texture should now be a red square with the number rendered into it.

I misread your code and you are correct that this:

modelEntity.model?.materials = [material, unlitMaterial]

is not appropriate.

The box generated by MeshResource.generateBox(size: 0.3) uses 1 material for all 6 faces, or a list of 6 materials, one for each face. A detailed discussion can be found here with the splitFaces parameter.

If you generate your texture with the desired background color and use the single unlitMaterial (i.e. modelEntity.model?.materials = [unlitMaterial]) parameter you should have a box with red faces and the string in your text value.

1

Answer 4

Tomato OP

2w

Thank you for your kind help, Vision Pro Engineer. With the lines of code you have provided, I get the following cube, which is quite close to what I need.

What I want to have is a cube

with a number on each surface. I guess that's cannot be achieved.

0