Solving Sticky UI: Head-Anchored Buttons in RealityKit

What if the UI wasn’t attached to the user at all? With a world-anchored HUD (Heads-Up Display), you place your UI elements at a fixed position in the physical environment. For example, a control panel might hover next to the virtual object it manipulates.

This approach grants the UI a powerful sense of presence and stability. It becomes part of the scene. The user can look away from it, focus on the main content, and then glance back when they need to interact. The major downside? The user can lose the UI if they turn or walk too far away. You might need to add subtle cues (like a glowing indicator at the edge of their vision) to guide them back.

Solution 3: The Hybrid Approach (The Gold Standard)

The hybrid approach combines the best of the lazy follow and world-anchored methods. This is the technique used in many polished, professional spatial applications (including some of Apple's own visionOS interfaces).

Here’s the logic:

1. The UI starts anchored in the world at a comfortable position.
2. As the user turns their head, the UI stays put, feeling solid and stable.
3. Here's the magic: If the UI element goes outside a certain angle from the center of the user’s vision (e.g., more than 30-40 degrees), it detaches from its world anchor.
4. It then smoothly animates (using a lazy follow) into the user's peripheral vision, finding a new comfortable spot.
5. Once it settles, it re-anchors itself to the world in that new location.

This method provides the stability of a world-locked object but ensures the UI is never truly lost. It’s accessible when you need it and unobtrusive when you don’t.

Comparison of UI Anchoring Techniques

Technique	Pros	Cons	Best For
Naive Head-Lock	Simple to implement; always visible.	Obtrusive, immersion-breaking, uncomfortable.	Quick prototypes, debugging overlays.
Lazy Follow	Feels organic; less obtrusive; still always accessible.	Can still be distracting during fast head movements.	Persistent status indicators; simple control panels.
World-Anchored	High immersion and stability; feels part of the world.	Can be lost or left behind by the user.	Contextual menus attached to specific objects.
Hybrid	The best of all worlds: stable, immersive, yet never lost.	Most complex to implement.	Primary application UI, global menus, control panels.

Implementation in RealityKit

Let's get our hands dirty with some code. We’ll focus on the lazy follow method, as it’s a fantastic starting point.

Setting Up a Basic Scene

First, you need an `ARView` and an entity to act as your button. We'll place it in an anchor that we will control manually, not one provided by ARKit.

import RealityKit
import ARKit

// In your ViewController or SwiftUI View
let arView = ARView(frame: .zero)

// 1. Create a stable anchor to hold our UI.
let uiAnchor = AnchorEntity()
arView.scene.addAnchor(uiAnchor)

// 2. Create the visual for our button.
let buttonMesh = MeshResource.generateBox(size: 0.1, cornerRadius: 0.02)
let buttonMaterial = SimpleMaterial(color: .systemBlue, isMetallic: false)
let buttonEntity = ModelEntity(mesh: buttonMesh, materials: [buttonMaterial])

// Add a collision shape for taps.
buttonEntity.generateCollisionShapes(recursive: false)
arView.installGestures([.tap], for: buttonEntity)

// 3. Add the button to our controllable anchor.
uiAnchor.addChild(buttonEntity)

Implementing the Lazy Follow Logic

To make the UI follow smoothly, we subscribe to the scene's update loop. In each frame, we'll find a point 0.5 meters in front of the camera and gently move our `uiAnchor` towards it.

// Somewhere in your setup code, subscribe to scene updates.
var updateSubscription: Cancellable?

self.updateSubscription = arView.scene.subscribe(to: SceneEvents.Update.self) { [weak self] event in
    self?.updateUIAnchorPosition()
}

func updateUIAnchorPosition() {
    guard let cameraTransform = arView.session.currentFrame?.camera.transform else { return }
    
    // 1. Define a target position in front of the camera.
    // The Z-axis is negative for "forward" in SceneKit/RealityKit's camera space.
    let cameraPosition = SIMD3(cameraTransform.columns.3.x, cameraTransform.columns.3.y, cameraTransform.columns.3.z)
    let forwardVector = -normalize(SIMD3(cameraTransform.columns.2.x, cameraTransform.columns.2.y, cameraTransform.columns.2.z))
    let targetPosition = cameraPosition + forwardVector * 0.5 // 0.5 meters in front

    // 2. Get the UI's current position.
    let currentPosition = uiAnchor.position
    
    // 3. Smoothly interpolate (lerp) towards the target.
    // A lower factor means a slower, "lazier" follow.
    let smoothingFactor: Float = 0.05
    let newPosition = simd_mix(currentPosition, targetPosition, SIMD3(repeating: smoothingFactor))
    
    // 4. Update the anchor's position.
    uiAnchor.setPosition(newPosition, relativeTo: nil)

    // 5. Bonus: Make the UI always face the camera.
    uiAnchor.look(at: cameraPosition, from: newPosition, relativeTo: nil)
}

// Helper for linear interpolation
func simd_mix(_ x: SIMD3, _ y: SIMD3, _ t: SIMD3) -> SIMD3 {
    return x * (1 - t) + y * t
}

By tweaking the `smoothingFactor`, you can control how tightly the UI follows the user’s gaze. A smaller value like `0.02` will feel very loose and floaty, while a larger value like `0.2` will feel much snappier.

Outlining the Hybrid Logic

Implementing the full hybrid approach is more involved, but the core idea is an extension of the above. In your update loop, you would:

1. Calculate the angle: Get the vector from the camera to the UI anchor and the camera's forward vector.
2. Compute the dot product: The dot product of these two (normalized) vectors will give you the cosine of the angle between them.
3. Check the threshold: If the angle is too large (i.e., the dot product is below a certain value), set a flag like `isRepositioning = true`.
4. Reposition: While `isRepositioning` is true, use the "lazy follow" code to move the UI to a new target position in the user's view.
5. Re-anchor: Once the UI is close enough to its new target, set `isRepositioning = false`. The UI will now stay at this new world position until the user looks away again.

A Note on Comfort and Ergonomics

No matter which technique you choose, remember the human on the other side of the screen. Follow these simple rules to create comfortable experiences:

• Positioning: Don't place UI directly at eye-level. A position slightly below the horizontal center is more natural and comfortable for long-term use.
• Depth: Avoid placing UI too close (less than 0.5m) or too far. A range of 0.75m to 2m is often a sweet spot.
• Scale: Ensure text is legible and buttons are large enough to be tapped easily, but not so large that they dominate the view.
• Consult the Experts: Read Apple's Human Interface Guidelines for Spatial Design. They are an invaluable resource for creating ergonomic and intuitive interfaces.

Conclusion: Unsticking Your UI

Creating great spatial UI is about balancing accessibility with immersion. While simply locking controls to the user’s head is the easiest path, it’s rarely the best one. By embracing techniques like the lazy follow or the more advanced hybrid model, you can build interfaces that feel less like an intrusive overlay and more like a natural, integrated part of the user's world.

The next time you start building an AR experience in RealityKit, take a moment to think about your UI. Don't let it get “stuck.” With a little extra effort, you can provide a smoother, more comfortable, and far more magical experience for your users. Happy coding!