Passthrough camera access is currently a buzzworthy subject in the XR community. We’ve seen how Meta, Apple, and Pico are approaching it, but what about Google and Android XR? After having a chat with someone at Google, I can share that their approach will closely resemble what we see on phones. Stick around for all the details!
The Camera Access Dilemma
For those not up to speed, let’s rewind a bit. Have you noticed that the latest standalone VR headsets double as MR headsets, using their front cameras to deliver an RGB passthrough view? This enables all those fabulous mixed reality apps like Cubism, Starship Home, Pencil, etc.
The OS processes camera frames to present us with passthrough, and as developers, we yearn for access to these frames. Why? With access, we could use AI and computer vision to enhance the user’s environment considerably. I’ve talked about this before, and I firmly believe unlocking camera access is crucial for a genuine mixed reality experience. It’s essential for applications to understand the world around us completely. For instance, utilizing a workaround in Quest, I’ve prototyped an AI+MR app aiding folks in home interior design. This wouldn’t have been feasible without camera access.
Sounds impressive, right? Sadly, there’s a major counterpoint: privacy concerns. Imagine if a shady developer gets access to the camera, capturing images around the user silently, running them through AI to extract sensitive data like personal documents or credit cards, possibly even faces or bodies for other malicious uses.
It’s a tightrope, balancing user privacy with unleashing mixed reality’s full potential.
The Behavior of XR Companies
Initially, full camera access wasn’t restricted, and during those early days, experiments with camera textures on gadgets like the Vive Focus were commonplace. Anyone remember the work we did with NTW around 2019? Projects like diminished reality, Aruco marker tracking, and sound reactivity were all the rage.
However, as mixed reality gained traction, companies hit the brakes for privacy reasons, restricting access to camera frames. Meta, Pico, HTC, soon followed by Apple, all tightened the reins.
This became the norm until XR developers rallied for change, pushing manufacturers to reconsider. Notable voices like Cix Liv, Michael Gschwandtner, and myself highlighted the stark contrast to how easy camera access was on phones, simply by requesting permission. We argued for a similar approach with XR devices, ensuring clarity and transparency for users who consented to camera access for tasks like object recognition or computer vision.
The movement bore fruit with Meta’s promised "Passthrough API" rollout. But where does Google and Android XR stand?
Android XR to Treat the Headset Like a Phone
Globally, Android powers most phones. When developing an Android app, a simple permission request grants camera stream access. If allowed, developers pick their desired camera – often ID “0” for the back camera – and do as they please with the footage.
Google’s aim is for Android XR to echo Android’s behavior as much as possible, especially concerning apps. Though unofficial, information circulating sparks curiosity, pushing me to inquire directly with Google. Here’s a snippet of what a company representative shared:
"Like any Android app, a developer can utilize existing camera frames with user consent for XR. The developer article delves into other permissions apps may request."
"Developers can request camera_id=0 for the world-facing camera. Similarly, request camera_id=1 for the selfie-camera. Both streams are managed via standard Android APIs: Camera2 and CameraX."
"For access to world-facing camera feeds, similar to a smartphone’s back camera, app developers should request permissions akin to phone apps. When requesting the selfie-camera, the stream received includes the user’s avatar, crafted from user tracking data via OpenXR APIs."
This means Android devs can rely on known methods, like CameraX, for managing XR cameras, enabling features from frame grabs to ML analysis on XR headsets and glasses. That’s big news!
Although accessible, note the rear camera is essentially a reconstructed avatar, a parallel to Apple’s Vision Pro strategy. This makes sense for consistency and functionality with existing Android apps that need camera access.
Google’s strategic choices ensure seamless functionality for apps on both phones and Android XR devices. The permission parity across devices also gets my personal thumbs up.
Still, a couple of lingering questions might remain. One being, what about all the raw camera streams?
"As of now, access to non-standard sensors, like forward-facing cameras or reconstructed inward cameras, isn’t provided."
In essence, cameras outside the front and back are off-limits. Let’s hope this changes for enterprise users in the future.
A Note on Android XR
Bear in mind, Android XR is still in preview; no headsets are officially out yet. This could imply changes before the official release. Unlikely, but it’s good to consider this eventuality.
The Opening Up of Camera Access
With Google and Meta paving the way for camera access, other companies might not be far behind. 2025 holds the promise of significant developments in mixed reality possibilities. I can’t wait to see the innovative creations from the developer community!
(Please note, this blog contains advertising and affiliate links. If you make a purchase via an affiliate link, I’ll earn a tiny commission. Check out the full disclosure here.)
