Apple Vision Pro and Samsung Galaxy XR Embrace PC VR Foveated Streaming

Apple’s Vision Pro is officially gaining PC VR foveated streaming capabilities, while Samsung’s Galaxy XR now offers the feature through the third-party application, Virtual Desktop. This advancement aims to enhance the quality and efficiency of wireless virtual reality experiences.

Understanding Foveated Streaming

It’s important to distinguish between foveated streaming and foveated rendering. Foveated rendering involves rendering the area the user is directly looking at with higher resolution. Foveated streaming, conversely, focuses on sending that same area to the headset with higher image quality than the rest of the frame. This technique is particularly useful given the limitations of video decoders in headsets regarding maximum resolution and bitrate, prioritizing quality where the user’s attention is focused.

Foveated streaming is a core feature of Valve’s Steam Link, utilized in its PC VR streaming offering with the USB PC wireless adapter.

Apple Vision Pro: visionOS 26.4 and Beyond

The visionOS 26.4 update will introduce foveated streaming to the Apple Vision Pro, enabling higher-quality wireless VR remote rendering from both local and cloud PCs. Apple’s developer documentation describes this new framework as low-level and host-agnostic, differing from the macOS Spatial Rendering introduced in visionOS 26, which is limited to local Mac hosts.

Nvidia’s CloudXR SDK is highlighted as an example host, but the framework is also designed to work with local PCs. Notably, Apple has released a Windows OpenXR sample on GitHub, marking the company’s first public engagement with the industry-standard XR API. This allows for the potential of foveated rendering within tools like ALVR.

Apple’s documentation also indicates support for displaying both rendered-on-device and remote content simultaneously. An example given is rendering the interior of a vehicle on the headset while streaming the detailed external environment from a powerful cloud PC, optimizing for latency and stability.

Samsung Galaxy XR: Virtual Desktop Integration

Samsung Galaxy XR users can now access foveated streaming through Guy Godin’s Virtual Desktop, a $25 application available on Google Play. This update also extends foveated streaming support to the Meta Quest Pro and Play For Dream MR, although those headsets previously achieved similar results through Valve’s Steam Link.

The latest Virtual Desktop update (version 1.34.16) includes the following changes:

• Added Foveated streaming with eye tracked headsets (Quest Pro, PFD & Galaxy XR)
• Improved color gradients and color accuracy with all codecs
• Added 96 fps and 100 fps support on Quest 2, 3/3S
• Added Gamepad vibration support
• Added adaptive quantization support with AMD GPUs using H.264/H.264+
• Improved initial connection reliability
• Added 21:9 resolutions for virtual monitor on macOS
• Improved thumbstick scroll on macOS
• Fixed distorted image with some laptop monitor resolutions
• Fixed rare black flash issue when playing some PCVR games
• Fixed hand joints offsets and interference with other drivers in SteamVR
• Fixed compatibility with Roblox anti-cheat
• Fixed more issues with AndroidXR invalid controller poses
• Fixed button support for some recent UE5 games
• Fixed reprojection stutters with some Unity (OVRPlugin) games

Looking Ahead

The integration of foveated streaming into both Apple Vision Pro and Samsung Galaxy XR represents a significant step forward in enhancing the quality and performance of wireless PC VR experiences. Continued development and adoption of this technology, along with advancements in eye-tracking capabilities, will likely play a crucial role in the future of immersive computing.