Apple’s LiTo: Reconstructing 3D Objects From a Single Image With Realistic Lighting

Apple researchers have developed a new AI model, LiTo (Surface Light Field Tokenization), capable of reconstructing a 3D object from a single image while maintaining consistent reflections, highlights, and other view-dependent effects. This advancement leverages the concept of latent space to represent both object geometry and appearance.

Understanding Latent Space

The concept of latent space, or embedding space, has gained prominence with the rise of transformer-based AI models and world models. Essentially, latent space involves reducing complex information into numerical representations and organizing these numbers in a multi-dimensional space. This allows for efficient measurement of relationships and prediction of generated content. For example, mathematical operations on word embeddings can demonstrate relationships like “king – man + woman = queen.”¹

LiTo: Surface Light Field Tokenization

In their study, titled “LiTo: Surface Light Field Tokenization,” Apple researchers propose a 3D latent representation that jointly models object geometry and view-dependent appearance.¹ Traditional methods often struggle to capture realistic lighting effects. LiTo addresses this by encoding a surface light field – samples of light interacting with an object – into a compact set of latent vectors. This unified 3D latent space reproduces effects like specular highlights and Fresnel reflections.

A key innovation is the model’s ability to reconstruct 3D objects from a single image, a departure from methods requiring multiple viewpoints.

How LiTo Works

The process involves an encoder that compresses object information into a compact latent space representation, capturing the shape and light interaction. A decoder then reconstructs the full 3D object, generating both geometry and lighting effects from different angles.¹

Training the Model

The model was trained using thousands of objects rendered from 150 different viewing angles and three lighting conditions. Instead of using all data at once, the system randomly selected subsets and compressed them into a latent representation. The decoder was then trained to reconstruct the full object and appearance from these subsets.¹

After initial training, another model was trained to predict the latent representation from a single image. The decoder then reconstructs the full 3D object, including view-dependent appearance changes. Comparisons with a model called TRELLIS demonstrate LiTo’s improved reconstruction quality.¹

Interactive comparisons between LiTo and TRELLIS are available on the project page.¹

Future Implications

Apple continues to advance machine learning and AI, with a focus on the Transformer architecture. The availability of optimized Transformer models for Apple devices, facilitated by the Apple Neural Engine (ANE), minimizes impacts on app performance and battery life while enhancing user privacy through on-device processing.¹ Further developments in foundation models, as showcased in Apple Intelligence, are integrating generative AI into everyday applications.², ³ The new Foundation Models framework provides developers with direct access to on-device foundation language models.³