[Vision Pro] and the Limitations of Location Awareness

The Vision Pro cannot perform navigation based on latitude/longitude and directional heading. This is because the device lacks built-in hardware for GPS and magnetic compass (geomagnetic sensor), making it unable to acquire such information.

Note: While approximate latitude/longitude data can be obtained via Wi-Fi—making some location-based navigation possible—it is not possible to determine the direction the Vision Pro is facing based on any built-in sensors.

However, for users to fully benefit from the unique potential of spatial navigation, such as overlaying virtual information on the real world and doing so in 360 degrees, leveraging accurate location data becomes essential.

For example, imagine a user wearing the Vision Pro: being able to intuitively understand the direction and distance to a destination—such as a store or landmark—from their current standing position.

While smartphone map apps already allow users to check directions, having a constant visual reference for cardinal directions (North, South, East, West) overlaid in the real world feels fundamentally different. It's like being able to see the North Star at all times, even during the day.

Can Vision Pro Acquire [Latitude/Longitude and Directional Heading]?

To reiterate, the Vision Pro has no way to acquire latitude, longitude, or directional heading. This is due to the absence of hardware such as GPS or magnetic sensors.

Still, we want to make spatial navigation based on real-world location data possible.

So how can we achieve that?

Fortunately, we already have a device capable of obtaining accurate latitude, longitude, and heading—that device is the iPhone, or any modern smartphone.

Can the iPhone Work in Tandem with Vision Pro?

If the Vision Pro lacks certain capabilities, why not let the iPhone fill in the gaps?
But first—can the two devices even communicate with each other?

If they can, the iPhone can provide the Vision Pro with the location data it gathers, allowing that information to be reflected in the MR space.

[LLD Compass]: A Spatial Compass Experience on Vision Pro

To explore this possibility, we created a spatial compass app called LLD Compass, which links the Vision Pro and iPhone to recreate directional orientation within a mixed reality (MR) space.
The name LLD stands for Latitude, Longitude, and Direction.

In this article, we’ll explore how LLD Compass was made possible—covering the technical architecture, key implementation strategies, and the potential use cases it unlocks.

LLD Compass – App Overview and Use Cases

By linking the Vision Pro with an iPhone, LLD Compass enables the visualization of cardinal directions (North, South, East, West) within a mixed reality environment.
This is the core concept behind the app.

• A red sphere represents North.

What Kind of Experience Does It Offer?

With the Vision Pro on, you can intuitively see spatial cues like:
“This direction leads to Tokyo Tower,”
“Haneda Airport is over that way,” or
“From Tokyo, Paris lies in this direction”—all visualized directly in your surrounding space.

While smartphone map apps can provide similar information, simply looking around to understand direction—without needing to glance at a screen—is a completely different experience.

By leveraging the Vision Pro’s unique ability to deliver spatial awareness, LLD Compass offers a new kind of value that traditional navigation tools can't provide.

Potential Use Cases

• Tourism Navigation: Instantly see which direction famous landmarks, shops, or attractions are located—just by looking around.
• Stargazing Guidance: Display constellations based on your orientation and current field of view.
• Venue Navigation: In theme parks or zoos, intuitively understand where your destination lies within the venue.
• Breaking News & Local Buzz: Get real-time directional cues for incidents, accidents, or trending reviews happening nearby.

Simply being able to see spatial and directional information at a glance greatly enriches the mixed reality experience.

System Architecture: Connecting Vision Pro and iPhone

Device Roles and Responsibilities
LLD Compass leverages the strengths of both devices by assigning them distinct roles:

• iPhone Acquires latitude, longitude, and heading data, then sends it to the Vision Pro Technologies used: SwiftUI / CoreLocation / MultipeerConnectivity
• Vision Pro Receives the location data and renders it spatially within the MR environment Technologies used: Unity / PolySpatial / Swift Bridge

How the Devices Communicate

Using Apple’s MultipeerConnectivity framework, the iPhone and Vision Pro communicate over a peer-to-peer (P2P) connection.
LLD Compass leverages this framework to send real-time location and heading data from the iPhone to the Vision Pro, where it’s immediately reflected in the mixed reality space.

• iPhone
  • Retrieves latitude, longitude, and heading information
  • Serializes the data into JSON
  • Sends it via MultipeerConnectivity
• Vision Pro (Swift Layer)
  • Receives the data using MultipeerConnectivity
  • Passes the JSON to Unity
• Unity
  • Deserializes the JSON into usable latitude, longitude, and heading values
  • Updates object positions and orientations accordingly in the spatial environment

App Structure

Given the system architecture described above, the solution requires two separate apps:

• The iPhone app: Responsible for acquiring latitude, longitude, and heading data, and sending it to the Vision Pro
• The Vision Pro app: Receives the location data and visualizes it within the MR environment

Implementation Challenges and Solutions

Bridging [iPhone], [Vision Pro], and [Unity]

The MR environment in this project was built using Unity.
One key challenge was how to reflect the data acquired on the iPhone within the Unity world running on Vision Pro.

Solution: Create a Swift Bridge on Vision Pro

To pass data from the native layer to Unity, we used UnitySendMessage.
A custom bridge was implemented in Swift, allowing the Swift layer on Vision Pro to forward received data directly into Unity.

As a result, the MultipeerConnectivity data exchanged between iPhone and Vision Pro can now be accessed and utilized within Unity.

Vision Pro Has No Concept of "North"

The Vision Pro has no way to determine true North.
While its internal sensors can detect relative rotation, it cannot identify any absolute orientation—such as the direction of magnetic North on Earth.

Solution: Perform an Initial Orientation Sync with iPhone

After the Vision Pro is worn, we initiate communication while both the Vision Pro and iPhone are facing the same direction.
This moment is treated as the reference point—a shared initial orientation.

From then on, the direction the iPhone is facing is assumed to be the same as that of the Vision Pro, effectively giving the Vision Pro a sense of absolute direction.

Determining the Direction of a Landmark

Suppose the user's current location—where they’re wearing the Vision Pro—is point A, and a landmark (e.g., Tokyo Tower) is point B.
We need to determine the relative direction from point A to point B.

Solution: Calculate the Bearing (Azimuth)

Note: A bearing (or azimuth) is the angle measured clockwise from a reference direction—typically true North—to the line connecting two points.
It’s widely used in fields such as geography, navigation, aviation, surveying, and GPS systems.

By using the latitude and longitude of both point A and point B, we can calculate the bearing from A to B, which tells us the direction in which the landmark lies.

double GetBearing(double lat1, double lon1, double lat2, double lon2)
{
    var phi1 = lat1 * Math.PI / 180;
    var phi2 = lat2 * Math.PI / 180;
    var deltaLambda = (lon2 - lon1) * Math.PI / 180;

    var y = Math.Sin(deltaLambda) * Math.Cos(phi2);
    var x = Math.Cos(phi1) * Math.Sin(phi2) -
            Math.Sin(phi1) * Math.Cos(phi2) * Math.Cos(deltaLambda);

    var theta = Math.Atan2(y, x);
    var bearing = (theta * 180 / Math.PI + 360) % 360;

    return bearing; // 北=0, 東=90, 南=180, 西=270
}

Future Outlook and Applications

Through the development of LLD Compass, we’ve come to truly appreciate the value and potential of bringing direction and location into MR space.

This technology can be applied to a wide range of services, such as:

• Integration with Map Services
  By projecting maps into spatial environments, users can experience the sensation of having a map unfold at their feet, making it easier to understand their surroundings.

• Real-Time Flight Tracking and Live Positioning
  Aircraft flying overhead can be displayed in real time—showing their direction and path—allowing users to visually understand where each plane is coming from and where it’s headed.

• Shared Location Awareness Among Multiple Users
  In fieldwork or at event venues, users can share their location data with others. This enables a shared experience where everyone can see each other’s position and orientation in the same MR space.

Final Thoughts

By combining the strengths of two distinct devices—Vision Pro and iPhone—LLD Compass delivers a simple yet powerful experience:
giving the Vision Pro a true concept of North.

• While Vision Pro excels at spatial expression, it has inherent limitations in acquiring location data
• The iPhone compensates for this, acting as a sensor to enable spatial navigation
• Real-time integration is achieved through Unity × Swift × MultipeerConnectivity
• From the user’s current position, it becomes possible to calculate the bearing to any external point

The true value of Vision Pro lies in how it overlays information onto the real world.
By embedding location and directional context into the spatial environment, we unlock entirely new and deeply immersive experiences.

LLD Compass is both a prototype and an experiment—a first step toward exploring that potential.

Moving forward, I hope to continue exploring what new kinds of experiences the Vision Pro can offer.