Evaluation on Software for Simulation in XR 

Introduction to XR Development

The most common approaches to XR development can be categorized into

• Native Development

• Game Engine

• WebXR

Native Development

Native development refers to building applications for a specific platform with platform-specific SDK. Native development provides deeper integration with OS-specific features by allowing developers to access advanced APIs to fully take advantage of the operating system and hardware.

The recent trend in the XR industry is shifting headsets from game consoles to real productivity tools, and companies have started to push for native applications to provide better user experience to appeal to a broader audience. One of the example is Apple Vision Pro, where developers use Swift and Reality to get native support for spatial understanding and eye & hand tracking.

Game Engine

Games have always been a popular category of applications in VR. It's also widely used for building VR applications. With the recent advancement in XR technologies, game engines have also supported XR features to support headsets.

Game engines are meant to help developers create video games with built-in toolkits for UI components, rendering, AI, etc., so developers can focus on the gameplay and the design without having to pay too much attention to technical details. They also allow you to port your application to a range of different platforms. One of the greatest thing about game engine for building simulation applications is its support for physics, such as collision detection, rigid body, and fluid/particle simulation.

WebXR

WebXR is a web standard that enables augmented reality and virtual reality experiences directly in a browser. What's worth noting is that WebXR is not limited to headsets but accessible on any WebXR-supported device, including computers and mobile phones.

WebXR provides essential features for XR applications, including controller input, hand tracking, and spatial anchors. Developers typically use WebXR alongside 3D JavaScript libraries to create lightweight, high-performance XR applications that are widely accessible across devices without requiring additional installation.

Game Engine

Unity

Unity is a popular cross-platform game engine released in 2005. Unity uses C# for scripting language, where memory is automatically managed by garbage collection.

AR Foundation is Unity’s cross-platform framework for building AR apps. It is essentially a wrapper over ARKit (iOS) and ARCore (Android), providing unified APIs to access native-level features and performance without writing platform-specific code.

Supported XR platforms include OpenXR, Meta Quest, Microsoft Hololens, PSVR2, visionOS, and AndroidXR.

Unreal

Released in 1998, Unreal is the "heavy truck" type of game engine, where advanced tools and settings are provided for complex and large scale projects, but requires a steep learning curve to overcome. A set of advanced rendering pipelines and features (e.g. global illumination), designed for ultra realistic visuals in games, require a good understanding of the engine to be able to fine-tune. In terms of gameplay scripting, Unreal uses Blueprint, a visual scripting system, and C++, where memory management is required.

Supported XR platforms include Meta Quest, HTC Vive, Valve Index, Apple ARKit, Google ARCore, and OpenXR.

Godot

Godot is an open-source and free game engine released in 2014. Godot has its own scripting language, GDScript, but also supports C# and C++ for scripting.

As the youngest game engine among three, Godot is still in its nascent stage and does not provide as much support for XR as Unity or Unreal. Godot’s XR capabilities are limited and are primarily supported through OpenXR, an industry-standard API that enables cross-platform XR development. While OpenXR provides a foundation for XR features like head tracking, controller input, and basic hand tracking, Godot’s implementation is limited compared to Unity and Unreal. Advanced XR features such as passthrough, depth sensing, and foveated rendering are either experimental or not yet fully supported.

JavaScript + WebXR

While WebXR can easily turn JavaScript 3D content into an XR experience, the limitations it comes with compared to native SDKs like ARKit or ARCore are notable. Performance is constrained by browser, leading to higher latency and lower graphical fidelity than native applications. WebXR also has limited access to device-specific features such as advanced hand tracking and deeper system integration. One of the most important XR features absent in WebXR is the ability to generate mesh from the user environment, which lays the foundation for virtual object placement and and occlusion.

Despite these drawbacks, WebXR remains a powerful tool for accessible, cross-platform XR experiences without requiring app installations.

Three.js

Three.js is a general purpose 3D JavaScript library, allowing developers to create 3D graphics and animations for the web without dealing with raw WebGL API calls. Three.js has full WebXR support. Even though there is no built-in physics in Three.js, relevant external libraries are available.

A-Frame

A-Frame is built on top of Three.js. It provides a higher-level framework by abstracts some of the complexity of WebGL rendering out. It also has simpler declarative HTML-like syntax and built-in event handling for interaction, making 3D and WebXR development easier.

Babylon.js

Babylon.js is another JavaScript library with 3D rendering features and XR capabilities, but compared to others, Babylon.js is more like a game engine because of its focus on performance, advanced rendering, and physics support. The built-in physics engines like Havok and Cannon.js make Babylon.js an ideal option for interactive simulations and dynamic environments.