Augmented Reality Frameworks: Building Immersive Experiences

Using software development tools called augmented reality (AR) frameworks, you can streamline AR development and craft your own immersive experiences. Frameworks are diverse and may offer a variety of unique features. They can help you manage object tracking, user interaction, and other core components of an AR project. 

Since Boeing researcher Thomas Preston Caudell coined the term in 1992, AR has evolved and grown popular with app developers in various industries. Gaming, education, health care, tourism, and retail have all begun embracing AR technology, opening up promising career paths to potential developers in these fields. 

AR lets your audience engage with digital information, such as 3D graphics or video, superimposed on their real-world environment by an AR-equipped device. For example, you might design a virtual try-on app for a clothing retailer or build complex software that provides sensory feedback to industrial maintenance trainees.

Uncovering the right framework for your AR project can be a worthwhile process. Some variables to consider are the availability of resources, your experience level, and the type of AR application you’re building.

Understanding augmented reality frameworks

AR frameworks offer valuable resources, such as libraries and application programming interfaces (APIs), to simplify AR development. These versatile tools can manage sensors and components for object tracking and important user interaction features such as raycasting and anchors. If you’d like to offload computationally strenuous tasks to remote servers, AR frameworks can help you achieve that.  

While augmented and virtual reality (VR) have similarly large potential, they offer unique experiences and may need different frameworks to run. VR technology renders a new, fully simulated world requiring a standalone or tethered headset. Both AR and VR fall under the extended reality (XR) umbrella, and you can use frameworks such as ARKit to run mixed reality (MR) applications blending elements of the two. 

Depending on your AR framework of choice, you’ll have support in developing native (including iOS and Android) or web-based applications. Next, consider whether a marker-based or markerless framework best fits your project:

• Marker-based frameworks: In marker-based AR, computer vision techniques detect a predefined marker (either a 2D object, 3D object, or QR code) in the environment where a virtual object appears. Markers are generally simple and specific, making them easy to recognize, even without particular processing skills.

• Markerless frameworks: Instead of detecting a predefined marker, markerless AR is sensor-based (either acoustic, magnetic, or inertial) or reliant on a natural, non-specific feature in the environment. Location-based AR is a subset of markerless AR where a GPS, accelerometer, or similar built-in tool collects data based on the device’s position.

Popular augmented reality frameworks

With a popular AR framework such as ARKit or ARCore, you can take full advantage of tutorials, community support, and extensive features that have only grown over the years.

ARKit

ARKit is Apple’s AR framework and runs on iOS, iPadOS, and visionOS. Some of its major features include:

• Motion capture: Tracks your motion in real-time using movements and gestures as input. 

• Scene understanding: Generates a detailed 3D reconstruction of your surroundings, allowing AR objects to interact with each other and the real-world environment.

• Light estimation: Estimates scene lighting information from a video frame, helping you render realistic graphics.  

Although ARKit was first launched in 2017, the latest version (ARKit 6) offers updated features, such as:

• Depth API: Uses per-pixel depth information to understand the environment, allowing virtual objects to blend in realistically

• Scene geometry: Builds a topological map of your real-world space, including labels for walls, ceilings, floors, doors, windows, and seats

• High-resolution video: Captures 4K video of AR experiences

To use ARKit, you’ll need to run iOS 11.0+, iPadOS 11.0+, or VisionOS 1.0+. 

Google Play Services for AR

Google Play Services for AR, formerly called ARCore, is Google’s AR software development kit (SDK). Its core features are similar to ARKit’s, and include:

• Motion tracking and anchors: Tracks your position and movements over time and displays those positions relative to the real world

• Environmental and depth understanding: Detects the location and size of different surfaces and measures how far apart they are from a set point 

• Light estimation: Gathers information about light intensity and color from the environment, realistically shading AR graphics 

Google Play Services for AR offers other helpful tools, including geospatial API (using Google Street View), scene semantics, recording and playback API, and persistence cloud anchors. 

While it can be used to develop Android apps, the kit also provides cross-platform APIs for iOS, the Unity engine, and the web. 

Cross-platform augmented reality frameworks

If you’d like your AR app to run on two or more platforms (iOS, Android, etc.), consider a cross-platform framework such as Unity 3D or Vuforia. 

Unity 3D

Unity is a game engine that supports end-to-end AR development across multiple platforms, including Android, iOS, Windows, and macOS. You can also use Unity to develop other types of 2D and 3D video games, including games with VR integration. Acclaimed developers often use the Unity engine to make popular 3D mobile games.

Some AR development tools that Unity offers are:

• XR Interaction Toolkit: This allows you to make AR apps interactive even before you’re comfortable coding. Supports haptic feedback, cross-platform controller input, visual responses, and more. 

• AR Foundation: This is a framework for building and deploying AR apps across multiple devices. It supports you in leveraging features such as body and image tracking, plane detection, point clouds, environment probes, and more. 

• Unity Mars: Provides specialized tools for intelligent AR interactions, helping streamline development.

You can use Unity Personal for free, but features such as Unity Mars are only available in Pro, Enterprise, and Industry plans.

Vuforia

Vuforia is a scalable AR development platform compatible with the Unity game engine. If you’re a business owner, it can help you build complex AR with practical, on-the-job applications. The Vuforia AR suite of tools includes:

• Vuforia Expert Capture: Constructs step-by-step AR guides and AI-enhanced inspections, empowering frontline workers to do their more accurately and efficiently.

• Vuforia Studio: Leverages existing Internet of Things (IoT) and computer-aided design (CAD) data to build immersive AR experiences. These resources may simplify improving worker proficiency and safety with AR.

• Vuforia Engine: Uses advanced computer vision technology to create cross-platform AR applications. Can help you build interactive brand campaigns, 3D product demonstrations, and more.

• Vuforia Chalk: Supports real-time collaboration and problem-solving between experts and technicians.

Web-based augmented reality frameworks

Consider web AR frameworks if you’re looking to build applications that run on internet browsers. That way, users won’t need to install an Android, iOS, or other app, making web AR a more accessible technology.

WebXR

WebXR, or the WebXR Device API, is an API for augmented and virtual reality experiences. Some features (and benefits) of WebXR are:

• Future proof code: With the rapid development of hardware used to access AR, WebXR can keep your experience functional across devices. 

• Rendering with WebGL: WebXR relies on Web Graphics Library (WebGL) for rendering, which is a popular JavaScript API to create high-performance interactive graphics. 

• Input and output capabilities: WebXR gathers pose information from controllers and AR headsets before generating output on the web. 

While you’ll need web access to launch apps, WebXR offers broad device support, including smartphones, desktop computers, standalone AR headsets, and standalone VR headsets (for VR applications). 

AR.js

AR.js is an open-source JavaScript library. It’s a lightweight tool that provides web AR features such as:

• Image tracking: When a web-enabled device's camera finds 2D images in the real world, the app displays content near or on top of those objects. That new content can be another image, an animated 3D model, a 2D video, or a GIF. 

• Marker tracking: When your camera spots a stable, predefined marker in the environment, the application displays a virtual object near or on top of it. 

• Location-based AR: If you’re building a location-based AR app for the web, AR.js has equipment specifically for this type of experience. Potential location-based experiences include history learning games and situated artwork, which artists might showcase at real-world locations.

The AR.js library is less complex than other frameworks, but the tools it offers are key to many engaging AR experiences. Apps made with AR.js can also run efficiently on smartphones. 

Advanced augmented reality frameworks

If you’re already comfortable building AR experiences and want more flexibility, you may consider advanced frameworks such as ARToolkit. Wikitude, another advanced option, became inactive in the fall of 2024—but its owners are developing a new project that you may still explore, especially if you’re a business owner. 

ARToolKit

ARToolkit is an open-source AR framework that offers SDKs for multiple platforms, including Android, iOS, PC, macOS, and Linux. It’s a versatile tool with possible uses in advertising, industry, media, academic research, and more. Some of ARToolKit’s features are:

• Natural feature tracking

• Camera calibration support

• Plugins for OpenSceneGraph and Unity 3D

• Mobile device optimization

• Stereo camera support with simultaneous tracking

• Optical see-through and video AR support

The source code for the latest evolution of ARToolKit, artoolkitX, is hosted on GitHub.

Wikitude

Wikitude was an SDK that supported object tracking, geolocation AR, and object recognition. Founded by Philipp Breuss-Schneeweis in 2008, it was a major platform in the AR sphere for over a decade before being discontinued in September 2024. Developers released over 40,000 mobile applications built with Wikitude. 

The owners of Wikitude are now focusing on the Snapdragon Spaces XR development platform, which supports headworn-device AR rather than mobile applications. Snapdragon is an enterprise AR ecosystem providing diverse tools and resources to companies in various industries. 

Practical applications of augmented reality frameworks

AR frameworks are flexible tools that help you craft various projects depending on your audience. For example, a chemistry teacher might ask their class to use an AR application that displays a virtual laboratory on a real-world surface, engaging remote learners in a simulated experiment. 

Orthopedic surgeons can use AR to project images and information onto the body pre-operation, improving their understanding of the patient’s anatomy and spatial relationship to their surgical tools. This process could reduce surgery time and decrease radiation exposure. Alternatively, medical professionals may prefer AR as a training tool, providing learners with visual or auditory feedback and gesture recognition. 

The furniture giant IKEA offers AR mobile apps that let customers preview furniture in their own living space before purchase. Other examples of successful AR projects include:

• Google Live View: A markerless AR navigation app built with ARCore (Android) and ARKit (iOS)

• Pokémon GO: A markerless AR mobile game released in 2016, built with ARCore (Android), ARKit (iOS), and the Unity game engine

• Virtual Try-On Technology: A genre of marker-based AR apps released by retail clothing sellers, built with frameworks such as ARCore (Android), ARKit (iOS), Vuforia, and Unity.  

• ARChem: An educational, markerless AR app to help learners better understand fundamental chemistry topics, built with ARCore (Android), ARKit (iOS), and the Unity engine.

Challenges and future directions

The AR landscape is evolving quickly, and developers are still uncovering new opportunities for improvement. Current challenges in the AR space include:

• Hardware compatibility: To run an AR app, you need AR-equipped hardware. AR applications can ideally run on multiple hardware programs with their own computational characteristics. However, its reach may be limited if new or established hardware is incompatible with an app.

• Performance optimization: The goal is to build reliable AR apps that respond to user input, render content at a consistent frame rate, and minimize battery drain. Consider exploring AR performance best practices, such as disabling central processing unit (CPU) intensive features when not in use. Additionally, consider pre-creating AR databases during development and/or monitoring device thermals. 

• User experience (UX) design: AR apps with accessible, intuitive designs are easier to use and open AR to a wider audience. Some UX design best practices for AR include natural interaction via multiple methods (e.g. gestures and speech), high object detail, and tolerance to camera movement; if a marker leaves the camera frame, it doesn’t vanish instantly. 

Although AR is still growing in these respects, the technology’s future is full of promise. Just a few recent developments and areas of research are:

• Hand tracking: Some AR-equipped devices can sense precise hand and finger movement, offering more information about the user than with a hand-held controller.  

• Real-time collaboration: Augmented reality can construct a shared virtual space for remote workers. It allows teams to communicate and problem-solve as if they were physically together.

• Machine learning (ML) and artificial intelligence (AI) integration: ML algorithms can extract and process information from images, while AI has an assortment of potential uses in AR technology, such as creating flexible, interactive learning environments. 

Getting started with augmented reality frameworks on Coursera

With the support of augmented reality frameworks, new and experienced developers can build complex, immersive experiences for customers, learners, gamers, or even their own workers. As AR technology evolves, so do the industries and businesses that leverage it.

To start your journey, check out Google’s Introduction to Augmented Reality and ARCore on Coursera. Once you’ve learned the fundamentals and want to dig even deeper into AR, consider the Virtual Reality Specialization from the University of London or the Extended Reality for Everybody Specialization from the University of Michigan.