AR/VR/MR, what can Android developers do?

By Yang Bo (Alpha), Google Developer Expert (GDE)

With ASUS announcing the first ZenFone AR with both Tango and Daydream capabilities at CES 2017, AR/VR/MR, which are super hot words in the global development circle recently, can finally make our domestic Android developers directly use Android Studio and convenient Android devices to explore virtual development!

Three types of interactions

AR

In augmented reality, the light received by the human eye includes real light and light emitted by the AR device screen, which are synthesized through the processing power of the human eye to enhance the sense of reality.

Phab 2 Pro is an AR device that allows you to take photos of cute kittens or puppies playing with babies without worrying about safety because the animals are virtual.

VR

In the virtual world, helmets or cardboards are used to completely block out the light from reality. The human eye only receives the light from the VR device, which is completely isolated from reality.

Oculus Rift, PlayStation VR, Samsung Gear VR and HTC Vive are all such devices.

MR

Mixed reality, MR devices themselves have the ability to perceive three-dimensional coordinates. As people move, the display of virtual objects also changes.

Hololens is a MR device.

Two major technologies

Tango (Dragon Slayer)

Just as Sumsung added a camera to the SCH-V200 phone in 2000, giving mobile phones the ability to see the world, the Tango camera now gives mobile phones the ability to perceive the real three-dimensional world.

Simply put, Tango has the following three functions:

• Motion tracking: Where am I, spatial position
• Area learning: what is ahead and what you see
• Depth perception: How far ahead is it?

Whether using ToF (Time of Flight), the sensor emits modulated near-infrared light, which is reflected after encountering an object. The sensor calculates the time difference or phase difference between the emission and reflection of the light to convert the distance of the photographed scene to generate depth information.

It is still structured light computing, which calibrates the space and assists in calculating the three-dimensional spatial position by projecting a coded laser spot array into the detection space.

The resulting cloud point data is shown in the figure below. It can be seen that a total of 12464 points with XYZ three-dimensional coordinates have been collected.

Daydream

The purpose is to provide users with the best virtual experience. By defining a set of best hardware operating effect standards and providing developers with an easy-to-use Daydream API, the screen delay that causes user dizziness is eliminated, providing the most cost-effective virtual experience.

hardware

Phab 2 Pro

The first consumer-grade Tango device uses Snapdragon 652, 4G memory and 64G storage. What’s more important is that both the domestic and foreign versions have complete Tango functions.

Android's "eyes" for perceiving the three-dimensional world - Tango component:

Typical application scenarios:

ZenFone AR

The first consumer Tango+Daydream device uses Snapdragon 821 to ensure that users get the Daydream experience. The first Daydream phones Google Pixel and Asus Zenphone 3 also use Snapdragon 821. Since the Daydream Viewer is now compatible with phones with screen sizes of 5.0 to 5.7 inches, ZenFone AR should be quite easy to use with a 5.7-inch screen on Android 7 (Nougat).

Tango Development

First, you need a Tango device, because there is no simulator that can simulate Tango-related camera hardware. The Phab 2 Pro in the previous section is a very good development machine!

Now let’s take Java API development as an example to enter Tango App development. If you are familiar with Unity 3D development, you can directly use the Unity 3D package, because 80% of Tango Apps are developed with Unity 3D, and of course the most common ones are games!

Environment Preparation

1. Android Studio 2.2

2. USB debugging

Phab 2 Pro turns on USB debugging in Developer option. If there is no Developer option, then you must be a novice in Android development. Under Settings->About phone, click Build number 7 times in a row to summon Developer option.

Get the sample code

 git clone https://github.com/googlesamples/tango-examples-java.git

Open the Java Cloud point example

Directly control + R

Choose Phab 2 Pro

If the following error occurs:

 Error:Execution failed for task ':app:transformNative_libsWithStripDebugSymbolForDebug' . 
 
 > java.lang.NullPointerException ( no error message)

Just make sure the compileSdkVersion and targetSdkVersion versions are consistent with the buildToolsVersion.

Installation Successful

You can see Cloud point, real-time total points and average depth of points on your phone.

More

https://github.com/lvonasek/tango

An APP that uses C API to process the collected point cloud and realize 3D reconstruction.

Approximate scanning results:

Daydream Development

Let's review the purpose of Daydream. It can be simply seen as an upgraded version of Cardboard, but it defines a hardware standard that can meet the user experience and develops a complete set of APIs to allow developers to easily develop high-quality experience applications.

The same Daydream App requires a Daydream ready phone to run or a phone that can run Cardboard apps and is greater than Android 4.4 (kitkat), and here again, the Phab 2 Pro is used.

Daydream is mainly used to complete the following development tasks:

• Lens distortion correction.
• Spatial audio.
• Head tracking.
• 3D calibration.
• Side-by-side rendering.
• Stereo geometry configuration.
• User input event handling.

Get the code

 git clone https://github.com/googlevr/gvr-android-sdk.git

Android Studio

Open gvr-android-sdk

Run (control + R) sample-sdk-treasurehunt

Choose Phab 2 Pro

Use Cardboard or Daydream Viewer on your phone (of course, various domestic boxes are also available. Since Daydream Viewer does not add computing power, it only adds an external nine-axis inertial measurement unit to the operating handle to assist in precise tracking)

Summarize

With ZenFone AR and Daydream Viewer, the total price is expected to be less than RMB 5,000. You can immerse yourself in Google Street View, YouTube, Netflix, Hulu, HBO and VR games such as Fantastic Beasts, Where to Find Them, Need for Speed ​​and Gunjack 2 at home without having to drag long cables or equip a high-performance host. Instead of sitting quietly, you can have a 360-degree free perspective.

*** VR hardware and API are ready, start your own development plan!