In this story, I will show how to use your HoloLens to scan your room and place a hologram in an empty space in the floor.

Introduction

One of the most interesting feature in the Mixed Reality Platform is the ability to interact with your environment placing holograms in your walls, ceiling and floor. This can be created with the HoloLen’s Spatials components, which can be viewed as a mesh representation of your surroundings.

‍

Spatial Mapping vs Spatial Understanding

Spatial Mapping

The main goal of this module is to detect solid surfaces in your room and transform them into a solid object which a GameObject (holograms) can interact and collide. When this scanning that place, it draw a mesh at the surface scanned and in each scan loop it improves the scanned data.

Spatial mapping provides a detailed representation of real-world surfaces in the environment around the HoloLens, allowing developers to create a convincing Mixed Reality experience. By merging the real world with the virtual world, an application can make holograms seem real. Applications can also be more naturally aligned with user expectations by providing familiar real-world behaviors and interactions.

The HoloLens device constantly scans its surroundings and refines its understanding of the world based on new sensor data. Updates occur frequently so that the device can pick up environmental changes such as people moving through the room or doors opening and closing. The world mapping data set is saved on the device and persists across multiple applications and device restarts.

‍

Spatial Understanding

When placing holograms in the physical world it is often desirable to go beyond spatial mapping’s mesh and surface planes. When placement is done procedurally, a higher level of environmental understanding is desired. This usually requires making decisions about what is floor, ceiling, and walls. In addition, the ability to optimize against a set of placement constraints to determining the most desirable physical locations for holographic objects.

Basically, this is an evolution and improvement of the Spatial Mapping component, where you can query for specific space and then place desired hologram in that space.

It’s enough theory. Let’s get some action!

‍

0. Create Base project

For this tutorial we’re using Unity3D 2017.3.03f, Mixed Reality Toolkit 2017.2.1f1 and Visual Studio 2017.

1. Open Unity3D and create a new project and name it SpatialDemo.

‍

‍

2. Import the MixedRealityToolKit package (Assets -> Import Package -> Custom Package). Once selected the package, click Import when is shown the files details.

‍

‍

3. Now, we need to configure the Project Settings. For that, open the Mixed Reality Toolkit menu > Configure > Apply Mixed Reality Project Settings windows and click Apply without any changes. This will change output target to run in Hololens.

‍

‍

4. Next, we should setup the Scene for work with the MR Platform. To do this, open the Mixed Reality Toolkit menu > Configure > Apply Mixed Reality Scene Settings windows and click Apply without any changes. This will prepare the actual scene with all the needed settings to answer the headset movement and the input (gestures and voice), and also add a default Gaze.

‍

‍

5. Following, lets configure the Capabilities of the app. So, open the dialog Mixed Reality Toolkit menu > Configure > Apply UWP Capability Settings, and ensure you’ve checked the Spatial Perception before you apply the changes.

‍

‍

6. Save the scene and choose a name of your like (i.e. General).

‍

1. Add Spatial Mapping

1. Locate the HoloToolKit\SpatialMapping\Prefabs folder and then drag the SpatialMapping file to the Hierarchy view. This prefab is the responsible to scan your environment and generate the mesh in which your GameObject can interact. With this, you can put a hologram on a surface you’ve scanned.

2. Select the SpatialMapping into your scene and uncheck the Draw Visual Meshes option in the Spatial Mapping Manager component. As the Spatial Understanding already draw a mesh, it unnecessary this.

‍

‍

3. Locate the HoloToolKit\SpatialUnderstanding\Prefabs folder and then drag the SpatialUnderstanding prefab. This component will give you a more sophisticated mesh and a finer scanning meshes than the ones SpatialMapping provides.

4. Create an empty GameObject named MappingOrchestator. There, you will attach your script which will scan and query for empty space in your room.

5. Create a new C# script named ScanManager in a new folder named Scripts. This script will start the spatial scan and determine when the user decides to complete the scan (doing AirTap). When this is achieved, this script will place a hologram in the first place it detected there is enough space. Remember to attach this script to the MappingOrchestator GameObject. You can leave this script empty for now.

‍

2. Improve the UI for the user

1. Add a new 3D Text GameObject to your scene. To do this, go to the GameObject menu, then choose 3D Object and next the 3D Text menu. Rename the object to something like InstructionText. This GameObject will show some helpful text to the user.

2. Change the Scale in the Transform component to 0.005 in the three dimensions. Then, in the Text Mesh component, change the Character Size to 0.5 and the Font Size to 48.

‍

‍

3. Because the 3D Text GameObejct will be fixed on the space world, it could be great if it moves along the user does while scanning. To make that possible, you simple need to add two components to it.

• The first is the Tagalong which make your GameObject follows the view of the user. So, if the user turns left, the GameObject which has attached this component will also be turned to the left smoothly. Same goes to each direction, of course.
• The second component is the Billboard which makes the GameObject always face to the user. For this example, you may change the Pivot Axis property the Y axis to only rotates the hologram in Y axis.

‍

‍

4. Select the Spatial Understanding GameObject and change the Mesh Material to SpatialMappingWireframe in the Spatial Understanding Custom Mesh in the SpatialUnderstanding GameObject.

‍

3. Create the Scanning Logic

1. Open the ScanManager.cs file in Visual Studio. You need to add a public variable named InstructionTextMesh, to pass from editor the Text Mesh’s 3D Text you created earlier. There is where the application will show the instructions to the user.

‍

‍

2. In the Start method, you need to call the RequestBegingScanning method from the SpatialUnderstanding instance to tell it to start the scan process. Additionally, you may handle the ScanStateChanged event to be notified when the scan is completed.

‍

‍

3. In the Update method, check the Scan State. If the user hasn’t decided to finish the scanning, you can display some metrics about the scanned surfaces to show some progress to the user like shown in the following code:

‍

‍

4. Additionally, you can implement the IInputClickHandler interface and its method to handle the Air Tap Gesture from the user to finish the scan process. To do this, in the OnInputClicked method, call the RequestFinishScan method from the SpatialUnderstanding instance. This could take some time, because it will take all the scanned surfaces and merge them together. Additionally, for all the areas that the user didn’t scan, it will join the nearest surface to get a closed room.

‍

‍

5. When the scan completes, your script will query for the scan result to get the first surface in the floor to put the hologram you configured. If it’s not enough surface, you may notify the user to re-try the scan (it’s your homework). You may also want to keep validating that the scanned space reached the minimum required for your hologram until user requests the scan finish (more homework here).

You code may look as the one bellow:

‍

‍

4. Test your code

1. Turn back to Unity Editor and select the MappingOrchestator GameObject. Note that the ScanManager component attached now accepts two parameters. In the Instruction Text Mesh, drag the InstructionText 3D text. For the Floor Prefab, you can use this chair or import any other you like to put on the floor.

2. Save your scene and Build your project (File > Build Settings) to test on your HoloLens Device or Emulator.

3. Once you have your application deployed and running, you can start scanning your room. You will see how the Spatial Understanding’s mesh is rendered on the surfaces. Try walking a considerable distance (you can view the amounts scanned on the 3D Text on the screen).

4. When you consider that the amount of space is enough, Air Tap to complete the scan. Next, if you scanned a surface of at least 0.25 by 0.25 meters of empty floor, the prefab will be put in the first space found by the mapping.

‍

‍

Conclusion

In this story you learned how is the general process required for placing hologram in your environment using the Spatial Understanding and the limit is only your imagination. Your applications will be welcomed for other user if it is fit his surroundings.

In the next story for this series, we will learn how to query another topology aside only the floor.

Thanks to Nicolás Bello Camilletti

‍

‍Originally published by Sebastian Gambolati for SOUTHWORKS on Medium 06 March 2018

‍