by

Holotoolkit Unity



-->

HTK 2017 MRTK v2; Setup: Delete MainCamera, add MixedRealityCameraParent / MixedRealityCamera / HoloLensCamera prefab to scene or use Mixed Reality Toolkit Configure Apply Mixed Reality Scene Settings menu item.: MainCamera parented under MixedRealityPlayspace via Mixed Reality Toolkit Add to Scene and Configure. HoloLens apps are created using either DirectX with C or Unity with C#. I found that it is generally quicker and easier to get an app up and running with Unity. In this article, we’ll set up. Let’s create an empty Unity project, clean the scene and import this package. Important: At this point we usually add some elements from HoloToolkit to use the Hololens Camera, InputManager and more. This new package already have a Prefab which include all this elements. It’s named Hololens location Assets / HUX / Prefabs / Interface.

Created by the team that produced Fragments and Young Conker, the HoloToolkit is a collection of scripts and components intended to accelerate the development of holographic applications targeting Windows Holographic. The toolkit is all free and comes in two varieties: one for Visual Studio and one for Unity 3D. I am a supporter of ensuring HoloToolkit-Unity compiles with target platforms in the same Unity project. Also it should work for all devices that support Windows Holographic or Mixed Reality. However, to burden the toolkit with ensuring everything works across different platforms (like iOS / oculus etc.) is overkill for it's current majority.

Note

The Mixed Reality Academy tutorials were designed with HoloLens (1st gen) and Mixed Reality Immersive Headsets in mind. As such, we feel it is important to leave these tutorials in place for developers who are still looking for guidance in developing for those devices. These tutorials will not be updated with the latest toolsets or interactions being used for HoloLens 2. They will be maintained to continue working on the supported devices. A new series of tutorials has been posted for HoloLens 2.

Unity

Spatial mapping combines the real world and virtual world together by teaching holograms about the environment. In MR Spatial 230 (Project Planetarium) we'll learn how to:

  • Scan the environment and transfer data from the HoloLens to your development machine.
  • Explore shaders and learn how to use them for visualizing your space.
  • Break down the room mesh into simple planes using mesh processing.
  • Go beyond the placement techniques we learned in MR Basics 101, and provide feedback about where a hologram can be placed in the environment.
  • Explore occlusion effects, so when your hologram is behind a real-world object, you can still see it with x-ray vision!

Device support

CourseHoloLensImmersive headsets
MR Spatial 230: Spatial mapping ✔️

Before you start

Prerequisites

  • A Windows 10 PC configured with the correct tools installed.
  • Some basic C# programming ability.
  • You should have completed MR Basics 101.
  • A HoloLens device configured for development.

Project files

  • Download the files required by the project. Requires Unity 2017.2 or later.
    • If you still need Unity 5.6 support, please use this release.
    • If you still need Unity 5.5 support, please use this release.
    • If you still need Unity 5.4 support, please use this release.
  • Un-archive the files to your desktop or other easy to reach location.

Note

If you want to look through the source code before downloading, it's available on GitHub.

Notes

  • 'Enable Just My Code' in Visual Studio needs to be disabled (unchecked) under Tools > Options > Debugging in order to hit breakpoints in your code.

Unity setup

  • Start Unity.
  • Select New to create a new project.
  • Name the project Planetarium.
  • Verify that the 3D setting is selected.
  • Click Create Project.
  • Once Unity launches, go to Edit > Project Settings > Player.
  • In the Inspector panel, find and select the green Windows Store icon.
  • Expand Other Settings.
  • In the Rendering section, check the Virtual Reality Supported option.
  • Verify that Windows Holographic appears in the list of Virtual Reality SDKs. If not, select the + button at the bottom of the list and choose Windows Holographic.
  • Expand Publishing Settings.
  • In the Capabilities section, check the following settings:
    • InternetClientServer
    • PrivateNetworkClientServer
    • Microphone
    • SpatialPerception
  • Go to Edit > Project Settings > Quality
  • In the Inspector panel, under the Windows Store icon, select the black drop-down arrow under the 'Default' row and change the default setting to Very Low.
  • Go to Assets > Import Package > Custom Package.
  • Navigate to the ..HolographicAcademy-Holograms-230-SpatialMappingStarting folder.
  • Click on Planetarium.unitypackage.
  • Click Open.
  • An Import Unity Package window should appear, click on the Import button.
  • Wait for Unity to import all of the assets that we will need to complete this project.
  • In the Hierarchy panel, delete the Main Camera.
  • In the Project panel, HoloToolkit-SpatialMapping-230UtilitiesPrefabs folder, find the Main Camera object.
  • Drag and drop the Main Camera prefab into the Hierarchy panel.
  • In the Hierarchy panel, delete the Directional Light object.
  • In the Project panel, Holograms folder, locate the Cursor object.
  • Drag & drop the Cursor prefab into the Hierarchy.
  • In the Hierarchy panel, select the Cursor object.
  • In the Inspector panel, click the Layer drop-down and select Edit Layers...
  • Name User Layer 31 as 'SpatialMapping'.
  • Save the new scene: File > Save Scene As..
  • Click New Folder and name the folder Scenes.
  • Name the file 'Planetarium' and save it in the Scenes folder.

Chapter 1 - Scanning

Objectives

  • Learn about the SurfaceObserver and how its settings impact experience and performance.
  • Create a room scanning experience to collect the meshes of your room.

Instructions

  • In the Project panel HoloToolkit-SpatialMapping-230SpatialMappingPrefabs folder, find the SpatialMapping prefab.
  • Drag & drop the SpatialMapping prefab into the Hierarchy panel.

Build and Deploy (part 1)

  • In Unity, select File > Build Settings.
  • Click Add Open Scenes to add the Planetarium scene to the build.
  • Select Universal Windows Platform in the Platform list and click Switch Platform.
  • Set SDK to Universal 10 and UWP Build Type to D3D.
  • Check Unity C# Projects.
  • Click Build.
  • Create a New Folder named 'App'.
  • Single click the App folder.
  • Press the Select Folder button.
  • When Unity is done building, a File Explorer window will appear.
  • Double-click on the App folder to open it.
  • Double-click on Planetarium.sln to load the project in Visual Studio.
  • In Visual Studio, use the top toolbar to change the Configuration to Release.
  • Change the Platform to x86.
  • Click on the drop-down arrow to the right of 'Local Machine', and select Remote Machine.
  • Enter your device's IP address in the Address field and change Authentication Mode to Universal (Unencrypted Protocol).
  • Click Debug -> Start Without debugging or press Ctrl + F5.
  • Watch the Output panel in Visual Studio for build and deploy status.
  • Once your app has deployed, walk around the room. You will see the surrounding surfaces covered by black and white wireframe meshes.
  • Scan your surroundings. Be sure to look at walls, ceilings, and floors.

Build and Deploy (part 2)

Now let's explore how Spatial Mapping can affect performance.

  • In Unity, select Window > Profiler.
  • Click Add Profiler > GPU.
  • Click Active Profiler > .
  • Enter the IP address of your HoloLens.
  • Click Connect.
  • Observe the number of milliseconds it takes for the GPU to render a frame.
  • Stop the application from running on the device.
  • Return to Visual Studio and open SpatialMappingObserver.cs. You will find it in the HoloToolkitSpatialMapping folder of the Assembly-CSharp (Universal Windows) project.
  • Find the Awake() function, and add the following line of code: TrianglesPerCubicMeter = 1200;
  • Re-deploy the project to your device, and then reconnect the profiler. Observe the change in the number of milliseconds to render a frame.
  • Stop the application from running on the device.

Save and load in Unity

Finally, let's save our room mesh and load it into Unity.

  • Return to Visual Studio and remove the TrianglesPerCubicMeter line that you added in the Awake() function during the previous section.
  • Redeploy the project to your device. We should now be running with 500 triangles per cubic meter.
  • Open a browser and enter in your HoloLens IPAddress to navigate to the Windows Device Portal.
  • Select the 3D View option in the left panel.
  • Under Surface reconstruction select the Update button.
  • Watch as the areas that you have scanned on your HoloLens appear in the display window.
  • To save your room scan, press the Save button.
  • Open your Downloads folder to find the saved room model SRMesh.obj.
  • Copy SRMesh.obj to the Assets folder of your Unity project.
  • In Unity, select the SpatialMapping object in the Hierarchy panel.
  • Locate the Object Surface Observer (Script) component.
  • Click the circle to the right of the Room Model property.
  • Find and select the SRMesh object and then close the window.
  • Verify that the Room Model property in the Inspector panel is now set to SRMesh.
  • Press the Play button to enter Unity's preview mode.
  • The SpatialMapping component will load the meshes from the saved room model so you can use them in Unity.
  • Switch to Scene view to see all of your room model displayed with the wireframe shader.
  • Press the Play button again to exit preview mode.

NOTE: The next time that you enter preview mode in Unity, it will load the saved room mesh by default.

Chapter 2 - Visualization

Objectives

  • Learn the basics of shaders.
  • Visualize your surroundings.

Instructions

  • In Unity's Hierarchy panel, select the SpatialMapping object.
  • In the Inspector panel, find the Spatial Mapping Manager (Script) component.
  • Click the circle to the right of the Surface Material property.
  • Find and select the BlueLinesOnWalls material and close the window.
  • In the Project panel Shaders folder, double-click on BlueLinesOnWalls to open the shader in Visual Studio.
  • This is a simple pixel (vertex to fragment) shader, which accomplishes the following tasks:
    1. Converts a vertex's location to world space.
    2. Checks the vertex's normal to determine if a pixel is vertical.
    3. Sets the color of the pixel for rendering.

Hsc syllabus 2018 bd. Build and Deploy

  • Return to Unity and press Play to enter preview mode.
  • Blue lines will be rendered on all vertical surfaces of the room mesh (which automatically loaded from our saved scanning data).
  • Switch to the Scene tab to adjust your view of the room and see how the entire room mesh appears in Unity.
  • In the Project panel, find the Materials folder and select the BlueLinesOnWalls material.
  • Modify some properties and see how the changes appear in the Unity editor.
    • In the Inspector panel, adjust the LineScale value to make the lines appear thicker or thinner.
    • In the Inspector panel, adjust the LinesPerMeter value to change how many lines appear on each wall.
  • Click Play again to exit preview mode.
  • Build and deploy to the HoloLens and observe how the shader rendering appears on real surfaces.

Unity does a great job of previewing materials, but it's always a good idea to check-out rendering in the device. Waring food dehydrator 11df21 manual.

Chapter 3 - Processing

Objectives

  • Learn techniques to process spatial mapping data for use in your application.
  • Analyze spatial mapping data to find planes and remove triangles.
  • Use planes for hologram placement.

Instructions

  • In Unity's Project panel, Holograms folder, find the SpatialProcessing object.
  • Drag & drop the SpatialProcessing object into the Hierarchy panel.

The SpatialProcessing prefab includes components for processing the spatial mapping data. SurfaceMeshesToPlanes.cs will find and generate planes based on the spatial mapping data. We will use planes in our application to represent walls, floors and ceilings. This prefab also includes RemoveSurfaceVertices.cs which can remove vertices from the spatial mapping mesh. This can be used to create holes in the mesh, or to remove excess triangles that are no longer needed (because planes can be used instead).

  • In Unity's Project panel, Holograms folder, find the SpaceCollection object.
  • Drag and drop the SpaceCollection object into the Hierarchy panel.
  • In the Hierarchy panel, select the SpatialProcessing object.
  • In the Inspector panel, find the Play Space Manager (Script) component.
  • Double-click on PlaySpaceManager.cs to open it in Visual Studio.

PlaySpaceManager.cs contains application-specific code. We will add functionality to this script to enable the following behavior:

Holotoolkit Unity

Using holotoolkit.unity
  1. Stop collecting spatial mapping data after we exceed the scanning time limit (10 seconds).
  2. Process the spatial mapping data:
    1. Use SurfaceMeshesToPlanes to create a simpler representation of the world as planes (walls, floors, ceilings, etc).
    2. Use RemoveSurfaceVertices to remove surface triangles that fall within plane boundaries.
  3. Generate a collection of holograms in the world and place them on wall and floor planes near the user.

Complete the coding exercises marked in PlaySpaceManager.cs, or replace the script with the finished solution from below:

Build and Deploy

  • Before deploying to the HoloLens, press the Play button in Unity to enter play mode.
  • After the room mesh is loaded from file, wait for 10 seconds before processing starts on the spatial mapping mesh.
  • When processing is complete, planes will appear to represent the floor, walls, ceiling, etc.
  • After all of the planes have been found, you should see a solar system appear on a table of floor near the camera.
  • Two posters should appear on walls near the camera too. Switch to the Scene tab if you cannot see them in Game mode.
  • Press the Play button again to exit play mode.
  • Build and deploy to the HoloLens, as usual.
  • Wait for scanning and processing of the spatial mapping data to complete.
  • Once you see planes, try to find the solar system and posters in your world.

Chapter 4 - Placement

Objectives

  • Determine if a hologram will fit on a surface.
  • Provide feedback to the user when a hologram can/cannot fit on a surface.

Instructions

  • In Unity's Hierarchy panel, select the SpatialProcessing object.
  • In the Inspector panel, find the Surface Meshes To Planes (Script) component.
  • Change the Draw Planes property to Nothing to clear the selection.
  • Change the Draw Planes property to Wall, so that only wall planes will be rendered.
  • In the Project panel, Scripts folder, double-click on Placeable.cs to open it in Visual Studio.

The Placeable script is already attached to the posters and projection box that are created after plane finding completes. All we need to do is uncomment some code, and this script will achieve the following:

  1. Determine if a hologram will fit on a surface by raycasting from the center and four corners of the bounding cube.
  2. Check the surface normal to determine if it is smooth enough for the hologram to sit flush on.
  3. Render a bounding cube around the hologram to show its actual size while being placed.
  4. Cast a shadow under/behind the hologram to show where it will be placed on the floor/wall.
  5. Render the shadow as red, if the hologram cannot be placed on the surface, or green, if it can.
  6. Re-orient the hologram to align with the surface type (vertical or horizontal) that it has affinity to.
  7. Smoothly place the hologram on the selected surface to avoid jumping or snapping behavior.

Uncomment all code in the coding exercise below, or use this completed solution in Placeable.cs:

Build and Deploy

  • As before, build the project and deploy to the HoloLens.
  • Wait for scanning and processing of the spatial mapping data to complete.
  • When you see the solar system, gaze at the projection box below and perform a select gesture to move it around. While the projection box is selected, a bounding cube will be visible around the projection box.
  • Move you head to gaze at a different location in the room. The projection box should follow your gaze. When the shadow below the projection box turns red, you cannot place the hologram on that surface. When the shadow below the projection box turns green, you can place the hologram by performing another select gesture.
  • Find and select one of the holographic posters on a wall to move it to a new location. Notice that you cannot place the poster on the floor or ceiling, and that it stays correctly oriented to each wall as you move around.

Chapter 5 - Occlusion

Objectives

  • Determine if a hologram is occluded by the spatial mapping mesh.
  • Apply different occlusion techniques to achieve a fun effect.

Instructions

First, we are going to allow the spatial mapping mesh to occlude other holograms without occluding the real world:

  • In the Hierarchy panel, select the SpatialProcessing object.
  • In the Inspector panel, find the Play Space Manager (Script) component.
  • Click the circle to the right of the Secondary Material property.
  • Find and select the Occlusion material and close the window.

Next, we are going to add a special behavior to Earth, so that it has a blue highlight whenever it becomes occluded by another hologram (like the sun), or by the spatial mapping mesh:

  • In the Project panel, in the Holograms folder, expand the SolarSystem object.
  • Click on Earth.
  • In the Inspector panel, find the Earth's material (bottom component).
  • In the Shader drop-down, change the shader to Custom > OcclusionRim. This will render a blue highlight around Earth whenever it is occluded by another object.

Finally, we are going to enable an x-ray vision effect for planets in our solar system. We will need to edit PlanetOcclusion.cs (found in the ScriptsSolarSystem folder) in order to achieve the following:

  1. Determine if a planet is occluded by the SpatialMapping layer (room meshes and planes).
  2. Show the wireframe representation of a planet whenever it is occluded by the SpatialMapping layer.
  3. Hide the wireframe representation of a planet when it is not blocked by the SpatialMapping layer.

Follow the coding exercise in PlanetOcclusion.cs, or use the following solution:

Build and Deploy

  • Build and deploy the application to HoloLens, as usual.
  • Wait for scanning and processing of the spatial mapping data to be complete (you should see blue lines appear on walls).
  • Find and select the solar system's projection box and then set the box next to a wall or behind a counter.
  • You can view basic occlusion by hiding behind surfaces to peer at the poster or projection box.
  • Look for the Earth, there should be a blue highlight effect whenever it goes behind another hologram or surface.
  • Watch as the planets move behind the wall or other surfaces in the room. You now have x-ray vision and can see their wireframe skeletons!

The End

Congratulations! You have now completed MR Spatial 230: Spatial mapping.

  • You know how to scan your environment and load spatial mapping data to Unity.
  • You understand the basics of shaders and how materials can be used to re-visualize the world.
  • You learned of new processing techniques for finding planes and removing triangles from a mesh.
  • You were able to move and place holograms on surfaces that made sense.
  • You experienced different occlusion techniques and harnessed the power of x-ray vision!

This article was first posted on gooroo.io

Developing applications and experiences for HoloLens is not hard to get started with, but after you have started a number of projects and worked through the same setup and plumbing a few times, you wish there was an easier way to do it. Well, there is! Created by the team that produced Fragments and Young Conker, the HoloToolkit is a collection of scripts and components intended to accelerate the development of holographic applications targeting Windows Holographic.

The toolkit is all free and comes in two varieties: one for Visual Studio and one for Unity 3D. Developing for HoloLens means spending a LOT of time in the Unity editor, and the HoloToolkit is a simple .unitypackage file you can download and import into your Unity project.

This will show a new menu item “HoloToolkit” in Unity.

These menu items form part of the visual change to Unity that HoloToolkit implements. Having menu items to perform the most trivial and basic configuration tasks for a new project gets you to the fun bits a lot quicker, and it makes sure that you haven’t missed anything.

Apart from the obvious menu item changes, the HoloToolkit also brings with it a LOT of scripts, prefabs, materials, shaders and plugins you can use in Unity3D.

While the sheer number of scripts in the toolkit makes it out of scope to go through them in this article, I’ll outline some of the capabilities they bring.

Gaze

Some of the issues with capturing a user’s gaze is that they don’t hold their head 100% still and they might try to look at objects that are hidden behind other objects. With the toolkit you get GazeManager and GazeStabiliser. These two scripts can help you determine if users are gazing at a particular object, if there are multiple objects in the gaze direction, stabilize the gaze for precision targeting based on general direction and interpolation algorithms and more.

Input Events

To deliver input to a mixed reality experience, you generally rely on GGV; gaze, gesture and voice. The HoloToolkit provides an InputManager script which have a large amount of functions and features to manage the various ways a user might input data to a HoloLens app using gaze and gesture. The script then also dispatches relevant events, such as hold for a gesture and change of focus for gaze. The InputManager script is 610 lines of genuine goodness that provide a bunch of plumbing for managing gaze and gestures in your app.

Voice

While voice commands are not difficult to implement (read my article on developing for HoloLens for more details), some of the process can be tedious. The KeywordManager script not only removes some of that plumbing code, but also provides ways to explicitly set keywords, or voice commands, in Unity rather than code. It also gives you the option to start the keyword recognizer manually, which is very handy in the scenario where you don’t want the app to listen for voice commands.

Sharing

The collaboration story on the HoloLens is one of the most compelling features for building real world experiences that actually will have some impact. Being able to see the same digital assets on two different devices in real time, while interacting with them, is incredibly powerful.

Holotoolkit Unitypackage

The HoloToolkit provides some enhancements to the sharing experience, which makes it, again, just a bit simpler. It provides everything from additional menu items in Unity, to pairing scripts and session management.

Holotoolkit Unity Package Download

Spatial Mapping

The defining factor of the HoloLens over other digital reality devices is the spatial mapping, which allows interaction between physical objects and digital assets. The spatial grid is a relatively high definition set of polygons that make out the space the user is in. The hard part comes when you need to identify what surfaces you can use to place objects, what is a wall, what is the floor etc. And again, HoloToolkit to the rescue!!

There are scripts like the SpatialMappingManager that lets you select materials, draw shadows, visualise the spatial mapping and set the spatial mapping source. The TapToPlace script will let users move objects around and placing them on real world surfaces.

Using Holotoolkit.unity

Use It. Now.

Holotoolkit Unity Package Download

The HoloToolkit really is a no-brainer when you start a new mixed reality project. The time saved both for setting up the project as well as for the learning curve of implementing the various event handlers and features, is significant. Because the project is on github, it is regularly updated, maintained and pull requests are regularly taken in.

Holotoolkit Unity

If you haven’t already got your first project on the way, check out my introduction course on Pluralsight.