Windows Holographic

File:Microsoft Windows Holographic.png

A depiction of a Microsoft HoloLens user navigating the shell, with an application window on the left, and the Start menu on the right.

Windows Holographic is a mixed reality platform developed by Microsoft, built around the API of Windows 10.

Holographic works by enabling applications in which the live presentation of physical real-world elements is incorporated with that of virtual elements (referred to as "holograms" by Microsoft) such that they are perceived to exist together in a shared environment. A variant of Windows for augmented reality computers (which augment a real-world physical environment with virtual elements) Windows Holographic features an augmented-reality operating environment in which any Universal Windows App can run. In addition, with Windows Holographic Platform APIs, which are part of the Universal Windows Platform, and supported as standard in Windows 10 (including versions for mobile devices and Xbox One), mixed reality features can be readily implemented in any Universal Windows App, for a wide range of Windows 10-based devices.

Microsoft announced Windows Holographic at its "Windows 10: The Next Chapter" press event on January 21, 2015. It is set to be introduced through the smart glasses headset Microsoft HoloLens, as part of the general rollout of Windows 10. The Windows 10 launch began July 29, 2015 with release of the PC version, with the Microsoft HoloLens Development Edition being released beginning March 30, 2016.

Microsoft HoloLens

Main article: Microsoft HoloLens

The premier device for Windows Holographic, Microsoft HoloLens is a smart-glasses headset that is a cordless, self-contained Windows 10 computer. It uses various sensors, a high-definition stereoscopic 3D optical head-mounted display, and spatial sound to allow for augmented reality applications, with a natural user interface that the user interacts with through gaze, voice, and hand gestures. Codenamed "Project Baraboo," HoloLens had been in development for five years before its announcement in 2015, but was conceived earlier as the original pitch made in late 2007 for what would become the Kinect technology platform. It was introduced with operating systems, such as Android, iOS and OS X on April 5, 2016.

Microsoft has targeted HoloLens for release "in the Windows 10 timeframe," with the Microsoft HoloLens Development Edition to begin shipping March 30, 2016, available by application to developers in the United States and Canada for a list price of US$3000. Although the Development Edition is considered to be consumer-ready hardware, as of February 2016 Microsoft has not set a time frame for consumer availability of HoloLens, with HoloLens chief inventor Alex Kipman stating that HoloLens will have a consumer release only when the market is ready for it. Companies such as Samsung Electronics and Asus had expressed interest in working with Microsoft to produce their own mixed-reality products based on HoloLens.

Design

The Microsoft HoloLens design is of a visor glasses unit connected to an adjustable padded inner headband. The unit can be tilted up and down, as well as adjusted forward and backward in relation to the headband. To wear the HoloLens, the user uses an adjustment wheel at the back of the headband to secure it around the crown of their head, supporting the weight of the unit. The user then tilts the visor down in front of the eyes.

In the brow of the unit is much of the sensors and related hardware, including the cameras and processors. The visor front is tinted; enclosed in the visor piece is a pair of transparent combiner lenses, in which the projected images are displayed in the lower half. The HoloLens must be calibrated to the interpupillary distance (IPD) of the individual user.

Along the bottom edges of the side arms by wearer's ears are a pair of small red stereo speakers. As opposed to headphones, the speakers do not obstruct external sound, allowing the user to hear virtual sounds together with the real-world environment. Using head-related transfer functions, the HoloLens generates binaural audio, which can simulate spatial effects, such that the user can perceive a sound as coming from a specific location.

On the top edge are two pairs of buttons: display brightness buttons above the left ear, and volume buttons above the right ear. Adjacent buttons are shaped differently—one concave, one convex—so that the user can distinguish them by touch.

At the end of the left arm is a power button and row of five tiny LEDs, which together are used to indicate system status, as well as for power management, indicating battery level and setting power/standby mode. A USB 2.0 micro-B receptacle is located along the bottom edge. A 3.5 mm audio jack is located along the bottom edge of the right arm.

Hardware

The HoloLens features an inertial measurement unit (IMU), four "environment understanding" cameras, an energy-efficient depth camera,, a 2-megapixel photographic video camera, a four-microphone array, and an ambient light sensor. The inertial measurement unit includes an accelerometer, gyroscope, and a magnetometer.

In addition to a CPU and GPU, HoloLens features a Microsoft Holographic Processing Unit (HPU), a custom coprocessor by Microsoft. The HPU processes and integrates data from the sensors—handling tasks such as spatial mapping, gesture recognition, and voice and speech recognition. According to Alex Kipman, the HPU processes "terabytes of information" from the HoloLens's sensors in real time. (see also Vision processing unit).

The combiner lenses of the HoloLens are optical waveguides consisting of three layers—for blue, green, and red—each with diffractive features. A "light engine" above each combiner lens projects light into the lens, a wavelength that then hits a diffractive element and reflects repeatedly along a layer until it is output to the eye. Similar to that of many other optical head-mounted displays, the display projection for the HoloLens occupies a limited portion of the user's field of view (FOV), particularly in comparison to virtual reality head-mounted displays, which typically cover a much greater field of view. Based on preliminary hardware, most observers have characterized the field of view of the HoloLens as small, though subjective estimates vary significantly. From the Build 2015 developer conference in May, one attendee estimated that the display field of view of the demonstration units was 30°×17.5°. In an interview at the 2015 Electronic Entertainment Expo in June, Microsoft corporate vice president of next-gen experiences Kudo Tsunoda indicated that the field of view is unlikely to be significantly different on release.

The HoloLens contains an internal rechargeable battery. Battery life is rated at 2-3 hours of active use, or 2 weeks of standby time. The HoloLens can be operated while charging.

HoloLens features IEEE 802.11ac Wi-Fi and Bluetooth 4.1 Low Energy (LE) wireless connectivity. The headset uses Bluetooth LE to pair with the included Clicker, a thumb-operated finger-sized input device that can be used for interface scrolling and selecting. The Clicker features a clickable surface for selecting, and an orientation sensor which provides for scrolling functions via tilting and panning of the unit. The Clicker features an elastic finger loop for holding the device, and a USB 2.0 micro-B receptacle for charging its internal battery.

Applications

As of 2016, a number of augmented-reality applications have been announced or showcased for Microsoft HoloLens. A collection of applications will be provided for free for developers purchasing the Microsoft HoloLens Developer Edition. Applications available at launch include:

• HoloStudio, a full-scale 3D modelling application by Microsoft with 3D print compatibility
• Actiongram, an application for staging and recording short video clips of simple mixed-reality presentations using pre-made 3D virtual assets, will be released in summer 2016 in the United States and Canada.
• An implementation of the Skype telecommunications application by Microsoft
• HoloTour, an audiovisual three-dimensional virtual tourism application
• Fragments, a high-tech crime thriller adventure game developed by Microsoft and Asobo Studio, in which the player engages in crime-solving
• Young Conker, a platform game developed by Microsoft and Asobo Studio, featuring a young version of Conker the Squirrel
• RoboRaid (previously code-named "Project X-Ray"), an augmented-reality first-person shooter game by Microsoft in which the player defends against a robot invasion, aiming the weapon via gaze, and shooting via the Clicker button or an air tap

Other applications announced or showcased for HoloLens include:

• An interactive digital human anatomy curriculum by Case Western Reserve University and Cleveland Clinic
• Architectural engineering software tools by Trimble Navigation
• A version of the Mojang video game Minecraft
• Extended functionality for the Autodesk Maya 3D creation application
• OnSight and Sidekick, software projects developed by a collaboration between NASA and Microsoft to explore mixed reality applications in space exploration
• FreeForm, a joint project between Autodesk and Microsoft integrating HoloLens with the Autodesk Fusion 360 cloud-based 3D development application
• Galaxy Explorer, an educational application about the Milky Way in development by Microsoft Studios, pitched and chosen by the developer community via the Share Your Idea campaign, and to be open-sourced upon completion
• A spacecraft design/visualization application in development by NASA's Jet Propulsion Laboratory (JPL)

Developed in collaboration with JPL, OnSight integrates data from the Curiosity rover into a 3D simulation of the Martian environment, which scientists around the world can visualize, interact with, and collaborate in together using HoloLens devices. OnSight can be used in mission planning, with users able to program rover activities by looking at a target within the simulation, and using gestures to pull up and select menu commands. JPL plans to deploy OnSight in Curiosity mission operations, using it to control rover activities by July 2015.

Interface

HoloLens features a natural user interface which the user interacts with using gaze, gesture, and voice inputs—sometimes referred to collectively as "GGV". Gaze tracking techniques such as head-tracking allows the user to bring application focus to whatever the user is looking at. Elements can be selected via an "air tap" gesture akin to clicking an imaginary mouse, with the hand raised in a pointing position, and tapping down on the element using the index finger The "tap" can be held for performing a "drag" function. Users can perform various actions via voice command.

The HoloLens shell carries over and adapts many elements from the Windows desktop environment. A "bloom" gesture for accessing the shell (performing a similar function to pressing a Windows key on a Windows keyboard or tablet, or the Xbox button on an Xbox One Controller) is performed by opening one's hand, fingers spread with the palm facing up. Windows can be dragged to a particular position, as well as resized. Virtual elements such as windows or menus can be "pinned" to locations, physical structures or objects within the environment; or can be "carried," or fixed in relation to the user, following the user as they move around. Title bars for application windows have a title on the left, and buttons for window management functions on the right.

See also

• Magic Leap
• Google Glass
• Meta (company)
• Daqri