In a blog post, Microsoft explained how they worked with car company VW to adapt HoloLens for use on a moving platform.
HoloLens uses two main types of sensors that measure its motion — visible light cameras and an inertial measurement unit, or IMU, that gauges acceleration and rotation speed. Together, the sensors mimic how humans see and move through the world.
But similar to how being in a car or boat can cause motion sickness when what appears to be a stable environment is actually moving, when the tightly coupled HoloLens sensors start disagreeing in a moving environment — with the inertial measurement unit recognizing motion and the cameras unable to — things fall apart.
Volkswagen hoped to use HoloLens for a research project, but when the device was put into a moving vehicle, its sensors lost tracking and the holograms it normally displays disappeared.
Dr. Andro Kleen, head of the data science team at Volkswagen Group Innovation, approached Microsoft for help and connected with Marc Pollefeys, Microsoft director of science and an expert in 3D computer vision and machine learning.
The two teams began collaborating around 2018 to develop the moving platform feature for HoloLens 2, which required solving a fundamental problem.
“We had extensive discussions,” says Pollefeys, now the director of Microsoft’s Mixed Reality and AI Lab in Zurich, Switzerland. “They presented their use cases and what they were hoping to enable. They were eager to work with us to find a solution and be able to use HoloLens in those situations.”
To address that issue, Pollefeys’ team developed an algorithm that models the discrepancies between the sensors and allows HoloLens to continue tracking.
The moving platform mode, combined with vehicle position data, enables HoloLens 2 to be used in new ways.
“We connected a positioning system that tracks the location of the vehicle. This way we were able to also place 3D elements such as information on point of interests outside of the car. This opens up completely new possibilities to not only display holograms within the driver’s forward-facing field of view, but also wherever the user wearing the glasses is looking,” says Michael Wittkämper, augmented reality expert at Volkswagen.
Microsoft rolled out the moving platform feature a few months ago and it is already attracting interest from maritime companies and organizations, which have been using HoloLens to remotely connect maritime workers with mechanical experts through Microsoft Dynamics 365 Remote Assist. The app allows an expert in another location to look through the other person’s HoloLens 2, share their field of view, diagnose a problem and provide input.
Previously only usable when a ship is in port, the capability is even more needed when vessels are at sea and a piece of equipment breaks down.
“The more remote the equipment or machine is, the harder it is to get the expert on site,” says Pollefeys, who is also a professor of computer science at ETH Zurich, a public research university. “This feature turned out to be critical to unlock HoloLens 2 for the maritime space.”
“We think of this as moving toward a mobility system where different products and mobility solutions will be connected,” Kleen says. “The basic assumption is that this technology will become lighter and smaller, and we think that as that happens, more people will get their hands on it and integrate it into their daily lives — and thus into their way of moving from A to B.”
Read more at Microsoft here.
