Real-Time Streaming of 3D Ultrasound Data to HoloLens

Ultrasound (US) imaging is a common method for interventional tasks as well as for diagnostic purposes. Currently, thephysician’s gaze is focused on the US workstation as generated images are displayed on-screen. Thus, the physician doesnot have visual information regarding the position of the US probe relative to the patient. Furthermore, the quality of 3Drendering is limited. HoloLens (Microsoft, USA) as an optical see-through head-mounted display can solve this problemby rendering 3D objects that can be displayed next to the US probe or following the gaze of the physician. We present anapproach that allows real-time streaming of US volumes to the HoloLens.A 3D US station (Vivid7, GE Healthcare) with an in-house modification, which enables access to the raw volume data,was used for US volume acquisition. Lossless data encoding was performed to reduce the bandwidth required and theresulting data was streamed to HoloLens using Google’s open-source gRPC framework. The application for HoloLenswas developed using Unity and C#. It includes the gRPC receiving module, the decoding module and the volume render-ing to display 3D US data. This approach was evaluated sending US volumes (size 67 x 92 x 120 voxels) from a computerto a router via Ethernet and then to the HoloLens via Wi-Fi with a frequency of 9.2 Hz (108.7 ms).After encoding, the data size was decreased by 63 %. Consecutive volumes were displayed with a time difference of 108.8± 42.3 ms (mean ± standard deviation) and a latency of 118.4 ± 27.9 ms. The results show that the proposed method iscapable of real-time streaming of US volumes to HoloLens. This will offer benefits for the physician during examinationsuch as a more intuitive view on anatomical 3D data and direct feedback of the position of the probe related to the patientas the rendered US volume is displayed in the physician’s visual field.