Pierrick Jouet

Multi-device mixed reality TV: a collaborative experience with joint use of a tablet and a headset

A multi-user experience extending a standard TV content with AR elements is presented. It runs with both a standard tablet and a premium MR headset, the Microsoft HoloLens. A virtual TV mosaic is displayed around the TV screen and used as a GUI to control both TV and MR content. This paper focuses on the collaborative and personalized dimension offered by the experience. Unlike most AR applications, it can be simultaneously run by several users using different devices. The users can share content with others while keeping a personalized display. The added-value of such an extended TV experience has been demonstrated through complementary types of content, and user feedback confirms a real interest in this new kind of home entertainment, at the same time immersive, interactive, collaborative and personalized.

[POSTER] MR TV Mozaik: A New Mixed Reality Interactive TV Experience

Technicolor has been investigating how Mixed Reality technology could impact the future of home entertainment. We have designed and implemented a system to extend a standard TV experience with AR content, using a consumer tablet or a headset. A virtual TV mosaic is displayed around the TV screen and used as a GUI to control both TV and MR content. Using this interface, the user is able to switch TV content, display meta-data in AR (subtitles, text information or program guide), enhance TV content with interactive 3D objects blended in the environment, or play a game in interaction with the real world. The interactions between the real and the virtual worlds are handled thanks to a scene analysis pre-processing stage, which provides information about both the geometry and the lighting of the real environment. The real-virtual interactions strongly contribute to reinforcement of the immersion feeling. User feedback shows that the concept is very promising.

Mixed Reality Extended TV

The Extended TV application allows to enhance an audiovisual content displayed on a TV, using Mixed Reality technology. During preprocessing, the close environment of the TV is scanned using a consumer depth camera. The captured RGB-D data are analyzed, providing models for both the 3D geometry and the lighting of the real scene. During runtime, the TV is watched through a tablet, and virtual objects can apparently come out of the screen and start populating the users environment. Virtual objects can be occluded by real objects, and virtual shadows are consistent with the real ones.