Dr. Ralf Schäfer

Ralf Schäfer ist Leiter der Videoabteilung am Fraunhofer Heinrich Hertz Institut (HHI). Nach seinem Studium des Electrical Engineering an der Technischen Universität Berlin (TUB), wurde er nach seiner Diplomprüfung im Jahre 1977 wissenschaftlicher Mitarbeiter im Heinrich-Hertz-Institut in Berlin. Im Jahre 1984 promovierte er an der TUB mit einer Arbeit aus dem Bereich der digitalen Codierung von Fernsehsignalen. Seit 1989 ist er Leiter der Abteilung Image Processing des HHI, wo er für 90 Wissenschaftler und 50 studentische Mitarbeiter verantwortlich ist. Seine Forschungsinteressen liegen in allen Bereichen, die Bilder und Video betreffen, von der Aufnahme bis zum Display und von Algorithmen bis hin zur Chiprealisierung. Neben seiner Rolle als Abteilungsleiter, trägt er die Verantwortung für 3 Technologiezentren: dem CINQ Center, a forum for knowledge and technology transfer related to Smart Data (www.cinq.de), dem Innovation Center for Immersive Imaging Technologies – 3IT (www.3it-berlin.de) und dem TiME-Lab (www.timelab-hhi.com), wo immersive Technologien entwickelt und demonstriert werden.

Synopsis

VOLUMETRIC VIDEO – CREATING REALISTIC DYNAMIC 3D MODELS OF PERSONS
R. Schäfer, O. Schreer, I. Feldmann, P. Kauff
Fraunhofer Heinrich Hertz Institute, Germany
ABSTRACT
Thanks to the availability of new head mounted displays for Virtual Reality, such as Oculus Rift and HTC Vive, the creation of fully immersive environments has gained a tremendous push. In addition, new Augmented Reality glasses and mobile devices reach the market that allow for novel mixed reality experiences. In addition, mobile devices are now are capable to register their environment and put CGI objects at fixed positions in the viewing space. There are many application in different fields for these technologies, such as gaming, entertainment, virtual prototyping, planning, architecture, construction, chemistry, environmental studies, energy, e-learning and edutainment. In addition, there is also a high potential for the cultural sector. For many of these applications a realistic and lively representation of human beings is desired. This can be achieved with a new technology called Volumetric Video. This technology allows to capture real persons with a specially designed system and to create naturally moving dynamic 3D models, which can then be observed from arbitrary viewpoints in a virtual or augmented real scene. 
The capture system developed by Fraunhofer HHI consists of an integrated multi-camera and lighting system for full 360 degree acquisition. The cylindrical capture studio uses 32 high resolution 20MPixel cameras and a large number (120-200) of LED panels that allow for arbitrary lit background. Hence, diffuse lighting from any direction and automatic keying is supported. The avoidance of green screen and provision of diffuse lighting offers best possible conditions for re-lighting of the dynamic 3D models afterwards at design stage of the VR experience. The 32 cameras are arranged in 16 stereo pairs, which are equally distributed in the cylinder. The stereo pairs create 3D information from all different viewpoints, similar as humans perceive with their two eyes. A 3D point cloud fusion algorithm creates a common and consistent dynamic 3D point cloud that is further transformed to a 3D mesh.
After the completion of the first prototype at Fraunhofer HHI in October 2017, a second, professional studio has been built up in Potsdam Babelsberg and is operated by Volucap GmbH, a joint venture between Studio Babelsberg, Arri, UFA, Interlake and Fraunhofer HHI.
This paper describes the construction of the volumetric studio, the capture and processing techniques and presents a number of production carried out for different field of application.