Volumetric video holds promise for virtual and augmented reality (VR/AR) applications but faces challenges in interactive scenarios due to high hardware costs, complex processing and substantial data streams. In this paper, we introduce Live4D, a cost-effective, real-time volumetric video generation and streaming system using an RGB-only camera setup. We propose a novel deep implicit surface reconstruction algorithm, that combined neural signed distance field with observed truncated signed distance field to generate the watertight meshes with low latency. Moreover, we achieve a robust non-rigid tracking method that provides temporal stability to the meshes while resisting tracking failure cases. Experimental results show that Live4D achieves a performance of 24fps using mid-range graphic cards and exhibits an end-to-end latency of 95ms. The system enables live streaming of volumetric video within a 20Mbps bandwidth requirement, positioning Live4D as a promising solution for real-time 3D vision content creation in the growing VR/AR industry.
Overview of the Live4D system. We directly take synchronized multi-view raw images as the system input. After ISP and stereo-rectification of calibrated RGB streams, several lightweight networks are employed to generate RGB-D frames, incorporating ROI information. These data are transmitted to the reconstruction server. It first performs TSDF fusion and applies the proposed volumetric completion method to achieve watertight implicit surface reconstruction. Simultaneously, historical frames are integrated and preserved in the key frame, finally fused to current data volume to ensure temporal stability of the signed distance field. Subsequently, mesh is generated through Marching Cubes, and further mesh processing includes geometry refinement and face reduction to improve quality and streaming efficiency. Ultimately, the mesh undergoes texture mapping, encoding and delivery to individual clients for rendering purposes.
@inproceedings{zhou2023live4d,
author = {Zhou, Yifeng and Wang, Shuheng and Li, Wenfa and Zhang, Chao and Rao, Li and Cheng, Pu and Xu, Yi and Ke, Jinle and Feng, Wenduo and Zhou, Wen and Xu, Hao and Gao, Yukang and Ding, Yang and Tang, Weixuan and Jiao, Shaohui},
title = {Live4D: A Real-time Capture System for Streamable Volumetric Video},
pages={1--4},
booktitle = {SIGGRAPH Asia 2023 Technical Communications},
year = {2023},
}