Live4D: A Real-time Capture System for Streamable Volumetric Video

Yifeng Zhou*, Shuheng Wang*, Wenfa Li*, Chao Zhang*, Li Rao, Pu Cheng, Yi Xu,
Jinle Ke, Wenduo Feng, Wen Zhou, Hao Xu, Yukang Gao, Yang Ding, Weixuan Tang, Shaohui Jiao

ByteDance, China
SIGGRAPH Asia 2023 Techinical Communications

*Indicates Equal Contribution.

Live4D Show Case

Abstract

Volumetric video holds promise for virtual and augmented reality (VR/AR) applications but faces challenges in interactive scenarios due to high hardware costs, complex processing and substantial data streams. In this paper, we introduce Live4D, a cost-effective, real-time volumetric video generation and streaming system using an RGB-only camera setup. We propose a novel deep implicit surface reconstruction algorithm, that combined neural signed distance field with observed truncated signed distance field to generate the watertight meshes with low latency. Moreover, we achieve a robust non-rigid tracking method that provides temporal stability to the meshes while resisting tracking failure cases. Experimental results show that Live4D achieves a performance of 24fps using mid-range graphic cards and exhibits an end-to-end latency of 95ms. The system enables live streaming of volumetric video within a 20Mbps bandwidth requirement, positioning Live4D as a promising solution for real-time 3D vision content creation in the growing VR/AR industry.

Pipeline

Live4D Show Case

Overview of the Live4D system. We directly take synchronized multi-view raw images as the system input. After ISP and stereo-rectification of calibrated RGB streams, several lightweight networks are employed to generate RGB-D frames, incorporating ROI information. These data are transmitted to the reconstruction server. It first performs TSDF fusion and applies the proposed volumetric completion method to achieve watertight implicit surface reconstruction. Simultaneously, historical frames are integrated and preserved in the key frame, finally fused to current data volume to ensure temporal stability of the signed distance field. Subsequently, mesh is generated through Marching Cubes, and further mesh processing includes geometry refinement and face reduction to improve quality and streaming efficiency. Ultimately, the mesh undergoes texture mapping, encoding and delivery to individual clients for rendering purposes.


Video

Data


BibTeX

@inproceedings{zhou2023live4d,
        author = {Zhou, Yifeng and Wang, Shuheng and Li, Wenfa and Zhang, Chao and Rao, Li and Cheng, Pu and Xu, Yi and Ke, Jinle and Feng, Wenduo and Zhou, Wen and Xu, Hao and Gao, Yukang and Ding, Yang and Tang, Weixuan and Jiao, Shaohui},
        title = {Live4D: A Real-time Capture System for Streamable Volumetric Video},
        pages={1--4},
        booktitle = {SIGGRAPH Asia 2023 Technical Communications},
        year = {2023},
        }