Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions Article Swipe

MAKE IMPACT

Guoxing Sun , Xin Chen , Yizhang Chen , Anqi Pang , Pei Lin , Yuheng Jiang , Lan Xu , Jingyi Yu , Jingya Wang · YOU? · · 2021 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2108.00362

4D reconstruction of human-object interaction is critical for immersive VR/AR experience and human activity understanding. Recent advances still fail to recover fine geometry and texture results from sparse RGB inputs, especially under challenging human-object interactions scenarios. In this paper, we propose a neural human performance capture and rendering system to generate both high-quality geometry and photo-realistic texture of both human and objects under challenging interaction scenarios in arbitrary novel views, from only sparse RGB streams. To deal with complex occlusions raised by human-object interactions, we adopt a layer-wise scene decoupling strategy and perform volumetric reconstruction and neural rendering of the human and object. Specifically, for geometry reconstruction, we propose an interaction-aware human-object capture scheme that jointly considers the human reconstruction and object reconstruction with their correlations. Occlusion-aware human reconstruction and robust human-aware object tracking are proposed for consistent 4D human-object dynamic reconstruction. For neural texture rendering, we propose a layer-wise human-object rendering scheme, which combines direction-aware neural blending weight learning and spatial-temporal texture completion to provide high-resolution and photo-realistic texture results in the occluded scenarios. Extensive experiments demonstrate the effectiveness of our approach to achieve high-quality geometry and texture reconstruction in free viewpoints for challenging human-object interactions.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2108.00362

PDF

https://arxiv.org/pdf/2108.00362

OA Status

green

Cited By

1

References

18

Related Works

10

OpenAlex ID

https://openalex.org/W3207655919

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W3207655919

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2108.00362

Digital Object Identifier

Title

Neural Free-Viewpoint Performance Rendering under Complex Human-object Interactions

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2021

Year of publication

Publication date

2021-08-01

Full publication date if available

Authors

Guoxing Sun, Xin Chen, Yizhang Chen, Anqi Pang, Pei Lin, Yuheng Jiang, Lan Xu, Jingyi Yu, Jingya Wang

List of authors in order

Landing page

https://arxiv.org/abs/2108.00362

Publisher landing page

PDF URL

https://arxiv.org/pdf/2108.00362

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2108.00362

Direct OA link when available

Concepts

Rendering (computer graphics), Artificial intelligence, Computer science, Computer vision, RGB color model, Object (grammar), Iterative reconstruction, Computer graphics (images)

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2021: 1

Per-year citation counts (last 5 years)

References (count)

18

Number of works referenced by this work

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload