PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF Article Swipe

MAKE IMPACT

Mohsen Gholami , Rabab Ward , Z. Jane Wang · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2312.14915

This paper proposes an end-to-end framework for generating 3D human pose datasets using Neural Radiance Fields (NeRF). Public datasets generally have limited diversity in terms of human poses and camera viewpoints, largely due to the resource-intensive nature of collecting 3D human pose data. As a result, pose estimators trained on public datasets significantly underperform when applied to unseen out-of-distribution samples. Previous works proposed augmenting public datasets by generating 2D-3D pose pairs or rendering a large amount of random data. Such approaches either overlook image rendering or result in suboptimal datasets for pre-trained models. Here we propose PoseGen, which learns to generate a dataset (human 3D poses and images) with a feedback loss from a given pre-trained pose estimator. In contrast to prior art, our generated data is optimized to improve the robustness of the pre-trained model. The objective of PoseGen is to learn a distribution of data that maximizes the prediction error of a given pre-trained model. As the learned data distribution contains OOD samples of the pre-trained model, sampling data from such a distribution for further fine-tuning a pre-trained model improves the generalizability of the model. This is the first work that proposes NeRFs for 3D human data generation. NeRFs are data-driven and do not require 3D scans of humans. Therefore, using NeRF for data generation is a new direction for convenient user-specific data generation. Our extensive experiments show that the proposed PoseGen improves two baseline models (SPIN and HybrIK) on four datasets with an average 6% relative improvement.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2312.14915

PDF

https://arxiv.org/pdf/2312.14915

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4390215536

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4390215536

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2312.14915

Digital Object Identifier

Title

PoseGen: Learning to Generate 3D Human Pose Dataset with NeRF

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-12-22

Full publication date if available

Authors

Mohsen Gholami, Rabab Ward, Z. Jane Wang

List of authors in order

Landing page

https://arxiv.org/abs/2312.14915

Publisher landing page

PDF URL

https://arxiv.org/pdf/2312.14915

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2312.14915

Direct OA link when available

Concepts

Computer science, Artificial intelligence, Estimator, Rendering (computer graphics), Pose, Robustness (evolution), Generalizability theory, Machine learning, Interpretability, Pattern recognition (psychology), Computer vision, Mathematics, Statistics, Chemistry, Biochemistry, Gene

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload