Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images Article Swipe

MAKE IMPACT

Nishant Jain , Suryansh Kumar , Luc Van Gool · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2311.04521

We introduce an improved solution to the neural image-based rendering problem in computer vision. Given a set of images taken from a freely moving camera at train time, the proposed approach could synthesize a realistic image of the scene from a novel viewpoint at test time. The key ideas presented in this paper are (i) Recovering accurate camera parameters via a robust pipeline from unposed day-to-day images is equally crucial in neural novel view synthesis problem; (ii) It is rather more practical to model object's content at different resolutions since dramatic camera motion is highly likely in day-to-day unposed images. To incorporate the key ideas, we leverage the fundamentals of scene rigidity, multi-scale neural scene representation, and single-image depth prediction. Concretely, the proposed approach makes the camera parameters as learnable in a neural fields-based modeling framework. By assuming per view depth prediction is given up to scale, we constrain the relative pose between successive frames. From the relative poses, absolute camera pose estimation is modeled via a graph-neural network-based multiple motion averaging within the multi-scale neural-fields network, leading to a single loss function. Optimizing the introduced loss function provides camera intrinsic, extrinsic, and image rendering from unposed images. We demonstrate, with examples, that for a unified framework to accurately model multiscale neural scene representation from day-to-day acquired unposed multi-view images, it is equally essential to have precise camera-pose estimates within the scene representation framework. Without considering robustness measures in the camera pose estimation pipeline, modeling for multi-scale aliasing artifacts can be counterproductive. We present extensive experiments on several benchmark datasets to demonstrate the suitability of our approach.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2311.04521

PDF

https://arxiv.org/pdf/2311.04521

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4388555580

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4388555580

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2311.04521

Digital Object Identifier

Title

Learning Robust Multi-Scale Representation for Neural Radiance Fields from Unposed Images

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-11-08

Full publication date if available

Authors

Nishant Jain, Suryansh Kumar, Luc Van Gool

List of authors in order

Landing page

https://arxiv.org/abs/2311.04521

Publisher landing page

PDF URL

https://arxiv.org/pdf/2311.04521

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2311.04521

Direct OA link when available

Concepts

Artificial intelligence, Computer science, Computer vision, Rendering (computer graphics), Artificial neural network, Robustness (evolution), Leverage (statistics), View synthesis, Chemistry, Biochemistry, Gene

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload