Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

Dynamic Scene Novel View Synthesis via Deferred Spatio-temporal Consistency

Beatrix-Emőke Fülöp-Balogh Eleanor Tursman James Tompkin Julie Digne Nicolas Bonneel 1
1 Origami - Origami
LIRIS - Laboratoire d'InfoRmatique en Image et Systèmes d'information
Abstract : Structure from motion (SfM) enables us to reconstruct a scene via casual capture from cameras at different viewpoints, and novel view synthesis (NVS) allows us to render a captured scene from a new viewpoint. Both are hard with casual capture and dynamic scenes: SfM produces noisy and spatio-temporally sparse reconstructed point clouds, resulting in NVS with spatio-temporally inconsistent effects. We consider SfM and NVS parts together to ease the challenge. First, for SfM, we recover stable camera poses, then we defer the requirement for temporally-consistent points across the scene and reconstruct only a sparse point cloud per timestep that is noisy in space-time. Second, for NVS, we present a variational diffusion formulation on depths and colors that lets us robustly cope with the noise by enforcing spatio-temporal consistency via per-pixel reprojection weights derived from the input views. Together, this deferred approach generates novel views for dynamic scenes without requiring challenging spatio-temporally consistent reconstructions nor training complex models on large datasets. We demonstrate our algorithm on real-world dynamic scenes against classic and more recent learning-based baseline approaches.
Document type :
Preprints, Working Papers, ...
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03374431
Contributor : Nicolas Bonneel Connect in order to contact the contributor
Submitted on : Tuesday, October 12, 2021 - 10:32:00 AM
Last modification on : Friday, October 15, 2021 - 12:35:24 PM

File

2109.01018.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03374431, version 1
  • ARXIV : 2109.01018

Citation

Beatrix-Emőke Fülöp-Balogh, Eleanor Tursman, James Tompkin, Julie Digne, Nicolas Bonneel. Dynamic Scene Novel View Synthesis via Deferred Spatio-temporal Consistency. 2021. ⟨hal-03374431⟩

Share

Metrics

Record views

53

Files downloads

632