3D Gaussian Inpainting with Depth-Guided Cross-view Consistency

Sheng-Yu Huang1, Zi-Ting Chou2, Yu-Chiang Frank Wang1,2
1 Graduate Institute of Communication Engineering, National Taiwan University   2 Nvidia
CVPR 2025
PDF Code(Coming Soon) Arxiv

Overview

Given a 3D Gaussian Splatting model G1:N pretrained on multi-view images I1:K at camera poses ξ1:K, our goal is to perform 3D inpainting based on the object masks M1:K (e.g., provided by SAM). With the rendered depth maps D1:K, the stage of Inferring Depth-Guide Inpainting Mask is able to refine the inpainting masks to preserve visible backgrounds across camera views. The stage of Inpainting-guided 3DGS Refinement then utilizes such masks to joint- ly update the new Gaussians G1:N′′ for both novel-view rendering and inpainting purposes.

Inferring Depth-Guided Inpainting Mask

Taking {I1,M1} at view ξ1 as an example reference view, the original background region I1B can be first produced. We then project the background region I2B from ξ2 to ξ1, updating I′1B and the associated inpainting mask M1′. By repeating this process across camera views, the final inpainting mask M1′ contains only the regions that are \textit{not} visible at any training camera views.

Difference of Inpainting Between Using Original Object Mask and Our Mask

Although the original object mask M1 can be used to inpaint the occluded regions, it may also include some background regions. This leads to a inpainted background conflicting existing one. In contrast, our mask M1′ is more accurate and can preserve visible background contents.

Inpainting Results on SPInNeRF Dataset

Ours
Original
Ours
Original
Ours
Original
Ours
Original

Ablation

We verify the effectiveness of our Inferring Depth-Guided Inpainting Mask and Inpainting-guided 3DGS Refinement.

BibTeX

@article{huang20253d,
  title={3D Gaussian Inpainting with Depth-Guided Cross-View Consistency},
  author={Huang, Sheng-Yu and Chou, Zi-Ting and Wang, Yu-Chiang Frank},
  journal={arXiv preprint arXiv:2502.11801},
  year={2025}
}