Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

ECCV 2026

Kwonyoung Ryu1,4 In-Jae Lee4 Jonghyun Jin2 Hyunjee Lee2 Jongmin Lee3 Jaesik Park4
1POSTECH 2POSCO DX 3Chung-Ang University 4Seoul National University

TL;DR: We extend a large view synthesis model into a reconstruction-free panoptic renderer
that propagates source-view labels to novel viewpoints.

Overview

Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We use this learned correspondence to propagate non-RGB signals.

PanopticLVSM introduction animation 1.
Binary-encoded panoptic labels follow the model's learned cross-view correspondence and appear consistently in novel views.

We present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation.

PanopticLVSM introduction animation 2.
Given multi-view panoptic labels, the same frozen rendering path propagates binary label encodings to a novel viewpoint and decodes them into target-view segmentation.

LVSM as a Signal Renderer

A large view synthesis model can serve as a renderer for panoptic label channels, not only RGB appearance.

Reconstruction-free Panoptic NVS

We propagate panoptic labels to novel views without NeRF, 3D Gaussians, or segmentation-specific training of the renderer.

Modular Source Segmenters

The source-view segmenter and the target-view renderer remain modular, so different segmenters can provide input labels.

Method

Given unposed source images, our method renders both RGB images and panoptic labels through the same large view synthesis model. We encode source-view panoptic maps as binary channels, propagate them with the frozen rendering path, and decode the rendered channels into target-view semantic and instance labels.

Overall pipeline for RGB rendering and panoptic propagation.
The view synthesis model is trained only for novel-view RGB synthesis, not for semantic or panoptic segmentation.
  1. Source view segmentation. Predict panoptic labels from unposed source images with a source-view segmenter.
  2. Encode labels. Convert discrete panoptic maps into binary channels compatible with RGB rendering.
  3. Reuse correspondence. Propagate label channels through the frozen scene encoder and render decoder.
  4. Decode panoptic views. Decode rendered channels into target-view semantic and instance labels.

Results

Novel View Panoptic Segmentation

Panoptic propagation preserves rendering quality while achieving competitive segmentation with a frozen novel view synthesis model. The learned correspondence remains effective even when source and target views have limited overlap.

Input Image 1 Input Image 2 Rendered RGB
Input Semantic 1 Input Semantic 2 Rendered Semantic
Input Image 1 Input Image 2 Rendered RGB
Input Semantic 1 Input Semantic 2 Rendered Semantic
Qualitative novel-view panoptic segmentation results.
Four low-overlap qualitative comparisons from the supplement.
Rendered ScanNet comparison table.
ScanNet benchmark: evaluates RGB novel-view synthesis and panoptic segmentation on standard-overlap view pairs.
Quantitative evaluation under different overlap ranges.
Overlap range evaluation: tests robustness when source and target views share limited geometric overlap.

Modular Composition

The propagation module does not depend on a specific source-view segmenter. It can use labels from trained panoptic models or off-the-shelf mask predictors without retraining the renderer.

Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
SAM2 mask propagation qualitative result.
Rendered modular composition table.
Modular composition: evaluates whether the frozen propagation path works with different source-view segmenters and transfers across datasets.
Additional result · not in the paper

SAM2 Masks with a Posed View Synthesis Model

We also test the same label-propagation pipeline with a posed LVSM. SAM2 provides source-view masks, and the frozen posed renderer propagates them to the target view without retraining.

Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask

References

  1. Jin et al., LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias, ICLR 2025.
  2. Wang et al., The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images without Any 3D Knowledge, ICLR 2026.
  3. Fan et al., Large Spatial Model: End-to-end Unposed Images to Semantic 3D, NeurIPS 2024.
  4. Xu et al., SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment, NeurIPS 2025.
  5. Cheng et al., Masked-attention Mask Transformer for Universal Image Segmentation, CVPR 2022.
  6. Ravi et al., SAM 2: Segment Anything in Images and Videos, ICLR 2025.
  7. Zust et al., PanSt3R: Multi-view Consistent Panoptic Segmentation, ICCV 2025.

BibTeX

@inproceedings{ryu2026extending,
  title     = {Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation},
  author    = {Ryu, Kwonyoung and Lee, In-Jae and Jin, Jonghyun and Lee, Hyunjee and Lee, Jongmin and Park, Jaesik},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgement

This work was supported by IITP grant (RS-2021-II211343: AI Graduate School Program at Seoul National Univ., RS-2025-25442338: AI star Fellowship Support Program at Seoul National Univ., and RS-2026-25517417: Development of Compression, Reconstruction, and Rendering Technologies for Free-viewpoint Media). POSCO DX Company, Ltd. provided generous support for this research.