Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

ECCV 2026

Kwonyoung Ryu1,4 In-Jae Lee4 Jonghyun Jin2 Hyunjee Lee2 Jongmin Lee3 Jaesik Park4
1POSTECH 2POSCO DX 3Chung-Ang University 4Seoul National University

TL;DR: A large view synthesis model can act as a renderer for panoptic signals,
propagating labels to novel viewpoints without explicit 3D reconstruction.

Overview

Large view synthesis models synthesize novel views through cross-view attention without explicit 3D representations, and recent studies have shown that they learn accurate spatial correspondence from RGB supervision alone. We observe that this correspondence generalizes beyond appearance.

PanopticLVSM introduction animation 1.
When non-photorealistic signals such as binary encoded panoptic labels are passed through the model, they are propagated to novel views with consistent spatial structure.

We present the first work to extend large view synthesis models beyond appearance rendering to 3D scene understanding. Given panoptic labels on the input views, we encode them into binary channel representations and pass them through the same model to render target-view segmentation.

PanopticLVSM introduction animation 2.
Given multi-view panoptic labels, the same frozen rendering path propagates binary label encodings to a novel viewpoint and decodes them into target-view segmentation.

LVSM as a Signal Renderer

We show that a large view synthesis model can render view-independent non-RGB signals, not only RGB appearance.

Reconstruction-free Panoptic NVS

We propagate panoptic labels to novel views without NeRF, 3D Gaussians, or segmentation-specific training of the renderer.

Modular Source Segmenters

We decouple source-view segmentation from target-view propagation, allowing different segmenters to supply the input labels.

Method

Our method operates in two parallel paths through the same large view synthesis model. Given unposed input views, the rendering path synthesizes the target-view image, while the segmentation path encodes the panoptic labels as binary channel encodings and passes them through the same novel view synthesis model to produce target-view segmentation.

Overall pipeline for RGB rendering and panoptic propagation.
The view synthesis model is trained only for novel-view RGB synthesis, not for semantic or panoptic segmentation.
  1. Source view segmentation. Predict panoptic labels from unposed source images with a source-view segmenter.
  2. Encode labels. Convert discrete panoptic maps into binary channels compatible with RGB rendering.
  3. Reuse correspondence. Propagate label channels through the frozen scene encoder and render decoder.
  4. Decode panoptic views. Decode rendered channels into target-view semantic and instance labels.

Results

Novel View Panoptic Segmentation

Panoptic propagation preserves rendering quality while achieving competitive segmentation with a frozen novel view synthesis model. Under sparse geometric overlap, the frozen NVS transformer retains a learned extrapolation capability.

Input Image 1 Input Image 2 Rendered RGB
Input Semantic 1 Input Semantic 2 Rendered Semantic
Input Image 1 Input Image 2 Rendered RGB
Input Semantic 1 Input Semantic 2 Rendered Semantic
Qualitative novel-view panoptic segmentation results.
Four low-overlap qualitative comparisons from the supplement.
Rendered ScanNet comparison table.
ScanNet benchmark: evaluates RGB novel-view synthesis and panoptic segmentation on standard-overlap view pairs.
Quantitative evaluation under different overlap ranges.
Overlap range evaluation: tests robustness when source and target views share limited geometric overlap.

Modular Composition

Our decoupled design lets us replace the source-view segmenter with an off-the-shelf predictor without retraining the rest of the system. The propagation stage can take labels from different source-view segmenters without retraining.

Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
Input Image 1 Input Image 2 Rendered RGB
Input Mask 1 Input Mask 2 Rendered Mask
SAM2 mask propagation qualitative result.
Rendered modular composition table.
Modular composition: evaluates whether the frozen propagation path works with different source-view segmenters and transfers across datasets.

References

  1. Jin et al., LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias, ICLR 2025.
  2. Wang et al., The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images without Any 3D Knowledge, ICLR 2026.
  3. Fan et al., Large Spatial Model: End-to-end Unposed Images to Semantic 3D, NeurIPS 2024.
  4. Xu et al., SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment, NeurIPS 2025.
  5. Cheng et al., Masked-attention Mask Transformer for Universal Image Segmentation, CVPR 2022.
  6. Ravi et al., SAM 2: Segment Anything in Images and Videos, ICLR 2025.
  7. Zust et al., PanSt3R: Multi-view Consistent Panoptic Segmentation, ICCV 2025.

BibTeX

@inproceedings{ryu2026extending,
  title     = {Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation},
  author    = {Ryu, Kwonyoung and Lee, In-Jae and Jin, Jonghyun and Lee, Hyunjee and Lee, Jongmin and Park, Jaesik},
  booktitle = {European Conference on Computer Vision (ECCV)},
  year      = {2026}
}

Acknowledgement

This work was supported by IITP grant (RS-2021-II211343: AI Graduate School Program at Seoul National Univ., RS-2025-25442338: AI star Fellowship Support Program at Seoul National Univ., and RS-2026-25517417: Development of Compression, Reconstruction, and Rendering Technologies for Free-viewpoint Media). POSCO DX Company, Ltd. provided generous support for this research.