Free-Form Scene Editor: Enabling Multi-Round Object Manipulation Like in a 3D Engine

Xincheng Shuai¹ · Zhenyuan Qin¹ . Henghui Ding ¹ . Dacheng Tao²

¹Fudan University · ²Nanyang Technological University, Singapore

🎯 Introduction

Recent advances in text-to-image (T2I) diffusion models have significantly improved semantic image editing, yet most methods fall short in performing 3D-aware object manipulation. In this work, we present FFSE, a 3D-aware autoregressive framework designed to enable intuitive, physically-consistent object editing directly on real-world images. Unlike previous approaches that either operate in image space or require slow and error-prone 3D reconstruction, FFSE models editing as a sequence of learned 3D transformations, allowing users to perform arbitrary manipulations, such as translation, scaling, and rotation, while preserving realistic background effects (e.g., shadows, reflections) and maintaining global scene consistency across multiple editing rounds. To support learning of multi-round 3D-aware object manipulation, we introduce 3DObjectEditor, a hybrid dataset constructed from simulated editing sequences across diverse objects and scenes, enabling effective training under multi-round and dynamic conditions. Extensive experiments show that the proposed FFSE significantly outperforms existing methods in both single-round and multi-round 3D-aware editing scenarios.

Figure 1. 3D-aware object manipulation results of our Free-Form Scene Editor (FFSE). 1) Object effects. FFSE can process a variety of 3D operations, including challenging transformations such as rotations. 2) Background effects. FFSE generates realistic environmental interaction resulting from object manipulations, such as shadows and occlusions. 3) Multi-round editing. FFSE maintains consistency of scene elements across multiple editing iterations. Moreover, the proposed FFSE provides a user-friendly interface without time-consuming 3D reconstruction.

Figure 2. Overall framework of Free-Form Scene Editor (FFSE) with dashed boxes indicating introduced learnable modules, where the middle blocks and convolutional layers from the base model are omitted for simplicity. Nd and Nu denote the number of down and up blocks respectively. Two 6-length editing sequences are shown as an example, where only DLsyn is active since the current training batch is sampled from Dsyn. Historical observations and operations are processed by frame encoder and operation encoder, respectively, to capture scene structure changes. The output of frame encoder is added to down block features, while the output from operation encoder is injected into the main branch via operation self-attention. Additionally, standard self-attention modules are enhanced by context self-attention to improve the appearance consistency of the edited object.

📋 TODO List

• Upload inference code and model weights of FFSE (in progress).
• Upload training code of FFSE.
• Upload 3DObjectEditor dataset.

✒️ Citation

If you find our work useful for your research and applications, please kindly cite using this BibTeX:

@inproceedings{SynFMC,
        title={{Free-Form Scene Editor}: Enabling Multi-Round Object Manipulation Like in a 3D Engine},
        author={Shuai, Xincheng and Qin, Zhenyuan and Ding, Henghui and Tao, Dacheng},
        booktitle={AAAI},
        year={2026}
      }