Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion

In the realm of 2D real-world data simulation, two predominant approaches have emerged: domain randomization and domain adaptation. The first line of works [60, 63, 61] randomly perturbs textures, lighting, and material of synthetic data, thereby modeling the synthetic-real domain gap as one kind of variation to endow models with robustness to real-world data. However, it brought the challenge of achieving a delicate balance between ensuring that the distribution of augmentations adequately covers that of real-world data and preventing the augmentation from being too extensive to hinder effective model training [24, 75, 39]. Domain adaptation, on the other hand, aims to minimize the domain gap by transferring knowledge from synthetic data to real-world scenarios, leveraging techniques such as adversarial training and self-supervised learning [52, 5, 74].

Acquiring real-world 3D data and their annotation is labor intensive and inefficient, deriving methods for the simulation of 3D data. Existing simulation methods primarily focus on enhancing the realism of depth information by grounding physical priors in simulators to close the domain gap with 3D data from the physical world [34, 30, 42, 78, 2]. However, these approaches are usually limited by their reliance on predefined and explicit physical modeling. Alternatively, data-driven methods turn to implicitly learn from the distribution in real-captured data, independent of physical modeling [4, 59, 19, 50], offering enhanced adaptability and generalization capabilities to various real-world scenarios. Nonetheless, due to the absence of extensive synthetic-real paired datasets and the reliance on traditional GANs [18], which suffer from training instability and mode collapse [29], these approaches have not achieved the expected performance. Consequently, the research progress of data-driven 3D Sim2Real techniques faces a recent stagnation.

We formulate the 3D Sim2Real task as learning the mapping from synthetic to real-world captured noise, based on depth images. Existing approaches in the domain of image generation and translation include a variety of generative models such as Generative Adversarial Networks (GANs) [18], Variational Autoencoders (VAEs) [28], Cycle-Consistent Adversarial Networks (Cycle-GAN) [80], and Pix2Pix [22]. While GAN-based methods, which have been predominant in image generation before diffusion models, achieved significant milestones in Sim2Real tasks [68, 74, 6], they often struggle with maintaining local details and semantic consistency, leading to sub-optimal performance [29, 13, 33, 38, 41].

Diffusion models, also known as denoising diffusion probabilistic models (DDPMs) [53], have emerged as a powerful alternative to GANs for image synthesis [13, 35, 37, 33, 77, 46, 47]. Unlike GANs, diffusion models offer a more stable training procedure and have been shown to generate images with higher fidelity and better preservation of fine details [21, 45]. Depth diffusion models recently track great attention. They mainly focus on the task of depth estimation in a monocular [48, 14, 49, 62, 26, 67] or multi-view [70, 27] manner. Other works also cover conventional image translation tasks, such as depth map super-resolution [51] and refinement [79], which produce clean and certain output images. Differently, our work leverages generative models to “generate noise”, where the expected output is highly uncertain. Thus, the distribution that needs to be learned is more complex, bringing fundamentally different challenges.

Given an initial data $x_{0}$, we gradually add random noise $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ to $x_{0}$, yielding a series of noisy data, which can be simply formulated as $x_{t+1}=x_{t}+\bm{\epsilon}_{t}$. After adding noise for multiple steps, the initial distribution of $p(x)$ is expected to be transformed into a standard Gaussian distribution.

According to DDPM [20], this process can be directly achieved by the following forward diffusion process:

$$q(x_{t})=\sqrt{\alpha_{t}}{x_{0}}+\sqrt{1-\alpha_{t}}\bm{\epsilon},x_{0}\sim p(x),t\in\{0,1,...,T\},$$

(1)

where $T$ denotes the number of the time step, t is the current time step, and $\alpha_{t}$ regulates the noise schedule deciding the pace at which the initial data distribution transforms into a standard Gaussian distribution.

Given a noisy $x_{t}$ from a standard Gaussian distribution as the initial state, ${x}_{t}$ is denoised to a cleaner $x_{t-1}$ at each time step, which can also be simply formulated as $x_{t-1}=x_{t}-\mu_{\theta}^{\bm{\epsilon}}(x_{t},t)+\bm{\epsilon}_{t}$ in which $\bm{\epsilon}_{t}\sim\mathcal{N}(\mathbf{0},\mathbf{\sigma_{t}I})$ and $\mu_{\theta}^{\bm{\epsilon}}$ is a learnable network to predict the injected noise. By repeating this reverse process until the maximum number of steps $T$, we can obtain the final generated data.

In DDPM [20], the backward diffusion process is proven to be:

$\displaystyle x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}x_{t}-\frac{1-\alpha_{t}}{\sqrt{\alpha_{t}(1-\Pi_{\tau=0}^{t}\alpha_{\tau})}}\mu_{\theta}^{\bm{\epsilon}}(x_{t},t;)+\sigma_{t}\bm{\epsilon},$

(2)

where $\sigma_{t}$ denotes the standard deviation of $q(x_{t-1}|x_{t},x_{0})$ in the backward diffusion to sample random noise. The loss for training $\mu_{\theta}^{\epsilon}$ is:

$$L_{\theta}=\mathbb{E}_{\bm{\epsilon},t}\left\|\bm{\epsilon}-{\mu_{\theta}}^{\bm{\epsilon}}(x_{t},t)\right\|^{2}$$

(3)

Different from unconditional diffusion models, text-to-image diffusion models focus on generating images with optional text prompts. A representative is Stable Diffusion (SD) [45]. SD trains a denoising U-Net architecture $\mu_{\theta}(x_{t},t,c)$ on the latent space of a pre-trained Variational Auto-Encoder (VAE). Here, $c$ represents the supplementary text prompt embedding, usually acquired through methods like CLIP [44].

We finetune SD model on synthetic-real paired depth data from LASA [31]. Specifically, given an initial synthetic depth map $\bm{S}$ and its paired real-captured depth map $\bm{R}$ from a pre-defined dataset, we first get the initial ground truth residual (i.e., difference) depth map $\bm{D_{res}^{gt}}=\bm{S}-\bm{R}$. Then we follow ControlNet [76] to add an additional encoder $En$ and decoder ${\bm{\mu}_{\theta}}^{\bm{\epsilon}}$ for finetuning SD. Specifically, we run the forward process to add noise on $\bm{D_{res}^{gt}}$ following Eq. 1 and encode with $En$ to obtain $x_{t}$. Next, we send $\bm{S}$ to a pretrained VAE encoder $En$, generating the latent condition signal. This latent signal $En(\bm{S})$ is further processed by an additional encoder $\bm{f_{\phi}}$ and sent to the decoder blocks of denoising U-Net. We train both the decoder blocks ${\bm{\mu}_{\theta}}^{\bm{\epsilon}}$ and encoder $\bm{f_{\phi}}$ using the following loss (based on Eq. 3):

$$L_{\theta,\phi}=\mathbb{E}_{\bm{\epsilon},\bm{S},t}\left\|\bm{\epsilon}-{\bm{\mu}_{\theta}}^{\bm{\epsilon}}(\bm{x}_{t},t,\bm{f_{\phi}}(En(\bm{S})))\right\|^{2}$$

(4)

Empirically, this residual represents the noise added to clean (i.e., synthetic) depth during real-world scanning. Moreover, compared to directly generating noisy real depth, adding noise to CAD depth, which is inherently clean and view-consistent, can produce more stable depth with smaller view variation and better preservation of the original geometry. From experimental results, learning residual indeed performs better (see Tab. 5).

First, assume that real depth is a sample from a Gaussian distribution, denoted by $\bm{R}\sim\mathcal{N}(\mathbf{\alpha_{R}},\mathbf{\vartheta})$, where $\alpha$ denotes the mean and $\vartheta$ represents the variance. Notably, $\vartheta$ can naturally represent real-captured noise. Therefore, synthetic depth can be denoted by $\bm{S}\sim\mathcal{N}(\mathbf{\alpha_{S}},0)$, where the variance is zero as there is no real-captured noise.

In our design, as the diffusion model is inherently designed to learn the transformation from a random noise $\mathcal{N}(\mathbf{0},\mathbf{I})$ to any other Gaussian distribution [56], we leverage it to model the distribution of the depth residual that can be denoted by $\mathcal{N}(\mathbf{\alpha_{R}-\alpha_{S}},\mathbf{\vartheta})$. Last, by adding the depth residual to synthetic depth, the output distribution can be formulated as: $\mathcal{N}(\mathbf{\alpha_{R}-\alpha_{S}},\mathbf{\vartheta})+\mathcal{N}(\mathbf{\alpha_{S}},0)=\mathcal{N}(\mathbf{\alpha_{R}},\mathbf{\vartheta})$, which is exactly identical to the previously assumed real-captured noise distribution.

Based on our observations, the noise captured in real data is spatially-variant, meaning different areas within a real depth map may display varying levels of perturbation. However, our Stage-I diffusion tends to introduce spatially-uniform noise across the entire input CAD depth. This discrepancy may stem from a bias within SD, as also discussed in [9]. As a result, although certain regions of the generated depth map align well with real perturbation patterns, there remain noticeable areas that deviate from reality. This leads to evident 3D geometric discrepancies between Stage-I outputs and real captures (as illustrated in Fig. 6).

Our design logic to discriminate unsatisfactory areas is that if the generated areas diverge significantly from real patterns, there should exist a distribution boundary of their 3D geometries. To this end, we propose employing a 3D binary classifier to discriminate this boundary. Initially, we split all generated and real-captured depth maps into different patches representing different areas, which are then projected to form 3D point patches. Subsequently, we leverage PointNet [43] as a 3D binary classifier to differentiate whether the input point patches come from model generation or real captures. The CAD depth maps are also concatenated in the input as additional guidance. The successful classification of a generated patch indicates that its corresponding area is distinct from real captures. Notably, the binary classifier is solely used during training.

After identifying these local patches, we introduce a second-stage depth refinement model that differs from the Stage-I diffusion in two crucial aspects: 1) The diffusion model is now conditioned on the concatenation of synthetic depth $\bm{S}$ and the previously generated depth residual $\bm{D}^{I}$ from Stage I. Here, $\bm{D}^{I}=\bm{D}_{res}^{I}+\bm{S}$, where $\bm{D}_{res}^{I}$ is the output residual depth from Stage-I model. Thus, the training loss defined in Eq. 4 is updated to:

$$L_{\chi,\psi}=\mathbb{E}_{\bm{\epsilon},\bm{D}^{I},\bm{S},t}\left\|\bm{\epsilon}-{\bm{\mu}_{\chi}}^{\bm{\epsilon}}(\bm{x}_{t},t,\bm{f_{\psi}}(En(\bm{D}_{res}^{I},\bm{S})))\right\|^{2},$$

(5)

2) More importantly, the diffusion loss is re-weighted to prioritize the enhancement of the identified unsatisfactory regions, making the final Stage-II loss $L_{\chi,\psi}$ to be:

$$L_{\chi,\psi}=\omega L_{\chi,\psi}^{sim}+\lambda L_{\chi,\psi}^{dis},$$

(6)

where $L_{\chi,\psi}^{sim}$ indicates the loss on similar regions, while $L_{\chi,\psi}^{dis}$ denotes the loss on previously identified distinct (i.e., unsatisfactory) areas. $\omega$ and $\lambda$ are weight coefficients, in which $\lambda>\omega$. In our experiment, setting $\omega=0.5$ and $\lambda=1.5$ produces the best result.

Our Stage-II diffusion model is specialized on learning the distribution of unsatisfactory areas in Stage-I generation. Meanwhile, by conditioning on Stage-I output and assigning less weight to satisfied areas, the generation on satisfied outputs is also preserved in Stage-II, as in Fig. 6.

In the initial stage, we sample a random distribution $\bm{\epsilon}$ from $\mathcal{N}(\mathbf{0},\mathbf{I})$ and feed it into the denoising U-Net $\bm{\mu}_{\theta}$ of Stage-I diffusion model. The model runs a backward diffusion process for $T$ steps to generate the initial depth residual $\bm{D}_{res}^{I}$, following Eq. 2 with the synthetic depth $\bm{S}$ serving as the condition. Now we can obtain the initial depth map $\bm{D}^{I}=\bm{D}_{res}^{I}+\bm{S}$.

Moving to the second stage, we repeat a similar backward diffusion process to generate the residual depth $\bm{D}_{res}^{II}$, using the denoising U-Net $\bm{\mu}_{\chi}$ of our Stage-II diffusion model, conditioned on both synthetic depth $\bm{S}$ and Stage-I output $\bm{D}^{I}$. The final depth map $\bm{D}^{II}=\bm{D}_{res}^{II}+\bm{S}$. DDIM sampler [54] is utilized during the whole inference phase.

Given the 3D synthetic object, we follow LASA [31] to embed CAD objects into 3D scenes. This allows us to render synthetic depth images, simulating the occlusions in real-world captures. Then we conduct the aforementioned two-stage real depth simulation on rendered synthetic depth maps, followed by a depth fusion algorithm to create simulated 3D data.

We train our model by finetuning the official StableDiffusion V2.1 [45]. The training dataset is a recent 3D synthetic-real paired dataset, LASA [31]. LASA is a large-scale dataset that contains 10,412 unique CAD models across 17 categories. Skilled artists are employed to meticulously craft aligned CAD models from 3D object real-world scans of [3]. LASA provides CAD-real depth map pairs, rendered based on camera information during the real-world collection. See the supplementary material for more implementation details.

Since our method operates on depth images, we select several image-to-image generation methods for comparison: 1) Cycle-Consistent Adversarial Networks (Cycle-GAN) [80], which is a representative generative network when paired training data are not available, so we train Cycle-GAN on the pair-shuffled LASA training data. 2) Pix2Pix [22], which is a classical image generation method. 3) Giga-GAN [25], a state-of-the art GAN-based method. 4) Stable Diffusion [45], our main baseline model. Both 2), 3), 4) are trained using the paired data from LASA.

3D instance reconstruction aims to recover the full geometry from real-captured partial point clouds or multi-view images. We utilize a recent state-of-the-art diffusion-based reconstruction network, DisCo, which is proposed in LASA [31]. It is originally pretrained on three synthetic datasets (i.e., ShapeNet [8], ABO [10], and 3D-Future [17]) and finetuned on LASA real training data. During pretraining, DisCo simply constructs partial point clouds by adding random perturbation to synthetic data. Instead, in our experiment, to simulate partial point clouds for pretraining, we perform simulation using our method and other baseline models based on the original three synthetic datasets used to pretrain DisCo, and replace half of the original partial point clouds with the simulated 3D data. The other configurations all follow the original DisCo.

Subsequently, we employ few-shot and zero-shot learning configurations to evaluate: We randomly sample only a few LASA real training data to finetune the pretrained network. In this way, the real-world knowledge of the network will mostly come from the pretraining data (i.e., the simulated data produced by our method and others) instead of the finetuning data, which can better reflect the performance gain from the simulated data itself. The pretrained model is finetuned with 20$\sim$30% of the original training samples for few-shot learning and is evaluated on the original LASA validation set. For zero-shot learning, we directly evaluate the pretrained network without fine-tuning.

Why not perform simulation on LASA synthetic data and compare the performance using the simulated data with that using LASA paired real GT? Since our StableSim2Real and other baseline techniques are trained on the LASA training set, it is unfeasible to conduct further inference on LASA, as this would not verify the actual model performance but only reflect the performance caused by overfitting. We can also not conduct simulation on a third-party paired dataset similar to LASA for 3D reconstruction, as there is currently no such paired dataset.

Following [31], we utilize mean Intersection over Union (mIoU), L2 chamfer distance, and 1% F-score metrics. Initially, we normalize both the results and the ground-truth CAD objects to fall within the range of -0.5 to 0.5, according to which we calculate the aforementioned metrics between them. As shown in Tab. 1, under all few-shot settings, the model pretrained using the simulated data from our method achieves the best performance, and significantly surpasses the model using no simulated data (i.e., only add random perturbation to synthetic data) for pretraining. Moreover, using the simulated data produced by Cycle-GAN or Pix2Pix does not improve the model performance compared to that using no simulated data, possibly attributed to their inferior simulation quality, introducing ambiguity and complexity during pretraining, making it hard for the model to learn useful real-world knowledge. Moreover, as listed in Tab. 2, under zero-shot setting, when solely using simulated data for pretraining without any finetuning, the model trained with our simulated data demonstrates more conspicuous effectiveness.

Fig. 3 shows the qualitative comparison, where our simulated data can help the model to gain the best reconstruction accuracy and diversity. More qualitative results are provided in the supplementary material.

We select OcCo [66] as our backbone, which is a self-supervised point cloud pretraining framework. OcCo is originally pretrained on ModelNet [71], and finetuned on real-scanned datasets – ScanObjectNN [64] for object classification, and S3DIS [1] for scene semantic segmentation. ModelNet contains 12,311 synthetic CAD objects from 40 classes. ScanObjectNN includes $\sim$15,000 real-scanned objects across 15 categories. S3DIS encompasses 271 indoor rooms and 13 classes. The classification task outputs the class of the whole 3D object, and the semantic segmentation predicts the category of each point in the 3D scene.

In our experiment, we apply our method and other baselines on ModelNet to generate simulated data that are further used to replace half of the original ModelNet synthetic data to pretrain OcCo. Next, the pretrained model is again evaluated under the few-shot learning setting. A typical setting is “$K$-way $N$-shot”, which indicates that $K$ classes are randomly selected during finetuning, and each category contains $N$ samples. The finetuned model is evaluated on the unseen samples picked from each of the $K$ classes. We select DGCNN [69] as the backbone for OcCo. For few-shot learning, we employ the “5-way, 10-shot” setting. For S3DIS, we follow the original OcCo to report the results using 6-fold cross-validation. All the other configurations follow the original OcCo.

We report the classification accuracy and segmentation mean Intersection over Union (mIoU) in Tab. 3. The model pretrained using our simulated data gets the best results across both 3D object classification and 3D scene understanding tasks. By pretraining with our simulated data, the model can acquire valuable real-world knowledge that helps downstream tasks. In contrast, pretraining with the low-quality simulated data from Cycle-GAN or Pix2Pix may even degrade the performance.

To evaluate the impact of each key design in our framework, we conduct ablation studies on the 3D instance reconstruction task, specifically on the table category. Similar to Tab. 2, we conduct experiments under zero-shot protocol. This allows for a focused evaluation of the module’s impact without the influence of additional factors. Other setups remain consistent with Sec. 4.2.

1) Residual generation. In Tab. 5 (w/o res.), directly generating the real-world depth map without using residual generation degrades the model performance, verifying the efficacy of our residual generation.

2) Loss re-weighting. In our Stage-II model, we re-weight the diffusion loss to focus more on unsatisfactory areas. In Tab. 5 (w/o re-w.), the result w/o loss re-weighting is provided, showing its effectiveness.

3) 3D binary discriminator. Instead of training a 3D binary discriminator, another simple way is to directly set a threshold of the absolute difference between Stage-I generation and real-world GT to decide distinct areas. We tried various difference thresholds and reported the best result in Tab. 5 (w/o bin.), which hurts the model performance. To explain, our output depth is gained by adding subtle noise to synthetic depth. Thus, the magnitude of the difference between our output and its corresponding real GT is similar at 2D/pixel space, where it is impractical to find an appropriate threshold to identify distinct/unsatisfactory areas. Instead, their difference in distribution space is relatively more significant, so using a 3D binary classifier to delineate their distribution boundary is a more feasible way.

4) Number of stages. Theoretically, we can iteratively improve the generation quality by adding more stages. However, as shown in Tab. 5, adding Stage-III model does not improve the model performance, and Stage-IV model even slightly degrades the result. To analyze, Stage-II model has been sufficient to identify most of the unsatisfactory areas and improve their generation quality. In addition, adding redundant stages may also cause overfitting issues that hurt the performance.

1) We compare simulation results with real-captured Ground Truth (GT) at the patch level. As illustrated in Fig. 6, our Stage-II model effectively refines previously unsatisfactory (i.e., dissimilar to GT) patches from Stage-I output to be similar to GT, while keeping previously satisfactory (i.e., similar to GT) patches almost unchanged.

2) When sending the newly generated output patches from Stage-II to our 3D binary discriminator, the classification accuracy notably drops from $\mathbf{67.8\%}$ to $\mathbf{32.5\%}$, suggesting that the generated patches successfully approximate the real-captured distributions, making it challenging for the classifier to discriminate. This again verifies the effectiveness of our Stage-II model. See the supplementary material for more ablations of the binary classifier.