Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesis

Abstract

In this paper, we present a novel framework for video-to-4D generation that creates high-quality dynamic 3D content from single video inputs. Direct 4D diffusion modeling is extremely challenging due to costly data construction and the high-dimensional nature of jointly representing 3D shape, appearance, and motion. We address these challenges by introducing a Direct 4DMesh-to-GS Variation Field VAE that directly encodes canonical Gaussian Splats (GS) and their temporal variations from 3D animation data without per-instance fitting, and compresses high-dimensional animations into a compact latent space. Building upon this efficient representation, we train a Gaussian Variation Field diffusion model with temporal-aware Diffusion Transformer conditioned on input videos and canonical GS. Trained on carefully-curated animatable 3D objects from the Objaverse dataset, our model demonstrates superior generation quality compared to existing methods. It also exhibits remarkable generalization to in-the-wild video inputs despite being trained exclusively on synthetic data, paving the way for generating high-quality animated 3D content. Project page: GVFDiffusion.github.io.

Figure 1: Our model is capable of creating high-fidelity 4D objects from in-the-wild video inputs. Best viewed with zoom-in.

1 Introduction

Recent advances in generative models have demonstrated remarkable capabilities across various modalities, including image [30, 31, 90, 45, 23, 55, 21], video [2, 3, 19], and 3D content [66, 95, 81, 93, 68, 24]. While these achievements mark important milestones, they naturally lead to the next frontier: 4D generation, which aims to create dynamic 3D content. The challenge of generating such content—a fundamental aspect of representing our inherently four-dimensional world—remains largely unexplored. This gap is particularly significant given that real-world phenomena inherently combine spatial and temporal dynamics, from the subtle object movements to complex character articulations.

Despite diffusion models [15, 45, 59] having demonstrated strong modeling capabilities in both 2D and 3D domains, training a robust 4D diffusion model for dynamic 3D content generation presents two main technical challenges. First, obtaining a large-scale 4D dataset is time-consuming. A straightforward approach involves fitting individual dynamic Gaussian Splatting (4DGS) representations [76] for each 3D animation sequence, but this solution typically requires tens of minutes per instance, making it computationally expensive and less scalable as the number of instances increases. Second, the higher-dimensional nature of the problem necessitates a large number of parameters (usually exceeding 100K tokens) to represent 3D shape, appearance, and motion simultaneously, making direct modeling with diffusion approaches extremely challenging. These limitations have significantly hindered the development of efficient and high-quality 4D generative models.

Motivated by the effectiveness of diffusion models applied to compact latent spaces in recent 2D and 3D generation works [59, 3, 81, 95, 58], we present a novel framework for 4D generative modeling that comprises a Direct 4DMesh-to-GS Variation Field VAE and a Gaussian Variation Field diffusion model. Our VAE framework encodes the canonical 3D Gaussian Splatting (3DGS) of objects and compresses each Gaussian’s attribute variations (i.e., Gaussian Variation Fields) into a compact latent space from 4D mesh data, thereby bypassing costly per-instance reconstructions. Inspired by previous works [91, 5], we employ a perceiver-style transformer network [27, 26, 72] with displacements of mesh points to effectively encode motion information. To bridge the gap between Gaussian Splatting representation and mesh-based ground truth motion, we introduce a mesh-guided loss that aligns the motion of Gaussian points with the corresponding mesh vertices. Our VAE is trained end-to-end with this mesh-guided loss and an image-level loss, enabling faithful compression of complex Gaussian Variation Fields. This approach reduces high-dimensional motion sequences to a compact 512-dimensional latent space, thus facilitating efficient diffusion modeling for 4D content generation.

Following the construction of our VAE, the 4D generative modeling naturally decomposes into canonical 3DGS generation and Gaussian Variation Field modeling. We leverage state-of-the-art 3D generative models [81] for the canonical component while focusing on modeling the Gaussian Variation Fields. To achieve this, we train a diffusion model to learn the latent space distribution of variation fields conditioned on the input video and canonical 3DGS, enabling controlled 4D content generation. Leveraging the compact nature of our latent space, we employ the Diffusion Transformer (DiT) architecture [52], augmented with temporal self-attention layers to capture smooth temporal dynamics across animations. The video frame features  [48] and the canonical 3DGS are taken as conditions for the diffusion model via cross-attention layers. Additionally, we incorporate positional priors into the diffusion model, enhancing its awareness of correspondences between canonical GS and their variation fields during the denoising process, thereby improving generation quality.

We train our model on a carefully curated diverse collection of animatable 3D objects from the Objaverse [13] and Objaverse-XL [12]. Extensive evaluations demonstrate the superior video-to-4D generation quality of our method compared to existing approaches. Despite being trained on synthetic data, our model exhibits remarkable generalization capabilities when applied to in-the-wild video inputs, effectively creating impressive animations from in-the-wild animation sequences. We believe that our approach represents a notable step toward narrowing the gap between static 3D generation and 4D content creation, paving the way for generating high-quality 4D content.

2 Related Work

3D generation. Early GAN-based 3D generation approaches [77, 100, 14, 7, 17, 64, 98, 80] laid the foundation for 3D content synthesis, while diffusion-based methods [74, 92, 93, 69, 44, 22, 6, 62, 29, 86, 9] advanced generation quality. Recent approaches have focused on latent space generation, either separating geometric modeling and appearance synthesis [97, 71, 91, 95, 37, 78, 99, 58] or jointly modeling both [81, 20, 83, 29, 35, 46, 84, 10]. Alternative methods [67, 53, 65, 8, 40, 75] leverage pretrained 2D models [59] through optimization techniques. Recent works [81, 95] have achieved high-quality 3D asset generation with detailed geometry and appearance, establishing a foundation for 4D content creation.

4D reconstruction. Early 4D reconstruction methods [50, 51, 16, 54, 4] extended neural volumetric techniques for dynamic scenes, while recent Gaussian Splatting-based approaches [76, 42, 11, 36, 25, 32] offer improved efficiency. Typical 4D reconstruction methods often require significant optimization time per instance (e.g., 6 minutes for 4DGaussians [76] and over 30 minutes for K-planes [16]), making them impractical to use as a preliminary step in fitting 4D representations for generation. In this paper, we explore an efficient approach to directly encode 4D mesh data for generative modeling in a single pass.

Figure 2: Framework of 4DMesh-to-GS Variation Field VAE. Our VAE directly encodes 3D animation data into Gaussian Variation Fields within a compact latent space, optimized through image-level reconstruction loss and the proposed mesh-guided loss.

Video-to-4D generation. Early attempts [28, 1, 56, 63] at video-to-4D generation predominantly relied on optimization-based approaches, utilizing pre-trained generative priors [40, 60, 61, 73] as guidance. These methods typically employ score distillation [53] techniques to optimize either neural volumetric representations [43] or 3D Gaussian Splatting [32]. These methods suffer from lengthy optimization times and SDS-related issues [75, 39] such as spatial-temporal inconsistency or poor input alignments. While some works [88, 87] use pseudo-labels for better consistency, recent approaches [57, 82, 49, 85, 94, 38] directly reconstruct 4D content from multiview images or videos. Notably, the introduction of large-scale 4D reconstruction models [57] has significantly reduced the generation time from hours to seconds. However, most of these approaches often struggle to maintain consistent quality across temporal sequences due to inherent multiview inconsistency in 2D generation results.

3 Method

Given an input video sequence $\bm{\mathcal{I}}=\{I_{t}\}_{t=1}^{T}$ of an object, our goal is to generate a sequence of 3DGS models $\bm{\mathcal{G}}=\{G_{t}\}_{t=1}^{T}$ that captures both the shape, appearance, and motion of the object. We decompose this task into canonical GS $G_{1}$ creation (using the first frame as canonical) and Gaussian Variation Fields $\bm{\mathcal{V}}=\{\Delta G_{t}\}_{t=1}^{T}$ generation, where $\bm{\mathcal{V}}$ describes each Gaussian’s attribute variations relative to $G_{1}$ over time. Our framework comprises two main components: (1) a direct 4DMesh-to-GS Variation Field VAE that efficiently encodes 3D animation sequences into a compact latent space, and (2) a Gaussian variation field diffusion model that learns the latent distribution of variation fields conditioned on the input video and canonical GS. The following sections detail each component.

3.1 Direct 4DMesh-to-GS Variation Field VAE

Extending 3DGS to generative modeling of dynamic content presents significant challenges. Fitting individual dynamic 3DGS representations for each animation instance is computationally expensive and scales poorly. Additionally, directly modeling temporal deformation of GS sequences with diffusion models is challenging due to the high dimensionality of both Gaussian quantities (e.g., typically over 100K in [32]) and the time dimension. Therefore, we propose an efficient autoencoding framework that directly encodes 3D animation data into Gaussian Variation Fields with a compact latent space, facilitating subsequent diffusion modeling.

Gaussian Variation Field encoding. Given a sequence of mesh animations $\bm{\mathcal{M}}=\{M_{t}\}_{t=1}^{T}$, we first convert them to point clouds $\bm{\mathcal{P}}=\{P_{t}|P_{t}\in\mathbb{R}^{N\times 3}\}_{t=1}^{T}$ through uniform surface sampling, where each point cloud contains $N$ points. The displacement fields $\{\Delta P_{t}|\Delta P_{t}\in\mathbb{R}^{N\times 3}\}_{t=1}^{T}$ are computed as temporal differences of corresponding points between frames:

$$\Delta P_{t}=P_{t}-P_{1},$$ (1)

where $P_{1}$ is the canonical frame’s point cloud. We then leverage a pretrained mesh-to-GS autoencoder $\mathcal{E}_{GS}$ and $\mathcal{D}_{GS}$ in [81] to obtain the canonical GS representation from canonical mesh $M_{1}$:

$$\begin{split}S_{1}=\mathcal{E}_{GS}(M_{1}),\\ G_{1}=\mathcal{D}_{GS}(S_{1}),\end{split}$$ (2)

where $G_{1}\in\mathbb{R}^{N_{G}\times 14}$ denotes the Gaussian parameters including positions $\bm{p}_{1}$, scales $\bm{s}_{1}$, rotation $\bm{q}_{1}$, colors $\bm{c}_{1}$, and opacity $\alpha_{1}$, with $N_{G}$ being the total number of canonical Gaussians. $S_{1}$ is the structured latent (SLat) representation for canonical GS (more details are included in the supplementary). We finetune $\mathcal{D}_{GS}$ to ensure coherent canonical GS reconstruction with their variation fields, while keeping $\mathcal{E}_{GS}$ frozen to leverage pretrained canonical GS diffusion models.

Inspired by 3DShape2VecSet [91], we employ a cross-attention layer to aggregate motion information from 3D animation sequences into a fixed-length latent representation. While directly using $G_{1}$ as query vectors is a straightforward approach, we find it leads to poor motion awareness. To enhance the network’s sensitivity to mesh deformation, we introduce a mesh-guided interpolation mechanism that generates motion-aware query vectors based on the spatial correspondence between $G_{1}$ and $P_{1}$.

Specifically, for each canonical Gaussian position $\bm{p}_{1}^{i}$, we identify its $K$ nearest neighbors in the canonical point cloud $P_{1}$ and compute their distances $\bm{d}_{i,k}$. To handle varying point densities across the mesh-sampled point cloud, we introduce an adaptive radius $r_{i}$ that adjusts the influence region based on the local point distribution. The interpolation weight $\bm{w}_{i,k}$ and adaptive radius $r_{i}$ are formulated as:

$$\bm{w}_{i,k}=\exp(-\frac{\beta\bm{d}_{i,k}}{r_{i}^{2}}),\quad r_{i}=\sqrt{\frac{1}{K}\sum_{k=1}^{K}\bm{d}_{i,k}},$$ (3)

where $\beta$ is a hyperparameter controlling the decay rate of interpolation weights with distance, with larger values producing more localized influence regions. We set $\beta=7.0$ in this paper.

We then interpolate the displacement fields $\Delta P_{t}$ for the $i$-th Gaussian at time $t$:

$$\Delta\bm{p}^{interp}_{t,i}=\sum_{k=1}^{K}\frac{\bm{w}_{i,k}}{\sum_{k}\bm{w}_{i,k}}\Delta P_{t,n(i,k)}$$ (4)

where $n(i,k)$ denotes the $k$-th nearest neighbor index. We perform farthest point sampling to $\Delta\bm{p}^{interp}_{t}$ based on their canonical positions to formulate our motion-aware query $\Delta\bm{p}^{fps}_{t}\in\mathbb{R}^{L\times 3}$ with reduced sequence length. The point cloud displacement fields $\Delta P_{t}$ serve as keys and values in the cross attention encoder. To preserve spatial relationships, we incorporate positional embedding $PE(\cdot)$ based on the canonical positions:

$$\begin{split}Q_{e}&=f_{disp}(\Delta\bm{p}^{fps}_{t})+PE(G_{1}),\\ K_{e}&=V_{e}=f_{disp}(\Delta P_{t})+PE(P_{1}),\\ \bm{z}&=\text{CrossAttn}(Q_{e},K_{e},V_{e}),\end{split}$$ (5)

where $f_{disp}$ is the displacement embedding layer. This process yields a latent representation $\bm{z}\in\mathbb{R}^{T\times L\times C}$, where $T$ is the number of temporal frames, $L$ is the latent size, and $C$ is the feature dimension. Notably, our encoding procedure compresses the sequence length from $N=8192$ to $L=512$, significantly reducing the subsequent diffusion modeling space.

Figure 3: Architecture of Gaussian Variation Field diffusion model. Our model is built upon diffusion transformer, which takes noised latent as input and gradually denoises it conditioned on the video sequence and canonical GS.

Gaussian Variation Field decoding. The decoding procedure first transforms the latent representation through $n$ layers of self-attention blocks to enable thorough motion information exchange. The decoder then maps this processed latent to a Gaussian Variation Field $\bm{\mathcal{V}}$, defined by the variations of Gaussian attributes $\Delta G_{t}=\{\Delta\bm{p}_{t},\Delta\bm{s}_{t},\Delta\bm{q}_{t},\Delta\bm{c}_{t},\Delta\alpha_{t}\}_{t=1}^{T}$. To ensure the decoder is aware of all canonical Gaussian attributes, we use all parameters of $G_{1}$ to query the latent output through a cross attention layer:

$$\begin{split}Q_{d}=f_{gs}(G_{1})&+PE(G_{1}),\quad K_{d}=V_{d}=\bm{z}_{n},\\ \Delta G_{t}&=\text{CrossAttn}(Q_{d},K_{d},V_{d}),\end{split}$$ (6)

where $f_{gs}$ is the embedding layer for the canonical Gaussians and $\bm{z}_{n}$ is the final self attention layer output. The final 3DGS sequence is obtained by:

$$\bm{\mathcal{G}}=\{G_{t}\}_{t=1}^{T}=\{{G_{1}+\Delta G_{t}}\}_{t=1}^{T}.$$ (7)

Table 1: Quantitative comparison of video-to-4D generation results. Our method demonstrates consistent performance improvements across all metrics while maintaining efficient generation speed. The generation time is measured on a single A100 GPU.

Method	PSNR$\uparrow$	LPIPS$\downarrow$	SSIM$\uparrow$	CLIP$\uparrow$	FVD$\downarrow$	Time$\downarrow$
Consistent4D [28]	16.20	0.146	0.880	0.910	935.19	$\sim$1.5 hr
SC4D [79]	15.93	0.164	0.872	0.870	833.15	$\sim$20 min
STAG4D [88]	16.85	0.144	0.887	0.893	1008.40	$\sim$1 hr
DreamGaussian4D [56]	15.24	0.162	0.868	0.904	799.56	$\sim$15 min
L4GM [57]	17.03	0.128	0.891	0.930	529.10	3.5 s
Ours	18.47	0.114	0.901	0.935	476.83	4.5s

Training objective. Our training objective consists of three main components. First, we employ image-level reconstruction loss between the rendered images $I_{t}^{render}$ from final predicted Gaussians and ground-truth images $I_{t}^{gt}$:

$$\mathcal{L}_{img}=\mathcal{L}_{1}+\lambda_{lpips}\mathcal{L}_{lpips}+\lambda_{ssim}\mathcal{L}_{ssim},$$ (8)

where $\lambda_{lpips},\lambda_{ssim}$ are loss weights for perception loss [96] and SSIM loss, respectively. To ensure faithful motion reconsturction, we introduce a mesh-guided loss that aligns the predicted Gaussian displacements with pseudo ground-truth $\Delta p_{t}^{interp}$ obtained through mesh-guided interpolation:

$$\mathcal{L}_{mg}=\sum_{t=1}^{T}\|\Delta\mathbf{p}_{t}-\Delta\bm{p}_{t}^{interp}\|_{2}^{2},$$ (9)

which we find is crucial for motion reconstruction quality. Finally, to facilitate subsequent diffusion training, we also regularize the latent distribution with a KL divergence loss $\mathcal{L}_{kl}$. The total loss is: $\mathcal{L}_{total}=\mathcal{L}_{img}+\lambda_{mg}\mathcal{L}_{mg}+\lambda_{kl}\mathcal{L}_{kl}$, where $\lambda_{mg},\lambda_{kl}$ are respective loss weights.

3.2 Gaussian Variation Field Diffusion

The diffusion process can be formalized as the inversion of a discrete-time Markov forward process. Let $\bm{z}^{0}\in\mathbb{R}^{T\times L\times C}$ denote our initial latent of Gaussian Variation Field from the distribution $p(\bm{z})$. During the forward phase, we progressively corrupt this latent sequence by adding Gaussian noise over diffusion steps $s\in[0,S]$, following $\bm{z}^{s}:=\alpha_{s}\bm{z}^{0}+\sigma_{s}\bm{\epsilon}$, where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\bm{I})$, and $\alpha_{s},\sigma_{s}$ define the noise schedule. After sufficient diffusion steps, $\bm{z}^{S}$ approaches pure Gaussian noise. Generation is achieved by reversing this process, starting from random Gaussian noise $\bm{z}^{S}\sim\mathcal{N}(\mathbf{0},\bm{I})$ and progressively denoising it to recover $\bm{z}^{0}$.

The compact latent space enables us to build our diffusion model upon the powerful Diffusion Transformer (DiT) architecture [52]. As illustrated in Figure 3, the model takes noise-corrupted latent as input, and processes them through a series of transformer blocks for denoising. Each transformer block incorporates diffusion timestep information through adaptive layer normalization (adaLN) and a gating mechanism. Beyond the standard spatial self attention layers, we introduce dedicated temporal self-attention layers to ensure coherent motion generation across the sequence.

To condition the generation process, we inject two types of features through cross-attention layers: (1) visual features $\bm{\mathcal{C}}^{v}=\{\bm{C}^{v}_{t}\}_{t=1}^{T}$ extracted from input video frames using DINOv2 [48], and (2) geometric features $\bm{\mathcal{C}}^{GS}=G_{1}^{fps}$ fartherest sampled from the static GS. We further incorporate positional embeddings based on canonical GS positions $\bm{p}_{1}^{fps}$ in our diffusion transformer, which strengthens the model’s awareness of correspondences between canonical GS and their variation fields during the denoising process, thereby effectively improving the generation quality.

We parameterize our diffusion model $\hat{\bm{v}}_{\theta}$ to predict the velocity $v^{s}:=\alpha_{s}\bm{\epsilon}-\sigma_{s}\bm{z}^{0}$ at each diffusion step $s$. The diffusion model is trained using:

$$\mathcal{L}_{\text{simple}}=\mathbb{E}_{s,\bm{z}^{0},\bm{\epsilon}}\left[\left\|\hat{\bm{v}}_{\theta}\left(\alpha_{s}\bm{z}^{0}+\sigma_{s}\bm{\epsilon},s,\bm{\mathcal{C}}\right)-\bm{v}^{s}\right\|_{2}^{2}\right],$$ (10)

where $\bm{\mathcal{C}}=\{\bm{\mathcal{C}}^{v},\bm{\mathcal{C}}^{GS}\}$ represents the conditional features of both $\bm{\mathcal{C}}^{v}$ and $\bm{\mathcal{C}}^{GS}$.

Figure 4: Qualitative comparison with previous state-of-the-art methods. Our model directly learns the distribution of Gaussian Variation Fields, enabling high-fidelity 4D generation with coherent temporal dynamics.

3.3 Inference Pipeline

During inference, our framework operates in a sequential pipeline. First, we obtain the canonical GS $G_{1}$ for the first frame using a pretrained 3D diffusion model [81]. Given an input video sequence $\{I_{t}\}_{t=1}^{T}$, we extract visual features and combine them with the farthest sampled canonical Gaussians as conditioning signals for our diffusion model. The diffusion model generates latent codes $\bm{z}$, which are subsequently decoded to obtain the Gaussian Variation Field $\bm{\mathcal{V}}$. The final animated Gaussian representation $G_{t}$ for each frame is obtained by applying these variations to the canonical Gaussians, effectively creating high-fidelity temporally coherent 4D animations.

4 Experiments

4.1 Dataset and Metrics

We conduct our experiments on Objaverse-V1 and Objaverse-XL [13], following previous work in 4D content generation. After filtering for objects with high-quality animations, we utilize 34K objects for training. To evaluate the video-to-4D generation quality, we construct a comprehensive test set of 100 objects by combining 7 instances from the widely-used Consistent4D [28] testset with 93 additional test instances from Objaverse-XL, ensuring a thorough evaluation with previous works. We render 4 novel views of each timestep for each instance. We assess the generation quality using multiple metrics: PSNR, LPIPS [96], and SSIM for frame-wise quality, and FVD [70] for temporal consistency of the generated sequences. All evaluations are performed on renderings at $512\times 512$ resolution.

4.2 Implementation Details

For our VAE implementation, the canonical Mesh-to-GS autoencoder is builds upon Trellis [81], with training conducted in two stages: finetuning the sparse GS decoder $\mathcal{D}_{GS}$ on canonical 3D only for 150K iterations, followed by joint training with other modules for 200K iterations on 4D animation data. The VAE architecture employs point cloud size $N=8192$, latent size $L=512$, and feature dimension $C=16$. We optimize the VAE using AdamW with a learning rate $5e-6$ and $5e-5$ for $\mathcal{D}_{GS}$ and other modules respectively, using batch size 32. The diffusion model is trained on 24-frame sequences using AdamW optimizer with identical learning rate and batch size over 1300K iterations. We apply the cosine noise schedule [45] with 1000 timesteps for training the diffusion model. We set $T=24$ for training, and $T=32$ during inference to compare with prior works. For more implementation details, please refer to the supplementary materials.

Figure 5: More generation results of our model including in-the-wild videos (left) and videos from test set (right).

4.3 Main Results

Quantitative comparisons. We compare the video-to-4D generation results of our model with previous state-of-the-art methods including both optimization-based approaches [28, 88, 79, 56] and feedforward approach [57]. As shown in Table 1, our method consistently outperforms existing approaches across all quality metrics, demonstrating both superior reconstruction fidelity and better temporal coherence. Unlike some prior works [28, 88, 79, 56] require minutes to hours of optimization, our approach is also more efficient, taking 4.5 seconds to generate a 4D animation sequence (3.0 seconds for canonical GS creation and 1.5 seconds for Gaussian Variation Field diffusion), only slightly slower than feedforward reconstruction method L4GM [57]. These quantitative results collectively validate both the effectiveness and efficiency of our proposed method.

Qualitative comparisons. We also provide qualitative comparisons with previous state-of-the-art methods in Figure 4. The SDS-based approaches [28, 88] turn to generate results with blurry textures and poor geometry. The feedforward method L4GM leverages multiview images generated from 2D generative prior [61] to reconstruct the 4DGS sequences. However, the results of L4GM suffer from 3D inconsistency of the generated multiview images. In contrast, our model directly generates the canonical GS and the Gaussian Variation Fields, capable of creating high-fidelity 3D consistent animations with coherent temporal dynamics.

More visualization of generated results. Figure 5 presents additional generation results from our method, including examples conditioned on both in-the-wild videos (left two cases) and test set videos (right two cases). Our model demonstrates high-quality generation capability with faithful motion reproduction. Despite being trained on synthetic data, the model exhibits strong generalization capability by effectively capturing motion patterns from in-the-wild video inputs. Furthermore, the model successfully handles challenging multi-object scenarios, highlighting the robustness of our approach.

Table 2: Ablation study of key factors in our VAE.

Config.	Encoder Query Type	Mesh-guided Loss	Variation Attrs.	PSNR$\uparrow$	LPIPS$\downarrow$	SSIM$\uparrow$
A.	$\bm{p}_{t}^{fps}$	✗	$\Delta\bm{p}_{t},\Delta\bm{s}_{t},\Delta\bm{q}_{t}$	23.25	0.0678	0.936
B.	$\bm{p}_{t}^{fps}$	✓	$\Delta\bm{p}_{t},\Delta\bm{s}_{t},\Delta\bm{q}_{t}$	26.17	0.0544	0.950
C.	$\Delta\bm{p}_{t}^{fps}$	✓	$\Delta\bm{p}_{t},\Delta\bm{s}_{t},\Delta\bm{q}_{t}$	28.58	0.0478	0.958
D. (Ours)	$\Delta\bm{p}_{t}^{fps}$	✓	$\Delta\bm{p}_{t},\Delta\bm{s}_{t},\Delta\bm{q}_{t},\Delta\bm{c}_{t},\Delta\alpha_{t}$	29.28	0.0439	0.964

Figure 6: Visual ablation of VAE.

Figure 7: Our model is also capable of creating animations for existing 3D assets with conditional videos.

Table 3: Ablation study of our diffusion model.

Method	PSNR$\uparrow$	LPIPS$\downarrow$	SSIM$\uparrow$	CLIP$\uparrow$	FVD$\downarrow$
Ours w/o Pos. Emb.	17.86	0.121	0.897	0.931	547.20
Ours	18.47	0.114	0.901	0.935	476.83

4.4 Ablation Study

Ablation of our VAE. In Table 2 and Figure 6, we analyze key components of our VAE. Our baseline (Config. A) starts with using positions of farthest sampled canonical GS $\bm{p}_{t}^{fps}$ as query for the encoder’s cross-attention layer, with variation attributes limited to positions $\Delta\bm{p}_{t}$, scaling $\Delta\bm{s}_{t}$, and rotation $\Delta\bm{q}_{t}$ following previous 4DGS works [76]. Since we do not have ground-truth GS motion for explicit supervision, the VAE initially struggles with motion learning. After equipped with our mesh-guided loss, the motion reconstruction capability is effectively improved through pseudo displacement supervision (Config. B). We then replace the encoding query with motion-aware $\Delta\bm{p}_{t}^{fps}$ using mesh-guided interpolation, which successfully handles most of the motion sequences (Config. C). Finally, to give the model more flexibility to handle complex motion sequence, we incorporate color $\Delta\bm{c}_{t}$ and opacity $\Delta\alpha$ of Gaussian attributes to the variation fields, which further enhance the reconstruction capability of VAE.

Ablation of our diffusion model. We examine the importance of positional embeddings in our diffusion model training in Table 3. By incorporating positional prior based on canonical GS positions $\bm{p}_{1}^{fps}$, the diffusion transformer better captures the correspondence between spatial positions and their variations. Removing these positional embeddings leads to a significant performance drop, demonstrating their crucial role in achieving high-quality results.

4.5 Application

Despite being trained on single video inputs, our model is effective at animating existing 3D models according to the motions depicted in conditioned videos. More details are included in the supplementary materials. As demonstrated in Figure 7, this approach produces high-quality animations that faithfully reproduce the target motions. Therefore, for real-world applications, the users can first generate 2D animations from the rendered images of their 3D models using off-the-shelf video diffusion models [47, 33, 34, 18], then employ our model to create corresponding 4D animations.

5 Conclusion

In this paper, we introduce a novel framework to address the challenging task of 4D generative modeling. To efficiently construct the large-scale training dataset and reduce the modeling difficulty for diffusion, we first introduce a Direct 4DMesh-to-GS Variation Field VAE, which is able to efficiently compress complex motion information into a compact latent space without requiring costly per-instance fitting. Then, a Gaussian Variation Field diffusion model that generates high-quality dynamic variation fields conditioned on input videos and canonical 3DGS. By decomposing 4D generation into canonical 3DGS generation and Gaussian Variation Field modeling, our method significantly reduces computational complexity while maintaining high fidelity. Quantitative and qualitative evaluations demonstrate that our approach consistently outperforms existing methods. Furthermore, our model exhibits remarkable generalization capability with in-the-wild video inputs, advancing the state of high-quality animated 3D content generation.

Acknowledgments. We extend our gratitude to all the reviewers for their constructive feedback. We also appreciate Jiaqi Lou for the assistance with chart refinement and the production of the supplementary video.

Appendix A Additional Implementation Details

Table 4: Detailed configuration of model architecture. SW and FFN denotes “Shifted Window” and “FeedForward Net”. MSA, MSSA, MTSA, MCA stand for “Multihead Self-Attention”, “Multihead Spatial Self-Attention”, “Multihead Temporal Self-Attention” and “Multihead Cross-Attention”, respectively.

Network	#Layer	#Dim.	#Head	Block Arch.	Special Modules	#Param.
$\bm{\mathcal{E}}_{GS}$	12	768	12	3D-SW-MSA + FFN	3D Swin Attn.	85.8M
$\bm{\mathcal{D}}_{GS}$	12	768	12	3D-SW-MSA + FFN	3D Swin Attn.	85.1M
VAE Transformer	12	768	12	MSA / MCA + FFN	-	125.21M
Diffusion	12	512	16	MSSA + MTSA + MCA + FFN	QK Norm.	105.51M

A.1 Model Architecture

We will detail the architecture of each model below, with the summary demonstrated in Table 4.

A.1.1 Gaussian Variation Field Encoder

Our encoder mainly comprises two parts: the canonical GS autoencoder $\mathcal{E}_{GS}$ and $\mathcal{D}_{GS}$ and a cross attention layer to create latent space for Gaussian Variation Fields.

For the canonical GS autoencoder, we adopt the model architecture from [81], which introduces a Structured Latent (SLat) representation for static 3D assets. This representation defines a set of local latents on a 3D grid, where each latent is associated with an active voxel intersecting with the surface of the 3D asset. The SLat representation effectively captures both the overall structure through active voxels and fine details through local latent codes. The canonical GS autoencoder is built using a transformer-based architecture. It first aggregates visual features from multiview images using a pre-trained DINOv2 [48] encoder to create voxelized features. These features are then processed through a sparse transformer encoder that handles variable-length tokens corresponding to active voxels. The transformer incorporates shifted window attention in 3D space to enhance local information interaction while maintaining computational efficiency. The encoder outputs structured latents that follow a regularized distribution through KL-divergence penalties, which are then decoded to various representations. For this work, we only leverage its Gaussian representation decoder for our canonical GS autoencoding. $\mathcal{D}_{GS}$ is set to resolution 64, and decode to 8 Gaussians per voxel. We finetune the decoder $\mathcal{D}_{GS}$ while keeping the encoder $\mathcal{E}_{GS}$ frozen.

For the cross attention layer, we adopt the vanilla full attention [72] implementation. we set the motion-aware $\Delta\bm{p}^{fps}_{t}\in\mathbb{R}^{512\times 3}$ using proposed mesh-guided interpolation mechanism as query and point displacement fields $\Delta P_{t}\in\mathbb{R}^{8192\times 3}$ from mesh as keys and values. Then the latent representation $\bm{z}\in\mathbb{R}^{512\times 16}$ is obtained after the cross attention layer.

A.1.2 Gaussian Variation Field Decoder

For the Gaussian Variation Field decoder, we first adopt 12 layers of vanilla self attention for thorough information exchange. For the last cross attention layer to decode Gaussian Variation Fields $\Delta G_{t}\in\mathbb{R}^{N_{G}\times 14}$ The output feature of last self attention layer is set to keys and values, and we adopt all parameters of $G_{1}\in\mathbb{R}^{N_{G}\times 14}$ as query, where $N_{G}$ is the total number of canonical GS.

A.1.3 Canonical GS Generation Model

We adopt the model architecture from [81] to generate structure latent representation for further decoding to canonical GS, which follows a two-stage process. First, a structure generator creates the sparse structure by denoising a low-resolution feature grid using transformer blocks with adaptive layer normalization and cross-attention for condition injection. Then, a latent generator $\bm{\mathcal{G}}_{\mathrm{L}}$ generates local latents for the given structure using a sparse transformer architecture with downsampling and upsampling blocks. These two generators both adopt RMSNorm [89] to the queries and keys (QK Norm.) in diffusion training. They are conditioned on image conditions through CLIP and DINOv2 features respectively, and are trained separately using a continuous flow matching objective. Since we freeze the $\mathcal{E}_{GS}$, we can directly leverage the pretrained image-to-3D model [81] to create canonical GS.

A.1.4 Gaussian Variation Field Diffusion Model

Our Gaussian Variation Field diffusion model builds upon the diffusion transformer architecture [52]. To enable temporal coherence in generation, we introduce a temporal self-attention layer that complements the existing cross-attention, spatial self-attention, and feedforward layers. For video sequence conditioning, we extract frame-wise features using DINOv2 [48] and incorporate the farthest-sampled canonical Gaussian Splatting to maintain awareness of the canonical 3D model. To enhance spatial consistency, we incorporate positional priors into the generation process. During training, we encode the Gaussian Variation Field latent along with their corresponding canonical GS positions to formulate positional embeddings. During inference, we directly utilize the positions from farthest-sampled Gaussian Splatting for positional embedding computation.

A.2 Additional Training and Inference Details

In this paper, we designate the first frame of each video as the canonical frame. For our Direct 4DMesh-to-GS Variation Field VAE training, we set the loss weights as follows: $\lambda_{lpips}=0.2$, $\lambda_{ssim}=0.2$, $\lambda_{mg}=1.0$, and $\lambda_{kl}=1e-6$. Computationally, the VAE training requires one day on 32 Nvidia Tesla V100 GPUs (32GB) for the first stage and two days on 8 Nvidia Tesla A100 GPUs (40GB) for joint training, while the diffusion model training spans approximately one week on 8 Nvidia Tesla A100 GPUs (80GB). During inference, we adopt the adaptive mode of DPM-Solver [41] with order 2, requiring approximately 18 steps per instance.

During inference, we address potential orientation misalignment between the generated canonical GS and input images through an azimuth alignment process similar to [57]. Specifically, we render the canonical GS from multiple azimuth angles and compute image-level losses between these renders and the first video frame. We then transform the canonical GS according to the azimuth angle that yields the minimal loss, ensuring better alignment with the input video.

The in-the-wild conditional videos shown in the teaser (Figure 1 in main paper) are created by Kling [34]. The walking astronaut and boxing rat video frames in Figure 5 of the main paper are sourced from consistent4D and Emu video [18], respectively.

A.3 Additional Details of Creating Animation for Existing 3D Model

To animate existing 3D models using our approach, users follow a simple pipeline: First, their 3D assets are rendered as multiview images. These images are then processed to extract and aggregate DINOv2 features. Using these features, we construct a canonical Gaussian Splatting representation through our $\mathcal{E}_{GS}$ encoder and $\mathcal{D}_{GS}$ decoder. Finally, animations are generated by our diffusion model, which takes both the canonical GS and a conditional video as input. Users can create these conditional videos using state-of-the-art video diffusion models [47, 33, 34, 18] to specify their desired motion for the 3D model.

Appendix B Data Preparation Details

Our training dataset consists of 34K 3D mesh animations sourced from Objaverse-V1 [13] and Objaverse-XL [12]. For Objaverse-V1, we utilize the curated set of 9K high-quality 3D animations from [38]. For Objaverse-XL, we apply two filtering criteria: first, following [81], we filter out samples whose average aesthetic score  across 4 rendered views of the first frame falls below 5.5; second, we remove sequences with minimal motion. This filtering process yields 25K additional animations from Objaverse-XL.

Appendix C Additional Ablation

Ablation of $\mathcal{D}_{GS}$ Joint Finetuning. We investigate the importance of jointly finetuning the canonical GS decoder during our Direct 4DMesh-to-GS Variation Field VAE training. Starting from a pretrained canonical 3D $\mathcal{D}_{GS}$ checkpoint, we compare two settings: freezing $\mathcal{D}_{GS}$ while training other modules, and jointly training all modules (our approach). As shown in Table 5, joint training allows $\mathcal{D}_{GS}$ to receive feedback from animation reconstruction rather than being limited to static data only. This ensures the canonical GS reconstruction coherent with its corresponding variation fields.

Ablation of hyper-parameters in mesh-guided interpolation. We ablate the hyper-parameters including nearest neighbors $K$, and distance decay rate $\beta$ of interpolation in Table 6. Our setting ($K=8,\beta=7.0$) yields optimal results. Performance is relatively stable for other values, showing reasonable robustness.

Table 5: Additional ablation of our proposed VAE.

Model	PSNR$\uparrow$	LPIPS$\downarrow$	SSIM$\uparrow$
Ours w/o $\mathcal{D}_{GS}$ Finetuning	28.80	0.0460	0.962
Ours	29.28	0.0439	0.964

Table 6: Additional ablation of hyper-parameters in our mesh-guided interpolation.

$K$	$\beta$	PSNR$\uparrow$	LPIPS$\downarrow$	SSIM$\uparrow$	$K$	$\beta$	PSNR$\uparrow$	LPIPS$\downarrow$	SSIM$\uparrow$
16	7.0	28.38	0.0464	0.960	8	10.0	28.55	0.0462	0.961
8	7.0	29.28	0.0439	0.964	8	7.0	29.28	0.0439	0.964
4	7.0	28.94	0.0451	0.963	8	4.0	29.04	0.0446	0.963
1	7.0	28.22	0.0465	0.960	8	1.0	28.64	0.0457	0.962

Figure 8: Sample of our autoregressive generation result.

Appendix D More Results

D.1 Autoregressive Generation Results for Temporal Generalization

Temporal generalization is a known challenge in 2D/3D video generation. In our case, we can employ an autoregressive approach during inference for videos exceeding our training length: the GS from the last frame of a generated segment serves as the canonical GS for inferring the next segment’s variation fields, which allows for coherent long animations. We show a 120-frame generated sample using such an approach in Figure 8.

D.2 VAE Reconstruction Results

As illustrated in Figure 11, we demonstrate the reconstruction capabilities of our proposed Direct 4DMesh-to-GS Variation Field VAE. Our method efficiently encodes both canonical GS and their temporal variations from 4D meshes in a single pass, eliminating the need for time-consuming per-instance fitting procedures. The results demonstrate our model’s effectiveness in preserving both geometric fidelity and motion dynamics.

Figure 9: More generation results of real-world input videos.

D.3 More Visual Comparison with SOTA Methods

As illustrated in Figure 12, we provide extensive visual comparisons with state-of-the-art methods. Our approach demonstrates consistent superiority across diverse test cases, achieving better results in terms of both visual fidelity and temporal motion coherence.

D.4 More Reults of Animating Existing 3D Models

As shown in Figure 13, we demonstrate additional results showcasing our method’s capability to animate existing 3D models using conditional videos. Our approach successfully extracts and transfers motion patterns from the input videos, generating high-fidelity animations that faithfully preserve both geometric and temporal characteristics.

D.5 Additional Results on Real-World Video Inputs

Although our model is trained on synthetic data, it effectively generalizes to real-world video inputs. Figure 9 presents additional results, demonstrating the model’s robust generalization capabilities.

Appendix E Borader Impact

Like all generative models, our video-to-4D generation framework requires careful consideration of societal implications. While we mitigate certain ethical concerns by training exclusively on synthetic 3D animations from the Objaverse dataset, thus avoiding privacy and copyright issues associated with real-world data, we acknowledge potential risks. The ability to generate animated 3D content from videos could be misused for creating misleading content. We therefore emphasize the importance of establishing clear guidelines for the responsible deployment of video-to-4D generation technology.

Appendix F Limitation Discussion and Future Work

Video condition frame 0	Video condition frame 1
Canonical GS frame 0	Our Animation frame 1

Figure 10: Failure case. When the pretrained static 3D generative model produces canonical GS that are not well-aligned with the conditional video frames, our Gaussian Variation Field diffusion model struggles to bridge this inconsistency, resulting in suboptimal animations.

While our model demonstrates impressive results in video-to-4D generation, it has certain limitations. Our two-stage generation process first employs a pretrained static 3D generative model to create canonical Gaussian Splatting representations, which then serve as conditions for our diffusion model to generate Gaussian Variation Fields. A notable limitation arises when the static 3D generative model [81] produces canonical GS that exhibits discrepancies with the conditional video, such as mismatched head pose, incorrect eyes or light effects in Figure 10, potentially creating inconsistencies in the final animation. To address this limitation, future work could explore either fine-tuning the static 3D model to ensure better image alignment or developing an end-to-end 4D diffusion framework that jointly generates both the canonical representation and its temporal variations.

Figure 11: Additional visual results of VAE reconstruction.

Figure 12: More visual comparison with SOTA Methods.

Figure 13: More results of animating existing 3D model input with conditional videos.