Video Diffusion Alignment via Reward Gradient

Training efficiency comparison against various baselines, when trained for longer. The base model used here is ModelScope. We compare VADER against DPO, DDPO, on-policy DPO, and on-policy DDPO. For implementing the on-policy version of the baselines, we simply reduce the UTD (update-to-data) ratio to 1, thus only doing a single gradient update for each datapoint sampled. We observe that VADER significantly outperforms all of them in terms of compute efficiency.

Diversity of generated videos for VADER. The base model is VideoCrafter2. We generate 500 videos for each model and prompt combination. We use 5 prompts resulting in a total of 2500 videos per model. The diversity is calculated using the variance of VideoMAE latent space embeddings across the 500 videos for each prompt. We then average the variances over all prompts. We find that VADER variants exhibit reduced diversity compared to the baseline model VideoCrafter2. Prior works Robert Kirk, et al. and Sonia K Murthy, et al. have found similar results, where aligning a model for a specific use case often results in reduced diversity. Some Visualization are exhibited in Diversity Gallery.

Ablation of memory usage for different components in ModelScope-based VADER. For this experiment, we offload the memory to the CPU main memory to prevent GPU out-of-memory error. Starting with the standard LoRA + Mixed Precision, each row represents an added component (Subsampling Frames, Truncated Backpropagation, and Gradient Checkpointing) applied incrementally to the previous row. The total RAM reduced is about 240 GB after implementing all the steps.

The base model is VideoCrafter2. In this Table, we study how optimizing for specific reward functions via VADER affects scores on other reward functions. We observe that the HPS score increases significantly after fine-tuning the base model via the PickScore model, indicating a strong correlation. We study correlation across different reward models for VADER. We find that there is a strong positive correlation between PickScore and HPS scores, while a strong negative correlation between ViCLIP and Aesthetic reward function.

EvalCrafter evaluation results for VADER. EvalCrafter calculates Temporal Coherence using Warping Error, Semantic Consistency (cosine similarity of the embeddings of consecutive frames), and Face Consistency, which assess frame-wise pixel and semantic consistency. Motion Quality is evaluated through Action-Score (action classification accuracy), Flow-Score (average optical flow between frames obtained from RAFT), and Motion AC-Score (amplitude classification consistency with the text prompt). We generate 700 videos from each model for this comparision. Results demonstrate that all the VADER-variants outperform the base model (VideoCrafter2).

VBench evaluation results VADER using EvalCrafter Prompts. The base model is VideoCrafter2. The metrics used in VBench include: Subject Consistency (consistency of the main subject across frames, evaluated using DINO feature similarity), Background Consistency (using CLIP feature similarity), Motion Smoothness (fluidity of motion, based on motion priors from a frame interpolation model), Dynamic Degree (extent of motion in the video, estimated with RAFT), Aesthetic Quality (assessed via the LAION aesthetic predictor), and Imaging Quality (using MUSIQ). The weighted average is calculated by assigning weights to each metric. All metrics are given a weight of 1, except for the Dynamic Degree metric, which is assigned a weight of 0.5. We generate 700 videos for each model using EvalCrafter prompts that are not seen during training. We find that VADER-Pick has the best consistency score, while VADER-HPS shows the best aesthetic and imaging quality. We find VADER-HPS performs the best overall.

VBench evaluation results VADER using standard prompt suite. The base model is VideoCrafter2, Pika (2023-9), Gen-2 (2023-12), and T2V Turbo (VC2). The metrics used in VBench include: Subject Consistency (consistency of the main subject across frames, evaluated using DINO feature similarity), Background Consistency (using CLIP feature similarity), Motion Smoothness (fluidity of motion, based on motion priors from a frame interpolation model), Dynamic Degree (extent of motion in the video, estimated with RAFT), Aesthetic Quality (assessed via the LAION aesthetic predictor), Imaging Quality (using MUSIQ), and Temporal Flickering (computing the mean absolute difference across frames). The Quality Score is calculated by assigning weights to each normalized metric. All metrics are given a weight of 1, except for the Dynamic Degree metric, which is assigned a weight of 0.5. This follows VBench metrics. We find VADER-HPS surpass all baselines in terms of Quality Score, Aesthetic Quality and Dynamic Degree.

VBench evaluation results for Image to Video diffusion models. The base model is Stable Video Diffusion. We compare Stable Video Diffusion and VADER-V-JEPA. VADER-V-JEPA demonstrates improvements across most metrics, particularly in consistency and aesthetic quality.

We ablate the number of truncated backpropagation steps (K) in VADER. For this experiment, we use VADER trained using Aesthetic and HPS Rewards. The base model is VideoCrafter2. We find that higher values of K result in more semantic level changes, while K=1 results in more fine-grained changes, specifically in the earlier steps of training. Further, we find as we train longer, both the models start exhibiting semantic level changes. We also find it is easier to optimize with a smaller value of K, as can be seen in the results below.

Training Curves for VBench Distilled Reward Model. The plot on the left shows the average training and validation loss curves over epochs, showcasing the model's convergence behavior. The figure on the right depicts the average validation accuracy for each epoch, highlighting the model's performance in ranking the preference of video pairs in terms of every metrics, including background consistency, dynamic degree, image quality, motion smoothness, aesthetic quality, PickScore value, HPS score, and subject consistency.

The reward curve when training VADER using the VBench Distilled reward model.