VideoVAEPlus Tactile

--- license: apache-2.0 tags: - video - vae - autoencoder - tactile - visuo-tactile - world-model library_name: pytorch base_model: VideoVerses/VideoVAEPlus pipeline_tag: video-to-video --- # VideoVAE+ 16z — Tactile Finetune A visuo-tactile finetune of [VideoVAE+](https://github.com/VideoVerses/VideoVAEPlus) (the `sota-4-16z`, 16-latent-channel, text-free variant). Starting from the released `sota-4-16z.ckpt`, the autoencoder was finetuned to encode/decode tactile sensor video (left/right tactile streams) alongside RGB view frames. ## Model - **Architecture:** `AutoencoderKL2plus1D_1dcnn` (factorized 2+1D KL autoencoder, 1D temporal CNN) - **Latent channels (`z_channels`):** 16 - **Spatial compression:** 8× (`ch_mult=[1,2,4,4]`, 3 downsamples) - **Temporal compression:** 4× (16 frames → 4 latent timesteps) - **Text conditioning:** none (`caption_guide: False`) - **Base checkpoint:** `sota-4-16z.ckpt` from VideoVerses/VideoVAEPlus - **Finetune objective:** `LPIPSWithDiscriminator3D` (KL weight 1e-6, disc weight 0.5), base LR 5e-5 ## Files - `videovae_plus_16z_tactile.ckpt` — final finetuned weights (Lightning checkpoint, ~5 GB). - Source code, configs, and inference scripts mirrored from the working repo. ## Usage ```bash # reconstruct a video with the finetuned autoencoder python inference_video.py \ --config configs/inference/config_16z_infer_noloss.yaml \ --ckpt videovae_plus_16z_tactile.ckpt \ --input examples/videos/gt/0510_episode_000_tactile_left.mp4 ``` See the included `configs/train/config_16z_tactile.yaml` for the exact finetuning recipe. Load the checkpoint into the `AutoencoderKL2plus1D_1dcnn` model defined in `src/models/autoencoder2plus1d_1dcnn.py`. ## License & attribution Code and base model from [VideoVerses/VideoVAEPlus](https://github.com/VideoVerses/VideoVAEPlus) (Apache-2.0). This repository adds tactile-finetuned weights and the corresponding training/inference configs.

Source context: 0 downloads · 0 likes · Pipeline video-to-video · Library pytorch · Repo WitneyWW/VideoVAEPlus-tactile