This page is an AI translation of the Japanese original. If anything differs, the Japanese version prevails.

4D

ASU Releases Open4DGS, an Open-Source 4DGS Pipeline for Consumer Cameras

The NEM program at Arizona State University has released a 4DGS pipeline that corrects rolling shutter without requiring genlock. It reports that a method that gives fast-moving segments more training raised PSNR on fast scenes by about 2.5 dB.

This article was generated by AI and published automatically without prior human review.

Use cases
Production, Research, Entertainment
Sources
1 (en) (1 primary)

Key points

  1. Arizona State University's NEM program has released an open-source pipeline that builds free-viewpoint 4DGS from multi-camera video
  2. It estimates sub-frame offsets between cameras and rolling shutter from visual information alone, and does not assume genlock-capable global shutter equipment
  3. By increasing the number of training visits in fast-motion segments, PSNR on fast frames in one specific dance sequence rose from 31.69 dB to 34.18 dB, according to the report (announced value)
Contents
  1. What happened
  2. How it works
  3. By the numbers
  4. Why it matters
  5. Examples
  6. How to try it
  7. Caveats and open questions
  8. Glossary
  9. Sources

What happened

The Narrative and Emerging Media (NEM) program at Arizona State University (ASU) has released "Open4DGS," a pipeline for 4D Gaussian Splatting (a method that represents dynamic scenes with time-dependent Gaussians). It turns multi-camera footage of human performances into volumetric video that can be viewed from free viewpoints. The project page describes it as "production-ready" open-source software developed over about one year. The listed authors are Arvin Tehrani, Rory Mitchell, Ryat Yezbick and Nonny de la Peña, with affiliations given as ASU NEM and Contrast Media Labs.\n\nIts main feature is that it does not require global shutter equipment synchronized by genlock (a camera sync signal). In preprocessing, it learns from the footage each camera's continuous sub-frame phase offset and the rolling shutter readout time. It is intended for shooting with GoPro cameras, Sony action cameras, or a mix of consumer cameras.\n\nThe other feature is a training method for fast motion. It automatically adds intermediate keyframes according to speed, so that fast-moving frames are visited more often during training. In a comparison on one dance sequence (27 cameras, 60 fps), PSNR on fast frames rose by about 2.

How it works

Synchronization and rolling shutter correction: The preprocessing script calculate-sync.py tracks flow along the epipolar direction for calibrated camera pairs and estimates a phase offset for each camera. The capture time of each pixel row is modeled as "nominal time + camera-specific offset + readout time proportional to row position." Consistency between cameras is checked for cycles in a graph, and if confidence does not meet the threshold, the system falls back to a timecode-only baseline.\n\nSpeed-aware sub-keyframes: Per-point velocity is obtained from dense spatial matching with RoMa v2 and optical flow from MEMFOFOF. For points whose speed exceeds a threshold (4.0 spacings per frame in the example), intermediate keyframes are added at one-frame intervals and a short temporal width (σt) is assigned. The aim is to suppress fast-moving parts from lingering in neighboring frames like afterimages.\n\nWeighted frame sampling: Short-lived Gaussians receive gradients only when their frame is selected. With uniform sampling, fast frames are visited only about 98 times out of 6,000 steps, so a setting (--config.frame-sampling-fast-visits 387) is provided that adds visits to fast frames, up to 387.\n\nMotion model: The translation and rotation of each primitive are optimized jointly as a Lie algebra twist in se(3), with a local second-order

By the numbers

PublisherArizona State University, NEM program (Contrast Media Labs also listed)
InputMulti-camera video (consumer action cameras, etc.; no genlock required)
SynchronizationSub-frame phase offsets estimated from video (calculate-sync.py)
Rolling shutterPer-row readout time modeled (15.0 ms for Sony RX0 II at 60 fps)
Sub-keyframe threshold--subkeyframe-min-speed 4.0 spacings/frame
Visits to fast framesStandard about 98 → up to 387 (out of 6,000 steps)
Long-sequence processing3.0-second chunks streamed to the GPU sequentially
Evaluation sequenceapt-dance-01, frames 66–126, 27 cameras, 60 fps, evaluated on 6 unseen viewpoints
PSNR, fast frames (81–102)31.69 dB → 34.18 dB (+2.50 dB, 108 views)
PSNR, fast hand motion (86–96)31.59 dB → 34.23 dB (+2.65 dB, 66 views)
Before fast segment (66–80)31.56 dB → 33.02 dB (+1.46 dB)
After fast segment (103–126)31.68 dB → 33.39 dB (+1.71 dB)
PlaybackA 60 FPS real-time viewer is referenced (detailed conditions unconfirmed)

Why it matters

Volumetric capture has tended to face an equipment-cost barrier because it requires dedicated genlock-capable cameras. Open4DGS aims to make up for this requirement through estimation from the footage. University labs, small production companies, and experimental stage and live-event projects may find it easier to test with consumer cameras they already own.\n\nRolling shutter distortion is also a noticeable problem when shooting fast motion such as dance. The design of handling correction inside the pipeline is a useful reference for creators trying 4D capture.\n\nWhether it is usable in practice in Japan depends on the license terms, the required GPU environment, and reproducibility on one's own equipment. A realistic first step is a small shoot to check whether synchronization estimation succeeds (that is, whether it does not fall back to the timecode-only baseline). How the viewer and data format connect with existing 3DGS-based tools is also something to check before adoption.

Examples

  • Project page demo: free-viewpoint rendering from a multi-camera setup using Sony RX0 II cameras, with 15 ms rolling shutter corrected
  • Free-viewpoint rendering (60 FPS) of dance and acrobatic motion. The page says finger and cloth motion is preserved
  • A demo of fast motion such as ball sports, said to show contours that blur less after synchronization (video only; no figures confirmed)

How to try it

  1. Open the Code Repository and Pipeline Guide from the project page and check the requirements and license
  2. Shoot a short motion sequence with multiple cameras you already have. Check each camera model's specifications for its rolling shutter readout time
  3. Estimate synchronization with calculate-sync.py (for example: --fps 60 --rs-readout-ms 15.0 --output-sync sync.json) and check whether a fallback occurs
  4. Vary --config.frame-sampling-fast-visits during training and compare how fast segments look

Caveats and open questions

  • The material we could obtain covers only part of the project page text. Content after the explanation of the Fixed Canonical Set could not be confirmed.
  • All figures are values announced by the developers. The quantitative evaluation covers one dance sequence (27 cameras, 6 unseen viewpoints), and reproducibility on other scenes or equipment cannot be confirmed from the material.
  • Claims such as "production-ready" and "+2.5dB" are statements from the project page. The material contains no third-party verification.
  • The license, supported OS, required GPU memory, training time, and the specific repository URL cannot be confirmed from the material.
  • Links to a Technical Report and a Real-Time Viewer exist, but their contents are not included in the material. The conditions for 60 FPS streaming playback (equipment and data size) are unconfirmed.
  • Within the scope of the material, the only comparison is with uniform sampling; there is no comparison with other 4DGS methods.

Glossary

4D Gaussian Splatting (4DGS)
A method that gives a set of 3D Gaussians variation over time, so that moving scenes can be played back from free viewpoints.
Genlock
A mechanism that aligns the capture timing of multiple cameras using an external sync signal.
Rolling shutter
A readout method that reads an image row by row from the top. It causes fast-moving subjects to appear distorted.
Sub-keyframe
An intermediate frame added between regular keyframes. It is used to represent fast motion in finer detail.
PSNR
A metric showing the difference between a reconstructed image and the ground-truth image. The unit is dB, and higher values indicate a closer match.
se(3)
A mathematical representation (Lie algebra) that handles 3D rotation and translation together.

Sources

Primary sources (publisher, paper, repository)

The X post that surfaced this story

This article was generated in Japanese by AI from the sources listed (primarily primary sources) and published automatically, without prior human review, after automated checks such as source matching (how we use AI). Figures and specifications are as stated by the publishers and have not been independently verified by us. Please consult the original sources. To report an error, use our contact form; we will review it promptly and correct it. This page is an AI translation of the Japanese original. If anything differs, the Japanese version prevails. Japanese original

From the operator4DGS.jp(IZUTSUYA Inc.)

Get it by email

We send the latest issue every 3 days at 7:00 JST (in Japanese).

The newsletter is sent in Japanese only. You can subscribe on the Japanese page.