Skip to content
forked from NoizAI/HelixWorld

About

🪐 HelixWorld: real-time interactive audio-visual world model.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

 
 

Repository files navigation

HelixWorld 1.0 — Sound and vision, born together

HelixWorld

HelixWorld 1.0 is a real-time interactive audio-visual world model from Noiz AI, with researchers from HKUST, Tsinghua, CMU, and Google DeepMind.

Give it an image and a prompt. Walk forward or turn around — picture and sound update together from the same Transformer. The spatial field (azimuth, distance, reverb) turns with the camera. Audio is not a soundtrack laid on afterwards.

Weights, inference code, and the technical report ship in the coming weeks. Open a feature request if you want something on the roadmap.

arXiv Hugging Face License

Highlights

HelixWorld 1.0
Roaming / camera navigation Yes
Real-time 24 FPS
Continuous generation > 3 minutes
Joint audio-video 48 kHz stereo, same Transformer
Spatial sound field Follows viewpoint
Weights and code Coming soon

Genie 3, WorldPlay, LingBot-World and others roam visually. HelixWorld's difference is native audio-video, plus a spatial field that turns with the camera.

Coming soon

  • Weights + inference — #1
  • Technical report — #2
  • Interactive demo — #3
  • Hugging Face checkpoints — #4

Training code may follow inference. Dataset is not planned for 1.0. Subscribe to releases.

Method

Four stages. Details, figures, and ablations ship with the report.

spatial AV data  →  joint Transformer + action
                 →  causal rollout (KV cache, self-generated history)
                 →  few-step distillation (trajectory + DMD) for realtime
  1. Spatial AV data. First-person real-world video (on-location sound) plus game-engine captures with known geometry and listener pose. Fake stereo, post-hoc scores, and narration are dropped.
  2. Joint generation + action. Audio and video keep their own latents, enter one Transformer, and predict the next picture and sound from the current state and the user's action.
  3. Causal rollout. Bidirectional video models can look ahead; interaction cannot. Converted to a causal generator with KV cache, then trained on its own generated history.
  4. Realtime. Trajectory distillation + DMD cut denoising to a handful of steps. Action, decode, and AV output run as a pipeline.

Roadmap

1.0 is real-time audio-video interaction. Next:

Tracker
Multi-agent #5
Multi-person presence #6
Long-horizon consistency #7
Autonomous evolution #8
Games / interactive film / education #10

Citation

@misc{helixworld2026,
  title        = {HelixWorld: A Real-Time Interactive Audio-Visual World Model},
  author       = {{Noiz AI}},
  year         = {2026},
  howpublished = {\url{https://github.com/NoizAI/HelixWorld}},
  note         = {Weights and code forthcoming}
}

License

Code is Apache 2.0. The weight license will be published with the checkpoint.

Related work from the team: AudioX, AudioX-Turbo, YuE.

About

🪐 HelixWorld: real-time interactive audio-visual world model.

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages