Skip to content

[CI Fix][ROCm] Refresh DeepSeek V3 CP MI350X golden - #5557

Merged
acisseJZhong merged 1 commit into
mainfrom
codex/refresh-mi350-deepseek-cp-golden
Oct 20, 2026
Merged

acisseJZhong merged 1 commit into
mainfrom
codex/refresh-mi350-deepseek-cp-golden

Conversation

@acisseJZhong

Copy link
Copy Markdown
Contributor

Summary

  • Refresh the MI350X DeepSeek V3 FSDP+CP+PP+EP loss and grad-norm golden.
  • Keep the existing strict equality check and ten-step coverage unchanged.

Why

#4760 switched DeepSeek V3 context parallelism to the compact MLA path and refreshed the A10G golden, but the MI350X golden was not updated. The ROCm workflow only runs after changes reach main, so the stale accelerator-specific file first surfaced there.

The new sequence is bitwise identical across consecutive main runs, including:

The first passing run before #4760 and the first failing run used the same ROCm image digest and PyTorch nightly, isolating the change to TorchTitan source. The same new sequence also persists across later dependency nightlies.

Test plan

  • SKIP=pyrefly-check pre-commit run --all-files (pass)
  • Full pre-commit run --all-files: all other hooks pass; local Pyrefly is blocked because the shared venv has PyTorch 2.14 and older torch_checkpointing APIs while current main targets PyTorch 2.16 nightly APIs.
  • MI350X execution is covered by the linked independent main runs; ROCm hardware is not available locally.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 20, 2026
@acisseJZhong
acisseJZhong marked this pull request as ready for review October 20, 2026 05:16
@pytorch-bot pytorch-bot Bot added the ciflow/fake-pg Run 1-GPU Fake PG integration tests label Oct 20, 2026

@tianyu-l tianyu-l left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#5196 might break it again

@acisseJZhong
acisseJZhong merged commit 481d622 into main Oct 20, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/fake-pg Run 1-GPU Fake PG integration tests ciflow/rocm CLA Signed This label is managed by the Meta Open Source bot. module: rocm

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants