Repository navigation
fix(metal): bound convolution workspaces and check completed commands - #12
Merged
Merged
Conversation
jamesbrink
added a commit
to utensils/mold
that referenced
this pull request
Sep 6, 2026
Z-Image could silently return partial or zero-filled 1024-pixel VAE output on Metal. This pins the bounded Metal convolution backend from utensils/candle#12, removes the temporary latent-120 proactive tiling cap, and completes every Metal decode attempt inside a bounded OOM recovery policy before accepting pixels. CPU and CUDA retain their existing decode and fallback behavior. The Metal recovery contract is isolated behind injected tests: recognized OOMs may retry with tiles, repeated OOM during cleanup cannot suppress the existing eager CPU fallback, and unrelated errors propagate. The Candle pin is synchronized across the workspace and desktop roots; mobile has no Candle packages in its lockfile. The H3 source marker records the newly compiled Candle identity without relabeling historical qualification evidence. Validation: - 11 recovery, 17 VAE-tiling, and 35 Z-Image pipeline tests pass with Metal enabled. - Metal inference Clippy, workspace formatting, desktop/mobile locked metadata, CI routing, and single-Candle-identity checks pass. - Candle PR #12 is green on macOS, Linux, Windows, Clippy, rustfmt, and secret scanning. - Whole real-weight VAE outputs from 768 through 1024 pixels match CPU globally and per quadrant (relative L2 2.35e-6 to 5.24e-6, gate 1e-3). - A real generated 1024 latent matches CPU at 3.59e-6; reference, CPU, and Metal images were individually inspected with no visible corruption. - Independent source reviews found no Candle or Mold blocker. The qualification does not claim 1152-pixel support: that attempt was refused by the 12 GiB native allocation ceiling. A final production-planner render was not launched because the safe source route is sequential, while the current local planner deliberately selects eager loading for Z-Image GGUF; a custom harness would not prove the shipped planner path. Details and retained evidence are in `docs/qualification/zimage-metal-vae.md`. Closes #1040 after the dependent Candle PR is accepted and this PR passes terminal review/CI.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large Metal conv2d operations can retain multi-gigabyte im2col and GEMM temporaries, while a command that fails during its completion wait can previously be accepted as successful. Bound both workspaces to 64 MiB per spatial tile, reclaim completed batches, and check terminal errors before accepting completion/readback. Each tile reads the original full input, preserving the convolution receptive field; small convolutions retain their existing path. CPU and CUDA implementations are unchanged.
Metal-gated tests cover partial tiles, batches, noncontiguous and offset kernels, workspace ratios and multiple reclamation batches. Independent review approved this exact six-file production commit; allocation ceilings, terminal diagnostics and qualification harness changes are excluded.
Validation on M4 Max: whole Z-Image VAE outputs at 768–1024 pixels match CPU raw F32 outputs globally and per quadrant (relative L2 2.35e-6–5.24e-6, gate 1e-3). An independent sd.cpp whole-1024 decoder passes at 3.48e-4. A real reference-generated 1024 latent passes Candle Metal versus CPU at 3.59e-6; reference/CPU/Metal PNGs were individually inspected with no visible corruption. Tiny maintained Metal convolution tests pass under bounded allocation guards. Evidence: utensils/mold#1040 (comment).
1152-pixel Metal decoding was refused by the qualification-only 12 GiB native allocation ceiling; it is not qualified. The original silent-corruption mechanism was not reproduced under safe guards, so this does not establish a residency/driver root cause. These results qualify the isolated decoder, not a full Mold render or Wan. Downstream Mold will pin this revision together with Metal-only completion/OOM recovery; coordinated merge remains pending.