Repository navigation
Fused CPU Adam step - #8841
Fused CPU Adam step#8841
Conversation
…ed buffers Signed-off-by: pengdurice <pengduhit@gmail.com>
…ide the CPU Adam pass Signed-off-by: pengdurice <pengduhit@gmail.com>
|
Hi @pengdurice, It's great to see this PR addressing CPU optimizer overhead, as it closely aligns with our work on ReplayOpt and ReFlow (#8719). Our research focuses on the memory bandwidth bottlenecks of CPU-offloaded optimizers. In ReplayOpt, accepted to IEEE Computer Architecture Letters (CAL), we explored reducing the critical-path overhead of optimizer-state updates. Building on this work, ReFlow further reduces memory traffic through register-resident SIMD fusion and overlaps CPU optimizer execution with GPU computation through decoupled scheduling. In particular, ReFlow already implements the two main optimizations in this PR: fusing gradient scaling/clipping into CPU SIMD kernels and directly generating FP16/BF16 parameters without a separate conversion pass. There are also opportunities for further optimization. For configurations using BF16 gradient accumulation, or without gradient accumulation, gradients can be transferred to the CPU in BF16 and promoted to FP32 only within SIMD registers during the optimizer update. This reduces not only PCIe traffic but also gradient-related CPU memory traffic, which is especially beneficial for memory-bandwidth-bound CPU optimizers. The in-register conversion is relatively inexpensive and avoids a separate FP32 conversion pass. With these optimizations and asynchronous scheduling, ReFlow achieves up to 4.0× higher training throughput than ZeRO-Infinity on a single B200 and 1.86× on four B200 GPUs. I originally implemented these optimizations in separate ReFlow kernels to minimize regression risks and preserve the stability of the existing DeepSpeed CPU optimizer path. However, since this PR is addressing the same underlying bottlenecks, I think this could be a good opportunity to incorporate ReFlow's non-asynchronous optimizations into the default CPU-offload path, while keeping asynchronous execution optional. Would you be willing to review my PR (#8719) and share your thoughts on this direction? I'd greatly appreciate your feedback and would be happy to help with the integration and validation. The ReFlow paper has also been accepted to ASPLOS 2027 and is expected to be publicly available soon. I'd be happy to share a copy directly if it would help with reviewing the design and evaluation. Thank you for your time and consideration! |
0z5a
left a comment
There was a problem hiding this comment.
Thanks for this. Most of it checks out: the clip factor comes from the same folding-aware scaled_global_grad_norm (partition group → MP reduce → _autoep_expert_parallel_group), overflow returns before the sub-group loop, IncrementStep is idempotent across pieces, pieces are aligned to SIMD_WIDTH*8, and the bf16 store is RNE with NaN→0x7fc0 like torch. The new kernel tests pass bitwise on AVX-512 for me. Small thing: the first commit is #8840's head, so "Builds on #TODO" should read #8840.
One blocking issue and a few should-fix items below.
1. [blocking] Params with no gradient this step diverge from the unfused path — stage3.py:3022-3024
Skipping the in-place unscale_and_clip_grads has two side effects:
- In offload mode a sub-group's fp32 grad buffer isn't zeroed between steps, and
_finalize_offload_gradient_accumulationonly refreshes params reduced in this window. On master,unscale_and_clip_gradsrescales that stale region in place every step; on the fused path it keeps its raw loss-scaled, unclipped value and is only multiplied by the current factor. Repro on the CPU accelerator (1 rank, fp32 model,offload_optimizer: {device: cpu, pin_memory: false},gradient_clipping: 1e-3, a layer used only on step 0, 3 steps): every parameter is bitwise equal between the fused and unfused paths except the unused layer, which differs by up to 5.9e-3. Under AutoEP this is any expert that receives no tokens in a step. - After
step(),safe_get_full_gradnow returns unscaled, unclipped grads (0.221 vs 2.97e-4 in the same repro).
Options: take the fused path only when every param of the sub-group was reduced this window (e.g. check against _offload_boundary_param_ids), or zero the non-reduced regions in _finalize_offload_gradient_accumulation. The second would also fix an existing master issue where those stale grads are applied by the step but left out of the norm. Either way, a regression test with a conditionally-unused layer would lock it in.
2. [should-fix] Scope is wider than the description — stage3.py:3103
The description says fp16 and pin_memory: false keep the current path, but _grad_scale_for_cpu_adam has no dtype or pin gate: DeepSpeed's own fp16 (dynamic loss scale) and pin_memory: false both take the fused grad-scale path (the repro above is pin_memory: false). Only the copy-back half is bf16 + pinned. Could you either gate it, or update the description and add an fp16 case with a forced overflow showing the step and the loss-scale update are skipped and nothing is written?
3. [should-fix] step_pieces bypasses Optimizer.step — cpu_adam.py:194
It doesn't go through torch's wrapped Optimizer.step, so step pre/post hooks don't run, the profiler loses the Optimizer.step#DeepSpeedCPUAdam.step range, and _opt_called is never set. A torch LR scheduler built on the basic optimizer then warns that lr_scheduler.step() was called before optimizer.step(), and with ratio 1.0 + bf16 + pinned optimizer.step() is never called at all. Could this go through a step(..., pieces=...) overload, or run the hooks and mark the step manually?
4. [should-fix] Copy-back test fails on the CPU accelerator — test_zero.py:2098
With 2 CPU ranks the destination is never on a device, so assert calls, "no sub-group took the fused copy-back path" fails. It needs a skip when get_accelerator().device_name() == "cpu" (same issue as in #8840). TestZero3CpuAdamGradScale is fine on CPU.
5. [question] ARM — cpu_adam_impl.cpp:274
On SVE builds (Grace) adam_copy_supported() is false, so only the grad-scale half runs, via the new svmul_f32_x. The description lists ARM as unvalidated. I have a GH200 and can run test_cpu_adam_grad_scale_matches_scaling_in_place and test_cpu_adam_step_pieces_matches_step there if that helps.
6. [nit] src.is_pinned() here vs accelerator.is_pinned(src) in #8840, and the staging allocation duplicates _copy_through_pinned_staging; a shared helper would keep them in sync. Also _optimizer_step(..., grad_scale=...) is now always passed by keyword while SuperOffload's override keeps the old signature. That's safe today only because SuperOffload overrides step(); passing it only when not None would be more robust.
…PU Adam step Rebuilt on master after deepspeedai#8840 was merged. Review changes: step_pieces is replaced by DeepSpeedCPUAdam.step(grad_scale=..., bf16_copy=...), so the Optimizer.step hooks run. The scale factor of a fused step is applied later to the gradients that the next window does not overwrite, and before every fp32-gradient read, so parameters and gradients stay bitwise equal to master when a parameter gets no gradient. grad_scale_supported() is false on RVV. The copy-back test skips on the CPU accelerator. New tests for fp16 overflow, pin_memory off and unused layers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: pengdurice <pengduhit@gmail.com>
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 787f617047
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| piece_numel = self._cpu_adam_copy_piece_numel | ||
| if self._cpu_adam_copy_buffers is None: | ||
| self._cpu_adam_copy_buffers = [ | ||
| get_accelerator().pin_empty(piece_numel, dtype=torch.bfloat16) for _ in range(2) |
There was a problem hiding this comment.
Use an MPS-safe allocator for copy-back buffers
When ZeRO-3 bf16 CPU offload runs on MPS with offload_optimizer.pin_memory=true and the default torch pin backend, _can_copy_back_from_cpu_adam() selects this NEON copy path, but MPS_Accelerator only makes _torch_pin_memory() a no-op; pin_empty() instead falls through to DeepSpeedAccelerator._torch_empty_pinned(..., pin_memory=True), which MPS explicitly cannot provide. The first optimizer step therefore raises during scratch allocation rather than training; use the MPS no-op allocation or gate this path for these buffers.
AGENTS.md reference: AGENTS.md:L19-L19
Useful? React with 👍 / 👎.
tohtana
left a comment
There was a problem hiding this comment.
Looks good to me, thank you!
ZeRO-3 CPU offload: apply the clip factor and write the bf16 copy inside the CPU Adam pass
Builds on: #8840 (merged).
Part of: #8672 (Q4 2026 roadmap), "Enable training of large scale models".
Summary
With the optimizer offloaded to the CPU, ZeRO-3's step on master makes three passes over every CPU sub-group:
grad.mul_), even when the factor is 1;DeepSpeedCPUAdam(reads gradients, parameters and both moments; writes parameters and both moments);PCIe, into an fp32 device temporary the size of the sub-group (3.73 GiB at the default
sub_group_sizeof 1e9).This PR folds 1 and 3 into 2. The CPU Adam kernel multiplies each gradient by the factor as it loads it. With
pin_memoryon and a bf16 device partition, it also writes the bf16 copy of each updated piece into one of two pinnedbuffers, and the buffer is copied to the device while the next piece is updated. Parameters after training, and
gradients read after
step(), are bitwise equal to master.The change
csrc/includes/cpu_adam.h_grad_scale,_params_copy,set_grad_scale,set_params_copy,params_copy_for: the step multiplies each loaded gradient by the factor and, when a copy buffer is set, stores the updated parameter in bf16 therecsrc/includes/sve.hcsrc/adam/cpu_adam_impl.cppds_adam_step_scaled,ds_adam_step_scaled_copy,ds_adam_copy_supported,ds_adam_grad_scale_supported,ds_adam_block_numelcsrc/adam/cpu_adam.cppadam_update_scaled,adam_update_scaled_copy,adam_copy_supported,adam_grad_scale_supported,adam_block_numeldeepspeed/ops/adam/cpu_adam.pystep(closure=None, grad_scale=1.0, bf16_copy=None):bf16_copy=(piece_numel, before_piece, after_piece)updates each parameter piece by piece;before_piecereturns the bf16 buffer for the piece andafter_piecestarts its copy. The piece size is rounded down to a multiple of the kernel's widest vector block (128 floats with AVX512), so every element takes the vector path.copy_supported(),grad_scale_supported()deepspeed/runtime/zero/stage3.py_grad_scale_for_cpu_adampasses the factor to CPU Adam instead ofmul_for CPU sub-groups;_cpu_adam_step_and_copy_backruns the pieces into two pinned bf16 buffers of 2^26 elements, waits for the previous copy out of a buffer before refilling it, and waits for the last piece before returning (the next forward's all-gathers do not wait for this stream);_apply_deferred_grad_scale(below)tests/unit/ops/adam/test_cpu_adam.pytests/unit/v1/zero/test_zero.pyTestZero3OffloadPinnedCopyBackturns off the fused copy-backGradients that are not overwritten. On master the offloaded fp32 gradient buffer is not cleared between steps.
A parameter that gets no gradient in a window (a skipped branch, an expert with no tokens) keeps its older gradient,
the step reads it again, and the in-place
mul_multiplies it by every step's factor. Because the fused step does notwrite the scaled gradient back, each fused sub-group records its factor. At the start of the next
step()(before theoverflow check), the slices that the new window did not overwrite are multiplied by it, which is the multiply master
did. Every function that reads or sets offloaded fp32 gradients (
safe_get_full_grad,safe_get_local_grad, the setfunctions,
get_fp32_grad_partitions,update_fp32_grad_for_param_vectorized) applies a recorded factor first. Whenevery parameter gets a gradient, this multiplies nothing.
Where the fused path runs. The clip factor is fused for CPU sub-groups with float32 gradients and master weights,
with DeepSpeed's bf16 or fp16 (including the dynamic loss scale) and
pin_memoryon or off. It is not used with atorch GradScaler, ZenFlow, NVMe offload or Muon, or on the RVV build (its kernel has no factor;
grad_scale_supported()is false). The bf16 copy also needspin_memoryon, a bf16 device partition and a CPU Adambuild with a bf16 store (AVX512, NEON and the scalar loop; AVX2, SVE and RVV keep the copy of #8840). SuperOffload
overrides
step()and is not affected. No configuration key changes.Measured
Qwen2.5-32B, real weights, PG-19 rows of 8,192 tokens, 1 node x 8 H200, ZeRO-3, all of the optimizer offloaded to the
CPU (
pin_memoryon), fp32 Adam states, bf16, activation checkpointing, 12 OpenMP threads per rank. Master and this PRalternate in one job on the same node; each run is 2 warm-up and 6 measured steps, and the time is the mean of the 6.
wall_clock_breakdownthe two master runs differ from each other (step 8: master 1.746211, 1.746275, 1.746414; this PR 1.746454,
1.746391, 1.746197).
optimizer_steptimer (rank 0, run 3, mean of the measured steps): 4414 ms on master, 4281 ms with this PR. It variesbetween steps from 4125 to 4686 ms on master and from 3737 to 4719 ms with this PR.
temporary of master's copy-back does not set the peak.
Earlier measurement, on the first version of this PR stacked on the first version of #8840 (master 5f266e2, before
#8840 dropped its staging buffers): GLM-5.2, real weights, 65,536-token rows, 1 node x 8 H200, first 8 layers, 20% of
the optimizer on the GPU: 15.95 s per step before, 14.61-14.67 s with both halves fused; 8 nodes x 8 H200, all 78
layers: 147.16 s before, 143.61 s after.
Tests
test_cpu_adam_grad_scale_matches_scaling_in_placestep(grad_scale=f)gives bitwise the parameters and moments ofgrad.mul_(f)thenstep()test_cpu_adam_bf16_copy_matches_stepstep(bf16_copy=...)with pieces of 1,000 elements (not a multiple of the vector block) and 524,289-element tensors gives bitwise the result of onestep(), and the bf16 copies equalparam.to(torch.bfloat16)test_cpu_adam_bf16_copy_runs_step_hooksstep(bf16_copy=...)runs the optimizer's step pre- and post-hooks, and a torchStepLRbuilt on it gives no "lr_scheduler.step()beforeoptimizer.step()" warningTestZero3CpuAdamGradScale(2 ranks; clip 1e-3 and 1e6; ratio 1.0 and 0.5;pin_memoryon and off)safe_get_full_gradafter them, bitwise equal to the in-place path; the fused path ran; factor below 1 when clipping, 1 otherwiseTestZero3CpuAdamCopyBack(2 ranks; ratio 1.0 and 0.5; skipped on the CPU accelerator)TestZero3CpuAdamUnusedLayer(2 ranks;pin_memoryon and off)TestZero3CpuAdamFp16Overflow(2 ranks)TestZero3CpuAdamUnusedLayerfails if the recorded factor is dropped without the multiply: the unused layer'sparameters then differ by up to 9.766e-04, and the gradients read after the last step by up to 5.429e-01 (2 H200;
4.961e-01 on the CPU accelerator with 2 ranks); the other layers' parameters stay bitwise equal.
TestZero3OffloadPinnedCopyBack(from #8840) counts the calls of #8840's copy. It now turns off the fused copy-back,because on AVX512 every bf16 CPU sub-group would take it; the fused copy-back is tested by
TestZero3CpuAdamCopyBack.Hardware: 1 node of 8 NVIDIA H200 (driver 580.126.09), Intel Xeon Platinum 8488C (AVX512), torch 2.8.0+cu126,
pytest 9.1.1:
(*) The 3 failures were in the tests:
TestZero3OffloadPinnedCopyBack[1.0] and [0.5] (above), andTestZero3CpuAdamFp16Overflow, which expected the first overflow to lower the loss scale although the defaulthysteresisis 2; it now setshysteresis: 1. Only test files changed after that run.After merging master
af5198984into the branch (it brings #8974, which changescpu_adam.py), on the same hardware:test_cpu_adam.py,test_invalid_grad_norm.pyand the five ZeRO-3 classes above: 177 passed, 5 skipped.The 5 skips in
test_cpu_adam.pyare the file's own fp16 cases. CPU accelerator (DS_ACCELERATOR=cpu, 2 ranks,gloo):
test_cpu_adam.py,test_invalid_grad_norm.pyand the five ZeRO-3 classes: 160 passed, 16 skipped (fp16 andcopies to a device).
pre-commit run --filespasses on every changed file.Compatibility
pin_memoryon and off, all of the optimizer on the CPU (ratio 1.0) and half (0.5)copy_supported()is false and the copy keeps the path of #8840; the clip factor is fused on every build except RVV)grad_scale_supported()is false there, so both halves keep the master pathRelated pull requests
pass, in separate kernels with asynchronous scheduling. This PR puts the two synchronous parts into the default
DeepSpeedCPUAdampath and does not schedule anything asynchronously, so it does not depend on Add ReFlow asynchronous CPU-offloaded optimizers for ZeRO-3 #8719 or replace it.go through
_offload_grad_partition_at_boundary, which records them here, so with both PRs every slice is overwrittenin each window and the deferred multiply does no work. The two PRs edit neighbouring lines in
stage3.py.unscale_and_clip_gradsfor bf16/fp16 gradients only; the fused path is fp32 only. Textual conflict oftwo lines (this PR moves the factor computation into
_combined_grad_scale).AI assistance
This change, its tests and this description were prepared with AI assistance (Claude).