Skip to content

Fused CPU Adam step - #8841

Merged
pengdurice merged 3 commits into
deepspeedai:masterfrom
pengdurice:peng-add-cpu-adam-fused-step-v1
Nov 13, 2026
Merged

pengdurice merged 3 commits into
deepspeedai:masterfrom
pengdurice:peng-add-cpu-adam-fused-step-v1

Conversation

@pengdurice

@pengdurice pengdurice commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

ZeRO-3 CPU offload: apply the clip factor and write the bf16 copy inside the CPU Adam pass

Builds on: #8840 (merged).
Part of: #8672 (Q4 2026 roadmap), "Enable training of large scale models".

Summary

With the optimizer offloaded to the CPU, ZeRO-3's step on master makes three passes over every CPU sub-group:

  1. multiply every fp32 gradient by the clip and loss-scale factor in place (grad.mul_), even when the factor is 1;
  2. run DeepSpeedCPUAdam (reads gradients, parameters and both moments; writes parameters and both moments);
  3. copy the fp32 master to the device and cast it there to bf16 (ZeRO-3: copy offloaded master weights back to the device through pinned buffer #8840): twice the bytes of the bf16 result over
    PCIe, into an fp32 device temporary the size of the sub-group (3.73 GiB at the default sub_group_size of 1e9).

This PR folds 1 and 3 into 2. The CPU Adam kernel multiplies each gradient by the factor as it loads it. With
pin_memory on and a bf16 device partition, it also writes the bf16 copy of each updated piece into one of two pinned
buffers, and the buffer is copied to the device while the next piece is updated. Parameters after training, and
gradients read after step(), are bitwise equal to master.

The change

File Lines What
csrc/includes/cpu_adam.h +74 -1 _grad_scale, _params_copy, set_grad_scale, set_params_copy, params_copy_for: the step multiplies each loaded gradient by the factor and, when a copy buffer is set, stores the updated parameter in bf16 there
csrc/includes/sve.h +4 -1 the SVE path takes the factor
csrc/adam/cpu_adam_impl.cpp +131 ds_adam_step_scaled, ds_adam_step_scaled_copy, ds_adam_copy_supported, ds_adam_grad_scale_supported, ds_adam_block_numel
csrc/adam/cpu_adam.cpp +17 bindings adam_update_scaled, adam_update_scaled_copy, adam_copy_supported, adam_grad_scale_supported, adam_block_numel
deepspeed/ops/adam/cpu_adam.py +62 -4 step(closure=None, grad_scale=1.0, bf16_copy=None): bf16_copy=(piece_numel, before_piece, after_piece) updates each parameter piece by piece; before_piece returns the bf16 buffer for the piece and after_piece starts its copy. The piece size is rounded down to a multiple of the kernel's widest vector block (128 floats with AVX512), so every element takes the vector path. copy_supported(), grad_scale_supported()
deepspeed/runtime/zero/stage3.py +143 -10 _grad_scale_for_cpu_adam passes the factor to CPU Adam instead of mul_ for CPU sub-groups; _cpu_adam_step_and_copy_back runs the pieces into two pinned bf16 buffers of 2^26 elements, waits for the previous copy out of a buffer before refilling it, and waits for the last piece before returning (the next forward's all-gathers do not wait for this stream); _apply_deferred_grad_scale (below)
tests/unit/ops/adam/test_cpu_adam.py +106 kernel tests below
tests/unit/v1/zero/test_zero.py +218 ZeRO-3 tests below; TestZero3OffloadPinnedCopyBack turns off the fused copy-back

Gradients that are not overwritten. On master the offloaded fp32 gradient buffer is not cleared between steps.
A parameter that gets no gradient in a window (a skipped branch, an expert with no tokens) keeps its older gradient,
the step reads it again, and the in-place mul_ multiplies it by every step's factor. Because the fused step does not
write the scaled gradient back, each fused sub-group records its factor. At the start of the next step() (before the
overflow check), the slices that the new window did not overwrite are multiplied by it, which is the multiply master
did. Every function that reads or sets offloaded fp32 gradients (safe_get_full_grad, safe_get_local_grad, the set
functions, get_fp32_grad_partitions, update_fp32_grad_for_param_vectorized) applies a recorded factor first. When
every parameter gets a gradient, this multiplies nothing.

Where the fused path runs. The clip factor is fused for CPU sub-groups with float32 gradients and master weights,
with DeepSpeed's bf16 or fp16 (including the dynamic loss scale) and pin_memory on or off. It is not used with a
torch GradScaler, ZenFlow, NVMe offload or Muon, or on the RVV build (its kernel has no factor;
grad_scale_supported() is false). The bf16 copy also needs pin_memory on, a bf16 device partition and a CPU Adam
build with a bf16 store (AVX512, NEON and the scalar loop; AVX2, SVE and RVV keep the copy of #8840). SuperOffload
overrides step() and is not affected. No configuration key changes.

Measured

Qwen2.5-32B, real weights, PG-19 rows of 8,192 tokens, 1 node x 8 H200, ZeRO-3, all of the optimizer offloaded to the
CPU (pin_memory on), fp32 Adam states, bf16, activation checkpointing, 12 OpenMP threads per rank. Master and this PR
alternate in one job on the same node; each run is 2 warm-up and 6 measured steps, and the time is the mean of the 6.

Run Master This PR Change
1 9.445 s per step 9.077 s −3.9%
2 9.544 s 9.213 s −3.5%
3, with wall_clock_breakdown 9.910 s 9.532 s −3.8%
  • Losses: equal at step 1 (0.022649) in all six runs. From step 2 they differ in the 4th to 6th digit, by as much as
    the two master runs differ from each other (step 8: master 1.746211, 1.746275, 1.746414; this PR 1.746454,
    1.746391, 1.746197).
  • optimizer_step timer (rank 0, run 3, mean of the measured steps): 4414 ms on master, 4281 ms with this PR. It varies
    between steps from 4125 to 4686 ms on master and from 3737 to 4719 ms with this PR.
  • Peak device memory during a step: 34,658,109,952 bytes in all six runs. In this configuration the fp32 device
    temporary of master's copy-back does not set the peak.
  • Pinned host memory: the two bf16 buffers add 256 MiB per rank, allocated at the first fused copy-back.

Earlier measurement, on the first version of this PR stacked on the first version of #8840 (master 5f266e2, before
#8840 dropped its staging buffers): GLM-5.2, real weights, 65,536-token rows, 1 node x 8 H200, first 8 layers, 20% of
the optimizer on the GPU: 15.95 s per step before, 14.61-14.67 s with both halves fused; 8 nodes x 8 H200, all 78
layers: 147.16 s before, 143.61 s after.

Tests

Test Checks
test_cpu_adam_grad_scale_matches_scaling_in_place step(grad_scale=f) gives bitwise the parameters and moments of grad.mul_(f) then step()
test_cpu_adam_bf16_copy_matches_step step(bf16_copy=...) with pieces of 1,000 elements (not a multiple of the vector block) and 524,289-element tensors gives bitwise the result of one step(), and the bf16 copies equal param.to(torch.bfloat16)
test_cpu_adam_bf16_copy_runs_step_hooks step(bf16_copy=...) runs the optimizer's step pre- and post-hooks, and a torch StepLR built on it gives no "lr_scheduler.step() before optimizer.step()" warning
TestZero3CpuAdamGradScale (2 ranks; clip 1e-3 and 1e6; ratio 1.0 and 0.5; pin_memory on and off) ZeRO-3 with CPU offload: parameters after 3 steps, and gradients read with safe_get_full_grad after them, bitwise equal to the in-place path; the fused path ran; factor below 1 when clipping, 1 otherwise
TestZero3CpuAdamCopyBack (2 ranks; ratio 1.0 and 0.5; skipped on the CPU accelerator) pieces of 1,000 elements reuse both buffers many times per step; parameters bitwise equal to the separate passes; the fused copy-back ran
TestZero3CpuAdamUnusedLayer (2 ranks; pin_memory on and off) a layer used only in step 0: every sub-group takes the fused path, and parameters and gradients after 3 steps are bitwise equal to the in-place path
TestZero3CpuAdamFp16Overflow (2 ranks) fp16 with a dynamic loss scale and an inf loss in step 0: the step is skipped and the loss scale lowered on both paths, and the parameters after the later fused steps are bitwise equal

TestZero3CpuAdamUnusedLayer fails if the recorded factor is dropped without the multiply: the unused layer's
parameters then differ by up to 9.766e-04, and the gradients read after the last step by up to 5.429e-01 (2 H200;
4.961e-01 on the CPU accelerator with 2 ranks); the other layers' parameters stay bitwise equal.

TestZero3OffloadPinnedCopyBack (from #8840) counts the calls of #8840's copy. It now turns off the fused copy-back,
because on AVX512 every bf16 CPU sub-group would take it; the fused copy-back is tested by TestZero3CpuAdamCopyBack.

Hardware: 1 node of 8 NVIDIA H200 (driver 580.126.09), Intel Xeon Platinum 8488C (AVX512), torch 2.8.0+cu126,
pytest 9.1.1:

pytest unit/ops/adam/test_cpu_adam.py                                   # 138 passed, 5 skipped
pytest unit/v1/zero/test_zero.py                                        # 120 passed, 1 skipped, 3 failed (*)
pytest unit/v1/zero/test_zero.py::TestZero3OffloadPinnedCopyBack \
       unit/v1/zero/test_zero.py::TestZero3CpuAdam*  (4 classes)        # 15 passed, after the fix (*)
pytest unit/v1/zero/test_invalid_grad_norm.py                           # 18 passed
pytest unit/runtime/zero/test_zero_offloadpp.py unit/v1/pin_memory/     # 53 passed, 1 skipped
pytest unit/runtime/zenflow/test_zf.py                                  # 95 passed

(*) The 3 failures were in the tests: TestZero3OffloadPinnedCopyBack [1.0] and [0.5] (above), and
TestZero3CpuAdamFp16Overflow, which expected the first overflow to lower the loss scale although the default
hysteresis is 2; it now sets hysteresis: 1. Only test files changed after that run.

After merging master af5198984 into the branch (it brings #8974, which changes cpu_adam.py), on the same hardware:
test_cpu_adam.py, test_invalid_grad_norm.py and the five ZeRO-3 classes above: 177 passed, 5 skipped.

The 5 skips in test_cpu_adam.py are the file's own fp16 cases. CPU accelerator (DS_ACCELERATOR=cpu, 2 ranks,
gloo): test_cpu_adam.py, test_invalid_grad_norm.py and the five ZeRO-3 classes: 160 passed, 16 skipped (fp16 and
copies to a device). pre-commit run --files passes on every changed file.

Compatibility

Cell Validated
ZeRO-3, CPU optimizer offload, bf16, pin_memory on and off, all of the optimizer on the CPU (ratio 1.0) and half (0.5) yes (tests; Qwen2.5-32B on 1 node)
fp16 with a dynamic loss scale, including an overflow step yes (test)
Parameters without a gradient in a step yes (test)
AVX512 build (Intel Xeon Platinum 8488C, the H200 nodes) yes
NEON / SVE builds, AVX2-only CPUs no (AVX2 and SVE have no bf16 store in this kernel, so copy_supported() is false and the copy keeps the path of #8840; the clip factor is fused on every build except RVV)
RVV no; grad_scale_supported() is false there, so both halves keep the master path
GradScaler, NVMe offload, ZenFlow, Muon, SuperOffload path unchanged by design
ZeRO-1 / ZeRO-2 not touched

Related pull requests

AI assistance

This change, its tests and this description were prepared with AI assistance (Claude).

…ed buffers

Signed-off-by: pengdurice <pengduhit@gmail.com>
…ide the CPU Adam pass

Signed-off-by: pengdurice <pengduhit@gmail.com>
@st-bang97

Copy link
Copy Markdown
Contributor

Hi @pengdurice,

It's great to see this PR addressing CPU optimizer overhead, as it closely aligns with our work on ReplayOpt and ReFlow (#8719).

Our research focuses on the memory bandwidth bottlenecks of CPU-offloaded optimizers. In ReplayOpt, accepted to IEEE Computer Architecture Letters (CAL), we explored reducing the critical-path overhead of optimizer-state updates. Building on this work, ReFlow further reduces memory traffic through register-resident SIMD fusion and overlaps CPU optimizer execution with GPU computation through decoupled scheduling.

In particular, ReFlow already implements the two main optimizations in this PR: fusing gradient scaling/clipping into CPU SIMD kernels and directly generating FP16/BF16 parameters without a separate conversion pass.

There are also opportunities for further optimization. For configurations using BF16 gradient accumulation, or without gradient accumulation, gradients can be transferred to the CPU in BF16 and promoted to FP32 only within SIMD registers during the optimizer update. This reduces not only PCIe traffic but also gradient-related CPU memory traffic, which is especially beneficial for memory-bandwidth-bound CPU optimizers. The in-register conversion is relatively inexpensive and avoids a separate FP32 conversion pass.

With these optimizations and asynchronous scheduling, ReFlow achieves up to 4.0× higher training throughput than ZeRO-Infinity on a single B200 and 1.86× on four B200 GPUs.

I originally implemented these optimizations in separate ReFlow kernels to minimize regression risks and preserve the stability of the existing DeepSpeed CPU optimizer path. However, since this PR is addressing the same underlying bottlenecks, I think this could be a good opportunity to incorporate ReFlow's non-asynchronous optimizations into the default CPU-offload path, while keeping asynchronous execution optional.

Would you be willing to review my PR (#8719) and share your thoughts on this direction? I'd greatly appreciate your feedback and would be happy to help with the integration and validation.

The ReFlow paper has also been accepted to ASPLOS 2027 and is expected to be publicly available soon. I'd be happy to share a copy directly if it would help with reviewing the design and evaluation.

Thank you for your time and consideration!

@0z5a 0z5a left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. Most of it checks out: the clip factor comes from the same folding-aware scaled_global_grad_norm (partition group → MP reduce → _autoep_expert_parallel_group), overflow returns before the sub-group loop, IncrementStep is idempotent across pieces, pieces are aligned to SIMD_WIDTH*8, and the bf16 store is RNE with NaN→0x7fc0 like torch. The new kernel tests pass bitwise on AVX-512 for me. Small thing: the first commit is #8840's head, so "Builds on #TODO" should read #8840.

One blocking issue and a few should-fix items below.

1. [blocking] Params with no gradient this step diverge from the unfused path — stage3.py:3022-3024

Skipping the in-place unscale_and_clip_grads has two side effects:

  • In offload mode a sub-group's fp32 grad buffer isn't zeroed between steps, and _finalize_offload_gradient_accumulation only refreshes params reduced in this window. On master, unscale_and_clip_grads rescales that stale region in place every step; on the fused path it keeps its raw loss-scaled, unclipped value and is only multiplied by the current factor. Repro on the CPU accelerator (1 rank, fp32 model, offload_optimizer: {device: cpu, pin_memory: false}, gradient_clipping: 1e-3, a layer used only on step 0, 3 steps): every parameter is bitwise equal between the fused and unfused paths except the unused layer, which differs by up to 5.9e-3. Under AutoEP this is any expert that receives no tokens in a step.
  • After step(), safe_get_full_grad now returns unscaled, unclipped grads (0.221 vs 2.97e-4 in the same repro).

Options: take the fused path only when every param of the sub-group was reduced this window (e.g. check against _offload_boundary_param_ids), or zero the non-reduced regions in _finalize_offload_gradient_accumulation. The second would also fix an existing master issue where those stale grads are applied by the step but left out of the norm. Either way, a regression test with a conditionally-unused layer would lock it in.

2. [should-fix] Scope is wider than the description — stage3.py:3103

The description says fp16 and pin_memory: false keep the current path, but _grad_scale_for_cpu_adam has no dtype or pin gate: DeepSpeed's own fp16 (dynamic loss scale) and pin_memory: false both take the fused grad-scale path (the repro above is pin_memory: false). Only the copy-back half is bf16 + pinned. Could you either gate it, or update the description and add an fp16 case with a forced overflow showing the step and the loss-scale update are skipped and nothing is written?

3. [should-fix] step_pieces bypasses Optimizer.step — cpu_adam.py:194

It doesn't go through torch's wrapped Optimizer.step, so step pre/post hooks don't run, the profiler loses the Optimizer.step#DeepSpeedCPUAdam.step range, and _opt_called is never set. A torch LR scheduler built on the basic optimizer then warns that lr_scheduler.step() was called before optimizer.step(), and with ratio 1.0 + bf16 + pinned optimizer.step() is never called at all. Could this go through a step(..., pieces=...) overload, or run the hooks and mark the step manually?

4. [should-fix] Copy-back test fails on the CPU accelerator — test_zero.py:2098

With 2 CPU ranks the destination is never on a device, so assert calls, "no sub-group took the fused copy-back path" fails. It needs a skip when get_accelerator().device_name() == "cpu" (same issue as in #8840). TestZero3CpuAdamGradScale is fine on CPU.

5. [question] ARM — cpu_adam_impl.cpp:274

On SVE builds (Grace) adam_copy_supported() is false, so only the grad-scale half runs, via the new svmul_f32_x. The description lists ARM as unvalidated. I have a GH200 and can run test_cpu_adam_grad_scale_matches_scaling_in_place and test_cpu_adam_step_pieces_matches_step there if that helps.

6. [nit] src.is_pinned() here vs accelerator.is_pinned(src) in #8840, and the staging allocation duplicates _copy_through_pinned_staging; a shared helper would keep them in sync. Also _optimizer_step(..., grad_scale=...) is now always passed by keyword while SuperOffload's override keeps the old signature. That's safe today only because SuperOffload overrides step(); passing it only when not None would be more robust.

…PU Adam step

Rebuilt on master after deepspeedai#8840 was merged. Review changes: step_pieces is replaced by DeepSpeedCPUAdam.step(grad_scale=..., bf16_copy=...), so the Optimizer.step hooks run. The scale factor of a fused step is applied later to the gradients that the next window does not overwrite, and before every fp32-gradient read, so parameters and gradients stay bitwise equal to master when a parameter gets no gradient. grad_scale_supported() is false on RVV. The copy-back test skips on the CPU accelerator. New tests for fp16 overflow, pin_memory off and unused layers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: pengdurice <pengduhit@gmail.com>
@pengdurice pengdurice changed the title Peng add cpu adam fused step v1 Fused CPU Adam step Nov 7, 2026
@pengdurice
pengdurice marked this pull request as ready for review November 9, 2026 17:19
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-09T17:26:06.429268Z 787f617 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 787f617047

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

piece_numel = self._cpu_adam_copy_piece_numel
if self._cpu_adam_copy_buffers is None:
self._cpu_adam_copy_buffers = [
get_accelerator().pin_empty(piece_numel, dtype=torch.bfloat16) for _ in range(2)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Use an MPS-safe allocator for copy-back buffers

When ZeRO-3 bf16 CPU offload runs on MPS with offload_optimizer.pin_memory=true and the default torch pin backend, _can_copy_back_from_cpu_adam() selects this NEON copy path, but MPS_Accelerator only makes _torch_pin_memory() a no-op; pin_empty() instead falls through to DeepSpeedAccelerator._torch_empty_pinned(..., pin_memory=True), which MPS explicitly cannot provide. The first optimizer step therefore raises during scratch allocation rather than training; use the MPS no-op allocation or gate this path for these buffers.

AGENTS.md reference: AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

@delock
delock self-requested a review November 10, 2026 09:03
@tohtana tohtana added the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 10, 2026

@tohtana tohtana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thank you!

@tohtana
tohtana added this pull request to the merge queue Nov 12, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 12, 2026
@pengdurice
pengdurice added this pull request to the merge queue Nov 13, 2026
Merged via the queue into deepspeedai:master with commit 84481c3 Nov 13, 2026
23 checks passed
@pengdurice
pengdurice deleted the peng-add-cpu-adam-fused-step-v1 branch November 13, 2026 07:07
@tohtana tohtana removed the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants