Skip to content

Add ReFlow asynchronous CPU-offloaded optimizers for ZeRO-3 - #8719

Open
st-bang97 wants to merge 26 commits into
deepspeedai:masterfrom
st-bang97:PR_branch
Open

st-bang97 wants to merge 26 commits into
deepspeedai:masterfrom
st-bang97:PR_branch

Conversation

@st-bang97

@st-bang97 st-bang97 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Implements #8697.

This PR adds ReFlow, an asynchronous CPU-offloaded optimizer path for DeepSpeed ZeRO-3. The motivation and detailed design are described in #8697.

ReFlow reduces the cost of the CPU-offloaded optimizer path and overlaps the remaining host-side work with GPU backward computation.

The ReFlow design and evaluation have been accepted to ASPLOS 2027 as “ReFlow: Exposing Parallelism in CPU-Offloaded LLM Training via Register-Resident SIMD Fusion and Decoupled Update Scheduling.”

Once the paper is publicly available, I will update this PR and #8697 with the paper link and publish a technical blog covering ReFlow's design, usage, and evaluation.

What this PR adds

  • ReFlow integration for ZeRO-3 CPU-offloaded training.
  • ReFlow CPU Adam/AdamW and Lion optimizers.
  • Fused AVX/SIMD optimizer kernels with BF16 gradient input and point-of-use FP32 promotion inside SIMD registers.
  • CPU-resident gradient accumulation for intermediate micro-steps, avoiding repeated blocking H2D gradient retrievals.
  • Final GPU-side accumulation and gradient-norm computation only at the terminal micro-step.
  • Asynchronous bucket-wise CPU optimizer execution overlapping GPU backward.
  • Ready/Persist scheduling that separates GPU-critical low-precision parameter generation from persistent FP32 optimizer-state updates.
  • Gradient clipping and overflow handling without FP32 optimizer-state rollback.
  • zero_optimization.reflow configuration.
  • Unit/training tests, CI coverage, configuration documentation, and tutorial.

Validation

ReFlow has been tested on:

  • NVIDIA B200
  • NVIDIA A100
  • NVIDIA RTX 5080
  • NVIDIA RTX A5000
  • NVIDIA RTX PRO 6000
  • AMD EPYC CPUs
  • Intel Xeon CPUs

Correctness testing covers Adam/AdamW and Lion, gradient accumulation, gradient clipping, overflow handling, checkpointing, learning-rate scheduling, high-precision optimizer-state access, and single-/multi-GPU execution.

Under deterministic GPU execution, ReFlow produces bit-identical training results to the baseline.

Representative performance results:

Workload ZeRO-Infinity ReFlow Improvement
OPT-30B, 1× B200 166.41 TFLOPS/GPU 671.29 TFLOPS/GPU 4.0×
OPT-30B, 4× B200 366.40 TFLOPS/GPU 681.18 TFLOPS/GPU 1.86×

On a single A100 at 15B, ReFlow also achieves 1.7× higher throughput than ZeRO-Infinity.

In the standard AdamW evaluation, separate from the clipping-heavy experiment, ReFlow reduces the exposed optimizer update tail by more than 96.8% across the evaluated gradient-accumulation configurations and reduces total iteration time by up to 74.3%.

Under a separate clipping OPT-30B AdamW workload, ReFlow reduces the critical-path host update time from 2440 ms to 850 ms (65.2%), while requiring no FP32 optimizer-state rollback.

ReFlow is also optimizer-agnostic: with Lion, it achieves 1.6×–4.4× higher throughput than ZeRO-Infinity-Lion across 8 to 1 GPUs.

All modified files pass the DeepSpeed pre-commit checks.

This commit introduces ReFlow, an asynchronous CPU-offloaded optimizer
path for DeepSpeed ZeRO-3. ReFlow reduces CPU optimizer overhead and
overlaps bucket-wise CPU optimizer execution with GPU backward
computation.

Highlights:
- ReFlow Adam/AdamW and Lion CPU optimizers.
- BF16 gradient offload with SIMD-local FP32 promotion and CPU-side
  gradient accumulation.
- Register-resident AVX/SIMD optimizer kernels.
- Ready/Persist scheduling for overlapping CPU optimizer execution
  without FP32 optimizer-state rollback.
- ReFlow configuration, unit/training tests, CI coverage, documentation,
  and tutorial.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Optimizer remapping changes supported client semantics, public optimizer steps can access unregistered native state, and failed asynchronous commits may still be checkpointed.

Review effort: Balanced
Findings: 4 High severity · 2 Medium severity · 3 Low severity

Open (9)
What changed in this PR

Adds Reflow, an asynchronous CPU-offloaded Adam/Lion path for ZeRO-3.

Changes:

  • Adds bucket-wise asynchronous optimization, gradient accumulation, state persistence, and NUMA scheduling.
  • Adds fused AVX kernels and accelerator builder integration.
  • Adds configuration, documentation, and correctness/launcher tests.
File Description
tests/​unit/​v1/​zero/​test_reflow.py Tests Reflow integration and correctness.
tests/​unit/​v1/​ops/​test_reflow_cpu_optimizers.py Tests native optimizer kernels.
tests/​unit/​launcher/​test_ds_arguments.py Tests numactl fallback behavior.
op_builder/​xpu/​cpu_adam.py Adds Reflow Adam sources.
op_builder/​supa/​cpu_lion.py Adds Reflow Lion sources.
op_builder/​supa/​cpu_adam.py Adds Reflow Adam sources.
op_builder/​sdaa/​cpu_adam.py Adds Reflow Adam sources.
op_builder/​npu/​cpu_lion.py Adds Reflow Lion sources.
op_builder/​npu/​cpu_adam.py Adds Reflow Adam sources.
op_builder/​mps/​cpu_adam.py Adds Reflow Adam sources.
op_builder/​mlu/​cpu_adam.py Adds Reflow Adam sources.
op_builder/​hpu/​cpu_adam.py Adds Reflow Adam sources.
op_builder/​cpu/​cpu_adam.py Adds CPU Reflow sources.
op_builder/​cpu_lion.py Adds default Lion sources.
op_builder/​cpu_adam.py Adds default Adam sources.
docs/​code-docs/​source/​zero3.rst Exposes Reflow configuration API.
docs/​code-docs/​source/​optimizers.rst Publishes optimizer classes.
docs/​_tutorials/​reflow.md Adds the Reflow tutorial.
docs/​_pages/​config-json.md Documents Reflow configuration.
docs/​_data/​navigation.yml Links the tutorial.
deepspeed/​utils/​timer.py Adds current-stream synchronization.
deepspeed/​utils/​numa.py Adds numactl permission fallback.
deepspeed/​runtime/​zero/​utils.py Registers supported optimizers.
deepspeed/​runtime/​zero/​partitioned_param_coordinator.py Orders all-gathers after Reflow writes.
deepspeed/​runtime/​zero/​config.py Adds compatibility validation.
deepspeed/​runtime/​reflow/​reflow_utils.py Implements affinity and range helpers.
deepspeed/​runtime/​reflow/​reflow_cpu_lion.py Implements Reflow Lion wrapper.
deepspeed/​runtime/​reflow/​reflow_cpu_adam.py Implements Reflow Adam wrapper.
deepspeed/​runtime/​reflow/​reflow_config.py Defines Reflow settings.
deepspeed/​runtime/​reflow/​reflow_stage3.py Implements asynchronous ZeRO-3 execution.
deepspeed/​runtime/​reflow/​README.md Documents internal design.
deepspeed/​runtime/​reflow/​engine.py Integrates and remaps optimizers.
deepspeed/​runtime/​reflow/​__init__.py Defines the Reflow package.
deepspeed/​runtime/​engine.py Selects Reflow engine components.
deepspeed/​runtime/​config.py Rejects incompatible configurations.
csrc/​lion/​reflow_cpu_lion_impl.cpp Implements fused Lion kernels.
csrc/​lion/​reflow_cpu_lion_bindings.cpp Exposes Lion native APIs.
csrc/​lion/​cpu_lion.cpp Registers Reflow Lion bindings.
csrc/​includes/​reflow_simd.h Adds SIMD conversion/store helpers.
csrc/​includes/​reflow_cpu_lion.h Defines Reflow Lion kernels.
csrc/​includes/​reflow_cpu_affinity.h Applies worker affinity.
csrc/​includes/​reflow_cpu_adam.h Defines Reflow Adam kernels.
csrc/​includes/​reflow_bindings.h Declares native bindings.
csrc/​adam/​reflow_cpu_adam_impl.cpp Implements fused Adam kernels.
csrc/​adam/​reflow_cpu_adam_bindings.cpp Exposes Adam native APIs.
csrc/​adam/​cpu_adam.cpp Registers Reflow Adam bindings.
ci/​tests_fetcher.py Selects Reflow tests in CI.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread deepspeed/runtime/reflow/engine.py Resolved
Comment thread deepspeed/runtime/reflow/engine.py Resolved
Comment thread deepspeed/utils/numa.py Resolved
Comment thread docs/_pages/config-json.md Resolved
Comment thread docs/_pages/config-json.md Resolved
Comment thread docs/_pages/config-json.md Resolved
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
@st-bang97

st-bang97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

I’ve addressed all nine review comments in separate signed-off commits, including maximize support, optimizer and precision guards, checkpoint error handling, target-host NUMA checks and fallback, and documentation fixes.
Please take another look when you have a chance.
Also, I added example file of Reflow with DeepSpeedExamples PR.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
@st-bang97

Copy link
Copy Markdown
Contributor Author

Hi @tjruwase, @tohtana, and @hwchen2017,

Just following up on this PR. All previous review comments have been addressed, and the implementation is ready for review.

Could you please take a look when you have time? I'd appreciate any feedback or guidance on the next steps.

Thank you!

@pengdurice pengdurice mentioned this pull request Oct 16, 2026
@st-bang97 st-bang97 mentioned this pull request Oct 18, 2026
34 tasks
@sfc-gh-truwase

Copy link
Copy Markdown
Collaborator

@st-bang97 thanks for so much for your PR and even more for your patience. Apologies for the delay. Taking a look now.

Comment thread deepspeed/launcher/multinode_runner.py Resolved
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>

@pengdurice pengdurice left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DeepSpeed will skip a step if the gradient from that step is invalid (e.g. nan or inf), is_invalid_grad_norm is used to check that. Does ReFlow call it and handle invalid clearly?

@st-bang97

Copy link
Copy Markdown
Contributor Author

DeepSpeed will skip a step if the gradient from that step is invalid (e.g. nan or inf), is_invalid_grad_norm is used to check that. Does ReFlow call it and handle invalid clearly?

Thanks for pointing this out, @pengdurice!

I checked the current implementation and confirmed that ReFlow handles FP16 gradient overflow but does not use "is_invalid_grad_norm()". I'll update the validation to use "combine_grad_norm_groups()" and "is_invalid_grad_norm()", consistent with the latest ZeRO-3 path, and add regression tests for NaN/Inf handling.

I'm currently attending a conference, so I plan to push the changes next week.

Thanks again for the review!

@pengdurice

Copy link
Copy Markdown
Contributor

DeepSpeed will skip a step if the gradient from that step is invalid (e.g. nan or inf), is_invalid_grad_norm is used to check that. Does ReFlow call it and handle invalid clearly?

Thanks for pointing this out, @pengdurice!

I checked the current implementation and confirmed that ReFlow handles FP16 gradient overflow but does not use "is_invalid_grad_norm()". I'll update the validation to use "combine_grad_norm_groups()" and "is_invalid_grad_norm()", consistent with the latest ZeRO-3 path, and add regression tests for NaN/Inf handling.

I'm currently attending a conference, so I plan to push the changes next week.

Thanks again for the review!

No worries, take your time and enjoy your conference. BTW, does the current implementation fuse the invalidity check into the SIMD kernel? If not, do you think it is a good idea to do so to avoid the cost of python's is_invalid_grad_norm? Is that an overkill?

@st-bang97

Copy link
Copy Markdown
Contributor Author

DeepSpeed will skip a step if the gradient from that step is invalid (e.g. nan or inf), is_invalid_grad_norm is used to check that. Does ReFlow call it and handle invalid clearly?

Thanks for pointing this out, @pengdurice!
I checked the current implementation and confirmed that ReFlow handles FP16 gradient overflow but does not use "is_invalid_grad_norm()". I'll update the validation to use "combine_grad_norm_groups()" and "is_invalid_grad_norm()", consistent with the latest ZeRO-3 path, and add regression tests for NaN/Inf handling.
I'm currently attending a conference, so I plan to push the changes next week.
Thanks again for the review!

No worries, take your time and enjoy your conference. BTW, does the current implementation fuse the invalidity check into the SIMD kernel? If not, do you think it is a good idea to do so to avoid the cost of python's is_invalid_grad_norm? Is that an overkill?

Hi Peng, thanks for the detailed review and suggestions!

1. SIMD invalidity check

That's a good suggestion! The invalidity check isn't currently fused into the SIMD kernel, but it could help reduce Python overhead.

Since ReFlow computes gradient norms on the GPU and checks the resulting global norm separately, I'm not sure how much additional benefit SIMD fusion would provide. I think it's worth profiling and analyzing further before deciding whether to integrate it.

I'll first update the invalidity handling to match the latest ZeRO-3 implementation.

2. Potential data race across micro-steps

Good catch! The current implementation waits for the GPU copy to finish before CPU accumulation, but it doesn't explicitly guarantee that the accumulation completes before the next micro-step overwrites src_slice.

I haven't encountered this issue in my tests, but I agree that we should guarantee correctness regardless of execution timing. I'll add synchronization to protect buffer reuse while avoiding unnecessary blocking, along with a regression test.

3. AutoTP and AutoEP compatibility

ReFlow was originally designed for dense models, and we've validated single- and multi-GPU dense training, but not specifically AutoTP or AutoEP.

When we initially ported ReFlow to DeepSpeed ZeRO-3, we based the implementation on the existing ZeRO-3 code. Since #8670 was merged afterward, its changes, including _grad_norm_counts_every_rank(), haven't been integrated into ReFlow yet. I'll update the implementation to incorporate these changes.

One related optimization I'd like to mention is ReFlow's batched gradient-norm reduction. Instead of reducing each subgroup separately, it collects their squared norms into a GPU tensor and batches collective operations for subgroups sharing the same process group. This reduces the number of collective calls, particularly when there are many small subgroups, and could potentially benefit standard ZeRO-3 offload as well.

However, this optimization was originally developed without AutoTP or AutoEP. I'd like to preserve it while incorporating the recent changes, and I'd appreciate your thoughts on whether this batching approach could be extended to these parallel configurations while maintaining correctness.

I currently don't have access to a multi-GPU setup to validate these specific configurations, so I'd also appreciate some help with testing. I'll review the existing optimizer offload restrictions for AutoEP+AutoTP folding as well.

Thanks again for your feedback!

@xylian86

Copy link
Copy Markdown
Collaborator

@st-bang97

1. Bucket sizes and worker configuration

We reran ReFlow at c4b6660 with the same setup as before (4× B200, one Xeon 8559C socket, 12 cores/rank).

reduce_bucket_size / sub_group_size, workers 32 × 2048 (8 per GPU) 16 × 4096 (4) 16 × 2048 (4)
4e8 / 4e8, 2 × 5 cores 6.98 7.00 6.96
2e8 / 2e8, 2 × 5 cores 6.97 7.00 6.96
1e8 / 1e8, 2 × 5 cores 6.99 7.03 7.00
4e8 / 4e8, 1 × 10 cores 7.00 7.03 7.00
SuperOffload 5.80 5.79 5.42

2. Hardware difference

One important difference is that our experimental system had roughly twice as many CPU cores available per GPU as your current setup, which likely contributed to the better overlap we observed.

I'm a bit confused about how you arrived at twice as many CPU cores per GPU. When running your experiments on 4× B200 GPUs, did you use two CPU sockets?

In a standard DGX B200 system, 8 GPUs share 2 CPUs, so for a fair comparison, a 4-GPU run should use only one CPU socket, right?

Put another way, when renting 4× B200 GPUs from a cloud provider, you would typically get half a node's CPU resources (one socket, 48 cores), rather than both sockets.

I'm happy to be corrected if I'm missing something.

3. CPU-only optimizer

To clarify, SuperOffload results presented here uses offload_optimizer.ratio=1.0 (heterogeneous optimizer computation disabled): all optimizer states and updates remain on the CPU, with no GPU optimizer computation.

@st-bang97

Copy link
Copy Markdown
Contributor Author

Hi @xylian86,

Thanks for running these additional experiments! It's interesting that the performance remains nearly unchanged across the different bucket sizes and worker configurations.

1. Hardware configuration

Let me clarify our hardware setup. The GPU resources we received through a government-funded computing program were provided as full 8-GPU HGX B200 servers, rather than DGX systems.

For the 4-GPU experiments, we used two GPUs from each NUMA node, utilizing the CPU resources across both sockets. Our system had two AMD EPYC 9365 CPUs, providing 72 physical cores in total, compared with 48 physical cores in your single-socket setup.

I agree that your single-socket configuration is more representative of a typical 4-GPU deployment, particularly in cloud environments.

I also looked more closely at the memory specifications and noticed another important difference between the two platforms:

Specification AMD EPYC 9365 Intel Xeon Platinum 8559C
Memory channels per socket 12 8
Maximum memory speed DDR5-6400 DDR5-5600
Theoretical bandwidth per socket 614.4 GB/s 358.4 GB/s

Even at the single-socket level, the EPYC platform has approximately 1.71x the theoretical memory bandwidth of the Xeon platform. Given the memory-intensive nature of CPU optimizer computation, this difference is worth investigating.

2. Further investigation

Thanks also for clarifying that your SuperOffload results use offload_optimizer.ratio=1.0, with all optimizer updates performed on the CPU.

Since tuning the bucket sizes and worker configuration had little impact, I'd like to take a closer look at the implementation itself.

I'll compare our research implementation with the current DeepSpeed integration to check whether any performance optimizations were missed or changed during the porting process. In particular, I'd like to investigate the Ready-phase bottleneck and identify further optimization opportunities for ReFlow.

I think the hardware differences and the implementation changes are both worth examining to better understand the performance gap.

@tohtana tohtana added the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 10, 2026
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Nov 13, 2026
# ZeRO-3 CPU offload: apply the clip factor and write the bf16 copy
inside the CPU Adam pass

**Builds on:** deepspeedai#8840 (merged).
**Part of:** deepspeedai#8672 (Q4 2026 roadmap), "Enable training of large scale
models".

## Summary

With the optimizer offloaded to the CPU, ZeRO-3's step on master makes
three passes over every CPU sub-group:

1. multiply every fp32 gradient by the clip and loss-scale factor in
place (`grad.mul_`), even when the factor is 1;
2. run `DeepSpeedCPUAdam` (reads gradients, parameters and both moments;
writes parameters and both moments);
3. copy the fp32 master to the device and cast it there to bf16 (deepspeedai#8840):
twice the bytes of the bf16 result over
PCIe, into an fp32 device temporary the size of the sub-group (3.73 GiB
at the default `sub_group_size` of 1e9).

This PR folds 1 and 3 into 2. The CPU Adam kernel multiplies each
gradient by the factor as it loads it. With
`pin_memory` on and a bf16 device partition, it also writes the bf16
copy of each updated piece into one of two pinned
buffers, and the buffer is copied to the device while the next piece is
updated. Parameters after training, and
gradients read after `step()`, are bitwise equal to master.

## The change

| File | Lines | What |
|---|---|---|
| `csrc/includes/cpu_adam.h` | +74 -1 | `_grad_scale`, `_params_copy`,
`set_grad_scale`, `set_params_copy`, `params_copy_for`: the step
multiplies each loaded gradient by the factor and, when a copy buffer is
set, stores the updated parameter in bf16 there |
| `csrc/includes/sve.h` | +4 -1 | the SVE path takes the factor |
| `csrc/adam/cpu_adam_impl.cpp` | +131 | `ds_adam_step_scaled`,
`ds_adam_step_scaled_copy`, `ds_adam_copy_supported`,
`ds_adam_grad_scale_supported`, `ds_adam_block_numel` |
| `csrc/adam/cpu_adam.cpp` | +17 | bindings `adam_update_scaled`,
`adam_update_scaled_copy`, `adam_copy_supported`,
`adam_grad_scale_supported`, `adam_block_numel` |
| `deepspeed/ops/adam/cpu_adam.py` | +62 -4 | `step(closure=None,
grad_scale=1.0, bf16_copy=None)`: `bf16_copy=(piece_numel, before_piece,
after_piece)` updates each parameter piece by piece; `before_piece`
returns the bf16 buffer for the piece and `after_piece` starts its copy.
The piece size is rounded down to a multiple of the kernel's widest
vector block (128 floats with AVX512), so every element takes the vector
path. `copy_supported()`, `grad_scale_supported()` |
| `deepspeed/runtime/zero/stage3.py` | +143 -10 |
`_grad_scale_for_cpu_adam` passes the factor to CPU Adam instead of
`mul_` for CPU sub-groups; `_cpu_adam_step_and_copy_back` runs the
pieces into two pinned bf16 buffers of 2^26 elements, waits for the
previous copy out of a buffer before refilling it, and waits for the
last piece before returning (the next forward's all-gathers do not wait
for this stream); `_apply_deferred_grad_scale` (below) |
| `tests/unit/ops/adam/test_cpu_adam.py` | +106 | kernel tests below |
| `tests/unit/v1/zero/test_zero.py` | +218 | ZeRO-3 tests below;
`TestZero3OffloadPinnedCopyBack` turns off the fused copy-back |

**Gradients that are not overwritten.** On master the offloaded fp32
gradient buffer is not cleared between steps.
A parameter that gets no gradient in a window (a skipped branch, an
expert with no tokens) keeps its older gradient,
the step reads it again, and the in-place `mul_` multiplies it by every
step's factor. Because the fused step does not
write the scaled gradient back, each fused sub-group records its factor.
At the start of the next `step()` (before the
overflow check), the slices that the new window did not overwrite are
multiplied by it, which is the multiply master
did. Every function that reads or sets offloaded fp32 gradients
(`safe_get_full_grad`, `safe_get_local_grad`, the set
functions, `get_fp32_grad_partitions`,
`update_fp32_grad_for_param_vectorized`) applies a recorded factor
first. When
every parameter gets a gradient, this multiplies nothing.

**Where the fused path runs.** The clip factor is fused for CPU
sub-groups with float32 gradients and master weights,
with DeepSpeed's bf16 or fp16 (including the dynamic loss scale) and
`pin_memory` on or off. It is not used with a
torch GradScaler, ZenFlow, NVMe offload or Muon, or on the RVV build
(its kernel has no factor;
`grad_scale_supported()` is false). The bf16 copy also needs
`pin_memory` on, a bf16 device partition and a CPU Adam
build with a bf16 store (AVX512, NEON and the scalar loop; AVX2, SVE and
RVV keep the copy of deepspeedai#8840). SuperOffload
overrides `step()` and is not affected. No configuration key changes.

## Measured

Qwen2.5-32B, real weights, PG-19 rows of 8,192 tokens, 1 node x 8 H200,
ZeRO-3, all of the optimizer offloaded to the
CPU (`pin_memory` on), fp32 Adam states, bf16, activation checkpointing,
12 OpenMP threads per rank. Master and this PR
alternate in one job on the same node; each run is 2 warm-up and 6
measured steps, and the time is the mean of the 6.

| Run | Master | This PR | Change |
|---|---|---|---|
| 1 | 9.445 s per step | 9.077 s | −3.9% |
| 2 | 9.544 s | 9.213 s | −3.5% |
| 3, with `wall_clock_breakdown` | 9.910 s | 9.532 s | −3.8% |

- Losses: equal at step 1 (0.022649) in all six runs. From step 2 they
differ in the 4th to 6th digit, by as much as
the two master runs differ from each other (step 8: master 1.746211,
1.746275, 1.746414; this PR 1.746454,
  1.746391, 1.746197).
- `optimizer_step` timer (rank 0, run 3, mean of the measured steps):
4414 ms on master, 4281 ms with this PR. It varies
between steps from 4125 to 4686 ms on master and from 3737 to 4719 ms
with this PR.
- Peak device memory during a step: 34,658,109,952 bytes in all six
runs. In this configuration the fp32 device
  temporary of master's copy-back does not set the peak.
- Pinned host memory: the two bf16 buffers add 256 MiB per rank,
allocated at the first fused copy-back.

Earlier measurement, on the first version of this PR stacked on the
first version of deepspeedai#8840 (master 5f266e2, before
deepspeedai#8840 dropped its staging buffers): GLM-5.2, real weights, 65,536-token
rows, 1 node x 8 H200, first 8 layers, 20% of
the optimizer on the GPU: 15.95 s per step before, 14.61-14.67 s with
both halves fused; 8 nodes x 8 H200, all 78
layers: 147.16 s before, 143.61 s after.

## Tests

| Test | Checks |
|---|---|
| `test_cpu_adam_grad_scale_matches_scaling_in_place` |
`step(grad_scale=f)` gives bitwise the parameters and moments of
`grad.mul_(f)` then `step()` |
| `test_cpu_adam_bf16_copy_matches_step` | `step(bf16_copy=...)` with
pieces of 1,000 elements (not a multiple of the vector block) and
524,289-element tensors gives bitwise the result of one `step()`, and
the bf16 copies equal `param.to(torch.bfloat16)` |
| `test_cpu_adam_bf16_copy_runs_step_hooks` | `step(bf16_copy=...)` runs
the optimizer's step pre- and post-hooks, and a torch `StepLR` built on
it gives no "`lr_scheduler.step()` before `optimizer.step()`" warning |
| `TestZero3CpuAdamGradScale` (2 ranks; clip 1e-3 and 1e6; ratio 1.0 and
0.5; `pin_memory` on and off) | ZeRO-3 with CPU offload: parameters
after 3 steps, and gradients read with `safe_get_full_grad` after them,
bitwise equal to the in-place path; the fused path ran; factor below 1
when clipping, 1 otherwise |
| `TestZero3CpuAdamCopyBack` (2 ranks; ratio 1.0 and 0.5; skipped on the
CPU accelerator) | pieces of 1,000 elements reuse both buffers many
times per step; parameters bitwise equal to the separate passes; the
fused copy-back ran |
| `TestZero3CpuAdamUnusedLayer` (2 ranks; `pin_memory` on and off) | a
layer used only in step 0: every sub-group takes the fused path, and
parameters and gradients after 3 steps are bitwise equal to the in-place
path |
| `TestZero3CpuAdamFp16Overflow` (2 ranks) | fp16 with a dynamic loss
scale and an inf loss in step 0: the step is skipped and the loss scale
lowered on both paths, and the parameters after the later fused steps
are bitwise equal |

`TestZero3CpuAdamUnusedLayer` fails if the recorded factor is dropped
without the multiply: the unused layer's
parameters then differ by up to 9.766e-04, and the gradients read after
the last step by up to 5.429e-01 (2 H200;
4.961e-01 on the CPU accelerator with 2 ranks); the other layers'
parameters stay bitwise equal.

`TestZero3OffloadPinnedCopyBack` (from deepspeedai#8840) counts the calls of
deepspeedai#8840's copy. It now turns off the fused copy-back,
because on AVX512 every bf16 CPU sub-group would take it; the fused
copy-back is tested by `TestZero3CpuAdamCopyBack`.

Hardware: 1 node of 8 NVIDIA H200 (driver 580.126.09), Intel Xeon
Platinum 8488C (AVX512), torch 2.8.0+cu126,
pytest 9.1.1:

```
pytest unit/ops/adam/test_cpu_adam.py                                   # 138 passed, 5 skipped
pytest unit/v1/zero/test_zero.py                                        # 120 passed, 1 skipped, 3 failed (*)
pytest unit/v1/zero/test_zero.py::TestZero3OffloadPinnedCopyBack \
       unit/v1/zero/test_zero.py::TestZero3CpuAdam*  (4 classes)        # 15 passed, after the fix (*)
pytest unit/v1/zero/test_invalid_grad_norm.py                           # 18 passed
pytest unit/runtime/zero/test_zero_offloadpp.py unit/v1/pin_memory/     # 53 passed, 1 skipped
pytest unit/runtime/zenflow/test_zf.py                                  # 95 passed
```

(*) The 3 failures were in the tests: `TestZero3OffloadPinnedCopyBack`
[1.0] and [0.5] (above), and
`TestZero3CpuAdamFp16Overflow`, which expected the first overflow to
lower the loss scale although the default
`hysteresis` is 2; it now sets `hysteresis: 1`. Only test files changed
after that run.

After merging master `af5198984` into the branch (it brings deepspeedai#8974, which
changes `cpu_adam.py`), on the same hardware:
`test_cpu_adam.py`, `test_invalid_grad_norm.py` and the five ZeRO-3
classes above: 177 passed, 5 skipped.

The 5 skips in `test_cpu_adam.py` are the file's own fp16 cases. CPU
accelerator (`DS_ACCELERATOR=cpu`, 2 ranks,
gloo): `test_cpu_adam.py`, `test_invalid_grad_norm.py` and the five
ZeRO-3 classes: 160 passed, 16 skipped (fp16 and
copies to a device). `pre-commit run --files` passes on every changed
file.

## Compatibility

| Cell | Validated |
|---|---|
| ZeRO-3, CPU optimizer offload, bf16, `pin_memory` on and off, all of
the optimizer on the CPU (ratio 1.0) and half (0.5) | yes (tests;
Qwen2.5-32B on 1 node) |
| fp16 with a dynamic loss scale, including an overflow step | yes
(test) |
| Parameters without a gradient in a step | yes (test) |
| AVX512 build (Intel Xeon Platinum 8488C, the H200 nodes) | yes |
| NEON / SVE builds, AVX2-only CPUs | no (AVX2 and SVE have no bf16
store in this kernel, so `copy_supported()` is false and the copy keeps
the path of deepspeedai#8840; the clip factor is fused on every build except RVV) |
| RVV | no; `grad_scale_supported()` is false there, so both halves keep
the master path |
| GradScaler, NVMe offload, ZenFlow, Muon, SuperOffload | path unchanged
by design |
| ZeRO-1 / ZeRO-2 | not touched |

## Related pull requests

- deepspeedai#8719 (ReFlow) also fuses the gradient scaling into its CPU kernels
and writes bf16 parameters without a separate
pass, in separate kernels with asynchronous scheduling. This PR puts the
two synchronous parts into the default
`DeepSpeedCPUAdam` path and does not schedule anything asynchronously,
so it does not depend on deepspeedai#8719 or replace it.
- deepspeedai#8981 zeroes the gradients of parameters that got none in a window.
All of its writes to the offloaded fp32 gradients
go through `_offload_grad_partition_at_boundary`, which records them
here, so with both PRs every slice is overwritten
in each window and the deferred multiply does no work. The two PRs edit
neighbouring lines in `stage3.py`.
- deepspeedai#8970 changes `unscale_and_clip_grads` for bf16/fp16 gradients only;
the fused path is fp32 only. Textual conflict of
two lines (this PR moves the factor computation into
`_combined_grad_scale`).

## AI assistance

This change, its tests and this description were prepared with AI
assistance (Claude).

---------

Signed-off-by: pengdurice <pengduhit@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@tohtana tohtana removed the needs-gpu-test GPU testing is recommended to validate the changes in this PR. label Nov 14, 2026
@pengdurice

Copy link
Copy Markdown
Contributor

@st-bang97 , feel free to fix the failing checks and then I can merge it.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread csrc/includes/reflow_cpu_adam.h Resolved

@hwchen2017 hwchen2017 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1. Please give ReFlow its own builder instead of adding it to cpu_adam/cpu_lion.
ReFlow is opt-in, but this PR compiles it into the existing cpu_adam and cpu_lion extensions (bind_reflow_adam(m) / bind_reflow_lion(m)). ReFlow's kernels require AVX2/AVX-512, while cpu_adam must also build on CPUs without them (NEON, SVE, RVV, scalar). This false dependency means a ReFlow build problem on such CPUs would break DeepSpeedCPUAdam/DeepSpeedCPULion (and ZeRO-Offload) for users who never enable ReFlow. Please add separate ReflowCPUAdamBuilder/ReflowCPULionBuilder, with is_compatible() requiring AVX2/AVX-512, and leave cpu_adam/cpu_lion unchanged.

2. Don't add ReFlow to other accelerators' builders unless it has been verified there.
The PR changes builders under op_builder/{hpu,klxpu,mlu,mps,npu,sdaa,supa,xpu}. The validation listed in the description covers only NVIDIA GPUs with x86 CPUs, and apart from MPS no vendor workflow ran on this PR. The runtime side is also CUDA-specific in places, for example the GPU→NUMA detection via torch.cuda.get_device_properties`.

@st-bang97

st-bang97 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Hi @pengdurice and @hwchen2017,

I've pushed 958fdc9 to address the recent feedback. ReFlow now has separate CPU optimizer builders, with the changes to other accelerator builders reverted. I also fixed the loss scaling issue and the previous CPU CI test failures, and added regression tests.

The latest CI checks are currently marked action_required. Could you help approve the workflows and take another look at the changes?

Thanks!

Comment thread op_builder/cpu/reflow.py Outdated
from cpuinfo import get_cpu_info
cpu_info = get_cpu_info()
except Exception:
cpu_info = self._backup_cpuinfo()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method no loner exist. Can you reuse the _cpu_info() function in builder.py?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will change this code.

@hwchen2017

Copy link
Copy Markdown
Contributor

@st-bang97 Also use pre-commit to fix the format issue.

@hwchen2017

Copy link
Copy Markdown
Contributor

Parameters that get no gradient in a step are trained on garbage (blocking). Plain ZeRO-3 now gives such parameters a zero gradient, a fix from #8981. ReFlow replaces the code that applies that fix, so it never runs. Each CPU buffer alternates between holding gradients and holding the newly computed weights. So a skipped parameter gets the previous step's weights applied as its gradient, and its GPU copy stops matching the FP32 master. In my repro the layer's FP32 master was off by 0.014 after one skipped step, and losses diverged one step later. The value the update read was bit-for-bit the layer's own weights. Gradient accumulation has the same problem when a parameter is first used after the first micro-step. MoE experts and conditional branches would hit this.

@st-bang97 Please double check on this issue.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Run Ready and Persist on one CPU worker pool while preserving subgroup
dependencies and worker affinity. Let gradient norm checks overlap Ready
work and propagate worker failures before committing optimizer state.

For unclipped, fixed-scale reentrant checkpoints, stage low-precision
weights in the existing CPU buffer until recomputation and overflow
checks finish. Publish disjoint shards as their workers complete and
wait for the accumulated gradient before running Ready on shared weights.

Cover checkpoint recomputation, local gradient edits, invalid steps and
asynchronous publication against standard ZeRO-3.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Use one optimizer worker per rank by default and share its OpenMP team across ready ranges. Publish each transfer chunk after its writers fence their streaming stores. Keep checkpointed ranges staged until publication is safe and make the H2D chunk size configurable.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Gather low-precision shards within each parameter's partition group for the FP32 callback. Write the result to the CPU buffer used by Ready and Persist, refresh gradient norms and overflow, and coordinate callback failures before writing a shard.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Share the local squared norm, clipping scale, overflow reduction and
parameter restore paths with ZeRO-3. Let NVMe swap-out take the
low-precision weights produced by Ready.

Simplify Ready submissions and native bindings. Reuse the AVX-512 BF16
conversion for cached and streaming stores. Exercise both parameter and
optimizer NVMe offload against standard ZeRO-3.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Allocate ReFlow's CPU gradient slots directly and reuse idle reduction
bucket storage for accumulation, norms and parameter restoration. Track
copies that borrow a bucket and retain their buffers through teardown.

Keep the standard ZeRO-3 allocation and copy paths intact. Exercise
delayed bucket reuse and pinned master restoration in regression tests.

Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
@st-bang97

Copy link
Copy Markdown
Contributor Author

Hi @pengdurice, @hwchen2017, and @xylian86,

I've pushed several updates addressing the review feedback and improving ReFlow's correctness, compatibility, and performance.

The main changes are:

  1. Correctness and compatibility: Thanks to @hwchen2017 for identifying the blocking issue with unused parameters. I've updated gradient handling to prevent stale data in reused CPU buffers from being interpreted as gradients, which could corrupt the FP32 master parameters and cause inconsistencies with GPU parameters. The changes also cover gradient accumulation, reentrant checkpointing, and overflow handling to align with recent ZeRO-3 updates. I also fixed the reported builder issues, including _cpu_info(), and addressed the formatting issues.

  2. Unified CPU workers: Thanks to @xylian86 for pointing out the potential contention between Ready and Persist. I tested this under deliberately increased contention and observed performance degradation when the two phases competed for CPU resources. Based on this, I consolidated Ready and Persist into a shared worker pool and improved Ready task batching to reuse the OpenMP worker team.

  3. Streaming stores: I found that the non-temporal streaming stores from the original ReFlow implementation had been lost during integration. On our original B200 setup, they provided only a few percent improvement. However, under heavier memory contention and more constrained memory bandwidth, I observed improvements exceeding 10%. I've restored streaming stores and added finer-grained synchronization and memory fences so that completed chunks can be safely transferred while subsequent Ready computation continues.

  4. Chunked H2D transfers: Inspired by the 16 MiB chunking approach in SuperOffload's PCIe optimization, ReFlow now transfers updated parameters to the GPU in configurable chunks (16 MiB by default). This provides more opportunities to overlap H2D transfers with CPU computation.

  5. Additional improvements: Reduced redundant gradient-buffer allocations and temporary copies, reused existing ZeRO-3 helpers for gradient norms, clipping, overflow handling, and parameter restoration, and improved gradient access and NVMe offload handling.

I've also added extensive regression tests covering these changes and various edge cases. For now, I've intentionally kept the tests comprehensive and conservative to catch potential correctness issues. Once the implementation stabilizes, I think we can consolidate some of the tests while preserving the important coverage.

Thanks again for the detailed feedback and suggestions!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants