Repository navigation
Conversation
This commit introduces ReFlow, an asynchronous CPU-offloaded optimizer path for DeepSpeed ZeRO-3. ReFlow reduces CPU optimizer overhead and overlaps bucket-wise CPU optimizer execution with GPU backward computation. Highlights: - ReFlow Adam/AdamW and Lion CPU optimizers. - BF16 gradient offload with SIMD-local FP32 promotion and CPU-side gradient accumulation. - Register-resident AVX/SIMD optimizer kernels. - Ready/Persist scheduling for overlapping CPU optimizer execution without FP32 optimizer-state rollback. - ReFlow configuration, unit/training tests, CI coverage, documentation, and tutorial. Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Optimizer remapping changes supported client semantics, public optimizer steps can access unregistered native state, and failed asynchronous commits may still be checkpointed.
Review effort: Balanced
Findings: 4
Open (9)
Adam remap drops maximize semantics · New ReflowCPULion inherits incompatible optimizer step · New ReflowCPULion inherits incompatible Lion step · New Checkpoint serialization ignores pending state update failures · New Lion remap drops fp32_optimizer_states · New NUMA validation runs on the wrong host · New Documentation disagrees with bucketwise cores default · New Reference documents incorrect worker core default · New Configuration reference omits public ReflowConfig fields · New
What changed in this PR
Adds Reflow, an asynchronous CPU-offloaded Adam/Lion path for ZeRO-3.
Changes:
- Adds bucket-wise asynchronous optimization, gradient accumulation, state persistence, and NUMA scheduling.
- Adds fused AVX kernels and accelerator builder integration.
- Adds configuration, documentation, and correctness/launcher tests.
| File | Description |
|---|---|
tests/unit/v1/zero/test_reflow.py |
Tests Reflow integration and correctness. |
tests/unit/v1/ops/test_reflow_cpu_optimizers.py |
Tests native optimizer kernels. |
tests/unit/launcher/test_ds_arguments.py |
Tests numactl fallback behavior. |
op_builder/xpu/cpu_adam.py |
Adds Reflow Adam sources. |
op_builder/supa/cpu_lion.py |
Adds Reflow Lion sources. |
op_builder/supa/cpu_adam.py |
Adds Reflow Adam sources. |
op_builder/sdaa/cpu_adam.py |
Adds Reflow Adam sources. |
op_builder/npu/cpu_lion.py |
Adds Reflow Lion sources. |
op_builder/npu/cpu_adam.py |
Adds Reflow Adam sources. |
op_builder/mps/cpu_adam.py |
Adds Reflow Adam sources. |
op_builder/mlu/cpu_adam.py |
Adds Reflow Adam sources. |
op_builder/hpu/cpu_adam.py |
Adds Reflow Adam sources. |
op_builder/cpu/cpu_adam.py |
Adds CPU Reflow sources. |
op_builder/cpu_lion.py |
Adds default Lion sources. |
op_builder/cpu_adam.py |
Adds default Adam sources. |
docs/code-docs/source/zero3.rst |
Exposes Reflow configuration API. |
docs/code-docs/source/optimizers.rst |
Publishes optimizer classes. |
docs/_tutorials/reflow.md |
Adds the Reflow tutorial. |
docs/_pages/config-json.md |
Documents Reflow configuration. |
docs/_data/navigation.yml |
Links the tutorial. |
deepspeed/utils/timer.py |
Adds current-stream synchronization. |
deepspeed/utils/numa.py |
Adds numactl permission fallback. |
deepspeed/runtime/zero/utils.py |
Registers supported optimizers. |
deepspeed/runtime/zero/partitioned_param_coordinator.py |
Orders all-gathers after Reflow writes. |
deepspeed/runtime/zero/config.py |
Adds compatibility validation. |
deepspeed/runtime/reflow/reflow_utils.py |
Implements affinity and range helpers. |
deepspeed/runtime/reflow/reflow_cpu_lion.py |
Implements Reflow Lion wrapper. |
deepspeed/runtime/reflow/reflow_cpu_adam.py |
Implements Reflow Adam wrapper. |
deepspeed/runtime/reflow/reflow_config.py |
Defines Reflow settings. |
deepspeed/runtime/reflow/reflow_stage3.py |
Implements asynchronous ZeRO-3 execution. |
deepspeed/runtime/reflow/README.md |
Documents internal design. |
deepspeed/runtime/reflow/engine.py |
Integrates and remaps optimizers. |
deepspeed/runtime/reflow/__init__.py |
Defines the Reflow package. |
deepspeed/runtime/engine.py |
Selects Reflow engine components. |
deepspeed/runtime/config.py |
Rejects incompatible configurations. |
csrc/lion/reflow_cpu_lion_impl.cpp |
Implements fused Lion kernels. |
csrc/lion/reflow_cpu_lion_bindings.cpp |
Exposes Lion native APIs. |
csrc/lion/cpu_lion.cpp |
Registers Reflow Lion bindings. |
csrc/includes/reflow_simd.h |
Adds SIMD conversion/store helpers. |
csrc/includes/reflow_cpu_lion.h |
Defines Reflow Lion kernels. |
csrc/includes/reflow_cpu_affinity.h |
Applies worker affinity. |
csrc/includes/reflow_cpu_adam.h |
Defines Reflow Adam kernels. |
csrc/includes/reflow_bindings.h |
Declares native bindings. |
csrc/adam/reflow_cpu_adam_impl.cpp |
Implements fused Adam kernels. |
csrc/adam/reflow_cpu_adam_bindings.cpp |
Exposes Adam native APIs. |
csrc/adam/cpu_adam.cpp |
Registers Reflow Adam bindings. |
ci/tests_fetcher.py |
Selects Reflow tests in CI. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
|
I’ve addressed all nine review comments in separate signed-off commits, including maximize support, optimizer and precision guards, checkpoint error handling, target-host NUMA checks and fallback, and documentation fixes. |
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
|
Hi @tjruwase, @tohtana, and @hwchen2017, Just following up on this PR. All previous review comments have been addressed, and the implementation is ready for review. Could you please take a look when you have time? I'd appreciate any feedback or guidance on the next steps. Thank you! |
|
@st-bang97 thanks for so much for your PR and even more for your patience. Apologies for the delay. Taking a look now. |
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
pengdurice
left a comment
There was a problem hiding this comment.
DeepSpeed will skip a step if the gradient from that step is invalid (e.g. nan or inf), is_invalid_grad_norm is used to check that. Does ReFlow call it and handle invalid clearly?
Thanks for pointing this out, @pengdurice! I checked the current implementation and confirmed that ReFlow handles FP16 gradient overflow but does not use "is_invalid_grad_norm()". I'll update the validation to use "combine_grad_norm_groups()" and "is_invalid_grad_norm()", consistent with the latest ZeRO-3 path, and add regression tests for NaN/Inf handling. I'm currently attending a conference, so I plan to push the changes next week. Thanks again for the review! |
No worries, take your time and enjoy your conference. BTW, does the current implementation fuse the invalidity check into the SIMD kernel? If not, do you think it is a good idea to do so to avoid the cost of python's is_invalid_grad_norm? Is that an overkill? |
Hi Peng, thanks for the detailed review and suggestions! 1. SIMD invalidity check That's a good suggestion! The invalidity check isn't currently fused into the SIMD kernel, but it could help reduce Python overhead. Since ReFlow computes gradient norms on the GPU and checks the resulting global norm separately, I'm not sure how much additional benefit SIMD fusion would provide. I think it's worth profiling and analyzing further before deciding whether to integrate it. I'll first update the invalidity handling to match the latest ZeRO-3 implementation. 2. Potential data race across micro-steps Good catch! The current implementation waits for the GPU copy to finish before CPU accumulation, but it doesn't explicitly guarantee that the accumulation completes before the next micro-step overwrites I haven't encountered this issue in my tests, but I agree that we should guarantee correctness regardless of execution timing. I'll add synchronization to protect buffer reuse while avoiding unnecessary blocking, along with a regression test. 3. AutoTP and AutoEP compatibility ReFlow was originally designed for dense models, and we've validated single- and multi-GPU dense training, but not specifically AutoTP or AutoEP. When we initially ported ReFlow to DeepSpeed ZeRO-3, we based the implementation on the existing ZeRO-3 code. Since #8670 was merged afterward, its changes, including One related optimization I'd like to mention is ReFlow's batched gradient-norm reduction. Instead of reducing each subgroup separately, it collects their squared norms into a GPU tensor and batches collective operations for subgroups sharing the same process group. This reduces the number of collective calls, particularly when there are many small subgroups, and could potentially benefit standard ZeRO-3 offload as well. However, this optimization was originally developed without AutoTP or AutoEP. I'd like to preserve it while incorporating the recent changes, and I'd appreciate your thoughts on whether this batching approach could be extended to these parallel configurations while maintaining correctness. I currently don't have access to a multi-GPU setup to validate these specific configurations, so I'd also appreciate some help with testing. I'll review the existing optimizer offload restrictions for AutoEP+AutoTP folding as well. Thanks again for your feedback! |
1. Bucket sizes and worker configurationWe reran ReFlow at c4b6660 with the same setup as before (4× B200, one Xeon 8559C socket, 12 cores/rank).
2. Hardware difference
I'm a bit confused about how you arrived at twice as many CPU cores per GPU. When running your experiments on 4× B200 GPUs, did you use two CPU sockets? In a standard DGX B200 system, 8 GPUs share 2 CPUs, so for a fair comparison, a 4-GPU run should use only one CPU socket, right? Put another way, when renting 4× B200 GPUs from a cloud provider, you would typically get half a node's CPU resources (one socket, 48 cores), rather than both sockets. I'm happy to be corrected if I'm missing something. 3. CPU-only optimizerTo clarify, SuperOffload results presented here uses |
|
Hi @xylian86, Thanks for running these additional experiments! It's interesting that the performance remains nearly unchanged across the different bucket sizes and worker configurations. 1. Hardware configurationLet me clarify our hardware setup. The GPU resources we received through a government-funded computing program were provided as full 8-GPU HGX B200 servers, rather than DGX systems. For the 4-GPU experiments, we used two GPUs from each NUMA node, utilizing the CPU resources across both sockets. Our system had two AMD EPYC 9365 CPUs, providing 72 physical cores in total, compared with 48 physical cores in your single-socket setup. I agree that your single-socket configuration is more representative of a typical 4-GPU deployment, particularly in cloud environments. I also looked more closely at the memory specifications and noticed another important difference between the two platforms:
Even at the single-socket level, the EPYC platform has approximately 1.71x the theoretical memory bandwidth of the Xeon platform. Given the memory-intensive nature of CPU optimizer computation, this difference is worth investigating. 2. Further investigationThanks also for clarifying that your SuperOffload results use Since tuning the bucket sizes and worker configuration had little impact, I'd like to take a closer look at the implementation itself. I'll compare our research implementation with the current DeepSpeed integration to check whether any performance optimizations were missed or changed during the porting process. In particular, I'd like to investigate the Ready-phase bottleneck and identify further optimization opportunities for ReFlow. I think the hardware differences and the implementation changes are both worth examining to better understand the performance gap. |
# ZeRO-3 CPU offload: apply the clip factor and write the bf16 copy inside the CPU Adam pass **Builds on:** deepspeedai#8840 (merged). **Part of:** deepspeedai#8672 (Q4 2026 roadmap), "Enable training of large scale models". ## Summary With the optimizer offloaded to the CPU, ZeRO-3's step on master makes three passes over every CPU sub-group: 1. multiply every fp32 gradient by the clip and loss-scale factor in place (`grad.mul_`), even when the factor is 1; 2. run `DeepSpeedCPUAdam` (reads gradients, parameters and both moments; writes parameters and both moments); 3. copy the fp32 master to the device and cast it there to bf16 (deepspeedai#8840): twice the bytes of the bf16 result over PCIe, into an fp32 device temporary the size of the sub-group (3.73 GiB at the default `sub_group_size` of 1e9). This PR folds 1 and 3 into 2. The CPU Adam kernel multiplies each gradient by the factor as it loads it. With `pin_memory` on and a bf16 device partition, it also writes the bf16 copy of each updated piece into one of two pinned buffers, and the buffer is copied to the device while the next piece is updated. Parameters after training, and gradients read after `step()`, are bitwise equal to master. ## The change | File | Lines | What | |---|---|---| | `csrc/includes/cpu_adam.h` | +74 -1 | `_grad_scale`, `_params_copy`, `set_grad_scale`, `set_params_copy`, `params_copy_for`: the step multiplies each loaded gradient by the factor and, when a copy buffer is set, stores the updated parameter in bf16 there | | `csrc/includes/sve.h` | +4 -1 | the SVE path takes the factor | | `csrc/adam/cpu_adam_impl.cpp` | +131 | `ds_adam_step_scaled`, `ds_adam_step_scaled_copy`, `ds_adam_copy_supported`, `ds_adam_grad_scale_supported`, `ds_adam_block_numel` | | `csrc/adam/cpu_adam.cpp` | +17 | bindings `adam_update_scaled`, `adam_update_scaled_copy`, `adam_copy_supported`, `adam_grad_scale_supported`, `adam_block_numel` | | `deepspeed/ops/adam/cpu_adam.py` | +62 -4 | `step(closure=None, grad_scale=1.0, bf16_copy=None)`: `bf16_copy=(piece_numel, before_piece, after_piece)` updates each parameter piece by piece; `before_piece` returns the bf16 buffer for the piece and `after_piece` starts its copy. The piece size is rounded down to a multiple of the kernel's widest vector block (128 floats with AVX512), so every element takes the vector path. `copy_supported()`, `grad_scale_supported()` | | `deepspeed/runtime/zero/stage3.py` | +143 -10 | `_grad_scale_for_cpu_adam` passes the factor to CPU Adam instead of `mul_` for CPU sub-groups; `_cpu_adam_step_and_copy_back` runs the pieces into two pinned bf16 buffers of 2^26 elements, waits for the previous copy out of a buffer before refilling it, and waits for the last piece before returning (the next forward's all-gathers do not wait for this stream); `_apply_deferred_grad_scale` (below) | | `tests/unit/ops/adam/test_cpu_adam.py` | +106 | kernel tests below | | `tests/unit/v1/zero/test_zero.py` | +218 | ZeRO-3 tests below; `TestZero3OffloadPinnedCopyBack` turns off the fused copy-back | **Gradients that are not overwritten.** On master the offloaded fp32 gradient buffer is not cleared between steps. A parameter that gets no gradient in a window (a skipped branch, an expert with no tokens) keeps its older gradient, the step reads it again, and the in-place `mul_` multiplies it by every step's factor. Because the fused step does not write the scaled gradient back, each fused sub-group records its factor. At the start of the next `step()` (before the overflow check), the slices that the new window did not overwrite are multiplied by it, which is the multiply master did. Every function that reads or sets offloaded fp32 gradients (`safe_get_full_grad`, `safe_get_local_grad`, the set functions, `get_fp32_grad_partitions`, `update_fp32_grad_for_param_vectorized`) applies a recorded factor first. When every parameter gets a gradient, this multiplies nothing. **Where the fused path runs.** The clip factor is fused for CPU sub-groups with float32 gradients and master weights, with DeepSpeed's bf16 or fp16 (including the dynamic loss scale) and `pin_memory` on or off. It is not used with a torch GradScaler, ZenFlow, NVMe offload or Muon, or on the RVV build (its kernel has no factor; `grad_scale_supported()` is false). The bf16 copy also needs `pin_memory` on, a bf16 device partition and a CPU Adam build with a bf16 store (AVX512, NEON and the scalar loop; AVX2, SVE and RVV keep the copy of deepspeedai#8840). SuperOffload overrides `step()` and is not affected. No configuration key changes. ## Measured Qwen2.5-32B, real weights, PG-19 rows of 8,192 tokens, 1 node x 8 H200, ZeRO-3, all of the optimizer offloaded to the CPU (`pin_memory` on), fp32 Adam states, bf16, activation checkpointing, 12 OpenMP threads per rank. Master and this PR alternate in one job on the same node; each run is 2 warm-up and 6 measured steps, and the time is the mean of the 6. | Run | Master | This PR | Change | |---|---|---|---| | 1 | 9.445 s per step | 9.077 s | −3.9% | | 2 | 9.544 s | 9.213 s | −3.5% | | 3, with `wall_clock_breakdown` | 9.910 s | 9.532 s | −3.8% | - Losses: equal at step 1 (0.022649) in all six runs. From step 2 they differ in the 4th to 6th digit, by as much as the two master runs differ from each other (step 8: master 1.746211, 1.746275, 1.746414; this PR 1.746454, 1.746391, 1.746197). - `optimizer_step` timer (rank 0, run 3, mean of the measured steps): 4414 ms on master, 4281 ms with this PR. It varies between steps from 4125 to 4686 ms on master and from 3737 to 4719 ms with this PR. - Peak device memory during a step: 34,658,109,952 bytes in all six runs. In this configuration the fp32 device temporary of master's copy-back does not set the peak. - Pinned host memory: the two bf16 buffers add 256 MiB per rank, allocated at the first fused copy-back. Earlier measurement, on the first version of this PR stacked on the first version of deepspeedai#8840 (master 5f266e2, before deepspeedai#8840 dropped its staging buffers): GLM-5.2, real weights, 65,536-token rows, 1 node x 8 H200, first 8 layers, 20% of the optimizer on the GPU: 15.95 s per step before, 14.61-14.67 s with both halves fused; 8 nodes x 8 H200, all 78 layers: 147.16 s before, 143.61 s after. ## Tests | Test | Checks | |---|---| | `test_cpu_adam_grad_scale_matches_scaling_in_place` | `step(grad_scale=f)` gives bitwise the parameters and moments of `grad.mul_(f)` then `step()` | | `test_cpu_adam_bf16_copy_matches_step` | `step(bf16_copy=...)` with pieces of 1,000 elements (not a multiple of the vector block) and 524,289-element tensors gives bitwise the result of one `step()`, and the bf16 copies equal `param.to(torch.bfloat16)` | | `test_cpu_adam_bf16_copy_runs_step_hooks` | `step(bf16_copy=...)` runs the optimizer's step pre- and post-hooks, and a torch `StepLR` built on it gives no "`lr_scheduler.step()` before `optimizer.step()`" warning | | `TestZero3CpuAdamGradScale` (2 ranks; clip 1e-3 and 1e6; ratio 1.0 and 0.5; `pin_memory` on and off) | ZeRO-3 with CPU offload: parameters after 3 steps, and gradients read with `safe_get_full_grad` after them, bitwise equal to the in-place path; the fused path ran; factor below 1 when clipping, 1 otherwise | | `TestZero3CpuAdamCopyBack` (2 ranks; ratio 1.0 and 0.5; skipped on the CPU accelerator) | pieces of 1,000 elements reuse both buffers many times per step; parameters bitwise equal to the separate passes; the fused copy-back ran | | `TestZero3CpuAdamUnusedLayer` (2 ranks; `pin_memory` on and off) | a layer used only in step 0: every sub-group takes the fused path, and parameters and gradients after 3 steps are bitwise equal to the in-place path | | `TestZero3CpuAdamFp16Overflow` (2 ranks) | fp16 with a dynamic loss scale and an inf loss in step 0: the step is skipped and the loss scale lowered on both paths, and the parameters after the later fused steps are bitwise equal | `TestZero3CpuAdamUnusedLayer` fails if the recorded factor is dropped without the multiply: the unused layer's parameters then differ by up to 9.766e-04, and the gradients read after the last step by up to 5.429e-01 (2 H200; 4.961e-01 on the CPU accelerator with 2 ranks); the other layers' parameters stay bitwise equal. `TestZero3OffloadPinnedCopyBack` (from deepspeedai#8840) counts the calls of deepspeedai#8840's copy. It now turns off the fused copy-back, because on AVX512 every bf16 CPU sub-group would take it; the fused copy-back is tested by `TestZero3CpuAdamCopyBack`. Hardware: 1 node of 8 NVIDIA H200 (driver 580.126.09), Intel Xeon Platinum 8488C (AVX512), torch 2.8.0+cu126, pytest 9.1.1: ``` pytest unit/ops/adam/test_cpu_adam.py # 138 passed, 5 skipped pytest unit/v1/zero/test_zero.py # 120 passed, 1 skipped, 3 failed (*) pytest unit/v1/zero/test_zero.py::TestZero3OffloadPinnedCopyBack \ unit/v1/zero/test_zero.py::TestZero3CpuAdam* (4 classes) # 15 passed, after the fix (*) pytest unit/v1/zero/test_invalid_grad_norm.py # 18 passed pytest unit/runtime/zero/test_zero_offloadpp.py unit/v1/pin_memory/ # 53 passed, 1 skipped pytest unit/runtime/zenflow/test_zf.py # 95 passed ``` (*) The 3 failures were in the tests: `TestZero3OffloadPinnedCopyBack` [1.0] and [0.5] (above), and `TestZero3CpuAdamFp16Overflow`, which expected the first overflow to lower the loss scale although the default `hysteresis` is 2; it now sets `hysteresis: 1`. Only test files changed after that run. After merging master `af5198984` into the branch (it brings deepspeedai#8974, which changes `cpu_adam.py`), on the same hardware: `test_cpu_adam.py`, `test_invalid_grad_norm.py` and the five ZeRO-3 classes above: 177 passed, 5 skipped. The 5 skips in `test_cpu_adam.py` are the file's own fp16 cases. CPU accelerator (`DS_ACCELERATOR=cpu`, 2 ranks, gloo): `test_cpu_adam.py`, `test_invalid_grad_norm.py` and the five ZeRO-3 classes: 160 passed, 16 skipped (fp16 and copies to a device). `pre-commit run --files` passes on every changed file. ## Compatibility | Cell | Validated | |---|---| | ZeRO-3, CPU optimizer offload, bf16, `pin_memory` on and off, all of the optimizer on the CPU (ratio 1.0) and half (0.5) | yes (tests; Qwen2.5-32B on 1 node) | | fp16 with a dynamic loss scale, including an overflow step | yes (test) | | Parameters without a gradient in a step | yes (test) | | AVX512 build (Intel Xeon Platinum 8488C, the H200 nodes) | yes | | NEON / SVE builds, AVX2-only CPUs | no (AVX2 and SVE have no bf16 store in this kernel, so `copy_supported()` is false and the copy keeps the path of deepspeedai#8840; the clip factor is fused on every build except RVV) | | RVV | no; `grad_scale_supported()` is false there, so both halves keep the master path | | GradScaler, NVMe offload, ZenFlow, Muon, SuperOffload | path unchanged by design | | ZeRO-1 / ZeRO-2 | not touched | ## Related pull requests - deepspeedai#8719 (ReFlow) also fuses the gradient scaling into its CPU kernels and writes bf16 parameters without a separate pass, in separate kernels with asynchronous scheduling. This PR puts the two synchronous parts into the default `DeepSpeedCPUAdam` path and does not schedule anything asynchronously, so it does not depend on deepspeedai#8719 or replace it. - deepspeedai#8981 zeroes the gradients of parameters that got none in a window. All of its writes to the offloaded fp32 gradients go through `_offload_grad_partition_at_boundary`, which records them here, so with both PRs every slice is overwritten in each window and the deferred multiply does no work. The two PRs edit neighbouring lines in `stage3.py`. - deepspeedai#8970 changes `unscale_and_clip_grads` for bf16/fp16 gradients only; the fused path is fp32 only. Textual conflict of two lines (this PR moves the factor computation into `_combined_grad_scale`). ## AI assistance This change, its tests and this description were prepared with AI assistance (Claude). --------- Signed-off-by: pengdurice <pengduhit@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
@st-bang97 , feel free to fix the failing checks and then I can merge it. |
There was a problem hiding this comment.
🟡 Changes recommended
SIMD paths skip unscaling for valid FP16 loss scales below 1, causing incorrect Adam moments and potentially reversing Lion updates while scalar tails behave correctly.
1 open issue
9 resolved since last review
Checkpoint serialization ignores pending state update failures ReflowCPULion inherits incompatible Lion step ReflowCPULion inherits incompatible optimizer step Adam remap drops maximize semantics NUMA validation runs on the wrong host Lion remap drops fp32_optimizer_states Configuration reference omits public ReflowConfig fields Reference documents incorrect worker core default Documentation disagrees with bucketwise cores default
Review effort: Balanced
There was a problem hiding this comment.
1. Please give ReFlow its own builder instead of adding it to cpu_adam/cpu_lion.
ReFlow is opt-in, but this PR compiles it into the existing cpu_adam and cpu_lion extensions (bind_reflow_adam(m) / bind_reflow_lion(m)). ReFlow's kernels require AVX2/AVX-512, while cpu_adam must also build on CPUs without them (NEON, SVE, RVV, scalar). This false dependency means a ReFlow build problem on such CPUs would break DeepSpeedCPUAdam/DeepSpeedCPULion (and ZeRO-Offload) for users who never enable ReFlow. Please add separate ReflowCPUAdamBuilder/ReflowCPULionBuilder, with is_compatible() requiring AVX2/AVX-512, and leave cpu_adam/cpu_lion unchanged.
2. Don't add ReFlow to other accelerators' builders unless it has been verified there.
The PR changes builders under op_builder/{hpu,klxpu,mlu,mps,npu,sdaa,supa,xpu}. The validation listed in the description covers only NVIDIA GPUs with x86 CPUs, and apart from MPS no vendor workflow ran on this PR. The runtime side is also CUDA-specific in places, for example the GPU→NUMA detection via torch.cuda.get_device_properties`.
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
|
Hi @pengdurice and @hwchen2017, I've pushed 958fdc9 to address the recent feedback. ReFlow now has separate CPU optimizer builders, with the changes to other accelerator builders reverted. I also fixed the loss scaling issue and the previous CPU CI test failures, and added regression tests. The latest CI checks are currently marked Thanks! |
| from cpuinfo import get_cpu_info | ||
| cpu_info = get_cpu_info() | ||
| except Exception: | ||
| cpu_info = self._backup_cpuinfo() |
There was a problem hiding this comment.
This method no loner exist. Can you reuse the _cpu_info() function in builder.py?
There was a problem hiding this comment.
I will change this code.
|
@st-bang97 Also use |
|
Parameters that get no gradient in a step are trained on garbage (blocking). Plain ZeRO-3 now gives such parameters a zero gradient, a fix from #8981. ReFlow replaces the code that applies that fix, so it never runs. Each CPU buffer alternates between holding gradients and holding the newly computed weights. So a skipped parameter gets the previous step's weights applied as its gradient, and its GPU copy stops matching the FP32 master. In my repro the layer's FP32 master was off by 0.014 after one skipped step, and losses diverged one step later. The value the update read was bit-for-bit the layer's own weights. Gradient accumulation has the same problem when a parameter is first used after the first micro-step. MoE experts and conditional branches would hit this. @st-bang97 Please double check on this issue. |
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Run Ready and Persist on one CPU worker pool while preserving subgroup dependencies and worker affinity. Let gradient norm checks overlap Ready work and propagate worker failures before committing optimizer state. For unclipped, fixed-scale reentrant checkpoints, stage low-precision weights in the existing CPU buffer until recomputation and overflow checks finish. Publish disjoint shards as their workers complete and wait for the accumulated gradient before running Ready on shared weights. Cover checkpoint recomputation, local gradient edits, invalid steps and asynchronous publication against standard ZeRO-3. Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Use one optimizer worker per rank by default and share its OpenMP team across ready ranges. Publish each transfer chunk after its writers fence their streaming stores. Keep checkpointed ranges staged until publication is safe and make the H2D chunk size configurable. Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Gather low-precision shards within each parameter's partition group for the FP32 callback. Write the result to the CPU buffer used by Ready and Persist, refresh gradient norms and overflow, and coordinate callback failures before writing a shard. Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Share the local squared norm, clipping scale, overflow reduction and parameter restore paths with ZeRO-3. Let NVMe swap-out take the low-precision weights produced by Ready. Simplify Ready submissions and native bindings. Reuse the AVX-512 BF16 conversion for cached and streaming stores. Exercise both parameter and optimizer NVMe offload against standard ZeRO-3. Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
Allocate ReFlow's CPU gradient slots directly and reuse idle reduction bucket storage for accumulation, norms and parameter restoration. Track copies that borrow a bucket and retain their buffers through teardown. Keep the standard ZeRO-3 allocation and copy paths intact. Exercise delayed bucket reuse and pinned master restoration in regression tests. Signed-off-by: st-bang97 <st.bang@dgist.ac.kr>
|
Hi @pengdurice, @hwchen2017, and @xylian86, I've pushed several updates addressing the review feedback and improving ReFlow's correctness, compatibility, and performance. The main changes are:
I've also added extensive regression tests covering these changes and various edge cases. For now, I've intentionally kept the tests comprehensive and conservative to catch potential correctness issues. Once the implementation stabilizes, I think we can consolidate some of the tests while preserving the important coverage. Thanks again for the detailed feedback and suggestions! |



Implements #8697.
This PR adds ReFlow, an asynchronous CPU-offloaded optimizer path for DeepSpeed ZeRO-3. The motivation and detailed design are described in #8697.
ReFlow reduces the cost of the CPU-offloaded optimizer path and overlaps the remaining host-side work with GPU backward computation.
The ReFlow design and evaluation have been accepted to ASPLOS 2027 as “ReFlow: Exposing Parallelism in CPU-Offloaded LLM Training via Register-Resident SIMD Fusion and Decoupled Update Scheduling.”
Once the paper is publicly available, I will update this PR and #8697 with the paper link and publish a technical blog covering ReFlow's design, usage, and evaluation.
What this PR adds
zero_optimization.reflowconfiguration.Validation
ReFlow has been tested on:
Correctness testing covers Adam/AdamW and Lion, gradient accumulation, gradient clipping, overflow handling, checkpointing, learning-rate scheduling, high-precision optimizer-state access, and single-/multi-GPU execution.
Under deterministic GPU execution, ReFlow produces bit-identical training results to the baseline.
Representative performance results:
On a single A100 at 15B, ReFlow also achieves 1.7× higher throughput than ZeRO-Infinity.
In the standard AdamW evaluation, separate from the clipping-heavy experiment, ReFlow reduces the exposed optimizer update tail by more than 96.8% across the evaluated gradient-accumulation configurations and reduces total iteration time by up to 74.3%.
Under a separate clipping OPT-30B AdamW workload, ReFlow reduces the critical-path host update time from 2440 ms to 850 ms (65.2%), while requiring no FP32 optimizer-state rollback.
ReFlow is also optimizer-agnostic: with Lion, it achieves 1.6×–4.4× higher throughput than ZeRO-Infinity-Lion across 8 to 1 GPUs.
All modified files pass the DeepSpeed pre-commit checks.