mismatched results of huggingface checkpoint on countdown

[ShijianXu]

Dear authors,

Thanks a lot for this nice work and I really like it.
However, I am having some problems with reproducing the Countdown results.

I loaded the released LoRA adapter (GSAI-ML/ESPO-Countdown), merged it onto GSAI-ML/LLaDA-8B-Instruct, and evaluated it using what I believe is your own eval protocol from the repo (eval/eval.py): remasking=low_confidence, diffusion_steps = gen_length // 2, block_length=32, temperature=0, on the 256-example dataset/countdown_cd3_test.jsonl test set (confirmed byte-identical to the copy in your repo).

My results:

Canvas	My accuracy	Table 1 (paper)
L=128	62.11%	81.6%
L=256	62.50%	82.0%
L=512	69.53%	79.3%

One thing I noticed that might be relevant: eval/run_eval.sh and the default path in eval/parse_and_get_acc.py both point at a checkpoint from a run named countdown_base_bs8_gspo_elbo_k2_mc2, step 9500, inside a directory literally called ablation/. That run name doesn't match what recipes/run_countdown_llada.sh itself would produce. Could you confirm whether the publicly released GSAI-ML/ESPO-Countdown adapter corresponds to that exact checkpoint/run, or to a different one? If it's a different (e.g., earlier or shorter) run than the one behind Table 1, that would explain the gap I'm seeing.

Thanks again for open-sourcing the code and checkpoints!
Best,

[JingyangOu]

Thank you for your attention to our work!

The original model was trained and evaluated on H-series GPUs. Unfortunately, I no longer have access to the original server or the corresponding checkpoints.

I just tried to reproduce the results on an A100 using GSAI-ML/LLaDA-8B-Instruct, together with the Countdown LoRA released on Hugging Face. The only change I made was renaming the checkpoint path in eval/run_eval.sh. The results are:

Setup (task_model_seqlen_diffusteps)         Accuracy      Avg Effective Tokens
--------------------------------------------------------------------------------
countdown_models_ESPO-Countdown_128_64         76.95%                    112.64
countdown_models_ESPO-Countdown_256_128        81.64%                    213.00
countdown_models_ESPO-Countdown_512_256        77.34%                    392.30

These results are slightly lower than those reported in the paper. This may be caused by differences in the hardware/software environment, which can lead to some inconsistency between training and inference. However, the performance drop in your reproduction is substantially larger than what I observe, and I do not fully understand the cause at the moment.

Regarding the checkpoint-path mismatch you mentioned: while organizing the repository, I renamed the model output directories to make the code structure and naming conventions more consistent. If training the model for 10,000 steps following the settings in the repository and evaluate the converged checkpoint, I believe you the results reported in the paper can be reproduced.