Fluent Alignment with Disfluent Judges

Code and data for Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages (ICLR 2026). We post-train NorMistral-11B for Norwegian without any Norwegian instruction data: a short English SFT stage on LIMA, followed by on-policy RL on machine-translated prompts with an LLM-as-a-judge reward. The policy only ever trains on its own samples, so it stays fluent even when the judge is not.

• data/ – No Robots and LIMA, original and machine-translated (Tower-Plus-72B, plus the MT systems of Section 4.5), with the download and translation scripts
• sft/ – supervised finetuning (English SFT stage, translated-SFT baseline) and the chat template of Appendix E.1
• rl/ – distributed on-policy RL with the judge prompt of Appendix E.2
• evaluation/ – generation, LLM-as-a-judge evaluation (NorRewrite, NorSummarize, NorQuAD) and fluency scoring; evaluation/outputs*/ contain the responses generated by the three compared models for these benchmarks and for the human annotation

Usage

The *.sh files are the SLURM scripts used on LUMI; fill in the <PROJECT_ID> placeholders and your own environment (requirements.txt). Default hyperparameters follow Appendix C.

# 1. English SFT
cd sft && sbatch train.sh --data_path ../data/lima_english.jsonl --output_dir checkpoints/english_sft

# 2a. translated SFT (baseline)
sbatch train.sh --model_path_or_name checkpoints/english_sft --data_path ../data/no_robots_nob_responses.jsonl --output_dir checkpoints/translated_sft --max_epochs 3

# 2b. on-policy RL (16 pipelines x 24 GPUs)
cd ../rl && sbatch run.sh --model_name ../sft/checkpoints/english_sft --judge_model_name mistralai/Mistral-Nemo-Instruct-2407

# evaluation
cd ../evaluation && python3 generate.py --model_name <checkpoint> --dataset norrewrite   # or norsummarize, norquad, fluency_prompts
python3 judge_pairwise.py --dataset norrewrite --predictions outputs/norrewrite_*.jsonl && python3 winrates.py --input norrewrite_judge_results.json

The human annotations are at ltg/normistral-fluency-annotation.

Citation

@inproceedings{samuel2026fluent,
  title     = {Fluent Alignment with Disfluent Judges: Post-training for lower-resource languages},
  author    = {Samuel, David and {\O}vrelid, Lilja and Velldal, Erik and Kutuzov, Andrey},
  booktitle = {International Conference on Learning Representations},
  year      = {2026}
}