RL from textual feedback

[zhujiem]

• [Arxiv'26] Reinforcement Learning via Self-Distillation

  GRPO只能采用scalar reward，信号很稀疏，浪费了一些高质量的textual feedback，比如跑的test cases或错误信息。本文提出SDPO，基于自蒸馏策略优化，即通过输入prompt+feedback作为teacher，只输入prompt作为student，跑OPD loss。其中feedback可以是自己生成的正确答案及当前的执行错误。不同之处是SDPO需要考虑对teacher进行EMA/inital加权，避免训练不稳定(漂移)。

• Text2grad: Reinforcement learning from natural language feedback.
• Optimizing generative ai by backpropagating language model feedback
• From words to rewards: Leveraging natural language for reinforcement learning
• Feedback descent: Open-ended text optimization via pairwise comparison
• Language models can learn from verbal feedback without scalar rewards.

[zhujiem]

Agent自进化/自主学习

Agent Loop

• [ICLR'23] ReAct: Synergizing Reasoning and Acting in Language Models

ReAct 提出将"推理"（思维链）与"行动"（调用外部工具）交错执行的统一范式：在同一个上下文中交替进行 Thought → Action → Observation，让推理链能引用环境反馈、并用行动获取真实信息。构成了Agent的基本任务循环。推理帮助规划行动、减少幻觉，行动则为推理提供接地与纠错。

• [NeurIPS'23] Reflexion: Language Agents with Verbal Reinforcement Learning

现有方法（如 ReAct）在处理编程、推理等复杂任务时，缺乏有效的自我评估与反馈机制：任务失败后只能在同一上下文中继续推理，残留的失败链会诱导模型重复同一错误，学习效率低。本文提出 Reflexion 框架，一种语言（verbal）强化学习方法：Agent 通过语言反馈自我改进，无需更新模型权重。
Reflexion 包含两类记忆：单次 trial 内的试错上下文（短期），以及跨 trials 持久化的反思记忆（长期）。基本思想是：每次尝试（episode）结束后先由评估器在结果级判定成败，若失败则生成策略级反思写入长期记忆，并以此条件化下一次尝试，形成"失败 → 反思 → 记忆 → 重试"的自主学习闭环。
与 ReAct 的分工：典型组合即 Reflexion 外层循环包裹 ReAct 内层执行。ReAct 是单次任务内的执行循环（Thought → Action → Observation，失败后沿原上下文继续推理，易顺着错误思路重复犯错）；Reflexion 是跨多次尝试的自我修正循环：尝试 → 失败 → 反思 → 重试。其关键洞察是：失败轨迹留在同一上下文中往往纠不回来，因此每次重试都开启一条干净的新轨迹，仅携带"别再做 X"的反思提示。

Prompt Optimization

• [Arxiv'24] TextGrad: Automatic "Differentiation" via Text
• [Nature'25] Optimizing Generative AI by Backpropagating Language Model Feedback

每个batch根据text feedback得到提示词的优化意见(梯度)，再结合minibatch的意见进行调整(gradient step)，再validation set验证是否有效，有效提升则保留，否则跳过。

• [ICLR'26] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

提出基于reflective text feedback的黑盒提示词优化方法，结合遗传算法和帕累托优化，通过选择candidate->繁殖mutates->评估evaluate->存档，在train set上保留最优prompt candidates再采样进行mutate，在train上评估是否有效，train上有效则替换原prompt，并在validation set上验证泛化性。

• [Arixv'25] Training-Free Group Relative Policy Optimization

Skills Evolution

• [Arxiv'26] WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

WikiSkill 提出一个框架，将 Agent 的技能演化与一个持久化知识库（wiki）协同进化，把原始的执行经验、积累的知识和可执行技能分离，并持续把经验固化为 wiki，供后续技能更新复用。相比已有工作，wiki作为一个经验知识库总结了agent执行pattern和完整的skills更新的成功/失败经验。

• EvoSkill

• SkillOpt

Harness Evolution

• Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity
• EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
• Continual harness: Online adaptation for self-improving foundation agents
• EXG: Self-evolving agents with experience graphs
• Meta-Harness
• HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Continual learning (RL from textual feedback)

• Text2grad: Reinforcement learning from natural language feedback

Memory-based augmentation

Context-and harness evoluation

• https://github.com/bojieli/ai-agent-book/blob/main/book/chapter9.md
• Agentic context engineering: Evolving contexts for self-improving language models
• Multi-agent design: Optimizing agents with better prompts and topologies
• Dynamic cheatsheet: Test-time learning with adaptive memory
• Meta context engineering via agentic skill evolution
• Learning from natural language feedback for personalized question answering
• Promptbridge: Cross-model prompt transfer for large language models
• promptolution: A unified, modular framework for prompt optimization
• Recursive multi-agent systems
• Adaptation of agentic AI: A survey of post-training, memory, and skills
• Understanding the challenges in iterative generative optimization with LLMs
• Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories
• Generalizable self-evolving memory for automatic prompt optimization
• Expanding the capabilities of reinforcement learning via text feedback
• UR2: Unify RAG and Reasoning through Reinforcement Learning

[zhujiem]

Agent Memory

• Memagent: Reshaping long-context llm with multi-conv rl-based memory agent
• [Arxiv'26] MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens

MSA 提出一种端到端可训练的稀疏注意力记忆框架，用文档级 RoPE 和 Top-k 路由检索，配合 KV 缓存压缩、MemoryParallel 推理引擎和 MemoryInterleave 多跳机制(类似多步CoT)。其中，文档处理是Independent Document Processing (文档间不互相 attend、采用独立RoPE编码)，可以并行。模型在 64K 文中训练，推理时外推到 100M tokens 的知识库(train-on-short / infer-on-long)。在offline阶段，MSA对每个文档进行一次前向提取chunk平均的KV状态(K̄, V̄, K̄ᴿ) ，缓存在CPU。在online推理阶段，通过文档级稀疏注意力select稀疏KV，加载到显存。它在长文本 QA 与 NIAH 基准上超越已有方法，100M 上下文仅衰减 8.8%。

• Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale

直接从上下文中做 in-context retrieval，业界已经有尝试，但是在 1M token（1 万 doc）量级的输入下并没有系统研究过。本文通过实验验证在 1M token 量级时因为上下文输入的干扰 token 太多，会出现注意力稀释现象——虽然 pre-softmax 分数最高的往往是 gold 文档（模型其实认对了），但无关文档对 softmax 分母的贡献随 N 增长更快，导致 softmax 之后 gold 的归一化注意力质量坍缩。因此关键不是纠正排序，而是放大信号，其中乘以 logN（SSMax）比较有效，而 sink 加分母基本无效。为了缓解这个问题，提出通过改进 length-aware softmax（SSMax 为主）和 Document-level sparse attention（用检索分数做 top-B 文档路由）的方式，提升检索召回率。引用：Scalable In-context Ranking with Generative Models，采用blockrank方式进行IC检索。

• [Arxiv'26] Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
• Memory Caching: RNNs with Growing Memory
• Metis: Memory foundation model
• TransMem: Transforming Hidden States into Memory for Large Language Models
• MEMO: Memory as a Model
• HISA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention
• InfLLM-V2 [4], MoBA [5], and SeerAttention [24]
• Nosa: Native and offloadable sparse attention
• Infllm-v2: Dense-sparse switchable attention for seamless short-to-long adaptation
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory