[bug] Reflection sometimes says there is no code to review

[sayhiben]

Summary

During the REFLECT state, after the UI prints # what did I learn?, the LLM sometimes responds with a non-lesson along the lines of:

I don't have any code to review and pull lessons from

That response is not actionable, and because _do_reflect() saves any non-empty response, it can be persisted into lessons.md as if it were a useful lesson.

What I expected

The reflection step should consistently produce one concise, technical lesson for the next generation cycle, or fall back to a deterministic generic lesson if there is not enough useful context.

What happens

Intermittently, the model appears to treat the prompt as a code-review request but notices that no code was provided, so it responds that it cannot review or pull lessons from code. This is visible in the TinyProgrammer UI immediately after # what did I learn?.

Investigation

I checked upstream/main at 1d467c7fe986f5745ec6aef6b39af436ad619bc3.

In programmer/brain.py, _do_reflect() builds only a result string:

• success: Result: Success.
• failure: Result: Failed. Error: ...

Then it calls:

prompt = self.llm.build_reflection_prompt(result)

In llm/generator.py, build_reflection_prompt() asks the model to "Review this Python code execution" and focus on syntax, libraries, or logic errors, but the prompt contains no generated source code and, on successful runs, no details beyond Success..

This seems related to #26. That issue closed by removing the unused code parameter because the reflection system was intended to produce lightweight one-line lessons from the execution result rather than full code reviews. The current behavior suggests that at least some models still need either bounded source/context or a differently worded prompt; otherwise they sometimes produce a refusal/meta-response instead of a lesson.

Possible fixes

A few options that would preserve the lightweight reflection behavior:

• Include a bounded excerpt of self.current_program.code in the reflection prompt, with an instruction to return only one short lesson.
• Include fuller structured context, such as program type, success/failure, error message, and maybe the active canvas constraints.
• If source code should intentionally remain excluded, reword the prompt so it asks for a lesson from the execution outcome only and explicitly says not to complain about missing code.
• Add a guard in _do_reflect() so obvious non-lessons like "I don't have code to review" are not saved into lessons.md, and optionally replace them with a fallback lesson.

Repro notes

This is intermittent and model-dependent. The simplest way to observe it is to let TinyProgrammer complete normal cycles until it reaches REFLECT, then watch the response after # what did I learn?.

[Michael3Dee]

Since the LLM doesn't get any feedback, how should it learn? Maybe favorite button, screenshot/gif as indicators of interest ("well done"). You could also take a screenshot 5 seconds after start and 115 seconds after start, compare for difference and assume more change = better.

[cuneytozseker]

Fixed in f42fe69 — build_reflection_prompt now takes code and program_type alongside result, and _do_reflect() passes the current program's code and type into it. After a few days of production runs on both Pi devices, the "I don't have any code to review" refusals stopped appearing in lessons.md and recent entries are concretely grounded in the actual program (e.g., "The tiny_canvas library does not have a move() method", "Use c.sleep() instead of time.sleep() to remain compatible with the TinyCanvas API").

PR #65 remains valuable as defense-in-depth (non-lesson guard + fallback handling), but the minimum-viable approach of providing source context turned out to be sufficient for the symptom described here.