You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
• Add configurable extra_instructions for /ask and line-level /ask prompts.
• Thread extra_instructions into Jinja prompt templates and tool variables.
• Document usage and add tests validating prompt rendering behavior.
Diagram
graph TD
U(["User PR comment (/ask)"]) --> T["/ask tool"] --> R["Render Jinja prompt"] --> L{{"LLM"}}
C[("pr_questions config") ] --> T
P["Prompt templates"] --> R
subgraph Legend
direction LR
_u(["User"]) ~~~ _p["Process"] ~~~ _cfg[("Config")] ~~~ _ext{{"External"}}
end
Loading
High-Level Assessment
The following are alternative approaches to this PR:
1. Separate settings keys for PR-wide vs line-level ask
➕ Allows more granular control (different guardrails for code-line questions vs general PR questions).
➕ Avoids surprising behavior if line-level prompts need stricter/looser constraints.
➖ More config surface area and documentation overhead.
➖ Higher chance of misconfiguration / user confusion.
2. Shared prompt macro/include for the extra_instructions block
➕ Avoids duplicating the same conditional block across multiple prompt templates.
➕ Easier future edits to the wording/format of the extra instructions section.
➖ Requires introducing/standardizing a template include mechanism in the prompt-loading path (if not already present).
➖ Slightly more indirection for prompt authors.
Recommendation: The PR’s approach (single pr_questions.extra_instructions value injected into both PR-wide and line-level /ask system prompts) is a good default: minimal configuration, consistent behavior, and easy to override per comment via settings. If future users request different constraints for line-level vs PR-wide asks, consider splitting the config key; otherwise keep the simpler unified setting.
Files changed (7) +102 / -1
Enhancement (4) +21 / -1
pr_line_questions_prompts.tomlInject extra_instructions block into line-level /ask prompt+9/-0
Inject extra_instructions block into line-level /ask prompt
• Adds a Jinja conditional section that includes user-provided 'extra_instructions' in the system prompt when non-empty, keeping the prompt unchanged when empty.
pr_questions_prompts.tomlInject extra_instructions block into PR-wide /ask prompt+9/-0
Inject extra_instructions block into PR-wide /ask prompt
• Adds a Jinja conditional section to the system prompt to include 'extra_instructions' when provided, enabling custom guardrails and response shaping for '/ask'.
pr_questions.pyPass extra_instructions into PR-wide /ask template variables+1/-0
Pass extra_instructions into PR-wide /ask template variables
• Adds 'extra_instructions' to the variables used by the PR-wide questions tool so the prompt templates can render the optional extra instructions block.
test_pr_questions_helpers.pyAdd tests for extra_instructions prompt rendering+48/-0
Add tests for extra_instructions prompt rendering
• Introduces unit tests verifying that 'extra_instructions' text appears in the rendered system prompt when set and that the block is omitted when empty. Covers both PR-wide and line-level '/ask' prompt templates.
ask.mdDocument /ask configuration including extra_instructions+32/-0
Document /ask configuration including extra_instructions
• Adds a new "Configuration options" section describing 'extra_instructions', 'enable_help_text', and 'use_conversation_history'. Includes examples for setting 'extra_instructions' in '.pr_agent.toml' and overriding it in a PR comment.
1. Line prompt forces answering always✓ Resolved📎 Requirement gap⛨ Security
Description
The line-level /ask system prompt still includes unconditional “must always answer” language that
can conflict with extra_instructions intended to require refusals for disallowed questions (e.g.,
PR rating requests). This direct prompt conflict can undermine policy enforcement by causing the
model to answer when it should refuse.
+{%- if extra_instructions %}+++Extra instructions from the user:+======+{{ extra_instructions }}+======+{%- endif %}
Evidence
PR Compliance ID 6 requires extra_instructions to steer behavior, including reliably refusing
disallowed questions. The cited /ask system prompt includes an extra_instructions section, but
it is preceded by an unconditional requirement to always answer / not avoid answering, which creates
a direct conflict with refusal-oriented extra_instructions and can prevent refusal behavior from
being applied as required.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
The `/ask` system prompt (including the line-level prompt) contains unconditional language requiring the model to always answer questions / not avoid answering, which can contradict `extra_instructions` that are meant to restrict behavior and require refusal for disallowed question types.
## Issue Context
PR Compliance ID 6 expects `extra_instructions` to be enforceable and to steer behavior, including refusing disallowed requests (e.g., “rate this PR 1–10”) with an explanation. Because the current `/ask` prompt includes “must answer” instructions alongside an `extra_instructions` block, the unconditional “always answer” requirement can override or conflict with refusal requirements, undermining compliance for line-level questions.
## Fix Focus Areas
- pr_agent/settings/pr_line_questions_prompts.toml[4-16]
- pr_agent/settings/pr_questions_prompts.toml[4-15]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
2. Extra instructions logged 🐞 Bug⛨ Security
Description
PRQuestions.run() logs the entire pr_questions config dict; with the new
pr_questions.extra_instructions field, any text configured there will be included in debug log
artifacts.
The PR adds extra_instructions under [pr_questions], and the /ask implementation logs the entire
pr_questions section as debug artifacts, which will now include that new field's value.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
`PRQuestions.run()` logs `dict(get_settings().pr_questions)` as an artifact. After this PR, that dict includes `extra_instructions`, so any sensitive internal guidance placed there will be written to logs.
### Issue Context
This PR introduces `pr_questions.extra_instructions` to support policy-like instructions for `/ask`. Those instructions may be more sensitive than typical boolean/tuning settings.
### Fix Focus Areas
- pr_agent/tools/pr_questions.py[54-58]
- pr_agent/settings/configuration.toml[129-133]
### Implementation notes
- When building `relevant_configs`, omit `extra_instructions` entirely, or replace its value with a redacted form (e.g., `"<redacted>"` or `"<redacted: N chars>"`).
- Keep the rest of the config dump intact for debugging.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
3. No token budgeting line-ask 🐞 Bug☼ Reliability
Description
PR_LineQuestions renders and sends prompts directly to the model without any token-budget
enforcement; with the newly-added extra_instructions block, long instructions can push the request
over the model context limit and fail the AI call.
The PR adds extra_instructions into the line-level system prompt and passes it via self.vars,
but PR_LineQuestions._get_prediction() calls the model with fully-rendered prompts and does not
use any token-budgeting/clipping logic before chat_completion.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
`PR_LineQuestions._get_prediction()` sends `system_prompt`/`user_prompt` to `chat_completion` without any clipping/token budgeting. This PR adds `extra_instructions` into the line-question system prompt, increasing the chance the combined prompt exceeds the model context window.
### Issue Context
Unlike `PRQuestions`, the line-level flow does not use `get_pr_diff(...)` or any other pruning step before calling the model.
### Fix Focus Areas
- pr_agent/tools/pr_line_questions.py[31-45]
- pr_agent/tools/pr_line_questions.py[152-167]
- pr_agent/settings/pr_line_questions_prompts.toml[1-20]
### Implementation notes
Implement a pre-flight token budget check in `PR_LineQuestions._get_prediction()` (or just before it calls `chat_completion`):
- Count tokens for `system_prompt` + `user_prompt`.
- If over limit, clip least-critical fields first (e.g., `conversation_history`, then `full_hunk`, and/or cap `extra_instructions` length) using existing clipping utilities.
- Ensure the truncation is deterministic and logged (without dumping full sensitive content).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
In pr_agent/settings/configuration.toml the new pr_questions.extra_instructions line uses a
different spacing style than the surrounding keys in the same section. This creates unnecessary
formatting drift in pr_agent/settings TOML defaults.
Compliance ID 14 requires preserving TOML formatting in pr_agent/settings/*.toml. In the
[pr_questions] section, existing keys use key=value style (no spaces), while the newly added
extra_instructions = "" introduces a different formatting style within the same section.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
`pr_questions.extra_instructions` is added with inconsistent spacing around `=` compared to adjacent keys in the same TOML section, creating avoidable formatting drift.
## Issue Context
This file is under `pr_agent/settings/`, where formatting/ordering should be preserved as much as possible to reduce noisy diffs.
## Fix Focus Areas
- pr_agent/settings/configuration.toml[129-133]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
5. Test Jinja env mismatch 🐞 Bug⚙ Maintainability⭐ New
Description
_render_jinja_template() enables Jinja autoescape=True, but production prompt rendering does
not, so rendered prompts can differ when extra_instructions contains escapable characters (e.g.,
<, &, quotes). This reduces fidelity of the new prompt-rendering tests because they may pass
while production would render different system prompt text.
The unit test helper enables autoescaping, while the runtime code paths render the same templates
with a non-autoescaping Jinja environment; therefore, extra_instructions can be escaped in tests
but not in production.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
### Issue description
The prompt-rendering tests use a Jinja2 environment with `autoescape=True`, while production uses `Environment(undefined=StrictUndefined)` (default `autoescape=False`). This means tests can render `extra_instructions` differently than the actual runtime prompt.
Concrete example: if `extra_instructions` contains `<b>no</b>`, tests render `<b>no</b>` while production renders the literal `<b>no</b>`.
### Issue Context
Production prompt rendering happens in `PRQuestions._get_prediction`, `PR_LineQuestions._get_prediction`, and in `TokenHandler._get_system_user_tokens`, all creating `Environment(undefined=StrictUndefined)` without autoescape.
### Fix Focus Areas
- tests/unittest/test_pr_questions_helpers.py[21-26]
- pr_agent/tools/pr_questions.py[105-110]
- pr_agent/tools/pr_line_questions.py[152-158]
- pr_agent/algo/token_handler.py[88-94]
### Suggested fix
Update `_render_jinja_template()` in the tests to match production:
- Remove `autoescape=True` (or set `autoescape=False`).
(Alternatively, if you intentionally want autoescaping everywhere, then change production environments consistently and update any expectations accordingly.)
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Author self-review: I have reviewed the code review findings, and addressed the relevant ones.
1. Line prompt forces answering always 📎 Requirement gap⛨ Security⭐ New
Description
The line-level /ask system prompt still includes unconditional “must always answer” language that
can conflict with extra_instructions intended to require refusals for disallowed questions (e.g.,
PR rating requests). This direct prompt conflict can undermine policy enforcement by causing the
model to answer when it should refuse.
+{%- if extra_instructions %}+++Extra instructions from the user:+======+{{ extra_instructions }}+======+{%- endif %}
Evidence
PR Compliance ID 6 requires extra_instructions to steer behavior, including reliably refusing
disallowed questions. The cited /ask system prompt includes an extra_instructions section, but
it is preceded by an unconditional requirement to always answer / not avoid answering, which creates
a direct conflict with refusal-oriented extra_instructions and can prevent refusal behavior from
being applied as required.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
The `/ask` system prompt (including the line-level prompt) contains unconditional language requiring the model to always answer questions / not avoid answering, which can contradict `extra_instructions` that are meant to restrict behavior and require refusal for disallowed question types.
## Issue Context
PR Compliance ID 6 expects `extra_instructions` to be enforceable and to steer behavior, including refusing disallowed requests (e.g., “rate this PR 1–10”) with an explanation. Because the current `/ask` prompt includes “must answer” instructions alongside an `extra_instructions` block, the unconditional “always answer” requirement can override or conflict with refusal requirements, undermining compliance for line-level questions.
## Fix Focus Areas
- pr_agent/settings/pr_line_questions_prompts.toml[4-16]
- pr_agent/settings/pr_questions_prompts.toml[4-15]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
2. Extra instructions logged 🐞 Bug⛨ Security⭐ New
Description
PRQuestions.run() logs the entire pr_questions config dict; with the new
pr_questions.extra_instructions field, any text configured there will be included in debug log
artifacts.
The PR adds extra_instructions under [pr_questions], and the /ask implementation logs the entire
pr_questions section as debug artifacts, which will now include that new field's value.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
### Issue description
`PRQuestions.run()` logs `dict(get_settings().pr_questions)` as an artifact. After this PR, that dict includes `extra_instructions`, so any sensitive internal guidance placed there will be written to logs.
### Issue Context
This PR introduces `pr_questions.extra_instructions` to support policy-like instructions for `/ask`. Those instructions may be more sensitive than typical boolean/tuning settings.
### Fix Focus Areas
- pr_agent/tools/pr_questions.py[54-58]
- pr_agent/settings/configuration.toml[129-133]
### Implementation notes
- When building `relevant_configs`, omit `extra_instructions` entirely, or replace its value with a redacted form (e.g., `"<redacted>"` or `"<redacted: N chars>"`).
- Keep the rest of the config dump intact for debugging.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
3. No token budgeting line-ask 🐞 Bug☼ Reliability⭐ New
Description
PR_LineQuestions renders and sends prompts directly to the model without any token-budget
enforcement; with the newly-added extra_instructions block, long instructions can push the request
over the model context limit and fail the AI call.
The PR adds extra_instructions into the line-level system prompt and passes it via self.vars,
but PR_LineQuestions._get_prediction() calls the model with fully-rendered prompts and does not
use any token-budgeting/clipping logic before chat_completion.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
### Issue description
`PR_LineQuestions._get_prediction()` sends `system_prompt`/`user_prompt` to `chat_completion` without any clipping/token budgeting. This PR adds `extra_instructions` into the line-question system prompt, increasing the chance the combined prompt exceeds the model context window.
### Issue Context
Unlike `PRQuestions`, the line-level flow does not use `get_pr_diff(...)` or any other pruning step before calling the model.
### Fix Focus Areas
- pr_agent/tools/pr_line_questions.py[31-45]
- pr_agent/tools/pr_line_questions.py[152-167]
- pr_agent/settings/pr_line_questions_prompts.toml[1-20]
### Implementation notes
Implement a pre-flight token budget check in `PR_LineQuestions._get_prediction()` (or just before it calls `chat_completion`):
- Count tokens for `system_prompt` + `user_prompt`.
- If over limit, clip least-critical fields first (e.g., `conversation_history`, then `full_hunk`, and/or cap `extra_instructions` length) using existing clipping utilities.
- Ensure the truncation is deterministic and logged (without dumping full sensitive content).
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
In pr_agent/settings/configuration.toml the new pr_questions.extra_instructions line uses a
different spacing style than the surrounding keys in the same section. This creates unnecessary
formatting drift in pr_agent/settings TOML defaults.
Compliance ID 14 requires preserving TOML formatting in pr_agent/settings/*.toml. In the
[pr_questions] section, existing keys use key=value style (no spaces), while the newly added
extra_instructions = "" introduces a different formatting style within the same section.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
`pr_questions.extra_instructions` is added with inconsistent spacing around `=` compared to adjacent keys in the same TOML section, creating avoidable formatting drift.
## Issue Context
This file is under `pr_agent/settings/`, where formatting/ordering should be preserved as much as possible to reduce noisy diffs.
## Fix Focus Areas
- pr_agent/settings/configuration.toml[129-133]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
In pr_agent/settings/configuration.toml the new pr_questions.extra_instructions line uses a
different spacing style than the surrounding keys in the same section. This creates unnecessary
formatting drift in pr_agent/settings TOML defaults.
Compliance ID 14 requires preserving TOML formatting in pr_agent/settings/*.toml. In the
[pr_questions] section, existing keys use key=value style (no spaces), while the newly added
extra_instructions = "" introduces a different formatting style within the same section.
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
## Issue description
`pr_questions.extra_instructions` is added with inconsistent spacing around `=` compared to adjacent keys in the same TOML section, creating avoidable formatting drift.
## Issue Context
This file is under `pr_agent/settings/`, where formatting/ordering should be preserved as much as possible to reduce noisy diffs.
## Fix Focus Areas
- pr_agent/settings/configuration.toml[129-133]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
I kept extra_instructions = "" because that's how every other extra_instructions line is written in configuration.toml. Happy to change we need section-local formatting instead. Maybe what's worth a follow-up is adding an /ask example to additional_configurations.md if needed in the same pr?
Fixed the CodeQL alerts, the new prompt tests now render Jinja templates with autoescape=True. These are plain-text LLM prompts, so behavior is unchanged, also no html involved, these are local tests, still updated them to pass ci.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/ask didn’t support extra_instructions like /review and /improve. This adds it for PR-wide and line-level ask.
Set it in .pr_agent.toml or per comment, eg. to refuse certain questions (like “rate this PR 1–10”) and tell the user why.
Fixes #2473