[Copilot]

Pi evals read the main workflow’s restored user.txt and system.txt instead of the BinEval prompt, producing UNKNOWN answers while reporting success.

• Prompt handoff: Write the BinEval prompt to both prompt.txt and user.txt; clear system.txt to remove stale workflow instructions.
• Regression coverage: Seed restored Pi prompts and assert evals setup replaces them with the question-bearing BinEval prompt and an empty system prompt.

• Fixes Pi engine: evals job runs the main workflow prompt instead of the BinEval prompt (all answers UNKNOWN) #71663

[Copilot]

🟡 Changes recommended

A reproduced answer-parsing error can record results that contradict the Pi judge's response.

1 open issue

• Avoid echoed prompt overriding judge response during parsing · New

What changed in this PR

Fixes gh-aw’s Pi evaluation prompt handoff so the judge receives the BinEval questions instead of restored workflow instructions.

Changes:

• Write the BinEval prompt to both prompt files and clear the stale system prompt.
• Add regression coverage for restored Pi prompts and clean up prompt files after tests.

File	Description
actions/​setup/​js/​run_evals.test.cjs	Tests replacement of restored Pi prompts and adds cleanup.
actions/​setup/​js/​run_evals.cjs	Prepares combined and split prompts for evaluation.

_{Review effort: Max}

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (actions/setup/js/run_evals.cjs:109): Prompt injection risk: setupMain writes the full BinEval prompt (which includes raw, model-generated agent output from /tmp/gh-aw/evals/agent_output.json via buildEvalPrompt) into aw-prompts/user.txt for Pi. A malicious or compromised agent output can embed instructions that the judge model may follow, biasing eval answers; consider stronger data-only encapsulation/escaping for (agent_output) and/or deterministic parsing/validation of judge output to reduce instruction-following from agent output. - Fix Pi evals reading stale workflow prompts #71664 (comment)
3. Review (actions/setup/js/run_evals.cjs:109): Pi echoes this prompt into its JSON log, and parseMain searches that raw log as well as the judge's response. With question ID tests, agent output - Tests: Yes overrides a judge response of tests: NO, incorrectly recording YES. Prefer extracted assistant text over the raw log, retain the fallback for plain-text logs, and add a parse regression test with an echoed user message. - Fix Pi evals reading stale workflow prompts #71664 (comment)
4. Fix failing check JS Tests (shard 4/4) (FAILURE): https://github.com/github/gh-aw/actions/runs/40107398508/job/120912608541.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 8e3cce5
Sous-chef work: 31f7e812d45f0792093269b7b90445aa9803225a204ec130694b9ceb0227d7cc 8727a4b44d87215c119662bcebf74d5fa004e38c05ca57b166be8bbfe92fcb52 a753ecaa2b569ceba751deb39ce078e8345b4556c21563523110f8268580467b
Sous-chef state: d7e6ab39e48c6cfceba08dc4ef1ea88e06bac4f1e8ddf9dade4535f417a68780

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · ~11 AIC · ⌖ ~10 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[Copilot]

Added an explicit guard that treats the serialized agent output as untrusted evidence and tells the evaluator to ignore instructions inside it, while retaining the existing JSON escaping. The regression test covers both. Commit: 5332633.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (actions/setup/js/run_evals.cjs:113): Prompt injection risk: untrusted agent_output.json content (model-generated) is embedded into the BinEval judge prompt via buildEvalPrompt(..., sanitizeAgentOutputForPrompt(agentOutputContent)) without an explicit guard instructing the judge to treat (agent_output) as data and ignore any instructions inside it, so a crafted agent output could steer YES/NO/UNKNOWN answers. - Fix Pi evals reading stale workflow prompts #71664 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 38daf54
Sous-chef work: df0e533b81a7da1348834f35972c5e362da19941727679fdae21b3c4e89e000d
Sous-chef state: 61055413b2fa01889fc71f2b8e08ccd5a67a5220daf160685f750cb0c3429822

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · ~9 AIC · ⌖ ~8 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[pelikhan]

@copilot resolve the merge conflicts on this branch.

[Copilot]

@copilot resolve the merge conflicts on this branch.

Resolved the conflict in actions/setup/js/model_pin_check.test.cjs, preserving main’s empty known-failure list, and completed the merge from main in b98a0b7.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (actions/setup/js/run_evals.cjs:123): Prompt injection risk: Untrusted agent output from /tmp/gh-aw/evals/agent_output.json is embedded into the BinEval judge prompt via buildEvalPrompt(... guardAgentOutputForPrompt(sanitizeAgentOutputForPrompt(agentOutputContent)) ...) (actions/setup/js/run_evals.cjs:123). Because the only defense is advisory text telling the model to ignore instructions, a malicious or attacker-influenced agent output string can still steer the judge to output incorrect YES/NO answers (or answer directives), corrupting eval results. - Fix Pi evals reading stale workflow prompts #71664 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 5332633
Sous-chef work: 0e44ce2468b8d1c7f42d0cb19666cc98f0e60e87a5b9b8bb77b3622da82d93e6
Sous-chef state: a111f010d02cc71e1043e571ca4cc075cce467566310ca8de7e09081d6675e29

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · ~4 AIC · ⌖ ~12 AIC · ⊞ 1.1K · ◷
_{Comment /souschef to run again}

[github-actions]

🎉 This pull request is included in a new release.

Release: v0.93.4