Skip to content

Route Terminal Stylist with model routing instead of pinning gpt-5.6-sol - #73387

Merged
SivaKesava1 merged 3 commits into
mainfrom
copilot/model-routing-terminal-stylist
Oct 31, 2026
Merged

SivaKesava1 merged 3 commits into
mainfrom
copilot/model-routing-terminal-stylist

Conversation

Copilot AI commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

Terminal Stylist pins copilot/gpt-5.6-sol. Its threat-detection job inherits that pin, so a run costs about 87 AIC (about 69 for the agent, 18 for detection). With goal: cost and mode: economy, the router is expected to pick gpt-5.6-luna at max effort for this task. That model scores about the same on the router's benchmarks for a fraction of the cost.

Frontmatter (.github/workflows/terminal-stylist.md)

engine:
  id: pi
  model-routing:
    goal: cost
    mode: economy
  • The pin is removed. There's no allowed-models and no model-provider, because Pi routes through GitHub Copilot without it.
  • The prompt, schedule, tools, permissions and safe outputs don't change.

Regenerated lock file (terminal-stylist.lock.yml)

  • The AWF router image ghcr.io/github/gh-aw-firewall/router:0.29.0@sha256:d054ab66… is added to the pinned and downloaded images.
  • A "Prepare model-routing conversation" step is added.
  • The agent's AWF config sets experimental.modelRouting: true. The model config is routing: {enabled: true, goal: cost, mode: economy}, with empty allowed and blocked lists.
  • The agent step no longer sets GH_AW_PI_MODEL or GH_AW_PI_NATIVE_PROVIDER.
  • The agent job outputs the routed model, model_effort and model_routing_status. Downstream jobs receive them as GH_AW_ENGINE_MODEL, GH_AW_ENGINE_MODEL_EFFORT and GH_AW_MODEL_ROUTING_STATUS.
  • Threat detection now uses its detection default model instead of gpt-5.6-sol. This workflow has no evals job.

No other workflows or lock files change.

Notes

  • Known failure on main: TestNoRawOctalPermissionLiteralsInOSCalls in pkg/constants fails because of pkg/cli/logs_harbor*.go. This PR doesn't touch those files.
  • Follow-up, not in this PR: the issue's "After merge" checks. That means auditing the first routed runs, comparing AIC per run with about 87 today, and comparing the success rate and report quality with recent runs.
  • Rollback: if reports get worse, switch to mode: balanced. To undo routing entirely, restore engine: pi and the model: pin, then recompile.
Copilot AI and others added 2 commits October 31, 2026 02:07
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Update Terminal Stylist to use model routing Oct 31, 2026
Copilot AI requested a review from SivaKesava1 October 31, 2026 02:28
@SivaKesava1
SivaKesava1 marked this pull request as ready for review October 31, 2026 02:28
Copilot AI auto review requested due to automatic review settings October 31, 2026 02:28
@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Design Decision Gate 🏗️

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Design Decision Gate 🏗️ · opus55 · ~11 AIC · ⌖ ~33 AIC · ◇ ~1 AIC · ⊞ 2.0K · ◷

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Ponytail Reviewer

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Ponytail Reviewer · gpt56 · ~8 AIC · ⌖ ~7 AIC · ⊞ 15.2K · ◷

Generated by Ponytail Reviewer for #73387

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel

Outcome: reported_no_action

The agent completed and reported no action. A no-op output alone does not verify that all required work was completed or that publication was possible.

Agent messages are omitted from this comment. The category uses structured run status, not interpretation of agent prose.

Generated from Test Quality Sentinel · sonnet50 · ~13 AIC · ⌖ ~16 AIC · ◇ ~5 AIC · ⊞ 12.7K · ◷

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: no blocking findings

The frontmatter change is schema-consistent, and the regenerated lockfile wires model routing through the router while propagating routed model metadata to reporting and safe-output jobs. I found no actionable correctness, security, or CI issue in the changed lines.

Review scope

Reviewed the workflow source and generated lockfile routing changes; no inline comments were necessary.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt50 · ~3 AIC · ⌖ ~1 AIC · ⊞ 1.7K
Comment /review to run again

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the cost-focused model-routing change.

No open issues 🎉

What changed in this PR

Moves Terminal Stylist from a fixed gpt-5.6-sol model to cost-focused routing, supporting #73380’s goal of reducing report-generation costs without changing its task or permissions.

Changes:

  • Keeps the Pi engine and enables model routing with goal: cost and mode: economy.
  • Regenerates the lock file with the pinned router image, routing setup, and downstream model metadata.
  • Removes the inherited threat-detection model pin, allowing the configured detection default; prompt, schedule, tools, permissions, and safe outputs remain unchanged.
File Description
.github/​workflows/​terminal-stylist.md Replaces the fixed model pin with cost-focused economy routing.
.github/​workflows/​terminal-stylist.lock.yml Adds generated routing configuration and metadata handling, and restores default detection model selection.

Review effort: Balanced (auto)

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /codebase-design per pr-triage (change_type: refactor_cleanup). No actionable issues found.

📋 Analysis

This is a minimal, well-precedented configuration change:

  • engine: pi / model: copilot/gpt-5.6-sol → engine: { id: pi, model-routing: { goal: cost, mode: economy } }
  • Both goal: cost and mode: economy are valid enum values enforced by validateModelRouting in pkg/workflow/model_routing.go (goal ∈ {cost, cost-speed}; mode ∈ {economy, balanced, robust, auto}).
  • The exact same goal: cost / mode: economy pairing with engine.id: pi is already in production use elsewhere (e.g. smoke-pi-routed.md, claude-code-user-docs-review.md, copilot-session-insights.md, safe-output-health.md), so this isn't a novel/untested pattern.
  • Omitting allowed-models/model-provider is consistent with how other pi-engine workflows configure routing — Pi routes through GitHub Copilot without an explicit provider.
  • The regenerated lock file correctly drops the hardcoded GH_AW_PI_MODEL/GH_AW_PI_NATIVE_PROVIDER env vars, adds the AWF router image/step, and threat detection now falls back to its own detection default model instead of inheriting the removed pin — all expected side effects of this change, not new risk surface.
  • No changes to prompt, tools, permissions, or safe outputs, so blast radius is contained to model selection/cost for one low-risk internal audit workflow.

Positive Highlights

  • ✅ PR description is thorough: documents expected cost impact (~87 → ~60 AIC), the follow-up audit plan, and a clear rollback path (mode: balanced or re-pin).
  • ✅ Change follows the established repo convention for migrating pinned-model workflows to router-based model selection.
  • ✅ Scope is tight — only the two files that needed to change were touched.

No inline comments posted: the diff is a 3-line, schema-valid config change with direct precedent elsewhere in the repo, and the PR body already covers the operational risk (cost regression, rollback) that would otherwise need separate flagging.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · ~65 AIC · ⌖ ~18 AIC · ⊞ 10.8K
Comment /matt to run again

@SivaKesava1
SivaKesava1 merged commit 633bb2c into main Oct 31, 2026
63 checks passed
@SivaKesava1
SivaKesava1 deleted the copilot/model-routing-terminal-stylist branch October 31, 2026 03:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants