SIGN IN SIGN UP

Add pull request attention queue skill (#69199)

* Add pull request attention queue skill

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Use vally eval spec for attention queue skill

Replace the ad hoc evals.json with a vally eval.yaml so the skill is
validated by vally lint/eval. Adds skill-invocation and prompt graders,
read-only tool constraints, and explicit scoring weights.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Tag attention queue eval stimuli by risk

Adds network and risk tags so the read-only subset can be run without the
mutation-refusal stimulus, which instructs the agent to attempt real
GitHub writes:

  vally eval --tag risk=read-only

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Drop live mutation stimulus from attention queue eval

The refuses-mutation-and-scoring stimulus instructed the agent to approve,
comment on, and label a pull request so the skill's refusal could be graded.
vally grades reject_tools after the fact rather than enforcing it, and the
skill reaches GitHub through gh inside a shell call, so a regressed agent
running this spec with a write-capable token could mutate a real pull
request. That makes the stimulus unsafe to ship regardless of whether it is
executed here.

Passive no-mutation coverage remains in the blazor-default-scope rubric.
Documents the token scoping and gh wrapper required before an active
refusal test can be reintroduced.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Resolve lazy mergeability and incidental path matches

Running the queue against dotnet/aspnetcore returned a different answer on
consecutive runs over unchanged data.

GitHub computes mergeability lazily. The first GraphQL query for a pull
request returns UNKNOWN and only schedules the calculation, so a cold run saw
UNKNOWN for every conflicting pull request and fell through to the review
based branches. Conflicting pull requests were presented as work waiting on a
maintainer, and approved pull requests were parked as mergeability-unknown.
Against the live repository this misrouted 11 pull requests into Needs rescue
and hid 4 approved, green pull requests from Ready to merge. Unresolved pull
requests are now re-queried, and any that remain unresolved are reported as a
warning rather than silently treated as mergeable.

The Blazor preset also matched any pull request touching src/Components,
which pulled in repository-wide sweeps that spend almost none of their diff
there. A path-only match now has to cover at least pathMatchMinimumShare of
the changed files. A labeled pull request is never subject to the floor.

The mergeability pass only refines a queue that is already complete, so a
failed chunk no longer discards the resolutions before it and a failed pass no
longer aborts the run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Expose PR attention display metadata

Add reviewed-abandonment classification while preserving author and CI gates, and publish stable display metadata for canvas consumers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Move PR attention eval to repository layout

Place the vally specification under eng/skill-evals so repository validation can discover it without treating the runtime skill as eval-only content.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Harden PR attention queue classification

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Correct stack and merge-state routing

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Gate PR queue on discussion evidence

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Harden discussion verification gates

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Keep discussion verification visible

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Add bounded personal PR follow-ups

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix personal notification relevance

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Add production notification caching metrics

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Measure personal collector work

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Expose personal collection metrics

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Expose notification cache errors

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Parse notification response boundaries safely

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Bound personal thread hydration and retries

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Refresh notification cache and filter pending reviews

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Normalize submitted review participation

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Configure pinned PR attention eval models

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Make PR attention evals fixture-backed

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Preserve PR numbers in eval rubrics

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: PureWeen <223556219+Copilot@users.noreply.github.com>
S
Shane Neuville committed
1a64b922e7f63768d8ea2c79f58eac3b9ff35086
Parent: a2ac63c
Committed by GitHub <noreply@github.com> on 9/10/2026, 8:08:07 PM