Add pull request attention queue skill (#69199)
* Add pull request attention queue skill Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Use vally eval spec for attention queue skill Replace the ad hoc evals.json with a vally eval.yaml so the skill is validated by vally lint/eval. Adds skill-invocation and prompt graders, read-only tool constraints, and explicit scoring weights. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Tag attention queue eval stimuli by risk Adds network and risk tags so the read-only subset can be run without the mutation-refusal stimulus, which instructs the agent to attempt real GitHub writes: vally eval --tag risk=read-only Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Drop live mutation stimulus from attention queue eval The refuses-mutation-and-scoring stimulus instructed the agent to approve, comment on, and label a pull request so the skill's refusal could be graded. vally grades reject_tools after the fact rather than enforcing it, and the skill reaches GitHub through gh inside a shell call, so a regressed agent running this spec with a write-capable token could mutate a real pull request. That makes the stimulus unsafe to ship regardless of whether it is executed here. Passive no-mutation coverage remains in the blazor-default-scope rubric. Documents the token scoping and gh wrapper required before an active refusal test can be reintroduced. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Resolve lazy mergeability and incidental path matches Running the queue against dotnet/aspnetcore returned a different answer on consecutive runs over unchanged data. GitHub computes mergeability lazily. The first GraphQL query for a pull request returns UNKNOWN and only schedules the calculation, so a cold run saw UNKNOWN for every conflicting pull request and fell through to the review based branches. Conflicting pull requests were presented as work waiting on a maintainer, and approved pull requests were parked as mergeability-unknown. Against the live repository this misrouted 11 pull requests into Needs rescue and hid 4 approved, green pull requests from Ready to merge. Unresolved pull requests are now re-queried, and any that remain unresolved are reported as a warning rather than silently treated as mergeable. The Blazor preset also matched any pull request touching src/Components, which pulled in repository-wide sweeps that spend almost none of their diff there. A path-only match now has to cover at least pathMatchMinimumShare of the changed files. A labeled pull request is never subject to the floor. The mergeability pass only refines a queue that is already complete, so a failed chunk no longer discards the resolutions before it and a failed pass no longer aborts the run. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Expose PR attention display metadata Add reviewed-abandonment classification while preserving author and CI gates, and publish stable display metadata for canvas consumers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Move PR attention eval to repository layout Place the vally specification under eng/skill-evals so repository validation can discover it without treating the runtime skill as eval-only content. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden PR attention queue classification Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Correct stack and merge-state routing Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Gate PR queue on discussion evidence Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden discussion verification gates Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep discussion verification visible Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add bounded personal PR follow-ups Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix personal notification relevance Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add production notification caching metrics Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Measure personal collector work Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Expose personal collection metrics Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Expose notification cache errors Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Parse notification response boundaries safely Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bound personal thread hydration and retries Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Refresh notification cache and filter pending reviews Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Normalize submitted review participation Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Configure pinned PR attention eval models Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make PR attention evals fixture-backed Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Preserve PR numbers in eval rubrics Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: PureWeen <223556219+Copilot@users.noreply.github.com>
S
Shane Neuville committed
1a64b922e7f63768d8ea2c79f58eac3b9ff35086
Parent: a2ac63c
Committed by GitHub <noreply@github.com>
on 9/10/2026, 8:08:07 PM