SIGN IN SIGN UP

Add validated Known Build Error support to test quarantine (#69039)

* Capture the real error signature in new quarantine issues

Quarantine issues recorded the agent's prose description of a failure rather
than the failure itself. That text is not matchable, and the Azure DevOps build
it came from ages out of public retention, so within a few weeks the issue is
the only surviving record of the failure and it no longer contains the actual
error. Anyone picking the test up later has nothing precise to work from.

The collector already fetched errorMessage and stackTrace per test to build the
agent prompt, so the data was always present; only the paraphrasing destroyed
it. It now derives an Arcade Known Issue payload from that same text and the
agent pastes it verbatim instead of describing it.

The signature is a literal substring of the real errorMessage. Volatile
fragments (ports, GUIDs, timings, addresses, paths, counters) are located and
the longest contiguous stable run between them is taken; fragments are never
stitched together, because Arcade matches with String.Contains and a stitched
string would never match anything. When no trustworthy signature can be derived
the payload is omitted and the reason recorded, rather than emitting one that
would match half the repository.

ExcludeConsoleLog is set because one xunit console log is shared by every test
in a Helix work item, so including it attributes unrelated failures to the
issue. The section uses the "### Known Issue Error Message" heading to avoid
colliding with the template's human-readable "## Error Message" prose.

The "Known Build Error" label is deliberately not applied. A Known Build Error
means a failure still blocks other people's builds, and a quarantined test no
longer blocks anything. Registering these is a separate human decision; this
change only prepares a valid payload.

The marker embeds a sha256 prefix of the ErrorMessage so any consumer can
detect paraphrasing or truncation without access to the originating run. This
detects mangling, not forgery, and the tests demonstrate that boundary.

Tests extract the derivation from the workflow between sentinel comments and
execute it, so they exercise the shipped code rather than a copy. They include
an independent reproduction of the signature a human hand-wrote for the real
Known Build Error on dotnet/aspnetcore#68708.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Create validated KBEs for quarantined tests

Generate quarantine issues through a deterministic safe-output handler that validates runtime-style matchers against captured test evidence and gates Build Insights enrollment.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Bind quarantine KBEs to deterministic eligibility

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Align quarantine output limits with gh-aw

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Harden quarantine KBE eligibility

Track assembly-level quarantine removals across project history, fail closed on ambiguous source associations, and cover renamed or inherited test layouts. Enforce a single custom issue call and use deterministic exact-title issue reuse.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Add quarantine workflow regression checks

Run quarantine fixtures in isolated pull request CI, document ordered substring matching, and fail clearly when RUNNER_TEMP is missing or empty.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Track quarantine history across partial test types

Resolve partial inherited runners within their project and apply logical type quarantine history only to methods and runners present at the transition. Preserve method-scoped freshness, cover the regression cases, and bound historical indexing work.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Validate historical inheritance and runner identity

Resolve inherited tests against the chain present at quarantine transitions, including historical-only intermediates. Reject cross-project runner ambiguity before direct matches and preserve method-scoped freshness policy. Add red/green regression coverage and document identity limits.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

* Clarify quarantine eligibility documentation

Explain the collector's responsibilities and use descriptive new-quarantine and re-quarantine terminology while preserving receipt identifiers and workflow behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723

---------

Co-authored-by: PureWeen <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723
S
Shane Neuville committed
d5aa21e4996d3ec4cbb5ea1691d62329010f4d86
Parent: f4bd503
Committed by GitHub <noreply@github.com> on 9/11/2026, 3:47:52 PM