Add validated Known Build Error support to test quarantine (#69039)
* Capture the real error signature in new quarantine issues Quarantine issues recorded the agent's prose description of a failure rather than the failure itself. That text is not matchable, and the Azure DevOps build it came from ages out of public retention, so within a few weeks the issue is the only surviving record of the failure and it no longer contains the actual error. Anyone picking the test up later has nothing precise to work from. The collector already fetched errorMessage and stackTrace per test to build the agent prompt, so the data was always present; only the paraphrasing destroyed it. It now derives an Arcade Known Issue payload from that same text and the agent pastes it verbatim instead of describing it. The signature is a literal substring of the real errorMessage. Volatile fragments (ports, GUIDs, timings, addresses, paths, counters) are located and the longest contiguous stable run between them is taken; fragments are never stitched together, because Arcade matches with String.Contains and a stitched string would never match anything. When no trustworthy signature can be derived the payload is omitted and the reason recorded, rather than emitting one that would match half the repository. ExcludeConsoleLog is set because one xunit console log is shared by every test in a Helix work item, so including it attributes unrelated failures to the issue. The section uses the "### Known Issue Error Message" heading to avoid colliding with the template's human-readable "## Error Message" prose. The "Known Build Error" label is deliberately not applied. A Known Build Error means a failure still blocks other people's builds, and a quarantined test no longer blocks anything. Registering these is a separate human decision; this change only prepares a valid payload. The marker embeds a sha256 prefix of the ErrorMessage so any consumer can detect paraphrasing or truncation without access to the originating run. This detects mangling, not forgery, and the tests demonstrate that boundary. Tests extract the derivation from the workflow between sentinel comments and execute it, so they exercise the shipped code rather than a copy. They include an independent reproduction of the signature a human hand-wrote for the real Known Build Error on dotnet/aspnetcore#68708. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Create validated KBEs for quarantined tests Generate quarantine issues through a deterministic safe-output handler that validates runtime-style matchers against captured test evidence and gates Build Insights enrollment. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Bind quarantine KBEs to deterministic eligibility Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Align quarantine output limits with gh-aw Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Harden quarantine KBE eligibility Track assembly-level quarantine removals across project history, fail closed on ambiguous source associations, and cover renamed or inherited test layouts. Enforce a single custom issue call and use deterministic exact-title issue reuse. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Add quarantine workflow regression checks Run quarantine fixtures in isolated pull request CI, document ordered substring matching, and fail clearly when RUNNER_TEMP is missing or empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Track quarantine history across partial test types Resolve partial inherited runners within their project and apply logical type quarantine history only to methods and runners present at the transition. Preserve method-scoped freshness, cover the regression cases, and bound historical indexing work. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Validate historical inheritance and runner identity Resolve inherited tests against the chain present at quarantine transitions, including historical-only intermediates. Reject cross-project runner ambiguity before direct matches and preserve method-scoped freshness policy. Add red/green regression coverage and document identity limits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 * Clarify quarantine eligibility documentation Explain the collector's responsibilities and use descriptive new-quarantine and re-quarantine terminology while preserving receipt identifiers and workflow behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723 --------- Co-authored-by: PureWeen <223556219+Copilot@users.noreply.github.com> Copilot-Session: d5905c51-843f-4d8c-9852-cbbcf77d7723
S
Shane Neuville committed
d5aa21e4996d3ec4cbb5ea1691d62329010f4d86
Parent: f4bd503
Committed by GitHub <noreply@github.com>
on 9/11/2026, 3:47:52 PM