feat(evals): add pi bench harness (integrations/pi-sdk) (#2808)
## What Adds `--harness pi` to evals, backed by a new private `@browserbasehq/stagehand-integrations-pi-sdk` package. - `packages/integrations/pi-sdk` — `runPiSession` over the programmatic `@earendil-works/pi-coding-agent` SDK (`createAgentSession`, hermetic tool allowlist so bash/edit/write are never exposed), in-process MCP bridge for facade tools, streamed `AgentSessionEvent`s → tool-call/result/usage, abort forwarding, redaction, status/stopReason normalization. - `packages/evals/framework/piToolAdapter.ts` — `via:"mcp"` mounts (stagehand_facade first, playwright_mcp, chrome_devtools_mcp); `via:"handles"` run-tool via the in-process bridge. - `packages/evals/framework/harnesses/piAdapter.ts` — events → `NormalizedToolCall[]`. - Registered via `defineExternalHarness`. ## Testing - Full unit gate green (evals + pi-sdk 12 + all other suites). - Connected smoke (`evals run b:webvoyager --harness pi --tool stagehand_facade -l 1 -t 1 -e browserbase`): **10 Stagehand tool calls** via the facade, final answer produced. Stacked on the mastra PR. Implemented with Codex (gpt-5.6) under supervision; review findings (redaction, stderr buffering, stop reasons, run-tool aborts) fixed in-branch. <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Adds a new `--harness pi` bench harness backed by a private `@browserbasehq/stagehand-integrations-pi-sdk`, enabling evals to run the PI coding agent with Stagehand tools while keeping tool exposure hermetic. PI stop reasons "aborted" and "length" now map to `sdk_error`; harness guidance strings derive from the registry dynamically. - Registers `pi` with default model `openai/gpt-5.4-mini`; surfaces: `stagehand_facade` (default), `playwright_mcp`, `chrome_devtools_mcp`, `stagehand_code`, `playwright_code`, `cdp_code`. - New pi-sdk: in-process session runner (turn budget, abort forwarding), token/cost extraction, transcript building, and a stdio MCP bridge with line-buffered, redacted stderr. - Custom-tool allowlist hosts the harness run-tool in-process and respects AbortSignal and timeouts. - Evals adapters: `piRunner` (marker contract, `pi_turns` metric), `piAdapter` (PI events to trajectories with reasoning, images, probe evidence), `piToolAdapter` (mounts, step observations, runtime/temp-dir cleanup, no unhandled rejections on late failures). - Redacts provider errorMessage fields and tool args/results so logs never leak secrets; CI/Turbo build and tests are wired for the new package. **Rollout** - Run `evals run <task> --harness pi --tool stagehand_facade` (or another supported surface). - Optional env: `EVAL_PI_MAX_TURNS` (or `AGENT_EVAL_MAX_STEPS`), `EVAL_PI_THINKING`, `EVAL_PI_RUN_TOOL_TIMEOUT_MS`, `EVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS`, `EVAL_CAPTURE_EVIDENCE_TIMEOUT_MS`. - For MCP usage, provide `mcpServers` in the mount; tools appear as `mcp__server__tool`. - No migration required. <sup>Written for commit 08c454b9f72464112cbd8388962de1a9f306bbe8. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2808?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
M
Miguel committed
bc70932b375d6983a26ddf90d711e04df5013433
Parent: 7adfd33
Committed by GitHub <noreply@github.com>
on 9/2/2026, 12:07:02 AM