SIGN IN SIGN UP

feat(evals): add pi bench harness (integrations/pi-sdk) (#2808)

## What

Adds `--harness pi` to evals, backed by a new private
`@browserbasehq/stagehand-integrations-pi-sdk` package.

- `packages/integrations/pi-sdk` — `runPiSession` over the programmatic
`@earendil-works/pi-coding-agent` SDK (`createAgentSession`, hermetic
tool allowlist so bash/edit/write are never exposed), in-process MCP
bridge for facade tools, streamed `AgentSessionEvent`s →
tool-call/result/usage, abort forwarding, redaction, status/stopReason
normalization.
- `packages/evals/framework/piToolAdapter.ts` — `via:"mcp"` mounts
(stagehand_facade first, playwright_mcp, chrome_devtools_mcp);
`via:"handles"` run-tool via the in-process bridge.
- `packages/evals/framework/harnesses/piAdapter.ts` — events →
`NormalizedToolCall[]`.
- Registered via `defineExternalHarness`.

## Testing

- Full unit gate green (evals + pi-sdk 12 + all other suites).
- Connected smoke (`evals run b:webvoyager --harness pi --tool
stagehand_facade -l 1 -t 1 -e browserbase`): **10 Stagehand tool calls**
via the facade, final answer produced.

Stacked on the mastra PR. Implemented with Codex (gpt-5.6) under
supervision; review findings (redaction, stderr buffering, stop reasons,
run-tool aborts) fixed in-branch.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds a new `--harness pi` bench harness backed by a private
`@browserbasehq/stagehand-integrations-pi-sdk`, enabling evals to run
the PI coding agent with Stagehand tools while keeping tool exposure
hermetic. PI stop reasons "aborted" and "length" now map to `sdk_error`;
harness guidance strings derive from the registry dynamically.

- Registers `pi` with default model `openai/gpt-5.4-mini`; surfaces:
`stagehand_facade` (default), `playwright_mcp`, `chrome_devtools_mcp`,
`stagehand_code`, `playwright_code`, `cdp_code`.
- New pi-sdk: in-process session runner (turn budget, abort forwarding),
token/cost extraction, transcript building, and a stdio MCP bridge with
line-buffered, redacted stderr.
- Custom-tool allowlist hosts the harness run-tool in-process and
respects AbortSignal and timeouts.
- Evals adapters: `piRunner` (marker contract, `pi_turns` metric),
`piAdapter` (PI events to trajectories with reasoning, images, probe
evidence), `piToolAdapter` (mounts, step observations, runtime/temp-dir
cleanup, no unhandled rejections on late failures).
- Redacts provider errorMessage fields and tool args/results so logs
never leak secrets; CI/Turbo build and tests are wired for the new
package.

**Rollout**
- Run `evals run <task> --harness pi --tool stagehand_facade` (or
another supported surface).
- Optional env: `EVAL_PI_MAX_TURNS` (or `AGENT_EVAL_MAX_STEPS`),
`EVAL_PI_THINKING`, `EVAL_PI_RUN_TOOL_TIMEOUT_MS`,
`EVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS`,
`EVAL_CAPTURE_EVIDENCE_TIMEOUT_MS`.
- For MCP usage, provide `mcpServers` in the mount; tools appear as
`mcp__server__tool`.
- No migration required.

<sup>Written for commit 08c454b9f72464112cbd8388962de1a9f306bbe8.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2808?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
M
Miguel committed
bc70932b375d6983a26ddf90d711e04df5013433
Parent: 7adfd33
Committed by GitHub <noreply@github.com> on 9/2/2026, 12:07:02 AM