feat(evals): add eve bench harness (integrations/eve-sdk) (#2809)
## What Adds `--harness eve` to evals, backed by a new private `@browserbasehq/stagehand-integrations-eve-sdk` package. - `packages/integrations/eve-sdk` — `runEveSession`: eve 0.29.4 has no in-process run-agent SDK and no stdio MCP support, so the session layer spawns `eve dev --no-ui --port 0` per row against a generated agent app and consumes the HTTP NDJSON event stream (`actions.requested` / `action.result` / turn boundaries); tool-step budget counted on completed results; abort → `session.cancel()` + dev-server SIGTERM; redaction. - `packages/evals/framework/eveToolAdapter.ts` — generates the eve agent app (`agent/tools/<server>__<tool>.ts` proxies over an `@modelcontextprotocol/sdk` stdio client) for `via:"mcp"` mounts; bounded `listTools`/cleanup; mkdtemp rollback; `load_skill` and other non-browser framework tools disabled. - `packages/evals/framework/harnesses/eveAdapter.ts` — events → `NormalizedToolCall[]`. - Registered via `defineExternalHarness`. Requires Node ≥ 24 (eve's own floor). ## Testing - Full unit gate green (evals + eve-sdk 16 + all other suites). - Connected smoke (`evals run b:webvoyager --harness eve --tool stagehand_facade -l 1 -t 1 -e browserbase`), run twice (before/after the budget fix): 5 and 3 Stagehand tool calls via the facade, final answer produced. Stacked on the pi PR. Implemented with Codex (gpt-5.6) under supervision; Claude + `codex exec review` findings (abort cancel, unredacted logs, unbounded listTools/cleanup, tmpdir leak, budget off-by-one) fixed in-branch. <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Adds an Eve external harness to evals via a new private `@browserbasehq/stagehand-integrations-eve-sdk`, enabling `evals run ... --harness eve`. Eve wasn't supported before; now tasks run with MCP-bridged tools, correct cancellation, redacted transcripts, and budget counted only on completed tool results. **Implementation** - Session layer spawns `eve dev` per task (no UI, random port), streams NDJSON, extracts usage/cost, and on abort cancels the Eve session exactly once and kills the dev server. - Tool adapter generates a temporary Eve agent app whose tools proxy to MCP servers, with bounded connect/listTools/close and cleanup timeouts, late-client close, and `mkdtemp` rollback. - Runner maps Eve events to normalized tool calls, usage, image evidence, and metrics, and registers `"eve"` with default model `openai/gpt-5.4-mini`. - Non-browser framework tools like `load_skill` are disabled; the Eve SDK is lazy-loaded to avoid import-time side effects. - Adds the package to Turbo/CI/Vitest with tests covering abort/kill/redaction, tool-adapter prep/cleanup/timeouts, `mkdtemp` rollback, lazy-load, and registry updates. **Rollout** - Requires Node 24+; provide MCP mounts via `STAGEHAND_EVE_MCP_SERVERS` (JSON array). - Run with `evals run b:<dataset> --harness eve --tool <tool> -e browserbase`. - Optional timeouts: `EVAL_EVE_MCP_LIST_TOOLS_TIMEOUT_MS` (default 60s) and `EVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS`. - Configure models with `EVAL_EVE_MODELS`; defaults to `openai/gpt-5.4-mini`. <sup>Written for commit 0407ad725405c148d348d37b928236295a86005a. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/browserbase/stagehand/pull/2809?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
M
Miguel committed
aaca8b0eb3105e7f8d98c19563db85c4c2588d5f
Parent: bc70932
Committed by GitHub <noreply@github.com>
on 9/2/2026, 12:07:02 AM