SIGN IN SIGN UP

feat(evals): add eve bench harness (integrations/eve-sdk) (#2809)

## What

Adds `--harness eve` to evals, backed by a new private
`@browserbasehq/stagehand-integrations-eve-sdk` package.

- `packages/integrations/eve-sdk` — `runEveSession`: eve 0.29.4 has no
in-process run-agent SDK and no stdio MCP support, so the session layer
spawns `eve dev --no-ui --port 0` per row against a generated agent app
and consumes the HTTP NDJSON event stream (`actions.requested` /
`action.result` / turn boundaries); tool-step budget counted on
completed results; abort → `session.cancel()` + dev-server SIGTERM;
redaction.
- `packages/evals/framework/eveToolAdapter.ts` — generates the eve agent
app (`agent/tools/<server>__<tool>.ts` proxies over an
`@modelcontextprotocol/sdk` stdio client) for `via:"mcp"` mounts;
bounded `listTools`/cleanup; mkdtemp rollback; `load_skill` and other
non-browser framework tools disabled.
- `packages/evals/framework/harnesses/eveAdapter.ts` — events →
`NormalizedToolCall[]`.
- Registered via `defineExternalHarness`. Requires Node ≥ 24 (eve's own
floor).

## Testing

- Full unit gate green (evals + eve-sdk 16 + all other suites).
- Connected smoke (`evals run b:webvoyager --harness eve --tool
stagehand_facade -l 1 -t 1 -e browserbase`), run twice (before/after the
budget fix): 5 and 3 Stagehand tool calls via the facade, final answer
produced.

Stacked on the pi PR. Implemented with Codex (gpt-5.6) under
supervision; Claude + `codex exec review` findings (abort cancel,
unredacted logs, unbounded listTools/cleanup, tmpdir leak, budget
off-by-one) fixed in-branch.

<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Adds an Eve external harness to evals via a new private
`@browserbasehq/stagehand-integrations-eve-sdk`, enabling `evals run ...
--harness eve`. Eve wasn't supported before; now tasks run with
MCP-bridged tools, correct cancellation, redacted transcripts, and
budget counted only on completed tool results.

**Implementation**
- Session layer spawns `eve dev` per task (no UI, random port), streams
NDJSON, extracts usage/cost, and on abort cancels the Eve session
exactly once and kills the dev server.
- Tool adapter generates a temporary Eve agent app whose tools proxy to
MCP servers, with bounded connect/listTools/close and cleanup timeouts,
late-client close, and `mkdtemp` rollback.
- Runner maps Eve events to normalized tool calls, usage, image
evidence, and metrics, and registers `"eve"` with default model
`openai/gpt-5.4-mini`.
- Non-browser framework tools like `load_skill` are disabled; the Eve
SDK is lazy-loaded to avoid import-time side effects.
- Adds the package to Turbo/CI/Vitest with tests covering
abort/kill/redaction, tool-adapter prep/cleanup/timeouts, `mkdtemp`
rollback, lazy-load, and registry updates.

**Rollout**
- Requires Node 24+; provide MCP mounts via `STAGEHAND_EVE_MCP_SERVERS`
(JSON array).
- Run with `evals run b:<dataset> --harness eve --tool <tool> -e
browserbase`.
- Optional timeouts: `EVAL_EVE_MCP_LIST_TOOLS_TIMEOUT_MS` (default 60s)
and `EVAL_AGENT_MOUNT_CLEANUP_TIMEOUT_MS`.
- Configure models with `EVAL_EVE_MODELS`; defaults to
`openai/gpt-5.4-mini`.

<sup>Written for commit 0407ad725405c148d348d37b928236295a86005a.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/browserbase/stagehand/pull/2809?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->
M
Miguel committed
aaca8b0eb3105e7f8d98c19563db85c4c2588d5f
Parent: bc70932
Committed by GitHub <noreply@github.com> on 9/2/2026, 12:07:02 AM