feat(streams): rollover block log and atomic stream → session cutover (#2216)
* bench(streams): experimental SQLite block strategies for stream→message cutover
Compares immutable packed rows, reusable generational slots and mutable
rollover blocks on billed rows, write time, page growth, replay, and the
cursor a restarted isolate recovers, with an atomic transactionSync
cutover to the message row. Numbers logged, invariants asserted.
* feat(streams): rollover block log and atomic stream → message cutover
- The Streams chunk log is mutable rollover blocks: an append grows the open
block row (UPDATE) until 256 KB, then opens the next. Same one billed row
per append; a stream of thousands of chunks is a handful of rows to
delete. Schema v2 folds existing cf_agents_stream_chunks rows into blocks.
- writer.close({ commit, discard }) / error(reason, { ... }) settle, run the
caller's synchronous writes and delete the stream's rows in one SQLite
transaction. Session.__DO_NOT_USE_WILL_BREAK__sync().upsert() is the
matching synchronous message write.
- ResumableStream.complete(id, { persist }) exposes the cutover;
discardCompleted() drops a completed stream's rows once its message is
persisted; start() reclaims completed streams a crash left behind.
- AIChatAgent and Think discard right after persisting (agent-tool child
turns excepted: the parent tails their stored chunks), so the retention
sweep no longer finds completed streams.
* test(streams): e2e crash matrix for the block log and cutover
ctx.abort() at each point that matters — before an append commits, right
after, during rollover (before/after commit), after persist before discard,
and inside/after the atomic cutover — then inspect the fresh instance. Each
restart finds either the exact committed prefix or the finished message.
* feat(chat): persist inside the cutover; remove the stream-buffer sweep
- ResumableStream.finish() leaves a finished stream live until cutover();
cutover() settles it, runs the message write and deletes its rows in one
transaction; finalizePending() settles when nothing is persisted;
reclaim() on every start() replaces the alarm-driven sweep.
- AIChatAgent persists the turn's messages inside the cutover
(persistMessages { _cutover }); Think does the same through
_persistAssistantMessageWithCutover. The session change feed and
auto-compaction run once the transaction commits.
- cleanupStreamBuffers / STREAM_CLEANUP_DELAY_SECONDS removed; the host
_cleanupStreamBuffers callbacks stay as no-ops for persisted alarms.
* fix(ci): drop cleanupStreamBuffers from experimental agents; test typing
* fix(streams): lazy per-stream v1 fold, idempotent cutover, events after commit
- Fold cf_agents_stream_chunks rows into blocks one stream at a time on
first touch (paged reads, whole-block inserts) instead of the whole log
at startup; drop the table once empty.
- A settle with commit/discard is a no-op on a stream already terminal or
deleted: commit does not run, nothing is discarded. ResumableStream's
cutover falls back to a plain persist in that case.
- Terminal, deleted events and reader wakeups fire after the cutover
transaction returns, never for a rolled-back commit.
* fix(ai-chat): cutover rides the instance, not a persistMessages argument
A subclass that overrides persistMessages and calls super with only the
messages dropped the internal _cutover option, so the message write and
the stream settlement fell back to two commits. The pending cutover now
lives on the instance keyed by the turn's message id: the first persist
that carries that message runs the cutover, a persist of other messages
takes the plain path, and the turn's end clears it. persistMessages'
public signature is unchanged. M
Matt committed
dd09d44913e60aa519aac51e53b13eac2111c732
Parent: 2847a28
Committed by GitHub <noreply@github.com>
on 9/8/2026, 4:37:04 PM