Testing your provider

Four layers, all reachable through the published SDK: the fake plugin host for the declaration, the host-entry harness for host RPC, the conformance kit and the real delta assembler for the bridge, and recorded replay for regressions on real traffic.

The layers #

The fake plugin host #

createFakePluginHost({ pluginId }) returns a bb that satisfies BbPluginApi and a harness exposing what the factory registered. It runs the same declaration validator the server does, so what it records is what a real install carries: defaults filled, paths normalized, invalid declarations refused at the call.

import { createFakePluginHost } from "@get-bb/plugin-sdk/testing";
import echoPlugin from "./server.js";

const host = createFakePluginHost({ pluginId: "echo-provider" });
echoPlugin(host.bb);
const declaration = host.harness.registrations.providerRegistrations
  .find((entry) => entry.id === "echo-agent");

expect(declaration.experimental_nativeSkillRoots).toEqual({
  user: [],
  project: [{ path: ".echo/skills", recursive: false, ancestors: false, namePrefix: "" }],
});
expect(declaration.experimental_resolvesNativeRoots).toBe(false);

Use it for anything the server decides at registration: that ids and roots validate, that deriveProviderOptions returns what you expect for a given settings snapshot, that a settings-driven re-registration disposes the old one first. The fake refuses what the server refuses at the register call, with the server's wording. experimental_hostEntry: false (a manifest with no bb.host entry) refuses bb.providers.register and bb.experimental_aiServices.register. experimental_declaredIconNames stands in for the names the manifest declares under bb.branding.experimental_icons (default none): a provider icon or a tool presentation.icon.glyph of the form "<pluginId>/<name>" that names another plugin or a name not in that list is refused on providers.register and agents.registerTool, so a test that registers a namespaced icon names the icons the manifest would. The rule itself (undeclaredIconProblem) lives in the SDK's host policy and the server applies the same function at register and at ingest; the echo declares a host glyph ("Zap") for its provider, which is why the example above passes without the option.

The host-entry harness #

Test one host entry in-process with the same validation, JSON transport, cancellation, lifecycle and output-size boundaries as the daemon worker (8 MiB per result). Process crashes stay an integration concern.

import { experimental_createHostEntryHarness } from "@get-bb/plugin-sdk/testing/host";
import hostEntry from "./host.js";

const harness = experimental_createHostEntryHarness(hostEntry, {
  experimental_paths: { dataDir: "/tmp/plugin-data", tempDir: "/tmp/plugin-temp" },
});
const roots = await harness.experimental_call("resolveNativeRoots", { providerId: "my-agent", cwd: "/repo" });
expect(roots.skills[0]).toMatchObject({ origin: "user", shape: "skills" });
await harness.experimental_dispose();

A one-line test worth keeping beside it: assert that the artifact exports both surfaces, because the daemon's two bootstraps look for them by name.

import hostEntry, { experimental_providerBridge } from "./host.js";
expect(experimental_providerBridge.experimental_apiVersion).toBe(1);
expect(typeof experimental_providerBridge.handleLine).toBe("function");
expect(hostEntry.experimental_apiVersion).toBe(1);

The conformance kit #

experimental_runBridgeConformance({ transport, session, providerId, timeoutMs? }) drives one bridge through JSON-RPC hygiene, the initialize handshake, then a shared session lifecycle (start → turn → grammar checks → release stop → resume → id uniqueness → fork, when the handshake declares one), using one transport for the whole run to mirror a real bridge lifetime, and releases the session it opened at the end, as the runtime releases a thread it detaches. It returns { results: [{ id, title, status: "pass" | "fail" | "skipped", detail }], passed }.

The transport is black-box at the message level and hands over raw wire messages only: send(line) delivers one raw line; takeMessages() returns every JSON-RPC message the bridge emitted since the last call, in order (responses, notifications, thread/delta included, and bridge-initiated requests, as parsed JSON); optional close(). Two implementations are expected: in-process (send is your exported handleLine, takeMessages is the captured output's own takeMessages) and a spawned binary (stdin write, stdout readline). The kit never sees which.

The grammar checks run over canonical events, so the kit assembles each thread/delta batch itself, through one stateful real assembler for the whole run under the providerId you name, and the ids the rules see are the ids the runtime would mint; the same reverse map names a turn back to the bridge for thread/stop { interrupt }. An invalid thread/delta throws out of the run rather than being dropped: a bridge must not pass conformance on output the runtime cannot read. (The transport no longer assembles: experimental_toConformanceMessages throws naming the replacement, resolveProviderTurnId is gone from the transport type, and CONFORMANCE_ASSEMBLED_EVENT_METHOD is retired.) The full test is in the Quickstart.

The session fixture: { cwd, promptInput, zeroWorkPromptInput?, interruptiblePromptInput?, options?, icons? }. Supply zeroWorkPromptInput (a prompt your provider completes without doing anything) to opt into the zero-work rule; supply interruptiblePromptInput (a prompt that opens a turn and never settles it on its own) to opt into the thread-independence and interrupt rules; supply icons: { pluginId, names } (what your manifest declares under bb.branding.experimental_icons) to opt into the declared-icon rule; options overrides the execution options, which default to full mode.

The rules #

The echo bridge pins fifteen green rules: the thirteen every bridge runs plus turn/settles-without-activity, which its zero-work prompt enables, and presentation/icon-namespaced-declared, which its icons declaration enables. The remaining four are gated: session/threads-independent and stop/interrupt-settles-before-result on an interruptible prompt, recovery/session-archived on archive support, and session/fork-identity on a handshake that declares fork; the echo has none of these, and a fixture that omits an opt-in produces no result for that rule rather than a failure, so a report stays fully green. A rule the fixture opted into but cannot exercise reports skipped with a reason. The three identity rules parse the thread/start, thread/resume and thread/fork results with the schema the runtime parses every session-construction result with, and a failure names the field: the result must be { providerThreadId, sessionRestorable? }, and the runtime adopts no session without providerThreadId on the result (a thread/identity notification does not substitute for it). session/resume-identity adopts the resumed identity for every later request, as the runtime does, and session/resume-id-uniqueness is skipped with it as the prerequisite; session/fork-identity forks the lifecycle session at its tip, the one shape every declaring bridge supports, parses the identity and releases the forked session; recovery/session-archived judges the resumes it performs the same way, so a resumed archived session must carry an identity and the retry after thread/unarchive must bring the session back with one. A bridge that answered resume or fork with a bare {} used to pass the whole suite while the runtime forgot the thread on its first resume or fork. The host applies the event-grammar rules (item/opens-before-delta, item/settles-once, turn/starts-once, turn/settles-once, turn/known) live at its event intake too, so a bridge that never ran the kit still meets them in production.

Asserting what rows become #

Conformance proves the protocol shape. A stream test proves every capability the bridge claims: what each delta becomes once the runtime has minted ids and built canonical events. The JSON-RPC harness drives the bridge; the collector assembles.

import { experimental_createBridgeDeltaEventCollector as createBridgeDeltaEventCollector,
         experimental_createBridgeJsonRpcTestHarness as createBridgeJsonRpcTestHarness }
  from "@get-bb/plugin-sdk/provider-bridge/testing";
import { handleLine } from "./src/provider-bridge.js";

const harness = createBridgeJsonRpcTestHarness(handleLine);
const collector = createBridgeDeltaEventCollector("echo-agent");

harness.sendRequest("1", "initialize", { protocolVersion: 2,
  client: { name: "test", version: "0.0.0" }, grammarVersions: [3, 3] });
await harness.waitForResponse("1");
harness.sendRequest("2", "thread/start", { threadId: "thr_test", cwd: "/workspace",
  instructionMode: "append", dynamicTools: [STAMP_TOOL_DEFINITION],
  options: { model: "echo-1", reasoningLevel: "medium", permissionMode: "full",
             permissionScope: "full", approvalReviewer: null, permissionEscalation: null,
             providerOptions: { shout: true, model: "echo-1", promptMode: null } } });
const { result } = await harness.waitForResponse("2");
harness.sendRequest("3", "turn/start", { threadId: "thr_test",
  providerThreadId: result.providerThreadId,
  input: [{ type: "text", text: "hello world", mentions: [] }],
  clientRequestId: "creq_ech2345678", options: FULL_OPTIONS });

// Play the runtime: answer the bridge's item/tool/call.
const call = harness.messages.find((m) => m.method === "item/tool/call");
handleLine(JSON.stringify({ jsonrpc: "2.0", id: call.id,
  result: { success: true, contentItems: [{ type: "inputText", text: "stamped: hello world" }] } }));

const events = harness.messages.flatMap((m) => collector.assembleMessage(m));
expect(events.map((e) => e.type).slice(0, 2)).toEqual(["turn/started", "turn/input/accepted"]);
for (const event of events.filter((e) => e.type === "item/completed")) {
  expect(event.item.presentation).toMatchObject({
    label: { pending: expect.any(String), completed: expect.any(String) },
    icon: { glyph: expect.any(String) },
  });
}
expect(collector.assembler.getBbItemId("thr_test", String(call.params.callId))).toBeDefined();

The collector throws on an invalid thread/delta rather than returning an empty list, so a bridge that emits garbage fails its suite loudly. Assert on the things the runtime cares about: that every item, opened and closed, carries a presentation; that a delegation's child turn links back through parentToolCallId; that the bb tool row carries server: "bb" and the definition's presentation; that an extension item arrives with the payload you sent (ingest validation is the server's job, and a separate server-level test); that usage and the context window ride the one usage dialect; that a zero-work turn assembles to exactly turn/started, turn/input/accepted, turn/completed.

A second block can drive the maintenance wire exactly as the runtime builds it (provider/health { providerId, cwd }) and parse the answer through providerHealthResultSchema, and check that undeclared maintenance methods answer METHOD_NOT_FOUND so declaration and bridge cannot disagree.

Recording a real session #

Set BB_PROVIDER_BRIDGE_RECORD_DIR=<dir> in the host daemon's environment and every bridge process tees the lines that cross its two boundaries into NDJSON files. The bootstrap records the runtime wire for every bridge; a bridge that spawns its provider child records the provider wire by calling experimental_recordProviderChildIo(child, { threadId }) right after spawn() (a no-op when record mode is off). A bridge whose provider pipe belongs to an SDK checks experimental_isProviderBridgeRecording() and takes the spawn over.

• Layout: <dir>/<providerId>/<threadId>/<direction>.ndjson, with _process for lines that belong to no thread (initialize, model/list, provider health). Directions: runtime→bridge, bridge→runtime, provider→bridge, bridge→provider.
• One entry per line: { ts, run, seq, dir, line }. seq is one counter across every lane of the process and run identifies the process, so the files merge back into exact order even across a bridge restart. Nothing buffers.
• A packaged recording adds a manifest.json (provider, cell, threadId, scope, cliVersion, recordedAt, description, note, bridgeRuns, lines). Redact before you commit: home paths, tokens and emails have no place in a fixture.

The cells every bridge is expected to reproduce (RECORDED_CONFORMANCE_CELLS): turn-tools, steer, stop-interrupt, approval-allow, approval-deny, user-question, resume, fork. A provider that legitimately refuses a cell (an ACP agent without session/fork) records no turns for it, and the replay must reproduce that rather than invent one.

Replay and parity #

A recording is replayed through the bridge the way the runtime spawns it: the recorded runtime lane is driven in, the recorded provider lanes are played by a replay child the bridge spawns in place of its real provider, and what the bridge writes back is assembled and diffed against the recording's own assembled events.

import {
  experimental_assembleRecordedEvents as assembleRecordedEvents,
  experimental_checkRecordedCellReplay as checkRecordedCellReplay,
  experimental_compareParity as compareParity,
  experimental_createBridgeDeltaEventCollector as createBridgeDeltaEventCollector,
  experimental_listRecordedCells as listRecordedCells,
  experimental_readBridgeRecording as readBridgeRecording,
  experimental_replayRecording as replayRecording,
  experimental_resolveProviderBridgeLaunch as resolveProviderBridgeLaunch,
  experimental_withCurrentBridgeLane as withCurrentBridgeLane,
} from "@get-bb/plugin-sdk/provider-bridge/testing";

const createAssembler = (providerId) => {
  const collector = createBridgeDeltaEventCollector(providerId);
  return { assembleMessage: (message) => collector.assembleMessage(message) };
};

for (const cell of listRecordedCells(RECORDINGS_ROOT)) {
  const recorded = assembleRecordedEvents(
    withCurrentBridgeLane(readBridgeRecording(cell.dir)), createAssembler, cell.provider);
  const run = await replayRecording({
    recordingDir: cell.dir,
    providerId: cell.provider,
    bridge: resolveProviderBridgeLaunch({ modulePath: BRIDGE_MODULE, pluginId: PLUGIN_ID }),
    createAssembler,
    planFromCurrentLane: true,
    timeoutMs: 60_000,
  });
  expect(run.stalls).toEqual([]);
  const comparison = compareParity(
    { events: recorded.events, rows: [], grammarViolations: recorded.grammarViolations },
    { events: run.events, rows: [], grammarViolations: run.grammarViolations },
    [], { provider: cell.provider, cell: cell.cell });
  expect(comparison.events).toEqual({ onlyInOld: [], onlyInNew: [] });
  const verdicts = checkRecordedCellReplay({ provider: cell.provider, cell: cell.cell,
    events: run.events, recordedEvents: recorded.events, stalls: run.stalls });
  expect(verdicts.filter((r) => r.status !== "pass")).toEqual([]);
}

• experimental_resolveProviderBridgeLaunch({ modulePath, pluginId }) builds the process exactly as the runtime spawns it, through the bootstrap the kit ships beside its bundle.
• A bridge with a provider child passes a ReplayProviderProfile whose env (or rewriteRuntimeLine, or prepareState) points the child at the kit's replay script. The replay child speaks json-rpc, claude-cli and pi-rpc dialects today; a CLI that speaks none of them cannot replay its provider lanes yet. The echo bridge has no child, so DEFAULT_REPLAY_PROFILE applies.
• The replay rewrites its temp workspace back to the recorded cwd in every line the bridge emits, so a bridge that derives paths from cwd compares with the recording.
• The recorded-cell verdicts are recorded/<cell>/replays (the bridge answers every recorded runtime request and needs no help), recorded/<cell>/events-schema-valid, recorded/<cell>/grammar, recorded/<cell>/turn-lifecycle (as many turns opened and settled as the recording), recorded/<cell>/not-empty.
• A recording is never rewritten. When a deliberate bridge change alters the stream, experimental_rerecordCurrentBridgeLane({ recordingDir, providerId, bridge, createAssembler }) writes the bridge's current output to bridge→runtime.current.ndjson beside the recorded lane, and experimental_withCurrentBridgeLane makes the comparison read it when it exists. The diff is the thing your pull request explains.

What "green" means #

• Conformance green: every scripted rule is pass (or skipped with a reason you accept). Your bridge is protocol-pure: hygiene, handshake, identity, turn lifecycle, schema-valid events, ordering, release semantics, id uniqueness across resume.
• Stream green: each row is what you intended, with presentation on every item, through the exact assembler the daemon runs.
• Parity green: zero event diffs, zero grammar drops, every recorded-cell verdict pass, zero stalls, exit code 0. A bridge change that alters real traffic fails here first.
• None of this tests your agent. It tests the translation between your agent and bb. A recording captured with a real agent makes the translation tests honest about the dialect.

The corpus #

Beyond the redacted recordings committed with each first-party bridge, bb's maintainers keep a larger private corpus of real recorded sessions and replay every bridge change against it, pinning per-cell event, row, provider/unhandled and grammar-drop counts so a regression in how a real dialect translates is caught before release. It is private and not shipped: nothing in the public kit depends on it, and your plugin's own recordings under your own repository are the equivalent for your provider.

Guarding the public-SDK rule #

Inside the bb monorepo a @bb/* import typechecks and runs, which is exactly why it needs a test: the workspace hides the privilege. The published scan does the walk: plugin code may import @get-bb/plugin-sdk and its /host, /app, /ai-services, /provider-bridge and /provider-bridge/acp subpaths, zod, node: builtins and relative files that stay inside the package root; tests may add /testing, /testing/app, /testing/host, /provider-bridge/testing and vitest; anything else goes in allow. A relative specifier that resolves outside the package root is reported as outside-package unless an allow pattern names it (inside the monorepo a climbing ../../../../packages/… import reaches a private package without naming it, which is exactly what an IDE auto-import plants), and an import() or require() whose argument is not a string literal is reported as dynamic-specifier with the argument text. It also reports the @bb/* names in package.json. The echo example and the bundled ACP plugin run it over themselves and name their one deliberate escape, the monorepo's shared vitest config, in allow:

import { dirname } from "node:path";
import { fileURLToPath } from "node:url";
import { describe, expect, it } from "vitest";
import { experimental_scanPublicSdkOnly as scanPublicSdkOnly } from "@get-bb/plugin-sdk/testing";

const scan = scanPublicSdkOnly(dirname(fileURLToPath(import.meta.url)), {
  // The one path that leaves this package: vitest.config.ts reads the
  // monorepo's shared test config. Named here so the scan admits nothing else.
  allow: [/^(?:\.\.\/)+vitest\.shared\.js$/u],
});

describe("echo-provider imports only the public SDK", () => {
  it("scans the plugin's source files", () => {
    expect(scan.files).toContain("server.ts");
    expect(scan.files).toContain("host.ts");
  });
  it("has no @bb/* import and stays inside the allowlist", () => {
    expect(scan.violations).toEqual([]);     // [{ file, specifier, reason }]; reason is
    // private-package | outside-allowlist | outside-package | dynamic-specifier
  });
  it("declares no @bb/* dependency in package.json", () => {
    expect(scan.privateDependencies).toEqual([]);
  });
});

Copy that test if you develop inside a fork of bb; outside it, the host artifact build enforces the rule for you. The scan finds specifiers with a regular expression over the source (from "…", import("…"), require("…")), not a parser: a non-literal import()/require() argument is reported rather than read, a string literal split across lines is missed, and a string that merely looks like an import (in a comment, say) is reported.