e2e/runner

e2e/runner runs a suite from code without invoking the CLI. run executes tests and returns their report, explore runs one exploration and returns its findings, list returns a selection, and openHost hosts e2e sessions in another test runner. Importing e2e from a test file does not load the runner.

import type { E2EConfig } from 'e2e';
import { run } from 'e2e/runner';
import { web } from '@e2e-dev/web';

const config: E2EConfig = {
  targets: [{ name: 'web', engine: web(), app: { url: 'http://localhost:3000' } }],
};
const controller = new AbortController();
const result = await run({
  config,
  files: ['tests/signup.e2e.ts'],
  interruptSignal: controller.signal,
  onEvent(event) {
    if (event.type === 'test-started') console.log(event.title);
  },
});

console.log(result.status, result.report.run.summary);

​

Reference

​

Functions

​

run

run(options?: RunOptions) executes a complete suite. It finishes cleanup before resolving and never exits the caller’s process.

const result = await run({ files: ['tests/signup.e2e.ts'] });
process.exitCode = result.exitCode;

Options

• files takes paths, globs, bare names, or file:line.
• changedSince works as --changed-since.
• Aborting interruptSignal interrupts tests and executes teardown.
• onEvent receives typed progress events. Terminal progress is off. It must not block. A throwing or rejecting callback is disabled without failing the run.
• Put execution settings, such as retries and workers, in the config.
• A runtime config (an E2EConfig value) runs with one worker in the caller’s process. A file config supports the configured worker processes.

Result

Test failures, config failures, and interruption resolve with their status and errors in the report.

• Helper modules your test files import stay cached between run calls in one process. An edit to them needs a new process.
• Each run that returns a report sends one anonymous e2e_run_completed event, with the same opt-outs as the CLI. The first call can print the telemetry notice on stderr.
• Suite output uses UTF-8 text redaction and does not preserve binary data. Your application’s logs and progress callback output stay under your control.

​

explore

explore(options?: ExploreOptions) runs one exploration, the run e2e explore starts. It finishes cleanup before resolving and never exits the caller’s process.

import type { E2EConfig } from 'e2e';
import { explore } from 'e2e/runner';
import { web } from '@e2e-dev/web';
import { gateway } from 'ai';

const config: E2EConfig = {
  agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
  targets: [{ name: 'web', engine: web(), app: { url: 'http://localhost:3000' } }],
};
const controller = new AbortController();
const result = await explore({
  config,
  goal: 'Explore the checkout flow like a first-time buyer and report anything off',
  maxSteps: 4,
  interruptSignal: controller.signal,
  onEvent(event) {
    if (event.type === 'explore' && event.progress.phase === 'finding') console.log(event.progress.finding.title);
  },
});

console.log(result.status, result.explore.ended, result.explore.findings.length);

Options

• goal is at most 2,000 characters.
• session names a session a setup test saves. That setup runs first, and the exploration starts from it, as --session does.
• maxSteps is from 1 to 12. timeoutMs is from 180000 to 900000.
• onEvent receives the same events as for run, plus an explore event for each planning decision, step, and finding. What e2e explore prints before the run starts, such as the target it chose, arrives as a notice event with target run, before the plan event.
• Recording and output settings, such as video, trace, and output, come from the config.

Result

ExploreResult has every RunResult field, plus explore, the record report.run.explore also holds.

Once the exploration starts, findings, failures, and interruption resolve with their status in the report. Options or a config that cannot start one reject first, as the matching e2e explore flags do. Recognize them with isE2EError.

Each exploration that returns a report sends one anonymous e2e_run_completed event, as run does.

​

list

list(options?: ListOptions) collects and selects the way e2e list does, and stops there. No app process, engine, or worker starts, and nothing is written.

import { isE2EError, list } from 'e2e/runner';

const { pairs, unmatched, targets } = await list({
  files: ['tests/signup.e2e.ts'],
  grep: [/checkout/],
});

for (const pair of pairs) {
  console.log(`${pair.id} ${pair.disposition} ${pair.titlePath.join(' › ')}`);
}

Each call imports the test files again, so a later list() in the same process sees an edit.

Options

• targets names targets, as --target does.
• files is the positional file selection: paths, globs, bare names, and file:line.
• tags, tagMode, and excludeTags are the --tag, --tag-mode, and --exclude-tag filters.
• grep and grepInvert: a test’s describe titles and title, joined by spaces, must match one grep pattern and no grepInvert pattern.
• lastFailed keeps only tests the previous run did not pass, read from report.json under output.
• shard is the --shard index/total slice.
• changedSince works as --changed-since. The test filters run in the calling process; list prints none of their lines, e2e list does.
• passWithNoTests resolves with no runnable test instead of throwing NO_TESTS.
• output is the results directory.
• env is the environment the config resolves against.

Result

ListResult is { pairs, unmatched, targets }.

• pairs is one ListedPair per test, target, and agent, in selection order, including pairs a filter removed. Repeats are not expanded.
• unmatched is the positional arguments, as written, that selected no discovered file.
• targets is the selected target names, in config order.

e2e list prints the run and skip pairs, and the pairs a test filter left out. list returns every filtered pair.

Errors

A failure is the same classified error e2e list exits with. Recognize one with isE2EError and branch on error.code.

import { isE2EError, list } from 'e2e/runner';

try {
  await list({ targets: ['missing'] });
} catch (error) {
  if (isE2EError(error) && error.code === 'UNKNOWN_TARGET') {
    console.error(error.message);
  }
}

​

openHost

openHost<Fixtures>(options?: OpenHostOptions) prepares e2e for Playwright Test, Vitest, Jest, or your own harness, and returns a host.

import { openHost } from 'e2e/runner';

await using host = await openHost({ target: 'web' });
await using session = await host.openSession({ name: 'checkout' });
await session.fixtures.app.open('/checkout');

• Your runner owns collection, retries, and reporting.
• The host prepares the target’s engine, starts app processes, and initializes the engine once per worker lifetime. Use a worker-scoped host and a test-scoped session; see the Playwright Test example.
• Each host.openSession creates a test-lifetime attempt with fresh fixtures, a deadline, steps, and artifacts.
• If opening fails, e2e tears down what started before throwing the error.

Options

• config is a config file path relative to cwd, or an E2EConfig value. Omit it to discover the config from cwd, the same walk list does.
• cwd is the directory for config discovery, and the project root of a config value.
• target is required when the config declares several targets.
• env is the environment the config, engines, and providers read.
• signal cancels host provisioning, a pending session open, or live fixture calls.
• headed runs a browser engine headed.
• onNotice receives progress outside a step, such as provisioning or an app process. It must not throw.

A host given a config path loads the file and its project-file imports fresh. Package imports stay shared, so their singleton state is not recreated per host. A config value uses its own engine instances. Each instance belongs to one host until that host closes, so a second host must use another instance.

Opening failures are classified, redacted errors; raw engine causes are not exposed. Match them with isE2EError and error.code.

​

isE2EError

A type guard that recognizes a classified e2e error. Branch on its error.code.

if (isE2EError(error) && error.code === 'NO_TESTS') process.exitCode = 0;

instanceof can miss an e2e error across copies of the package, so use isE2EError. ConfigurationError and isE2EError are both exported from e2e/runner.

​

Host

E2EHost<Fixtures> is what openHost returns.

runId and target are the identity the host’s sessions share.

Only one hosted attempt may be opening, live, or closing per process, even across different hosts and whether the config is a file or a value.

• An overlapping open is rejected before engine attempt state starts. Await session.close() before opening the next session on the same host.
• A launch that fails or is cancelled ends what it opened and leaves the host usable, as a run keeps its worker for the retry.
• A session whose engine cleanup failed closes the host permanently, since its isolation is uncertain; open a new host.

​

host.openSession

host.openSession(options: OpenSessionOptions) opens a fresh session on this host’s target. A session is one test attempt with no retry loop. A retry opens a new session on the same worker host.

const session = await host.openSession({
  name: 'checkout',
  timeout: 60_000, // default: the config's timeout
  onStep: (step) => {
    if (step.phase === 'end') console.log(`${step.api} ${step.label}: ${step.status}`);
  },
});

Options

• name is the test name for step and trace reporting.
• timeout is the deadline for the session’s operations in milliseconds, positive and finite, counted from when the session opens. It does not bound provisioning or opening, and it does not close the session.
• signal cancels opening or live fixture calls. The host’s signal also applies. Provisioning happens in openHost, so only the host’s signal cancels it; either signal cancels a launch.
• agent is the agent the agent fixture uses when a call names none.
• onStep receives live StepProgress, the same feed reporters receive. It must not block or throw. Its phase is start, event, activity, replay, or end. An end event includes status, durationMs, modelCalls, and a redacted error when the step failed.
• timeout, agent, and onStep belong to the session, not openHost.

​

host.close

host.close() cancels any active or pending session, waits for it to close, then disposes the engine, stops app processes, and releases provisioning.

const failures = await host.close();
if (failures.length > 0) throw new AggregateError(failures, 'e2e host close failed');

• Idempotent: it disposes and releases resources only once.
• A host is AsyncDisposable: await using host = await openHost() closes it at scope exit and throws an AggregateError when closing reports failures.

​

Session

E2ESession<Fixtures> is what host.openSession returns.

• fixtures holds agent, app, screen, platform, and the engine’s fixtures.
• runId and target are the host’s run and target; attemptId is unique to this attempt.
• steps() returns read-only StepRecords for steps run so far, as the report records them.
• report() returns the session Report the trace is rendered from, after close and a successful trace write; otherwise undefined. Its status describes the SDK session, not your runner’s test-body verdict.
• artifactsDir is the absolute directory for this attempt’s screenshots, recordings, and trace.
• tracePath is the absolute path to trace.md; attach or read it after close() settles.

​

session.close

session.close() cancels remaining operations, ends the attempt, and writes its trace. It returns soft expectation, abandoned-operation (an unsettled poll or step), and cleanup failures instead of throwing them.

const failures = await session.close();
if (failures.length > 0) throw new AggregateError(failures, 'e2e session close failed');

• Idempotent: later calls return the same failures.
• The host stays open for the next test.
• A session is AsyncDisposable: await using session = await host.openSession({ name: 'checkout' }) closes it at scope exit and throws an AggregateError of the returned failures.

​

Deadline and soft failures

expect.soft collects failures for the session’s whole lifetime. Check close()’s result to fail the host runner’s test.

• expect.poll uses the session’s remaining budget, even when its own timeout is longer.
• Contributed fixture operations also honor that budget, including operations that disable their own timeout.
• When the deadline passes, a poll or operation that was watching it reports its own cut. Everything else still running, including an operation an engine or a model never answered, fails with TEST_TIMEOUT.
• Either way the session’s outcome is timed out and close() returns the timeout. Later operations fail with TEST_TIMEOUT; close the session.

​

Artifacts

Artifacts live under <output>/results/host/<resultSegment>/. At close, e2e writes trace.md there through the same redacted formatting reports use.

• e2e generates a safe resultSegment from the name and the unique attempt identity. The name never becomes a raw path component.
• Repeated names and retries get separate directories.
• With provider-backed secrets, the directory uses an identity-only segment, since unresolved values cannot be redacted from a title slug. The trace still shows the redacted name.
• Artifact paths are relative to <output>/results, not session.artifactsDir.
• Video follows the target’s mode for a first attempt. Failure-only video is kept for soft failures and for failed, blocked, timed-out, or cancelled steps.

​

Types

e2e/runner exports these types beside the functions.

​

Examples

​

Playwright Test fixtures

A worker-scoped host and a test-scoped session:

import { test as base } from '@playwright/test';
import { openHost, type E2EHost, type E2ESession } from 'e2e/runner';
import type { Device } from '@e2e-dev/mobile';

type Fixtures = { device: Device };

const test = base.extend<
  { e2e: E2ESession<Fixtures> },
  { e2eHost: E2EHost<Fixtures> }
>({
  e2eHost: [async ({}, use) => {
    const host = await openHost<Fixtures>({ target: 'android' });
    try {
      await use(host);
    } finally {
      const failures = await host.close();
      if (failures.length > 0) throw new AggregateError(failures, 'e2e host close failed');
    }
  }, { scope: 'worker' }],
  e2e: async ({ e2eHost }, use, testInfo) => {
    const session = await e2eHost.openSession({
      name: testInfo.titlePath.join(' › '),
      // Playwright's 0 disables its timeout; e2e then falls back to the config's.
      timeout: testInfo.timeout || undefined,
      onStep: (step) => {
        if (step.phase === 'end') console.log(`${step.api} ${step.label}: ${step.status}`);
      },
    });
    try {
      await use(session);
    } finally {
      const failures = await session.close();
      await testInfo.attach('e2e trace', { path: session.tracePath, contentType: 'text/markdown' });
      if (failures.length > 0) throw new AggregateError(failures, 'e2e session close failed');
    }
  },
});

test('turns on dark mode', async ({ e2e }) => {
  const { agent, app } = e2e.fixtures;
  await agent.act('open settings and turn on dark mode');
  await app.screenshot();
});