Agents

In Millstream an LLM agent is an action like any other. It declares its prompt, its model, the schema of its answer and what it may touch, and the platform does the rest: rendering, sandboxing, validation, cost, transcripts and tests.

The manifest

An agent action is a directory with an action.yaml, a prompt.md and JSON Schemas for its input, output and config.

name: lab/score-listing
version: 2.1.0
description: Scores a listing for renovation potential
kind: agent
effects: none
input:  { $ref: schema/input.json }
output: { $ref: schema/output.json }
runtime:
  agent:
    harness: claude-code
    model: claude-sonnet-5-5
    prompt: prompt.md
    tools: [WebFetch]
    mcp: []
    network: [api.anthropic.com]
    maxTurns: 20
    maxCostUsd: 0.50

For each item, the agent adapter renders prompt.md with the payload, the config, the variables and the step's lessons, runs the harness, obtains a structured answer and validates it against the output schema. An answer that does not match is a retryable error and never reaches the next step. Each answer is emitted with its metrics: tokens in and out, cost, turns and duration, labelled with the model.

Answers are untrusted input

Crawled pages and documents reach agents, and a page can steer a model. Millstream treats every answer as data from outside the system.

• Validated before it goes anywhere. Every answer is checked against the action's output schema. A failed answer is retried, and its transcript kept.
• Abstaining is an answer. An agent unsure of an item can return {"abstain": "<why>"}, which hands the item to the step's fallback instead of the next step.
• Effects need a person. millstream check warns when an agent's output can reach an action declared effects: external, such as an email or a CRM write, with no review in between.
• Decisions stay in the pipeline. The agent returns a measurement, a score or a category, and a filter step holds the threshold, where it is visible and changed without touching the prompt.

Any harness, any model behind it

The engine never knows which model ran. The adapter speaks the action protocol on one side and drives a harness on the other.

Agents can also abstain. An output schema that allows {"abstain": "<why>"} lets the model hand an item it is unsure of to the step's fallback, such as a stronger model or a person.

Open-ended work, inside one step

Some work has no fixed plan: searching, reading documents, writing and running code until a result holds. With the claude-code harness, an agent step does that work with the tools its manifest lists, such as Bash, Read and WebFetch, inside its sandbox and with the interpreters its sandbox image holds, for up to maxTurns turns and maxCostUsd. The pipeline around it stays a fixed graph that millstream check validates, and the step's final answer passes the output schema like any other. The api harness makes one model call per item; its tool loop is on the roadmap.

A sandbox per call

Agents run per item, in a fresh container, so no context leaks between items.

• A read-only file system except the delivery's working directory.
• Only the tools, MCP servers and network destinations the manifest lists.
• No secrets beyond the step's config, and no variable from the node's environment.
• No capabilities, no-new-privileges, the node's own user.

Budgets that stop spending

Cost is taken from the harness when it reports it, otherwise computed from token counts and the price table in the environment file.

• Per delivery: maxCostUsd stops the agent; an attempt that ends above it is refused.
• Per run and per day: budget.runUsd and budget.dayUsd are enforced by reservation at dispatch, so an agent call starts only when the spend so far plus its own reservation fits.
• When a budget is reached, the pipeline's agent steps pause with the cause recorded, and resuming is a decision someone makes.

budget:
  runUsd: 25
  dayUsd: 120

Transcripts for every attempt

The full transcript of each attempt, prompts, tool calls, results and the final answer, is stored compressed in the artifact store and shown on the item's page. A failed attempt keeps its transcript too, and its error points to it. Every secret value of the pipeline is redacted from every string, the harness's own records included. Transcripts have their own retention.

Evals, and tests that never call a model

An agent action ships an evals/ directory of cases: an input, assertions in CEL, and optionally an LLM judge with a rubric. millstream eval scores each case, records the scores against the action's digest, and shows the previous score beside each one, so a prompt or model change is compared before it ships.

name: renovated flat
input: { url: https://example.com/listings/2 }
assert:
  - output.score < 30
judge:
  rubric: The score reflects the recent renovation.
  model: claude-haiku-5-5

Every harness call can be recorded and replayed. A recording is keyed by the harness, the model, the rendered prompt and the output schema, so tests and evals in CI run offline, need no API key, and are deterministic. Editing a lessons file changes the prompt and misses the old recording, so a stale answer is never replayed.

millstream eval ./actions/score-listing
millstream test action ./actions/score-listing --harness-mode replay

Lessons from past runs

Each agent step can carry a lessons file of rules and corrected examples, each citing where it came from: a reviewer's edit, an insight, an accepted proposal. Lessons belong to the step, not the action, so a shared action learns what is specific to each pipeline. The file is part of the revision, so replay stays deterministic.

rules:
  - text: Listings without a surface area are not scored; return score 0 and reason "missingSurface".
    source: proposal/p-0142
examples:
  - input: { … }
    output: { … }
    why: "Reviewer lowered the score: ground-floor flats on a main road."
    source: review/r-8812

Next

How pipelines learn Human review Security