Protocol

The op stream

What the core hands a renderer as an answer streams: one JSON operation per line, for drawing answers on a platform of your own.

View as Markdown

The SDKs never parse the format: they apply the core's operations. output: "ops" on /v1/ui and /v1/render gives you the same operations, for drawing answers yourself, without the core.

The core turns chunks of model text into a stream of operations, one JSON object per line (served as SSE data: lines by the API). Every renderer, on every platform, only applies ops; none of them parses the format.

Nodes

The document is a tree. A node's id is its position: the indexes from the root joined with dots ("3", "3.0", "3.0.2"). Ids never shift, because nodes only ever arrive after the ones before them.

Node type What it is Holds
A component name (Card, Chart, Camera, a customer's own) a component, aliases already resolved children, spans, or data, by its content kind
p paragraph spans
h heading; attrs.level 1 to 6 spans
ul, ol list li children
li list item; attrs.task is "open" or "done" for task items spans
table table row children
row table row; attrs.head is true for the header row cell children
cell table cell; attrs.align is "right" when its column is numeric spans
code fenced code; attrs.lang when given spans (one text span)
quote block quote spans
hr thematic break nothing

Ops

Op Fields Meaning
open id, type, attrs (left out when empty) A node starts; its parent is its id up to the last dot. A component's attributes are final when it opens (the tag was complete). Attribute values are strings, or true for a bare attribute.
spans id, spans The whole inline content of an open leaf so far (a string when it's plain text). Replaces the previous spans of that node. Sent at most once per chunk of input, and before the node closes.
node id, type, attrs, spans, data (each left out when empty) A leaf that opened and finished within one chunk: open, spans and close in one op.
close id, optional data The node is final and never changes again. Components with table, list or image content carry their parsed content in data.
diag line, cls, detail, severity A problem (error) or a coercion the core made (note), tied to the line, for repair.
end complete, truncated The stream is over. Anything still open was closed first, with a diagnostic.

data by content kind:

  • table (Chart, Map): {"columns": [...], "rows": [[...], ...]}, cells as the model wrote them (renderers parse numbers for charts).
  • list (FollowUps): {"items": [spans, ...]}.
  • images (Gallery): {"images": [{"src": "...", "alt": "...", "credit": [spans]}]} (credit only when the image has one).
  • A Question (interactive pack): {"items": [spans, ...], "answer": index or null}; the core matches answer to a choice by its letter, number or words.
  • A Checklist: {"checks": [{"done": true or false, "c": [spans]}]} ([x] marks done).
  • A What-if Result: {"formula": [tokens], "value": number or null}: the formula in postfix tokens (numbers, the Sliders' names, + - * / ^ %, neg, min, max, round, floor, ceil, abs, sqrt) and its value at the Sliders' starting values. A Slider's name is its name, or its label as one word ("Unit price ($)" is unit_price). Renderers run the tokens on a stack as the Sliders move; nothing else runs.

Spans

Inline content is an array of spans:

Span Fields
{"t": "text", "v": "..."} plain text
{"t": "strong" | "em" | "s" | "u" | "small" | "sub" | "sup" | "mark" | "kbd", "c": [spans]} formatting
{"t": "code", "v": "..."} inline code
{"t": "link", "href": "...", "c": [spans]} link
{"t": "img", "src": "...", "alt": "...", "credit": [spans]} inline image; credit is its title, parsed, when it has one
{"t": "br"} line break
{"t": "el", "type": "Badge", "attrs": {...}, "c": [spans]} a component inside a sentence or a table cell

The core strips emojis (keeping ©, ® and ™), reads the HTML models slip in as formatting, and keeps a dash between words from starting a line, so every renderer shows the same text.

Example

Text
{"op":"open","id":"0","type":"h","attrs":{"level":"1"}}
{"op":"spans","id":"0","spans":[{"t":"text","v":"Revenue overview"}]}
{"op":"close","id":"0"}
{"op":"open","id":"1","type":"Card","attrs":{"title":"Weekly revenue"}}
{"op":"open","id":"1.0","type":"Chart","attrs":{"type":"line","x":"Week","y":"USD"}}
{"op":"close","id":"1.0","data":{"columns":["Week","Revenue"],"rows":[["W1","24100"],["W2","30800"]]}}
{"op":"close","id":"1"}
{"op":"end","complete":true,"truncated":false}

Actions

Components never run code. When the user acts, the renderer sends an action, and the API turns it into the model's next turn.

User does Action Next model turn
Taps a Button with ask, or one with no href outside a Form {"type": "ask", "text": ask or the label, "node": id} a user message with that text
Taps a FollowUps item {"type": "ask", "text": the item, "node": id} a user message with that text
Taps a Button with href nothing is sent; the link opens none
Submits a Form (any Button inside it) {"type": "submit", "node": form id, "form": its name, "button": the label, "fields": {...}} a user message listing each field's label and value, with photos and files attached
Switches a Tab, opens Details nothing; local state none
Answers the last question of a Quiz {"type": "quiz", "node": id, "score": n, "total": n, "answers": [{"question", "choice", "right": true, false or null}]} a user message with the score and each answer
Votes in a Poll {"type": "vote", "node": id, "name": its name, "choice": the option} a user message with the vote
Ticks a Checklist item, marks a Flashcard known {"type": "state", "node": id, "name": the Checklist's name or "known", "value": what's ticked or known} none: it's component state (below)
Moves a What-if Slider nothing; the Results are worked out on the device none

fields is keyed by each field's name (or its label as a slug when it has none):

Field Value
Input, TextArea, Select, Radio, Date, Scan a string
Slider a number
Switch true or false
Checkboxes an array of the chosen options
Camera, File an array of the platform's file objects (File on the web)
Location {"lat", "lng", "accuracy"} in degrees and meters

Photos and files come to the app in the action as the platform's own file objects (a File on the web), for the app to upload where it likes; with AG-UI they go to the agent as content parts. The renderer checks required before sending. Device values exist only after the user acted: a tap took the photo, a tap shared the location.

Component state

What the reader has done to the interactive components and forms is also kept as plain JSON per node, the same shape on every platform, so it can be saved, restored and shared with an agent. With AG-UI it is the shared state's morphscreen key, by message id and node id, sent with every run; an agent that sets it sets the components, and that sends no action.

Component State
Quiz {"answers": [index or null per question]}
Flashcards {"known": [card index]}
Poll {"vote": option or null}
Checklist {"done": [item index]}
WhatIf {"values": {slider name: number}}
Form {"fields": {name: value}}, typed as above; uploads and locations stay on the device

The words of an action's user message come from the core (morphscreen_core::action::message), so every SDK and the API send the same turn. docs/agui.md has the rest.