Portable Media

Selected native implementation in progress. The approved R1 Media panel connects an admitted native document to explicit Run/Stop, color camera preview, completed analysis and immutable recording artifacts. The built-in macOS camera flow has local verification; SDK distribution, Linux/RTSP and packaged Desktop verification remain separate work. This is not a public packaging claim.

Media describes sources, typed tracks, transforms, sampled analysis and session outputs. Evaluating a description performs no capture, recording or AI request. A source binding names a device or approved network source configured by the model owner. Credentials, source addresses and backend settings stay outside model text; formulas cannot supply native elements or arbitrary plugin settings.

VIDEO_RESIZE, VIDEO_CROP and VIDEO_RATE require compiler-lowered Media descriptions. An eager native call with already evaluated arguments returns MEDIA/CONTEXT; it cannot construct an admitted graph or acquire a source.

Authoring

source = MEDIA_SOURCE("entrance")
video = MEDIA_VIDEO(source)
analysis = MEDIA_ANALYZE(
  video,
  (image) => {regions: ROWS(IMAGE_COMPONENTS(IMAGE_THRESHOLD(IMAGE_GRAYSCALE(image), 0.6)))},
  {rateHz: 5, width: 640, height: 360}
)
entrance = MEDIA_SESSION({video: video, analysis: analysis})
latest = MEDIA_LATEST(analysis)

The namespace form constructs the same session from ordinary assignments:

MEDIA Entrance
  source = MEDIA_SOURCE("entrance")
  video = MEDIA_VIDEO(source)
  analysis = MEDIA_ANALYZE(
    video,
    (image) => {regions: ROWS(IMAGE_COMPONENTS(IMAGE_THRESHOLD(IMAGE_GRAYSCALE(image), 0.6)))},
    {rateHz: 5, width: 640, height: 360}
  )
END MEDIA
latest = MEDIA_LATEST(Entrance.analysis)

Blocks preserve lambda and LET lexical scope. Newline, CRLF and semicolon separators are accepted. Empty, duplicate and unterminated blocks are rejected. A block supplies a namespace; it supplies no evaluator, clock or activation.

Native Media inspection

A Media inspection configuration can select one exported session:

Review!type = "media"
Review!config = <toml>
  kind = "media"
  version = 1
  title = "Built-in camera"
  [session]
  name = "Camera"
</toml>

Camera must export a MEDIA_SESSION in the same native model. See the complete camera example. The authenticated document owner also needs a private source grant and a host packaged with media-analysis and the selected connector. Neither the session name nor its inspection configuration grants device authority. Hosted model names and missing native ownership cannot substitute for that grant.

Opening the panel reads the admitted catalog without starting a source. Run starts one generation shared by preview, analysis and recording. Stop fences results, drains recording within the admitted deadline and releases acquisition. The panel reports failures and distinguishes incomplete recordings. Refresh status can retry artifact finalization without acquiring the source again. Recordings remain immutable artifacts of the current native incarnation; native File archive/reopen handoff is not yet connected. Download recordings before closing that incarnation when a portable copy is needed. Downloads up to 64 MiB verify content size and SHA-256 before handing bytes to the browser. Larger recordings remain retained and require another admitted export adapter. Matroska playback depends on the viewer's codec support.

Preview uses bounded binary RGBA transport and the existing Render miss/repeat policy. One preview request is in flight per visible panel; no preview loop exists before Run or after Stop. Raw frames never enter Workbook cells, Stream or JSON/base64. Latest analysis retains generation, sequence, control revision and distinct media/ arrival timestamps. Missing capture time stays unknown. The camera example admits no microphone and sends no AI images.

Types and graph boundaries

The inferred opaque types are Media, Video, Audio, MediaAnalysis and MediaSession. A description carries a graph identity rather than continuously changing pixels. Shared source descriptions use one acquisition pipeline within a session. Graphs must be acyclic, and each operation must receive the correct track type.

Native analysis receives an immutable image and settled scalar controls. It must return a fixed, bounded scalar or object shape, with a maximum result size of 64 KiB. An image intermediate cannot escape as an unbounded result. Asynchronous AI requests belong to the separate MEDIA_OBSERVE branch and require explicit permission to send images.

MEDIA_LATEST has no completed result before the first successful analysis. Reading it starts no acquisition and waits for no frame. MEDIA_EVENTS is the bounded projection into the existing native Stream collection. XLEN, XREAD, XRANGE and consumer groups retain their existing semantics. Durability and wall-clock event-time mapping require a separate private admission into the existing Stream source/checkpoint owner. Raw frames remain transient.

Inferred types and results

The compiler preserves distinct Media, Video and Audio descriptions through immutable aliases and named-function signatures. MediaAnalysis<T> preserves the callback's inferred result T. MediaSession<Outputs> preserves each named output, so Entrance.analysis retains its exact analysis result. These identities are static compiler facts; a record with a binding field grants no source authority. Lowering still validates graph shape, callback effects and bounded result schemas.

MEDIA_LATEST(analysis) has a closed completed-result shape: {identity: {...}, value: T}. As with other workbook value types, Blank can inhabit this shape: before completion the whole value is Blank. Unavailable timestamp fields project as Blank. No frame is acquired by a read, and no default result is invented. The identity fields generation, sequence, ptsNs, dtsNs, runningTimeNs, durationNs, capturedAtMs, arrivedAtMs and controlRevision use decimal text to preserve unsigned 64-bit values. Capture time remains unavailable for the synthetic source. Media time and capture/arrival wall time remain separate.

MEDIA_STATUS(session) preserves the closed generation, state, counters, failedOutputs and failureCode record. Generation uses decimal text; counters and output node IDs use numbers. A session without a published status is Blank. MEDIA_EVENTS preserves completed-result typing and publishes a native Stream; each entry's fields.result contains the same {identity, value} as Latest. MEDIA_OBSERVE has a closed result {text: String, completedAtMs: Number, validUntilMs: Number} from the existing read-only AI owner. Native images remain scoped inputs. A declaration grants neither acquisition nor image-send authority.

The recording constructor admits known Media, Video or Audio inputs. An unresolved wrapper parameter remains polymorphic until graph lowering checks its instantiated track; it is not narrowed to Media-only or granted authority. Marked builtin spellings such as MEDIA_SOURCE.(binding) retain the same types.

The synthetic analysis example uses one admitted acquisition, a bounded image callback, a settled threshold, Latest/Status and recording. A host must privately bind synthetic to the test pattern (and explicitly admit a test tone if audio is wanted), install its owner grant, and activate the session. Model evaluation alone remains inert.

Connected outputs

preview = RENDER(video, {rateHz: 30, blockSize: 1})
events = MEDIA_EVENTS(analysis)
completedCount = XLEN(events)
completed = XREAD(events, "0-0", 32)

The compositor explicitly attaches one presenter to the active session's Preview node. It pulls immutable packed RGBA leases through the existing Render driver, ExternalVsync clock boundary, deadline accounting and repeat-last miss policy. The shared Render epoch starts at preview admission before capture, preserving first-frame staleness. Its callback does not resolve model cells or lock Host. Native adapters can use detach_media_display_renderer_as with the admitted owner/session generation, and render_to_lease publishes into the existing bounded binary presentation lease using caller-owned scratch memory. A second presenter cannot steal the same branch; Stop fences detached presenters. This connects the native presentation boundary and retains the current inspection UI. The first Media Render configuration supports rateHz and blockSize: 1.

Events append at completion, independently of coalesced Latest publication. The current history retains at most 256 results and 4 MiB of encoded result payload, with explicit prefix reclamation and a selected event_trimmed(node) inspection counter. This is bounded history, not guaranteed durable delivery. Entry IDs order completions using nondecreasing arrival coordinates and sequence; PTS and running time remain metadata. Before activation the projection is Blank.

A private MediaEventDeliveryAdmission can bind an authored Events node to an approved coordination event source, a Capture or Arrival timestamp mapping, and mandatory per-generation event/byte limits. Capture mapping refuses unknown capture time; Arrival explicitly means arrival event time. Only completed bounded results enter the existing checksum-framed source log. An ordinary FROM_PIPE declaration can select its existing windows, ordering, receipts and checkpoints. Capacity or I/O failure terminates that analysis output and degrades the session. Re-evaluating Latest never journals a result or a frame.

observation = MEDIA_OBSERVE(video, "scene-description", {intervalSeconds: 5, maxCalls: 12})
observed = MEDIA_LATEST(observation)
monitored = MEDIA_SESSION({video: video, preview: preview, analysis: analysis, observation: observation})

Before activation, the authenticated source owner separately authorizes image sending and binds scene-description to an already admitted, fresh, dedicated Observe task for this model. authorizeMediaObservationV1(modelId, taskBinding, {runId, allowSendImages: true, expiresAtMs, maxAgeMs}) references the existing AI run; it cannot provide a provider endpoint, credentials or executable. The run must be volatile, single-task and within both the graph's call allowance and a maximum ten-minute authorization. Binding is exclusive and cannot reuse an input that has already received observations. The task owns its provider, prompt, call reservations, freshness, output bounds and cancellation.

A sampled image must have trustworthy capturedAtMs. Arrival and PTS never fill that field. Unknown time refuses the observation output without sending an image; other outputs may continue. Current synthetic GStreamer frames have unknown capture time. Positive observation verification uses a deterministic producer that timestamps its own generated frame and an in-process provider. Physical source timestamp extraction/acceptance remains separate work.

Selected frames are bounded to 640×360, encoded once at AI ingress, and admitted by the existing image contract. Observation text retains the existing AI owner's 1 MiB output bound; native callback results retain their 64 KiB bound. One operation is in flight with one newest pending sample. Completed text publishes through Latest and Events only while fresh and in the same generation/control revision. Stop cancels the dedicated AI run and wakes completion waits. Cancelling the existing agent task also withdraws the observation branch. Restart and recovery never renew that run or its exhausted authorization.

Options and limits

Analysis options use rateHz, width and height. Observation options use intervalSeconds (2, 5 or 10) and maxCalls (6, 12, 24 or 32); a bound task may admit fewer calls. Recording uses durationSeconds and maxBytes. A deployment may admit tighter limits. The initial recording profile is Matroska with VP8 video and optional, explicitly approved Opus audio. Selecting a video source never implicitly opens a microphone.

recording = MEDIA_RECORD(
  source,
  {durationSeconds: 600, maxBytes: 1073741824}
)
 
preview = RENDER(video, {rateHz: 30, blockSize: 1})
entrance = MEDIA_SESSION({video: video, preview: preview, recording: recording})

Activation and controls

Activation is a separate owner-authorized action. The desktop activation workflow and unattended edge startup remain in progress. An edge deployment may approve bounded recovery. Changing graph topology, source capabilities, backend or AI selection requires session replacement. Settled scalar controls update at frame boundaries.

The session states are starting, running, degraded, stopping, stopped and failed. Preview keeps the newest pending frame. Native analysis allows one operation in flight and one newest pending sample. AI observation retains its bounded request window and freshness checks. Recording uses bounded buffering; a recording failure marks that output degraded while preview and analysis may continue. Queue bounds do not bound all decoder or process memory.

Stopping releases acquisition and drains recording within a deadline. A failed teardown remains visible. Edge recovery starts a new generation and recording segment, and does not renew exhausted AI authorization.

Native document activation

An already admitted native document exposes mediaSession through the existing authenticated document RPC. The owning Desktop gateway signs the exact path and body with its authenticated principal. Private startup grants additionally bind documentId, model locator and owner; the same model name in another document cannot borrow them. Hosted grants cannot activate native documents.

The bounded command selects catalog, start, status, latest, settle, stop or finalize. Sessions and analyses use their authored exported names; generation and control revisions use canonical unsigned decimal text. This adapter requires selected media-analysis Host packaging; absent capability and the stronger coordination lifecycle profile fail closed. Start settles the existing native image before acquisition. Stop drains and returns immutable recording artifacts. File Close/source publication confirm worker cessation before custody release; a new teardown failure keeps the File open for explicit retry. Catalog and status never activate a camera. The frontend's nativeMediaSession transport preserves the exact document/model incarnation and refuses stale replies or hosted fallback. This supplies explicit activation plumbing; the current Media inspection view has not been redesigned or connected to a new camera Run control.

The private macOS source selector mac_camera_uid resolves its admitted device UID through AVFoundation during selected worker activation. An absent UID fails with MEDIA_CAMERA_UID_UNAVAILABLE; it never chooses another camera index. Source addresses, UID, resource scopes and SDK/plugin selection stay outside Grid text and public commands. No microphone is opened by camera selection.

Finite local QA now covers the built-in camera at 1280×720/30 requested capture, actual Image analysis in the compiled model, formula Latest/Status and native Stream, authenticated control updates, Stop, immutable VP8/Matroska recording and RGBA decode. Negative checks cover signatures/body substitution, nonce replay, same-name foreign documents, stale generations, closed owners, active Close/failed-teardown retry and absent devices. Camera wall-clock capture time remains unknown, so this does not enable the live AI branch.

Time and completed outputs

Each result identifies its source generation, frame sequence and control revision. Media timestamps, decoding timestamps when available, running time, duration, arrival time and trustworthy capture time remain distinct. Capture time stays unknown when the source supplies no reliable evidence. Event-time windows require an approved mapping from media time to wall-clock time.

Every start, reconnect and replacement receives a new generation. Late frames and completed analysis cannot publish into a newer session. Recordings preserve source, graph, timing, codecs and completeness. Recoverable partial files are retained separately and never presented as completed recordings.

Acknowledged recovery retains the same partial artifact on retry, including after explicit cleanup of its temporary source. An interrupted recording cannot later be upgraded to completion. Sampled Image buffers exist only for their admitted callback; intermediate pixels do not accumulate in workbook state.

Further adapters

Network publishing, Windows capture, file seeking and replay, rendered-frame ingress and GPU interoperability are later increments. Simulation poses alone are not rendered video frames. Inference engines and GPU composition remain separately selected capabilities. Destination and image-sending permissions remain explicit when those adapters are added.