Studies

Statistical models, experiments, and calibration

Statistical models, experiments, and calibration

These notations support assignment and declaration headers with identical meaning. Use plain, unqualified names. A parenthesized FIT header defines a reusable callable, also available in DO/function bodies. Other study blocks remain top-level bindings. Blocks cannot be marked input, default, or output. Formatting preserves the authored header; source-free rendering uses assignment spelling.

Statistical model formulas

input observations = {sales: [4, 5, 7, 8, 11, 13], price: [1, 2, 3, 4, 5, 6]}
 
demand = FIT ON observations
  RESPONSE sales
  PREDICTORS price
  METHOD linear
  MISSING error
END
 
forecast = MODEL.PREDICT(demand, {price: [7, 8]})

FIT demand ON observations is the alternative header. The bound value is a reactive fitted statistical model. Changing its data recalculates the model and its dependent predictions through the compiled model.

Data may be a column record, header matrix, native row-record vector, declared Table or materialized model-local Frame. Selected numeric columns have equal lengths; the shared input contract below applies to every estimator. Column names are exact, case-sensitive keys, local to the dataset; they are not references to model cells. PREDICTORS price, advertising, price*advertising adds three explicit terms. price*price is a quadratic term. * means a product only: it does not implicitly add main effects. Duplicate terms are rejected, including reversed products. Fixed transformations and categorical contrasts are described below. Arbitrary term functions, weights and model selection remain outside this profile.

RESPONSE, PREDICTORS, METHOD, and MISSING are required exactly once. METHOD linear, logistic, and probit use the existing REGRESS, LOGIT.FIT, and PROBIT.FIT estimators. INTERCEPT TRUE is the default; FALSE removes it. An estimator's rank, convergence, response-domain and sample-size errors remain errors. No fallback estimator is substituted.

MISSING error rejects missing values. MISSING drop removes training rows with null/blank entries in selected columns, and returns their retained zero-based rowIndices. Invalid numeric strings, booleans and nonfinite numbers are errors in both policies; categorical columns accept their declared label forms. Prediction preserves input row positions, returning blank for a dropped row, including a batch where every row is dropped. A missing required column or mismatched length among required columns is always an error.

The model exposes terms, ordered coefficients, namedCoefficients, rowIndices, droppedRows, the validated spec, and the native estimator report. The intercept label is (intercept); product labels use *. MODEL.PREDICT(model, data) needs only predictor columns. Logistic/probit predictions are probabilities. MODEL.FIT(data, spec) is the equivalent explicit function; the spec contains response, nested-array terms, method, missing and boolean intercept.

Bounds: 128 terms, eight factors per term, 100,000 rows, and 1,000,000 row/factor operations before constructing a design matrix. These are admission limits, not statistical guarantees.

Reusable FIT definitions

fit_demand = FIT(rows, confidence_level) ON rows
  RESPONSE sales
  PREDICTORS price
  METHOD linear
  CONFIDENCE confidence_level
  MISSING error
END
 
baseline = fit_demand({sales: [1, 3, 5, 7], price: [0, 1, 2, 3]}, 0.95)
forecast = MODEL.PREDICT(baseline, {price: [4]})

FIT fit_demand(rows, confidence_level) ON rows is the equivalent declaration header. Parentheses distinguish a callable from an immediately bound result; FIT() ON fixed_rows is a zero-argument constructor. Defining or importing it performs no fit or data read. A call fits its actual arguments through ordinary reactive dependency tracking; subsequent field reads use that result.

Up to 64 unique parameters are inferred through the ordinary callable system. Calls accept named arguments (0.95 AS confidence_level), aliases, partial application and callable composition. USE can import and re-export these constructors. A local definition in a DO block becomes a lexical callable; it creates no global result cell. Callable values remain subject to Grid's ordinary rules against escaping into cell data, records, collections or STATE.

All existing linear, transformed/category and nonlinear FIT clauses remain available. Data parameters accept the same columns, Table and model-local Frame inputs as other FIT calls. Numeric settings can use parameters. Column names remain dataset keys; nonlinear coefficient names must not duplicate constructor parameters. Portable nonlinear results keep their existing prediction program and can be transferred without the constructor or its training data.

Pass a declared Table at the call site, for example fit_demand(Observations, 0.95). Ordinary parameter forwarding, named arguments and partial application retain its revision dependency. Automatic adaptation refuses a shared local Table initializer or a helper that returns a captured Table: rewriting that shared occurrence could change unrelated reads. Pass the Table directly or provide an explicit Table/Frame handle in those cases.

Definitions can capture immutable constants and pure helper identities. Mutable inputs, cells, defaults, STATE, declared Tables and rule-written values must be passed explicitly, including dependencies hidden behind aliases or helper functions. Each actual argument retains its ordinary reactive dependency. This capture restriction applies to reusable constructors; existing name = FIT ON data blocks keep their reactive behavior.

See the reusable SOLVE/FIT example and reusable optimization definitions.

Linear covariance and confidence

demand = FIT ON observations
  RESPONSE sales
  PREDICTORS price
  METHOD linear
  COVARIANCE cluster BY store
  CONFIDENCE 0.90
  MISSING drop
END

COVARIANCE classical (the default), hc1, cluster BY column, and newey_west LAGS n select the existing REGRESS covariance implementations. CONFIDENCE defaults to 0.95 and must evaluate to a finite number strictly between zero and one. COVARIANCE applies to linear fits. Logistic/probit reject both clauses; nonlinear FIT retains CONFIDENCE under its existing UNCERTAINTY contract. Covariance changes inference, not the fitted coefficients. Native rank, degrees-of-freedom and cluster-count refusals remain errors.

Cluster labels are nonempty strings of at most 128 bytes or finite numbers. Numbers use REGRESS's text-label convention; numeric 1 and string "1" name the same cluster. Null labels participate in the same MISSING row mask as the response and predictors. Invalid labels fail even when another selected field in that row is missing. A cluster column must have the same row count. Prediction requires only predictor columns, never cluster labels or response values.

Newey–West uses retained row order; Grid does not infer or sort a time column. LAGS evaluates to an integer in 0–100,000. The native estimator caps effective lags at retained sample size minus one and reports requested/effective lags. HAC covariance is additionally limited to 10,000,000 scalar score-product updates, counting retained observations, fitted coefficients (including the intercept), and effective lag pairs. Oversized requests fail before estimation; reduce rows, predictors or lags. For the explicit MODEL.FIT spec, optional fields are covariance, cluster, lags, and confidence; cluster belongs only to "cluster", and lags only to "newey_west". Existing five-field specs retain their defaults and shape. See the complete clustered-fit example.

Nonlinear models

decay = FIT ON observations
  RESPONSE response
  PREDICTORS time
  PARAMETER amplitude IN [0, 20] INITIAL 8
  PARAMETER rate IN [0.01, 2] INITIAL 0.3
  PREDICT amplitude * EXP(-rate * time)
  METHOD least_squares
  MISSING error
  UNCERTAINTY local_iid
  CONFIDENCE 0.95
END
forecast = MODEL.PREDICT(decay, {time: [6, 7]})

FIT decay ON observations is equivalent. Each PARAMETER names a coefficient local to the formula. INITIAL is required; optional finite IN [lower, upper] bounds may be equal to fix a coefficient. Multiple plain PREDICTORS columns are supported. Put transformations and interactions in PREDICT. Formula names may be declared predictors, parameters, lexical LET locals or immutable closed scalar captures. Ordinary numeric helpers, including import aliases and imported private helper dependencies, use their defining lexical scope. Arithmetic, comparison, IF, boolean operators and common pure elementary numeric functions are supported. Callable aliases retain ordinary expression/import scope rules; a cell cannot store a function.

Mutable captures (including aliases to inputs and rule-written values), nested lambdas, unresolved higher-order calls, recursion, arrays/records and effectful calls are refused. Pass a changing quantity through a declared predictor or coefficient. Constructor settings keep their surrounding-scope rules.

saturation(amount, maximum, half) = LET(fraction, amount / (half + amount), maximum * fraction)
 
fit_kinetics = FIT(rows) ON rows
  RESPONSE measured_rate
  PREDICTORS concentration
  PARAMETER maximum IN [0.1, 10] INITIAL 1
  PARAMETER half IN [0.01, 10] INITIAL 0.5
  PREDICT saturation(concentration, maximum, half)
  METHOD least_squares
  MISSING error
END

The complete example uses this same helper in a reaction. Numeric LET initializers see preceding bindings and outer locals; nested bindings can shadow names. Computed helper arguments are evaluated once. Lazy IF and unused native error values keep ordinary compiled semantics. Helpers admit at most 64 arguments and 32 nested calls, within the 4,096-work/depth-64 formula budget. Imports retain ordinary USE capture admission: checked immutable scalar initializers remain in the defining module's scope. A consumer cannot replace them with same-named locals or values. Reusable FIT definitions can also carry their checked immutable helper closures.

The compiler emits an ordinary compiled lambda and the existing CURVE_FIT.REPORT operation. Unbounded fits use its existing least-squares algorithm; bounds, weights and uncertainty select its existing corresponding implementations. ITERATIONS defaults to 200 (1–10,000); TOLERANCE defaults to 1e-10 and must be positive and finite. There are at most 64 coefficients, 128 predictors, 100,000 rows and 1,000,000 selected data elements. Kernel work limits may be tighter.

STANDARD_DEVIATION sigma supplies positive known observation standard deviations. The objective is the sum of ((predicted - observed) / standardDeviation)^2. Optional UNCERTAINTY local_iid assumes equal independent observation variance; local_known requires STANDARD_DEVIATION. Local intervals inherit the native rank, active-bound and degrees-of-freedom checks; an unavailable interval is not silently replaced by another method. UNCERTAINTY bootstrap_normal SAMPLES 1000 SEED 42 selects the native parametric normal bootstrap; the seed is required, with 100–5,000 samples (default 1,000). The native estimator requires at least 10 expected draws in each confidence tail: samples * (1 - confidence) / 2 >= 10. For 95% confidence, use at least 400 samples. CONFIDENCE 0.95 requires an uncertainty method.

The result is a grid.statistical-model.curve.v2 record with terms (coefficient names), coefficients, namedCoefficients, rowIndices, droppedRows, the native report, and a self-contained prediction program. MODEL.PREDICT accepts the fitted value through function parameters, records, conditional selection, or a JSON round trip into another model. It needs no original FIT binding, source text, training data, or resident model handle.

predict_with(fitted, rows) = MODEL.PREDICT(fitted, rows)
 
saved = {model: decay}
forecast = predict_with(saved.model, {time: [6, 7]})

The portable model example shows this composition. Saving/loading uses ordinary value transport or storage; reading or predicting from a model starts no job. Fitting stays reactive; an exported value is a snapshot of that fitted result. Replacing an imported value or its prediction data recalculates dependent predictions normally.

The prediction program retains ordered predictor/parameter names and the missing policy. Existing formulas emit program version 1; lexical bindings use version 2 with private local slots and ordered initializers. Import checks fields, names, coefficients, supported pure operations, backward operand references, and expression budgets before executing. It prepares ordinary native compiled code once per prediction batch, using the existing numeric specialization for admitted polynomials and the same general compiled lambda fallback for other formulas. Solver-only intermediate-number conventions do not change prediction semantics. Limits are 4,096 expanded expression nodes, depth 64, and 10,000,000 expanded node visits per batch, alongside the shared row/data limits. Missing prediction rows remain blank; mathematical domain errors retain native row error values. Diagnostic reports are not traversed to predict.

This is a bounded numerical model value. It does not make general closures, outer mutable bindings, arbitrary user-defined functions, effects, or host handles portable. Admitted numerical helper bodies become closed expressions in the fitted value; prediction does not look up or execute helper source. Program-v2 values require a consumer supporting numerical program version 2; older consumers reject that nested version explicitly. String-concatenating formulas and incomplete builtin calls are refused; use a numeric formula so prediction does not depend on text or blank coercion settings. Linear/logistic/probit values retain their existing prediction contracts. Old grid.statistical-model.curve.v1 records remain inspectable but lack a formula; refit their original definition to obtain a portable value. Previously compiled models with declaration-expanded predictions retains its existing execution.

MODEL.COLUMNS(data, columns, missing[, keepRows]) is the shared strict column preparation primitive. It returns selected columns, one-based row indices, zero-based source rowIndices, row valid flags and droppedRows/totalRows. keepRows = TRUE keeps missing row positions for prediction. Booleans and strings are never silently converted to observations.

Experiments and parameter sweeps

input growth = 0.04
revenue = 100 * (1 + growth)
 
sensitivity = EXPERIMENT
  VARY growth OVER [0.02, 0.04, 0.06]
  SIMULATE FROM 0 TO 4 STEP 1
  OBSERVE revenue
  RESULTS FRAME sensitivity_results
END

EXPERIMENT sensitivity is the alternative header. The value is an inert grid.experiment.definition.v1 record containing the existing ExperimentSpec. Multiple VARY clauses form a Cartesian product in declaration order by default. DESIGN zip pairs equally positioned values; DESIGN cartesian makes the default explicit. Parameters must name model inputs; observations name model outputs. Targets are binding references in the owning AST, so a checked rename updates them.

Reading the record does not create jobs or Frames. Explicitly call the existing Host createExperiment(modelId, "sensitivity") operation, or the embedded host's create_experiment_from_definition(model, "sensitivity"). The typed client accepts the same definition name; HTTP callers POST a JSON string to /api/models/:id/experiments. The operation also continues to accept the existing structured request. It resolves the definition, checks bindings, then uses the established snapshot, job, cancellation, result Frame and replay owners. Native-document createExperiment accepts the same name under its existing document ownership checks.

Sweep values and simulation settings can be expressions evaluating to finite numbers. SIMULATE requires a positive step, an ordered interval, and an exact integral step count; both endpoints are included. Models may be algebraic: state cells are not required. SIM_TIME() and SIM_STEP() use the requested clock even without state. Up to 64 parameters, 128 distinct observations, 10,000 matrix rows and 10,000 simulation steps are accepted by the notation. Existing Host capacity and operation budgets may reject a smaller request. RESULTS FRAME and SIMULATE are required exactly once; one or more OBSERVE clauses must select at least one output. A definition without VARY or SCENARIO runs one baseline setting. Without REPLICATE, the existing deterministic simulation path applies.

Scenarios, ranges and seeded replications

outlook = EXPERIMENT
  VARY growth FROM 1 TO 3 COUNT 3
  SCENARIO baseline WITH starting = 100, noise_scale = 1
  SCENARIO stress WITH starting = 80, noise_scale = 2
  SIMULATE FROM 0 TO 4 STEP 1
  REPLICATE 32 SEED 42
  REDRAW per_step
  QUANTILES [0.05, 0.5, 0.95]
  OBSERVE population
  RESULTS FRAME outlook_results
END

The complete scenario example defines population with STOCK and a native NORMAL uncertainty source.

  • VARY input FROM first TO last COUNT n uses the existing evenly spaced range expansion. Both endpoints are included when n > 1; descending ranges are supported. COUNT 1 uses first. Counts evaluate to whole numbers from 1 to 10,000.
  • SCENARIO name WITH input = value, input = value groups finite numeric overrides. A comma permits continuation on the next line. SCENARIO name alone names a baseline with no overrides. Scenarios combine with every sweep tuple, in declaration order; none is implicitly added when scenarios are authored. At most 256 distinct scenario names and 256 overrides per scenario are accepted. All overridden names must be declared model inputs. A scenario cannot override a swept input or repeat an input. The combined matrix remains limited to 10,000 rows. Scenario-only comparisons are supported.
  • REPLICATE n SEED seed selects 1–100,000 stochastic simulation samples for each matrix row. The seed is required. Numeric seeds must be exact integers from zero to 9,007,199,254,740,991; canonical decimal strings support the entire u64 range, for example SEED "18446744073709551615". Seeds travel through the compiled model as strings without rounding. This controls the existing simulation uncertainty sources; it does not turn arbitrary external or volatile calls into seeded random generators.
  • REDRAW per_run is the default: each replication keeps its uncertainty draw for the whole trajectory. REDRAW per_step draws again at each time point. Matrix rows use the same base seed and corresponding draws, allowing paired comparisons of settings. Changing the seed changes those draws. Both modes use the existing native seed derivation and replay owner.
  • QUANTILES [q, ...] selects final-time distribution summaries, with values in [0, 1]. It requires REPLICATE, as does REDRAW. Defaults are 0.05, 0.25, 0.5, 0.75 and 0.95. At most 101 values are accepted; values must remain distinct after rounding to integer percentages because Frame columns use names such as population_p05, population_p50, and population_p95.

Result Frames retain run_id, scenario, sweep parameter columns, and each output's final, min, max, and mean, plus the requested quantiles. For replications, the main trajectory is the pointwise sample mean; min, max, and mean summarize that trajectory over time. Quantiles describe the sample population at the final time. They are not confidence intervals for the mean. Retained native output and verified replay expose the full time-varying bands. Scenario overrides and sweep values apply only to isolated runs; live model inputs remain unchanged. Existing Host work, storage, timeout and cancellation budgets still apply. Other existing experiment engines retain their structured APIs. Random and Latin-hypercube designs are described below; adaptive searches remain future work.

Reactive settings, paired axes, and named outputs

paired = EXPERIMENT
  DESIGN zip
  VARY growth OVER growth_choices
  VARY starting OVER starting_choices
  SIMULATE FROM 0 TO horizon STEP increment
  INTEGRATOR rk4
  TIME_UNIT "days"
  SENSITIVITY growth
  OBSERVE population
  OBSERVE gain = population - starting
  RESULTS FRAME paired_results
END

The complete paired example defines these inputs. Zip axes must have equal nonzero lengths. There is no truncation or broadcasting; scenarios still cross every paired tuple. No axes means one tuple. The existing 10,000-row limit applies after scenarios.

OBSERVE name = expression declares an ordinary model calculation. It is visible outside the study and cannot collide with an existing input, calculation, function, or another named observation. Use a distinctive name when multiple studies share one model. The expression evaluates in each isolated simulation; its name is the result/Frame column prefix and is retained for replay. Multiple OBSERVE clauses append outputs, with duplicates refused. Calibration also accepts observations: they are retained as output trajectories after fitting, without adding to the objective. Calibration retains at most 16 distinct outputs including trajectory FIT targets; experiments allow 128. All names participate in checked renaming.

Numeric settings use surrounding Grid expressions: confidence, covariance lags, parameter bounds and initial values, iteration budgets, tolerances, clocks, range endpoints/counts, scenario overrides, replication counts, and quantiles. IN keeps its explicit [lower expression, upper expression] structure. VARY OVER and QUANTILES also accept a computed numeric vector or one-row/one-column array; matrices are refused. Numeric settings do not coerce strings or booleans. Known invalid literals fail at compilation; dynamic errors fail when the FIT resolves or before a retained job is created. Existing finite, integral and work limits still apply.

FIT remains reactive. Nonlinear setting expressions resolve in the surrounding model, including INITIAL amplitude when amplitude is also a local coefficient; only PREDICT resolves coefficient and predictor locals. See configured FIT. Experiment/calibration settings are captured when the definition is explicitly submitted. Later model edits affect a new submission; retained runs keep their snapshot and settings. Reading a definition still starts no work.

Native simulation controls

Both experiment and calibration accept INTEGRATOR euler (default) or rk4, and TIME_UNIT expression for bounded nonempty text. Calibration requires SIMULATE when these options are supplied. These select the existing simulation settings; they add no integrator or time-unit conversion. See configured calibration.

Experiments additionally expose:

  • SENSITIVITY input, ... selects up to 64 distinct declared numeric inputs for the existing deterministic trajectory sensitivity calculation. It cannot be combined with REPLICATE.
  • ENGINE auto or monte_carlo, and RETAIN_SAMPLES booleanExpression, require REPLICATE. Retained sample trajectories are available through native replay.
  • CORRELATE leftSource WITH rightSource = rhoExpression selects existing uncertainty-source correlations, with finite rho in [-1, 1]. Self/duplicate pairs are refused; at most 256 pairs are accepted. Native source and correlation matrix validation still applies. This also requires REPLICATE.

See correlated samples. Simulation engine/integrator names are literal choices. Unsupported analytic ensembles and time correlation remain unsupported; their behavior is not approximated. Native sensitivity and uncertainty result structures remain the structured API/replay outputs.

Model calibration

input growth = 0.04
predicted_revenue = 100 * (1 + growth)
observed_revenue = 110
 
estimate_growth = CALIBRATION
  ADJUST growth IN [0, 0.2] INITIAL 0.04
  FIT predicted_revenue TO observed_revenue
  LOSS squared_error
  ITERATIONS 100
  TOLERANCE 0.0000001
END

CALIBRATION estimate_growth is the alternative header. The inert grid.calibration.definition.v1 record contains a CalibrationRunRequest. ADJUST requires finite, strictly ordered bounds. INITIAL is optional: omit it to use the input’s current value when the definition is read for execution. An explicit initial value must lie within the bounds; the Host checks the captured value against the same bounds before optimizing. The compiler requires each parameter to be a declared model input; explicit operation ingress resolves it against the resident compiled model. Multiple FIT clauses sum the selected residual loss; each pair can be a numeric scalar or equal-shaped numeric arrays. Missing or invalid values fail.

The compiler emits an ordinary compiled loss cell named __calibration_estimate_growth_loss. That generated name is reserved by this block; a conflicting declaration is rejected. MODEL.LOSS(predicted, observed) is the strict numeric squared-error primitive. No calibration evaluator is added.

Explicitly call createCalibrationRun(modelId, "estimate_growth"), or the embedded host's create_calibration_from_definition(model, "estimate_growth"). The typed client accepts the same name; HTTP callers POST a JSON string to /api/models/:id/calibration/runs. Existing retained runs supply progress, cancellation, immutable execution context, result and trace retention, and explicit evidence publication. Reading the definition does not optimize; completing a run does not write fitted values back to model inputs.

Optional SIMULATE FROM 0 TO 52 STEP 1 uses the existing simulation runner. The objective is the loss cell at the final simulated step. Without this clause, calibration evaluates the initial point (zero steps). This is not an implicit sum over a time series: represent an observed series as equal-shaped arrays, or author a cumulative loss explicitly in the model.

Bounds: 32 adjusted inputs, 16 FIT clauses, 10,000 iterations, positive finite tolerance, and the same simulation interval limits. Defaults are 100 iterations and tolerance 1e-7. The existing Host evaluation, timeout and lifecycle budgets still apply. FIT … TO … supports squared error, Huber and soft-L1 losses. The existing Nelder–Mead solver confirms its objective-spread tolerance across two successive iterations, so a single pair of equal-cost trial points does not prematurely end the search. This remains a local numerical search.

Custom calibration objectives

estimate = CALIBRATE
  ADJUST growth IN [0, 5] INITIAL 1
  MINIMIZE validation_error + complexity_penalty
  ITERATIONS 150
  TOLERANCE 0.00000001
END

MINIMIZE expression replaces all FIT and LOSS clauses. It cannot be combined with them or with ON observations. It is an ordinary Grid expression referencing model bindings, including reusable formulas and functions. Checked binding renames update its references. The existing compiled loss cell and retained calibration runner handle evaluation, bounds, progress, cancellation, failures and results.

The evaluated objective must be one finite numeric scalar; arrays, blanks, nonfinite values and formula errors make that candidate evaluation invalid. The existing optimizer retains the best finite candidate; a run with no finite evaluations fails with CALIBRATE_NO_FINITE_EVALUATIONS. Negative objective values are valid. SIMULATE retains the same final-step meaning, and omission evaluates the initial point. Penalties and validation scores can be authored in the model; this syntax adds no implicit time aggregation or global-optimum guarantee. Completing a run does not overwrite model inputs. See the complete custom-objective example.

Trajectory calibration

input growth = 1
population = growth * SIM_TIME()
observations = {time: [0, 0.5, 1.5, 3], measured: [0, 1, 3, 6], sigma: [1, 1, 1, 1]}
estimate = CALIBRATION ON observations
  ADJUST growth IN [0, 4] INITIAL 1
  FIT population AT time TO measured
  SIMULATE FROM 0 TO 3 STEP 1
  ALIGN linear
  STANDARD_DEVIATION sigma
  MISSING error
  LOSS squared_error
END

CALIBRATION estimate ON observations and the shorter CALIBRATE spelling are equivalent. Each FIT output AT timeColumn TO observedColumn joins a simulated scalar output to observed rows. Multiple targets can use different time and observed columns. Alignment defaults to exact: times must match simulation grid points, allowing only floating-point rounding tolerance. ALIGN linear interpolates between grid points. Neither mode extrapolates. Simulation endpoints are included. Observation order and repeated times are preserved; each row contributes separately.

A block-level STANDARD_DEVIATION sigma supplies a default positive error column. Each target can override it on the same line:

  FIT population AT time TO people STANDARD_DEVIATION population_sigma
  FIT revenue AT time TO money STANDARD_DEVIATION revenue_sigma

Without either setting, that target uses unit standard deviations. Each target's missing-row mask includes its own error column; a missing value in another target has no effect. Retained deviations must be positive; nonfinite entries are errors even under drop. The override is explicit and independent of clause order. See the complete weighted trajectory example. MISSING error is the default; drop removes incomplete rows separately per target and retains original row indices. The native objective sums ((predicted - observed) / standardDeviation)^2 over every retained observation and target. Bounds are 16 targets and 100,000 total observations, with the existing simulation and calibration operation budgets. Retained runs additionally cap estimated per-observation residual storage at 4 MiB before creating a job. Observations and alignment are validated before creating the retained job.

The definition remains inert and uses the same explicit Host submission API. Each candidate is evaluated by the existing compiled simulation runner. Completion adds trajectoryResiduals to the calibration result: symbol, original row index, time, observed value, aligned prediction, raw residual and standard deviation. The retained simulation outputs still show the winning candidate's full grid. Inputs are not overwritten. This form needs SIMULATE (structured requests must supply t0, dt and steps explicitly); the original form without ON observations continues to minimize its loss at the final simulated step.

Transformed predictors and categories

Linear, logistic and probit FIT accept fixed feature encoders directly in PREDICTORS. They are dataset-local terms, not calls that capture model globals:

response_model = FIT ON observations
  RESPONSE outcome
  PREDICTORS LN(income), age^2, CATEGORY(region, "west"), age*CATEGORY(region, "west")
  METHOD linear
  MISSING drop
END

LN(column), LOG10(column), SQRT(column), ABS(column), EXP(column), and POWER(column, exponent) produce one numeric term. column^exponent is the same power term. Exponents are finite numeric literals. CENTER(column) subtracts the complete-training-row mean; STANDARDIZE(column) also divides by the population standard deviation. Prediction retains these learned values. A zero standard deviation, transform domain error or overflow on a retained row is an error even with MISSING drop. Only blank/null values are missing. Nested transform calls, arbitrary formulas, model references and user-defined functions are outside this fixed encoder grammar; prepare a derived column or use nonlinear FIT's PREDICT for a general formula. The response remains a named numeric column in its original units.

CATEGORY(column[, reference]) accepts uniformly text or finite numeric labels. With an intercept it emits treatment contrasts: one indicator per retained level other than the reference. The optional reference is a nonempty text or finite numeric literal and must occur among complete training rows. Otherwise the first sorted retained level is the reference (numeric ascending or text lexicographic). With INTERCEPT false, all levels are encoded and an explicit reference is refused. Levels are learned after the shared missing-row mask, never from dropped rows. Each category allows at most 128 levels; the expanded design allows at most 128 terms and the existing one-million-operation design budget. Products expand the cross-product of factor columns and add no implicit main effects.

The fitted value uses grid.statistical-model.v2 and includes a portable design with learned levels, reference, centers and scales. terms, coefficients and namedCoefficients follow the expanded order. MODEL.PREDICT reuses that design without relearning from prediction rows; unseen levels are errors. Existing numeric fits retain the v1 value and prediction contract. Covariance options apply to the expanded design through the same native REGRESS engine. Nonlinear FIT continues to list plain predictor columns and put formulas in PREDICT.

The direct MODEL.FIT specification retains terms as column-name products and may include a matching features array. Each factor is a record with column and transform; power takes its numeric power field, and category optionally takes reference. For example terms: [["income"]], features: [[{column: "income", transform: "ln"}]]. The fitted design is output metadata, not an input to fitting.

Shared study data input

FIT ON data, MODEL.FIT, MODEL.PREDICT and MODEL.COLUMNS share named-column admission. This also supplies nonlinear FIT and trajectory CALIBRATION. Accepted carriers are column records, matrices with a distinct text header row, native vectors of row records, declared model Tables, and materialized model-local Frames. The shared path retains the source row order and existing zero-based rowIndices; missing-row masks and strict numeric validation are identical across carriers. Selected column names must be ASCII identifiers of at most 128 bytes; columns must exist and have equal lengths. Reads are complete or fail; there is no silent truncation. The limits are 100,000 rows and one million selected cells before estimation, with additional design expansion limits described above.

table Observations = {schema: "Observations"}
 
response_model = FIT ON Observations
  RESPONSE outcome
  PREDICTORS exposure^2, CATEGORY(region, "west")
  METHOD linear
  MISSING error
END

Declared Tables use the existing Table revision dependency, so committed mutations invalidate the study. For a Table handle supplied as an input value, refresh that input with the current handle/revision after a mutation; direct declared Table references track revisions automatically. Table order is not a statistical ordering guarantee: use an explicitly ordered relation for order-sensitive analysis such as Newey–West. A Frame input is an already materialized native handle supplied by the model's existing ETL or result owner. Reading it does not start an ETL job. Lazy, missing, foreign-model and unreadable Frames fail; no weaker read path is substituted. Changing the input handle causes the normal reactive recomputation. A bare compiled-model session without a host Table/Frame provider still accepts in-memory carriers and rejects resident handles explicitly.

Robust losses and multiple calibration starts

estimate = CALIBRATE ON observations
  ADJUST level IN [0, 40] INITIAL 0
  FIT prediction AT time TO measured STANDARD_DEVIATION sigma
  SIMULATE FROM 0 TO 3 STEP 1
  LOSS huber SCALE loss_scale
  STARTS start_count SEED 42
  ITERATIONS 100
END

LOSS squared_error preserves the existing sum of squared residuals. LOSS huber and LOSS soft_l1 reduce the effect of large residuals; both accept an optional positive finite SCALE expression (default 1). The scale is the transition size, not an estimated noise parameter. For scalar/array FIT predicted TO observed, r = predicted - observed. For trajectory targets, r is that residual divided by the target's standard deviation (default 1). With s = SCALE:

Loss Contribution of one residual
squared_error r²
huber r² when abs(r) <= s; otherwise 2*s*abs(r) - s²
soft_l1 2*s²*(sqrt(1 + (r/s)²) - 1)

These conventions match squared error near zero. Contributions are summed, not averaged. For scalar/array FIT, SCALE is part of the generated compiled loss formula and evaluates against each candidate, like the predicted and observed expressions. For trajectory FIT it is captured alongside the observation settings. Use a scale independent of ADJUST inputs when it should remain fixed throughout calibration. SCALE on squared error is refused. Missing/nonfinite values retain existing errors. MODEL.LOSS(predicted, observed, "huber", scale) exposes the same strict scalar/array primitive to custom objectives; its last two arguments are optional. A MINIMIZE objective cannot also carry LOSS or FIT clauses. The frozen V1 extension function catalog keeps its existing two-argument MODEL.LOSS contract; the robust arguments are available in Grid source and compiled models.

STARTS count SEED seed accepts 1..64 starts. The first uses authored INITIAL values; subsequent starts sample independently and uniformly inside each parameter's bounds, using the fixed retained seed. Settings can be model expressions and are captured at submission. Seeds use the same exact integer/decimal-string rules as REPLICATE. Each start receives ITERATIONS; the host's composed work budget includes starts × iterations × simulation points. All starts run sequentially through the existing compiled simulation and bounded Nelder–Mead solver, sharing cancellation, progress, one bounded trace and one overall best finite result. Equal best losses retain the earliest candidate. A start with no finite evaluations does not displace a successful start; all-invalid execution fails.

For multiple starts, results include zero-based bestStart and starts, with initial points, evaluation ranges, iteration counts, best parameters and losses. An invalid start has null best parameters/loss. Counts in the overall result are totals across starts. A single start preserves the existing result shape. This improves exploration but does not guarantee a global optimum or estimate uncertainty. See robust calibration.

Sampled experiment designs

coverage = EXPERIMENT
  DESIGN latin_hypercube COUNT design_size SEED 42
  VARY growth IN [0, 4]
  VARY starting IN [10, 20]
  SIMULATE FROM 0 TO 3 STEP 1
  OBSERVE population
  RESULTS FRAME coverage_results
END

DESIGN random COUNT count SEED seed samples independent uniform points. DESIGN latin_hypercube COUNT count SEED seed divides each input interval into count equally sized strata, independently shuffles each axis and samples once within each stratum. Both require one or more VARY input IN [lower, upper] clauses with finite strictly increasing bounds. Discrete OVER axes and counted FROM ... TO ... COUNT grids remain Cartesian/zip forms and cannot be mixed with a sampled design. Bound endpoints, count and seed may be model expressions.

COUNT is the number of parameter points, not points per axis. Each point is crossed with every scenario. The existing 10,000-total-run, memory and simulation work limits still apply before job creation. Sampling happens when the bounded matrix is expanded, before model evaluation. The retained specification and fixed algorithm reconstruct the same points for resumption and replay; result rows also retain their exact chosen inputs. A fixed seed, parameter order and design reproduce the same matrix; changing any of them may change it. The sampling algorithm is fixed SplitMix64 with 53-bit uniforms and an unbiased Fisher–Yates shuffle.

The design seed is separate from REPLICATE ... SEED ...: the former chooses parameter points, while the latter governs simulation randomness at each point. Existing named observations, scenarios, quantiles, Frames and replay remain available. Sampled designs currently cover bounded continuous uniform inputs; nonuniform distributions, categorical mixtures and Sobol sequences are later work. See sampled experiment.