Datamodel

Copy Markdown View Source

Statifier's datamodel is predicator (predicator-ex, ~> 8.0). This is a commitment, not a stopgap (ADR-0004): we do not chase the ECMAScript datamodel, and we never evaluate raw Elixir code from a document. Documents declare datamodel="predicator" (accepted alias: elixir for continuity with v1's converted W3C tests).

Why not ECMAScript

  • Embedding a JS engine trades away the security story that makes an Elixir SCXML engine worth having. Predicator is non-evaluative by design: no eval, no side channels, safe for end-user-authored documents.
  • The W3C tests that genuinely require ECMAScript are a bounded, known set; the corpus tooling rewrites or excludes them. The conformance ceiling this imposes is accepted and documented in the test manifest.
  • Real computation belongs in the host application, reached through <invoke> handlers and external <send> - controlled, typed, supervised. See docs/extending.md for how a host registers its own <invoke> handlers.

What the datamodel provides

  • <datamodel> / <data> with expr, child content, and src - binding="early" and late both supported. src is lowered, validated, and compiled like any other attribute, but it is never fetched, at any binding time - a decided, permanent deviation from spec 5.3.2's fetch MUST (ADR-0024): a binding-time fetch is I/O inside the pure core (ADR-0003), and dereferencing document-named URIs contradicts the security posture above (ADR-0004). A <data> with an src raises error.execution and leaves the id as an empty (nil) data element, the shape of 5.3.2's failure clause. Embedders that need the data fetch it themselves and supply it via environment-provided values for top-level <data> ids. test552 is the one corpus file this reddens, kept failing and visible by design.
  • The starting datamodel is on the effect stream, not only reachable through Session.snapshot/1: one {:datamodel_init, %Statifier.Effect.DatamodelInit{}} baseline is emitted once per session, before any <data> value is evaluated, followed by one {:datamodel_change, %Statifier.Effect.DatamodelChange{}} per <data> actually bound. A subscriber that folds the stream alone - seeding from the baseline and applying each binding at its location_path - reconstructs the whole datamodel under both binding="early" and binding="late", including a state-scoped <data> bound arbitrarily far into the run.
  • <assign> with deep paths (user.profile.name, items[0].sku), including auto-vivification of intermediate maps (ECMAScript-like assignment behavior; v1 refused to create intermediates). The root segment of the path must already exist in the datamodel - an undeclared root fails with error.execution rather than being created, so vivification only ever extends a path under a name the document already declared.
  • cond on transitions and <if>/<elseif>, expr everywhere the spec allows.
  • System variables per spec 5.10: _event, _sessionid (a sess_ id, stable for the session's lifetime), _name, _ioprocessors, and the In(stateId) predicate.

Evaluation contract

Every evaluation goes through one module with one context type:

  • Expressions are compiled once at Machine-build time into {:compiled, %Predicator.Compiled{}, source}; static attribute values are {:static, value}. One evaluator function handles both. The %Predicator.Compiled{} envelope is upstream's (predicator ADR-0009), not a statifier-side wrapper; we keep owning source because only statifier knows where the expression sat in the document (see ADR-0014 item 2).

  • The evaluation context is built once per evaluation site (once per executable-content block today), never once per expression, and never scoped to a whole macrostep: _event is rewritten on every internal-event round and In(stateId) reads a configuration that moves at every microstep, so a snapshot spanning the whole macrostep would already be stale before a later evaluation site in that same macrostep read it.

  • Every evaluation returns {:ok, value} | {:error, reason}. The interpreter maps errors to error.execution internal events per spec. Leaves never swallow errors.

  • An expression that fails to compile is rejected at load time everywhere except <data expr>, <assign expr>, <script>, and a namelist entry on <send>/<invoke>, which defer to runtime. Spec 5.9.4 permits either ("The SCXML Processor MAY reject documents containing syntactically ill-formed expressions at document load time, or it MAY wait and place 'error.execution' in the internal event queue at runtime"), so both halves conform - but the clause frames the choice as one processor-wide policy, and this engine currently makes it per element class. The asymmetry is deliberate, not an oversight: test/scion_tests/data/data_invalid_test.exs declares an unparseable <data expr="{p1: 'v1'"/> and asserts pass by catching the resulting error.execution, so load-time rejection would make that document unloadable and the test unpassable. test/scion_tests/assign/assign_invalid_test.exs requires the identical treatment for <assign expr="{p1: 'v1'"/>. A <data expr> or <assign expr> that will not compile is therefore captured as {:invalid, error} on the compiled node and raised at binding/execution time. <script> gets the same treatment for a body outside predicator's statement grammar (ADR-0026). cond, <log expr>, <content expr>, <foreach array>, and a <param> element's expr/location still fail Compiler.compile/1.

    If this is ever unified, it unifies toward deferral rather than away from it: deferral loads strictly more documents, and no corpus file requires load-time rejection. The trigger to watch for is a corpus document with an unparseable cond plus an error.execution handler - none exists today, and test344 is not one (its cond="1" compiles, then fails boolean coercion at evaluation). The trigger did fire for namelist, in st-vwdg: test553 (test/scxml_tests/mandatory/send/test553_test.exs) and test554 (test/scxml_tests/mandatory/invoke/test554_test.exs) are corpus documents with an unparseable location expression in a <send>/<invoke> namelist and an expected pass that depends on catching the resulting error.execution rather than failing to load at all. The response was to move that one element class to the deferring side, not to unify the whole policy - <param> keeps rejecting at load time even though ADR-0036 and ADR-0031 give its expr/location the same runtime treatment as a namelist entry, because the runtime rule and the load-time rule are separate axes and no corpus file forces <param> onto the deferring side. The cond half of the trigger is still unfired.

  • Datamodel keys are strings at every level, top-level and nested alike; an embedder-supplied :datamodel option is checked for this at MachineState.new/2 construction time rather than trusted (see the @doc on Statifier.Evaluator.context/1 for the property this backs).

  • Type coercion to/from event data has one normalization function with defined rules, Statifier.EventData.coerce/1, shared by <param>, <content>, namelist, and <donedata>. It implements B.2.8.1's key-value-pairs and space-normalized-string rungs, plus a predicator-literal rung standing in for the JSON rung; it does not implement the indicated-format or XML-DOM rungs (Statifier.EventData's moduledoc states why for each).

Statement sequences and <script>

<script> is supported (ADR-0026, amending ADR-0004 in part): the statement layer this section once asked for landed in predicator 5.0.0 - parse_program/2 parses sequences of assignments and expression statements over the existing expression language, and Predicator.execute/3 runs the program and returns the resulting context. A <script> body is a predicator statement program, not ECMAScript, and the no-eval security posture is unchanged.

  • A body that fails mid-program keeps every write up to the failing statement ({:error, error, %Context{}}'s third element is the partial context) and raises error.execution - spec 4.9's stop-and-keep error model, the shape IRP test156 observes. All-or-nothing rollback is a decided non-option.
  • A <script> child of <scxml> is evaluated at document load time, before the initial state is entered (spec 5.8) - a separate interpreter path from executable content.
  • <script src> is rejected at load with a named unsupported error; external fetch stays the unresolved question st-322 owns (see ADR-0024 on <data src>).
  • The ceiling: corpus script bodies are ECMAScript and predicator's statement grammar is not - object literals, typeof, and function definitions do not parse, and if/else/while/undefined/null are reserved words - so only a subset of the once-excluded :needs_script corpus files joins the ratchet.

Upstreaming to predicator

Seams found in v1 that belong in predicator rather than in statifier's glue:

  1. Persistent bound context: build a context once (data + host functions like In/1), evaluate many expressions against it, rebind cheaply when data changes. v1 rebuilt the full context map per expression. Predicator.Context.bind/3 is the cheap-rebind path that would let the once-per-block interval above widen again, once a caller needs to. Landed in predicator 5.0.0: Predicator.FunctionProvider (a module supplying named functions), Context.new/2's providers: and host: options, and Context.put_host/2 (an O(1) %{context | host: host} refresh). Taken in two steps. First, in the within-block form only (ADR-0028): measurement showed context construction is the majority of one macrostep's cost at realistic datamodel scale, and <assign> and <foreach> bind into the context an executable-content block already threads rather than rebuilding it per write. Then, taken for In/1 itself (ADR-0030): In/1 is a Predicator.FunctionProvider reading host rather than an inline functions: closure, so the resolved functions map holds no captured configuration and is identical for every context this library ever builds - Statifier.Evaluator.Functions.base_context/0 resolves it once at compile time, and context/1 refreshes host with put_host/2 and binds each datamodel root with bind/3, never calling Predicator.Context.new/2 on this path. This is a per-site cost reduction, not a widened interval: the "built once per evaluation site" commitment above (docs/datamodel.md:54-59) is unchanged - the site is still the whole block or selection round, not the individual write, and not the whole macrostep - and the seam is still not taken across blocks: no context is stored on MachineState, and widening the interval that far remains future work, per ADR-0030's grounds. The fixed term base_context/0 hoists is memoized upstream as of predicator 8.0, and the compile-time hoist is kept anyway - the memo removes re-validation, not the per-call stamp-and-allocate cost new/2 still pays, so the constant stays cheaper (ADR-0030's amendment). The normalize: false seam that memoization sits beside also exists as of 8.0 but is not taken here: it would help a whole-map new/2 call, and this path binds per root rather than constructing per build.
  2. Auto-vivifying path assignment: path resolution exists (context_location); assignment-with-creation should live beside it. Landed in predicator 3.6.0: Predicator.context_assign/4 and ContextLocation.put/3. Vivification is ECMAScript-like; a container collision raises :not_a_container; list assignment past the end pads with :undefined; a negative index raises :invalid_index. The statifier-side consumer landed in st-af3.4: Statifier.Machine.Content.Assign resolves the path with Predicator.context_location/3 and writes with ContextLocation.put/3 (split rather than the combined context_assign/4, so the resolve reads the normalized context and the write lands on the raw machine_state.datamodel - see that plan's Decisions 1 and 2).
  3. A typed undefined: predicator's :undefined currently leaks into hosts as a bare atom that every embedding normalizes ad hoc. Landed in predicator 5.0.0: the undefined literal (upstream px-ocp). Consumed here by st-unt: conf_predicator.xsl's seven boundness templates emit === undefined / !== undefined (non-strict == propagates :undefined rather than returning a boolean), and the 24 affected W3C corpus files were regenerated. No ratchet update was owed - none of those files is in test/passing_tests.json. The literal does not rescue a genuinely unbound root, so a Var<n> boundness cond still waits on st-af3.3 seeding the declared <data> it names. This repo's own nil -> :undefined shim is retired as of docs/adr/0037-unbound-spelled-undefined-at-the-writer.md: writers spell :undefined themselves, and nil means only predicator's null.
  4. Statement sequences (above). Landed in predicator 5.0.0: parse_program/2, the store/pop instructions, and Predicator.execute/1,2,3 returning the resulting context (with the partial context on error). Consumed here per ADR-0026; the statifier-side <script> implementation is st-af3.17. Predicator 8.0.0's protected_roots: option on Predicator.execute/3, consumed here under this repo's current ~> 9.0 pin, is the 5.10 half of this same seam - a write to a protected root now fails at the attempt rather than needing a post-hoc diff to catch it - consumed here by st-i9d, which replaces the post-hoc root diff as the primary enforcement mechanism.
  5. String prefix/substring: landed in predicator 3.7.0 (starts_with/2, ends_with/2, substring/2,3, index_of/2); conf:varPrefix (test224) no longer needs an exclusion.
  6. List concatenation: landed in predicator 3.7.0 (concat/2, and list + list); conf:extendArray (test525) no longer needs an exclusion.
  7. Numeric-type builtins: Math.pow (and possibly Math.sqrt) return floats even for integer-exact results, while === stays type-strict, so Math.pow(2, 3) === 8 is false under ECMAScript's single-Number-type assumption. ADR-0023 keeps the fix upstream in predicator rather than coercing numeric types at the statifier boundary; a mirror bead is filed in predicator-ex. Not yet landed: the three test/scion_tests/targetless_transition files stay failing until the dependency is bumped.

Each of these gets a beads issue here and a mirrored issue in predicator-ex when we hit the seam in implementation.