Where definitions live

Definitions are resolved per (tenant, datastore) pair. A schema source returns raw YAML bytes for that pair, the loader parses them into one schema, and the layers that apply to the tenant fold onto it. Swapping the source changes nothing downstream — the parse, the fold and the engine build are identical whether the bytes came from the metadata service, a directory on disk, or the binary itself.

Definitions on disk

A product is one repository, one folder per block — see Product repository layout. Entity definitions live under data/. The one file the Data API reads outside that folder is content/contentstores/contentstores.yaml, the content-store catalogue — see File fields and the content store.

data/
  access.yaml                    # product-level authorization
  datastores/
    datastores.yaml              # the manifest: defaultdatastore + datastores[]
    main/                        # one folder per datastore (matches datastores[].path)
      datastore.yaml             # the datastores: declaration — see the note below
      types.yaml
      entities/
        billing/invoice.yaml
        billing/invoice_line.yaml
        admin/audit.yaml         # arbitrary nesting — the walk is recursive
    charts/
      charts.yaml
  code/js/                       # scripted-endpoint sources
  code/go/
  plugins/<name>/plugin.yaml

Path	What reads it
data/datastores/datastores.yaml	The manifest — datastore names and folders
data/datastores/<path>/	Definition YAMLs for one datastore, walked recursively
data/access.yaml	Product-level access rules
data/code/js, data/code/go	Scripted-endpoint sources
data/plugins/<name>/	Data plugins

The datastore manifest

data/datastores/datastores.yaml is required. Without it no datastore resolves and the service gets an empty datastore map.

defaultdatastore: main
injectdatastore: main
datastores:
  - name: main
    path: main
    inject: true
    injectservices:
      - iam
  - name: charts
    path: charts

Key	Type	Required	Notes
defaultdatastore	string	yes	The datastore a blank ?datastore= resolves to
injectdatastore	string	no	Datastore injected into other services; defaults to defaultdatastore
datastores[].name	string	yes	Logical datastore name. Empty fails with datastore name is empty
datastores[].path	string	yes	Folder under data/datastores/. Empty fails with <name> datastore path is empty
datastores[].inject	bool	no	Marks the datastore injectable into other services. Default false
datastores[].injectservices	list	no	Service names this datastore is injected into

The manifest keys are spelled without hyphens (defaultdatastore), and they are a different surface from the default-datastore key inside a definition YAML. Both exist, and both matter.

Inside a datastore folder

Every *.yaml and *.yml under the folder is a definition document; nesting is free and files are read in lexical order by full path. One of them must declare the datastore itself, or the engine build fails with schema has no datastore — see Datastores and database pools.

default-datastore: main
datastores:
  - name: main

Note

The metadata-service source is the exception: it receives only the entity files and synthesises the datastores: declaration from the manifest metadata before parsing. A local-directory or embedded source gets no such treatment and must ship the declaration itself.

Schema sources

Source	Selected by	Reads
Metadata service	Default; no configuration	The tenant’s product tree through meta.svc, per tenant, per datastore
Local directory	schemasource: {type: localdir, path: …} in boot config	Every YAML under the directory, recursively
Embedded filesystem	Go-level assignment in a service binary	A //go:embed tree compiled into the binary
Arbitrary filesystem	Go-level assignment	Any fs.FS
Composed	Go-level assignment	Several sources concatenated

localdir is the only source type selectable from configuration. There is no type: embed and no type: compose — the other three are assignments a service binary makes in its own boot code, not operator-facing options.

Reading from a local directory

The boot-config block replaces the metadata-service call entirely. It is the local-development and test-harness path: no metadata service needs to be running.

schemasource:
  type: localdir
  path: ./erp-product

Key	Required	Notes
type	yes	localdir is the only implemented value
path	yes	Relative paths resolve against the process working directory; absolute paths are used verbatim

The directory is walked recursively and every YAML under it is loaded, sorted lexically by full path. The requested datastore name is used as a subdirectory selector: a datastore named main loads <path>/main/ when that directory exists, and otherwise falls back to walking the whole root.

Caution

That fallback is silent. With a flat directory every datastore request loads every YAML, and a misspelled datastore name loads everything instead of failing. Keep one subdirectory per datastore if the product has more than one.

Caution

A malformed schemasource: block never fails boot. An empty or unknown type, an empty path, a path that does not exist, and a path that is not a directory all log a warning and leave the metadata-service source in place. If that service is then unreachable, the visible failure is a downstream one, not the typo. Confirm the override took effect from the boot log line reporting the active source type and the resolved absolute path, and the follow-up line logged on the first resolution with the tenant, datastore and file count.

Pointing meta.svc at a folder on disk — meta serve <customer>:<product>:<path>, covered in Meta operations — is a different mechanism. It also serves definitions from a local directory, but the call still goes over HTTP through the metadata client, so the tenant still needs meta.url and the metadata service still has to be running. schemasource: localdir removes the call.

Composed sources

Compose concatenates the byte output of several sources and hands the whole list to one loader.

Caution

Composition is not layering. Collisions are resolved by the loader’s intra-layer merge rules — entities keep the first definition, most other collections are last-write-wins. There is no layer identity, no access override, no access-lock or final enforcement, and no diagnostics. Reaching for composition to get “embedded default plus per-tenant override” gets different semantics from the layer model below.

When sourcing fails

Message fragment	Cause
response missing data.datastore	The product has no data/datastores/datastores.yaml, or the metadata deployment resolved no product repository
dsName empty and no default datastore in response	A blank datastore was requested, the manifest declares no defaultdatastore, and the response holds anything other than exactly one datastore
datastore "<name>" not present in response	The requested datastore is not one the product publishes
datastore "<name>" has no files.data	The datastore folder exists but contains no definition YAMLs
product publishes no datastore files for this service	Normal for a block with embedded base entities — the base serves alone
parse <file>: …	A definition YAML failed to decode

Caution

A source returns raw bytes, not named files, so every YAML body is handed to the loader under a synthetic name — file-000.yaml, file-001.yaml, and so on. This holds for every source, local directory and embedded filesystem included. A parse error therefore names a file that does not exist in the product repository; the index is the body’s position in the list, which for a directory source is its position in the lexically sorted walk.

Layers

Three layers exist in the model. Which of them run depends on the block.

Layer	Where it comes from	May add	May override access
Service base	Entities embedded in the block’s binary	entities, fields, indexes, relations	no
Product	YAML published for the product	entities, fields, indexes, relations	yes, per tier
Tenant	Rows in the tenant’s own tenant_extensions table	tenant-scoped entities and fields only	no

data.svc ships no base entities of its own, so the product’s data/datastores/ YAML is the base and only one fold runs: the tenant layer onto it. The service-base row applies to blocks that embed their own definitions; for those, the product layer arrives from that block’s own folder in the product repository, and the fold is base ⊕ product, then ⊕ tenant.

A consequence worth stating plainly: in data.svc, access-lock: parses and is carried, but nothing supplies a product layer for it to refuse. final: true still bites, because it blocks the tenant layer too.

What the fold carries

Collection	Behaviour
Entities present in both layers	Folded field by field
Entities only in the layer	Added, subject to the plane rules
Fields	Additive; a name collision keeps the lower layer’s field
Indexes, entity relations	Appended
Schema-level references:	Added; a name collision keeps the base’s
customtypes:, enums:	Added; a layer entry that shadows the base is dropped
access:	The one override — product layer only, per tier, wholesale
datastores:, default-datastore	Taken from the base only; a layer cannot introduce a datastore
Traits, embeddables, stateflows, functions, views, syncs, seeds, caches, unions, symbols, encryption keys	Base’s carried; the layer’s ignored
actions:, queries:, endpoints:, migrations:, archives:	Dropped from both layers

No fold is ever fatal. Every conflict is dropped and recorded; the schema still loads and the engine still builds.

Caution

The last row is a defect, not a design. A fold discards schema-level named actions, scripted endpoints, migration steps, archive targets and materialized-view refresh dependencies — the base’s as well as the layer’s. In data.svc this fires for any tenant that has at least one YAML row in tenant_extensions; a tenant with no rows short-circuits before the fold and is unaffected. For a block that folds a product layer over a service base, it fires whenever a product layer exists. Do not rely on tenant extensions in a product that declares actions:, endpoints:, migrations: or archives:.

Isolation planes ride along with the fold: no layer may introduce a superadmin entity, the tenant layer may introduce only tenant-scoped entities and may not add to a shared or superadmin base entity, and a scope: change on an existing entity is dropped — the plane is fixed by the layer that declared the entity. Seals are covered on Entities and Access rules.

Duplicates within one layer

Duplicate handling is not uniform across collections, and the difference is easy to trip over when definitions are split across many files.

Collection	On a duplicate name
entities:, traits:	Keep the first, ignore the later one, log at error level
enums:, references:, embeddables:, customtypes:, stateflows:, functions:, views:, syncs:, seeds:, caches:, unions:, encryptionkeys:, subscriptions:, datastores:, actions:, queries:, queryhashes:	Last-write-wins by name
endpoints:	Last-write-wins by path
augments:, migrations:	Appended, never deduplicated

An entry whose name is the empty string is never a merge key — it is always appended. Unnamed references: therefore accumulate across files rather than collapsing into one.

Two duplicate cases are hard failures rather than a keep-first: an action name that collides across actions:, queries: and queryhashes:, and an entity-scoped action colliding with a schema-level one. Action names are globally unique per schema.

Caution

augments: parses into the schema and is then never applied. The block that copies augment fields onto their target entity is not part of the deployable service, so an augments: entry is inert — the fields never appear. To contribute fields from a higher layer, re-declare the entity in that layer and list the new fields.

Diagnostics

Every fold decision is recorded. Only the ignored ones are logged, at error level, with the layer, entity, field and tier.

Diagnostic	Meaning
override-applied	A product tier override was applied (informational; not logged)
ignored-duplicate	A field or relation already declared by a lower layer was dropped
ignored-tenant-access	A tenant layer’s access: block was dropped
ignored-locked-tier	An override of an access-locked tier was dropped
ignored-final-entity	A contribution to a final: true entity was dropped
ignored-scope-change	A plane change or plane escalation was dropped
ignored-shadow	A layer type or enum shadowing the base was dropped

These are the only signal that a layer’s YAML was silently not applied. Grep the logs for them after changing any layered definition.

The tenant layer

A tenant’s own definitions are rows in that tenant’s database, in the tenant_extensions table — not in the product repository and not in the metadata service.

Column	Purpose
path	Virtual path, rooted at /. Unique
content_type	MIME type. Only YAML types are treated as definitions
content_text	The YAML body
content_blob	Binary payloads (non-definition rows)
updatedon	Audit column
deletedon	Soft delete — rows with a value set are skipped

The tree is walked from /, and a row counts as a definition when content_type is one of application/x-yaml, application/yaml, text/yaml or text/x-yaml — case-insensitive, with any ; charset=… suffix trimmed. Non-YAML rows are read and discarded, so keep unrelated assets out of the same table if it is large.

The deployable service does not create this table. Blocks that ship base entities declare it in their own definitions; a product that wants tenant extensions declares it in data/datastores/. When the table does not exist, the tenant layer is skipped and the base schema serves — a missing-table error is recognised on PostgreSQL, SQLite, MySQL and DuckDB and swallowed deliberately, so deploying a newer binary never bricks a tenant that predates the table.

An overlay row holds an ordinary definition fragment:

entities:
  - name: invoice            # declared by the product; this fragment adds one field
    fields:
      - name: cost_centre
        type: string
        nullable: true
  - name: site_visit         # tenant-only entity — must declare its own key
    fields:
      - name: id
        type: ulid
        defaultvalue: ulid()
      - name: note
        type: text
        nullable: true

Three things happen to an overlay that do not happen to product YAML:

• The surrogate id backfill is suppressed. A standalone entity with no primary-key: and no inherits: id normally gets an id ULID column injected. A fragment does not, because it usually re-declares an entity the product owns. A tenant-only entity must therefore declare its own id field or a primary-key:, or the emitted DDL names a primary key over a column that does not exist.
• Audit columns are stripped. createdby, createdon, updatedby, updatedon, deletedby and deletedon are removed from every entity in the overlay before the fold, with no diagnostic. Those columns belong to the layer that owns the entity — see Traits.
• A top-level endpoints: block is removed from the raw YAML before parsing. Scripted endpoints are product-only. There is no error, no warning and no diagnostic.

Caution

Malformed YAML in an overlay row is the one failure in the whole layering path that refuses the engine build. Serving a tenant without the extensions it declared would silently change behaviour, so the build fails instead. Every other conflict is dropped and logged.

Reload

A tenant’s composed schema is built on the first request for a (tenant, datastore) pair and cached until the tenant’s whole scope is drained. Nothing watches the bytes themselves — no source is polled for content changes.

Trigger	Effect
Metadata service reports a configuration update	Drains the tenant’s scope
Tenant configuration change	Drains the tenant’s scope
Tenant removal	Drains the scope and drops the metadata client
DELETE /data/cache	Drains the calling tenant’s scope and drops its metadata client
DELETE /data/superadmin/tenant/:tenant/cache	Same, for another tenant; superadmin only
Process shutdown	Drains every tenant

The first trigger is the only automatic one. Each tenant with a metadata client runs a config-change poll on a fixed 15-second interval; when the metadata service answers that its service or product revision moved, the tenant’s scope is drained and the next request rebuilds.

The cache-bust endpoints run the same path as tenant removal, so they also stop that poll. It restarts when the next request rebuilds the tenant’s metadata client, which is the same request that rebuilds the schema — but a tenant that is busted and then left idle is not being polled.

Caution

That poll is the metadata client’s, not the schema source’s. Under schemasource: localdir — or an embedded or composed source — editing a definition file changes nothing until something on the list above drains the tenant. Restart the process or call DELETE /data/cache.

A drain is all-or-nothing for the tenant: engines, cached schemas, materialized-refresh runners, script runtimes and connection pools go for every datastore that tenant holds, not just the one whose definitions changed. The rebuild cost is paid lazily, on the next request that touches each datastore.

Caution

The first request after a drain re-resolves the tenant config, re-reads and re-parses the definitions, opens the pool, folds the tenant layer and runs migrations before it executes anything. The chassis per-request timeout defaults to one second, which a cold build routinely exceeds. Raise it in the boot config:

timeouts:
  defaultcontexttimeout: 30s

Confirming what took effect

GET /data/schema returns the composed schema for the requesting tenant — the result of every layer that applied, after every drop. It is the check that a product or tenant contribution is actually live, and it pairs with the error-level diagnostics for the ones that are not. The entity list, per-entity detail and generated SDL are covered in Schema export and discovery; the endpoint-level reference is at /api/data/.