Embedding the engine in a Go service

data.svc is one host of the Data API engine, not the engine itself. The engine is a Go library, and a service can open it in-process: same YAML definitions, same hook chain, same compiler, no HTTP hop. Every sibling platform service that owns entities does this.

The engine package’s import path is internal to the platform and is not published here. The Go fences below alias it as dataapi; substitute the real path from your product module.

What you get and what you lose

	Embedded	data.svc over HTTP
Entity definitions	Same YAML, loaded from an fs.FS or a loader you build	Sourced per tenant by the service
Hook chain	Same	Same
Access rules, RLS, field protection	Enforced only if you wire the providers	Wired by the service
Migration	Blind CREATE ... IF NOT EXISTS sweep	Reviewed plan pipeline with a ledger and a risk gate
REST / GraphQL surface	Not included	Included
Multi-tenancy	The engine factory, wired by you	Built in

Registering a backend

Backends register themselves in their own init. A consumer must blank-import the backend package it wants; nothing else pulls it in. Opening a DSN whose scheme has no registered provider fails with an unknown-scheme error that quotes the scheme and flags a forgotten import.

Five schemes, four backends — postgres and postgresql register the same provider.

DSN scheme	Backend
postgres	PostgreSQL
postgresql	PostgreSQL (alias of the same provider)
duckdb	DuckDB
clickhouse	ClickHouse
sqlite	SQLite

import (
    dataapi "<data-api-engine>"
    _ "<data-api-engine>/backend/postgres" // registers postgres:// and postgresql://
)

The DSN must contain ://. Everything before it is lowercased and used as the registry key; a DSN without a scheme separator is rejected before any connection is attempted. dataapi.Providers() returns the scheme names registered in the current process — useful in a boot diagnostic.

dataapi.RegisterProvider(scheme, p) is exported, so a consumer can add a custom backend. It panics on a duplicate scheme.

Caution

data.svc serves PostgreSQL only for tenant datastores. ClickHouse, DuckDB and SQLite are complete engine backends — pool, driver, transactions, compiler, dialect — and they work when you embed the engine, but they are not reachable from tenant configuration in the deployable service. See Backends.

Opening a handle

func Open(dsn string, schema *types.Schema, opts ...Option) (*DB, error)
func OpenCtx(ctx context.Context, dsn string, schema *types.Schema, opts ...Option) (*DB, error)
func OpenFromFS(ctx context.Context, dsn, namespace string, fsys fs.FS, root string, opts ...Option) (*DB, error)

Open is OpenCtx with context.Background(). Use OpenCtx anywhere boot has a deadline — a wedged database returns the context error instead of hanging the process, which is the difference between a failed readiness probe and a restart loop.

OpenFromFS is sugar for building the schema from a filesystem and calling Open. It is the go:embed path: the definitions travel in the binary, so the service boots with no runtime dependency on the metadata service. root is walked recursively, and every .yaml / .yml file under it is loaded — other extensions are ignored. For anything more involved — several sources, an in-memory test schema, extra per-file accumulators — build the loader yourself and pass its output to Open.

Caution

namespace becomes the schema’s name, which is the physical PostgreSQL schema every statement is qualified with — not a label. Passing "warranty" makes migration emit CREATE SCHEMA IF NOT EXISTS "warranty" and the compiler emit "warranty"."orders". Pass "" to stay unqualified. See Physical schema qualification; the mechanism is PostgreSQL-only, and the other backends suppress the prefix.

OpenCtx runs in this order:

1. Reject a nil schema.
2. Parse the DSN scheme and look up the provider.
3. Run the capability gates — history, soft delete, partitioning — against the backend’s declared capabilities. A schema that asks for something the backend does not support is refused before the pool opens.
4. Apply the options.
5. Create the pool, unless WithPool supplied one.
6. Build the hook chain: your hooks, then the baseline hooks, then the script-runtime hooks if a script engine was supplied.

The returned *DB embeds the engine interface, so it satisfies every signature that takes an engine.

Method	Purpose
Query(ctx, *QueryRequest) (*Result, error)	Read
Create / Update / Delete / Restore(ctx, *MutationRequest) (*Result, error)	Mutations
Transaction(ctx, func(txe Engine) error) error	Callback transaction
TransactionWith(ctx, *TxOptions, fn) error	Transaction with read-only, isolation, schema
Close(ctx) error	Drains the engine, then releases the pool
Dialect() types.SQLDialect	The dialect the handle was opened with
Pool() pool.Pool	The underlying connection pool
DBSchema() *types.Schema	The schema the handle was opened with
Migrate(ctx) error	Library-tier DDL sweep
MigrateDetailed(ctx, onError) (*MigrateResult, error)	Same, with per-failure detail
Drift(ctx, ...DriftOptions) (*migrate.Diff, error)	Read-only comparison against the live database

Close honours the context for the engine drain: in-flight operations get until the deadline to settle, after which Close returns and the pool closes underneath them. When the handle was opened with WithPool, Close skips the pool — the caller owns it.

The option catalogue

Seventeen constructors. All are variadic arguments to Open, OpenCtx and OpenFromFS.

Option	Effect
WithHooks(hooks ...hook.Hook)	Adds hooks to the lifecycle chain, ordered by each hook’s phase and priority
WithTenant(id string)	Sets the tenant ID on the engine context; read by RLS and multi-tenant hooks
WithUser(id string, roles ...string)	Sets the principal and roles; read by access control and by the audit hook that stamps createdby / updatedby
WithDebug()	Installs the stderr debug logger
WithDebugLogger(l exec.DebugLogger)	Installs your own debug logger
WithMaya(eng *engine.Engine)	Supplies the script engine. Wires it into every hook that evaluates expressions, and registers the engine’s own service functions as script callables
WithAccess(ac servicefn.AccessChecker)	Registers the access script namespace (access.check, access.checkWithContext)
WithACL(p servicefn.ACLProvider)	Registers the acl namespace (acl.check, acl.list, acl.grant, acl.revoke)
WithRBAC(p servicefn.RBACProvider)	Registers the rbac namespace (rbac.hasRole, rbac.getRoles, rbac.hasPermission, rbac.assignRole)
WithABAC(p servicefn.ABACProvider)	Registers the abac namespace (abac.evaluate, abac.getPolicies)
WithReBAC(p servicefn.ReBACProvider)	Registers the rebac namespace (rebac.check, rebac.listObjects, rebac.listSubjects)
WithContextProvider(fn func() context.Context)	Supplies the context that script-issued engine and ACL calls run under. Without it those calls fall back to context.Background()
WithPointcutScheduler(s *hook.PointcutScheduler)	Worker pool for pointcuts declaring async: true. Without one, async pointcuts run synchronously on the request goroutine
WithAuditSink(sink sqlcompiler.AuditSink)	Receives one event per successful mutation and every access-denial event from the compiler’s security layer, in one stream
WithoutDefaultScriptHooks()	Suppresses auto-registration of the default hooks
WithObserver(o exec.Observer)	Receives OnOperation, OnTransaction, OnMigrate, OnHookError. Nil is a no-op observer with no hot-path cost
WithPool(p pool.Pool)	Uses a caller-owned pool. The DSN still resolves the provider (compiler, dialect, driver) but no pool is created, and Close does not close it

The five authorization options all require WithMaya — they are registered as script namespaces, so without a script engine there is nothing to register them on.

WithObserver implementations run on the hot path. Keep them cheap and non-blocking; push expensive work onto a buffered channel.

The default hooks

Unless suppressed, Open registers a baseline of three hooks — defaults, validate, audit — skipping any whose name you already supplied via WithHooks. When a script engine is configured it also registers the script-runtime hooks: entity validation (cross-field rules), business constraints (cross-entity rules), and the pointcut dispatchers. Registration is skip-when-present there too, so supplying your own instance of one keeps your customisation.

Caution

WithoutDefaultScriptHooks suppresses all of the above, including the three baseline hooks — its name understates its reach. Without the defaults hook, a field’s declared defaultvalue: is parsed at boot and never written. Without the audit hook, createdon / updatedon / createdby / updatedby stay empty, and because the common trait the loader applies to every entity declares createdon and createdby non-nullable, every insert fails on a constraint violation. Use this option only when you are composing the entire chain yourself.

Migrating in-process

func (db *DB) Migrate(ctx context.Context) error
func (db *DB) MigrateDetailed(ctx context.Context, onError func(MigrationFailure)) (*MigrateResult, error)

This is a blind, non-introspecting sweep: it emits CREATE ... IF NOT EXISTS for everything the schema declares, in a fixed order, without reading the live database first. It is idempotent and safe to re-run. It has no plan, no review step, no ledger, and no rollback.

Step	What runs
1	CREATE SCHEMA — PostgreSQL only, and only when the schema carries a name
2	The six engine-owned system tables, first, so later steps can record into them
3	CREATE TABLE per entity
4	CREATE INDEX per index
5	ALTER TABLE ADD FOREIGN KEY per relation
6	Side tables for typeaudit and type4 entities only

Every step runs before the call decides whether to return an error, so one pass surfaces every problem rather than one per restart. Migrate discards the result and returns only the error. Step 6 covers only the two strategies that use a side table — a type2, type6 or typenoversion entity gets nothing here beyond its own table. See Row history for what each strategy stores.

MigrateResult carries TablesCreated, TablesSkipped, IndexesCreated, FKsCreated, and Failures []MigrationFailure. The result is always non-nil. The error is non-nil exactly when Failures is non-empty, and it wraps Failures[0] so errors.As finds the typed cause. Callers who want to tolerate specific failures inspect Failures and discard the error.

MigrationFailure is {Step, Object, DDL, Err}. Step is one of namespace, system-table, system-index, table, index, fk, audit-table, audit-index, history-table, history-index. Object is the entity, index or relation name; DDL is the rendered statement. The onError callback fires once per failure as the sweep proceeds — use it for metrics and structured logs. It runs inside the loop, so keep it cheap.

Caution

This is the library tier, meant for tests, development and single-tenant embedded use. It is not how a deployed tenant database is migrated. The service uses a diff-based pipeline — discover, compare, plan, apply — where destructive changes are refused unless explicitly allowed and an operator confirms a risk level before anything executes. A blind sweep can create but never alter or drop, so a schema change that needs an ALTER will silently do nothing here. See Migrations.

Detecting drift

func (db *DB) Drift(ctx context.Context, opts ...DriftOptions) (*migrate.Diff, error)

Drift introspects the live database, compares it against the handle’s schema, and returns the structural difference. It never executes DDL. An empty diff means the live schema matches the definitions; a non-empty one means someone ran an out-of-band ALTER, a migration failed quietly, or two deployments disagree.

DriftOptions field	Default	Meaning
SystemTablePrefixes []string	data_, _pg_, _information_, pg_, information_	Table-name prefixes excluded from the live scan
IncludeAuditTables bool	false	Whether <entity>_audit / <entity>_history side tables participate. They are engine-owned and managed in lockstep with the parent entity, so including them produces noise
AllowDrops bool	false	Whether a live column absent from the definitions becomes a drop operation. Left false it is reported as a refusal instead

Drift requires the backend provider to expose schema discovery. All four in-tree backends do; a custom provider that does not returns ErrNoDiscovery. When the provider also exposes its dialect — the in-tree ones do — the comparison runs in physical type space rather than logical, which is what stops a freshly migrated database from reporting phantom differences.

data.svc has no drift verb; the operator equivalent is a generated plan. See Drift detection.

Seeing the SQL

The debug logger is the only facility that shows compiled SQL, and it is in-process only.

db, err := dataapi.OpenCtx(ctx, dsn, schema, dataapi.WithDebug())

Each operation produces a record with Operation (query, create, update, delete, restore), Entity, AST (the parsed query rendered in pipe syntax), SQL, Args, IncludeSQL (one entry per relation eager-load subquery), QueryCount, RowCount and DurationMs. WithDebugLogger takes your own implementation of the two-method interface (LogQuery, LogMutation) to route this into structured logging.

Caution

The stderr logger does not print Args unless you set ShowArgs on it. Bound arguments carry plaintext user input — personal data, credentials, values not yet rewritten by the encryption hook, which can run after the logger depending on hook order. Keep ShowArgs false outside a developer machine.

There is no EXPLAIN surface: no route, no query DSL keyword, and no request flag exposes the compiled SQL or a database query plan. The debug logger is the whole story.

One engine per tenant

A service serving more than one tenant does not open a handle per request. The engine ships a manager that keys cached handles on (tenantKey, datastoreName), opens them lazily on first use, and coalesces concurrent openers for the same key onto a single open. It owns every handle it creates.

mgr, err := tenant.New(tenant.Deps{
    Schemas: tenant.StaticSchema(schema),
    DSNs: tenant.DSNFunc(func(ctx context.Context, tenantKey, ds string) (string, error) {
        return dsnFor(ctx, tenantKey, ds)
    }),
    Options: []dataapi.Option{dataapi.WithMaya(scriptEngine)},
})
if err != nil {
    return err
}
defer mgr.Close()

db, err := mgr.Get(ctx, tenantKey, "main")

Providers

Two interfaces supply everything that varies per tenant.

type SchemaProvider interface {
    Schema(ctx context.Context, tenantKey, datastoreName string) (*types.Schema, error)
}

type DSNProvider interface {
    DSN(ctx context.Context, tenantKey, datastoreName string) (string, error)
}

Adapter	Shape
SchemaFunc	Function adapter for SchemaProvider
DSNFunc	Function adapter for DSNProvider
StaticSchema(s)	One schema for every tenant and datastore — the common shape for a service shipping one embedded definition set and isolating tenants by physical namespace
StaticDSN(dsn)	One DSN for everything; in-memory tests and single-tenant deployments
DatastoreSchemas(map[string]*types.Schema)	Routes by datastore name; tenants share the per-datastore schema
DatastoreDSNs(map[string]string)	Routes DSNs by datastore name

The two map-based routers return *UnknownDatastoreError for a name they do not hold, so a caller can distinguish “no such datastore” from a connection failure.

CloneSchema(s, name) returns a copy of a schema with its Name replaced. This is how one source-of-truth schema serves many tenants while emitting a different physical qualification for each. The clone is shallow: entity and relation maps are shared with the original, which is safe because the compiler treats them as immutable, and unsafe the moment anything mutates them.

A provider is consulted on the first Get for a key and again after an invalidation — never on a cache hit. Rotating a tenant’s schema therefore requires an explicit invalidation from outside, typically from whatever subscribes to tenant configuration changes.

Deps

Field	Required	Meaning
Schemas SchemaProvider	Yes	Resolves the schema for a key
DSNs DSNProvider	Yes	Resolves the DSN for a key
Options []dataapi.Option	No	Options applied to every engine
OptionsFor func(ctx, tenantKey, dsName) []dataapi.Option	No	Additional per-key options, appended after Options. The seam for a per-tenant script runtime, paired with OnEvict for lockstep teardown
SchemaFinalize func(ctx, tenantKey, dsName, dsn string, schema *types.Schema) (*types.Schema, error)	No	Runs between schema resolution and the open, with the resolved DSN in hand — the seam for a composition layer that needs the tenant’s own database before an engine exists. An error refuses the build
OnEvict func(tenantKey, dsName string)	No	Runs after an entry’s handle has closed. Tear down entry-scoped auxiliaries here so a refresh can never leave a stale one bound to a rebuilt engine
Sharing *Sharing	No	Enables shared endpoint pooling. Nil keeps one pool per key

OptionsFor and SchemaFinalize are consulted on every open — first Get and every rebuild after an invalidation. OnEvict runs synchronously, once per evicted entry, outside the cache lock; it must not call back into Get for the same key.

New returns an error only for missing required fields. It never opens a connection.

Shared pools

Sharing routes tenants whose DSN resolves to the same endpoint — host, database and a poolable scheme — through one underlying connection pool. This is the “many small tenants on one PostgreSQL instance” case. Tenants whose DSN cannot be shared safely, including file-backed backends, fall back to per-tenant pooling silently; the setting is best-effort and never blocks an open.

Sharing field	Meaning
Registry	Holds the shared pools, keyed by endpoint. Required whenever Sharing is non-nil — New refuses otherwise. One per process. Manager.Close drops each entry’s refcount and the registry closes a pool when its count reaches zero; call Registry.Close yourself at shutdown to drain any pool whose count never got there
SchemaFunc func(tenantKey string) string	The physical schema name for a tenant. Nil uses the tenant key directly
StatementTimeout time.Duration	Overrides the default statement timeout for tenants on the shared path. Zero keeps the default; a negative value disables the statement timeout entirely
Eligible func(ctx, tenantKey, dsName) bool	Per-key gate. Return false to keep that pair on its own pool even though sharing is configured. Nil routes every shareable DSN through the registry

Caution

On the shared path, what isolates one tenant from another is SQL qualification, not the per-checkout session settings. The schema handed to the open — after SchemaFinalize — must carry the tenant’s own physical namespace in its Name. The engine’s non-transactional path uses the pool’s raw handle and never sees the per-checkout search_path; those settings are defence in depth on acquired and transactional connections only. Getting this wrong lets two tenants write into one schema.

Lifecycle

Call	Effect
Get(ctx, tenantKey, datastoreName)	Returns the cached handle, opening it on first call. An empty tenant key is allowed; an empty datastore name is an error
Invalidate(tenantKey, datastoreName)	Evicts and closes one entry, waiting for any in-flight open to finish first, then fires OnEvict
InvalidateTenant(tenantKey)	Evicts every datastore for one tenant
InvalidateAll()	Evicts everything, each eviction running the full close and OnEvict path
Close()	Evicts everything and refuses further Get calls. Returns the first close error
Stats()	{Cached int, Closed bool}. Cached counts entries that errored on open as well as live ones

Concurrent Get calls for the same key block on one open; different keys open in parallel. An entry that failed to open stays cached with its error until invalidated — retrying Get returns the same error, not a fresh attempt.

A complete service boot

package main

import (
    "context"
    "embed"
    "log"

    dataapi "<data-api-engine>"
    _ "<data-api-engine>/backend/postgres" // registers postgres:// and postgresql://
)

//go:embed data
var definitions embed.FS

func main() {
    ctx := context.Background()

    db, err := dataapi.OpenFromFS(ctx,
        "postgres://app:secret@localhost:5432/appdb?sslmode=disable",
        "warranty",           // the PostgreSQL schema every statement is qualified with
        definitions, "data",  // walked recursively for *.yaml / *.yml
        dataapi.WithUser("boot"),
    )
    if err != nil {
        log.Fatalf("open: %v", err)
    }
    defer db.Close(ctx)

    res, err := db.MigrateDetailed(ctx, func(f dataapi.MigrationFailure) {
        log.Printf("migrate %s %q: %v", f.Step, f.Object, f.Err)
    })
    if err != nil {
        log.Fatalf("migrate: %d failures", len(res.Failures))
    }
    log.Printf("tables created=%d skipped=%d indexes=%d fks=%d",
        res.TablesCreated, res.TablesSkipped, res.IndexesCreated, res.FKsCreated)

    diff, err := db.Drift(ctx)
    if err != nil {
        log.Fatalf("drift: %v", err)
    }
    if !diff.IsEmpty() {
        log.Fatalf("live schema does not match definitions")
    }
}

Swap the DSN for sqlite://:memory: and the blank import for the SQLite backend to get the same program with no external database.

What embedding does not give you

• No authorization by default. Access rules, row-level security and field protection are enforced by hooks and by compiler-side security components. Open registers the baseline three hooks and, with a script engine, the script-runtime hooks; the compiler’s security components are wired separately. An embedded engine with none of that wired evaluates queries with no enforcement while looking configured.
• No HTTP surface. REST and GraphQL belong to the host, not the engine.
• No reviewed migration. See the caution above.
• No tenant configuration. Datastore and pool resolution, tenant key derivation and configuration refresh are the host’s job. The engine factory gives you the caching and the provider seams, not the sources behind them.

Continue with

• Datastores and database pools — backends, dialect resolution, capability gates, physical schema qualification
• Migrations — the reviewed pipeline the sweep is not
• Your first datastore — the same journey through data.svc
• Query DSL — what goes into a query request