Observability

data.svc emits four things: metric instruments on the engine and on every route, one span per request, structured logs on stdout, and a set of admin routes that report live runtime state. Only the last two reach you without extra wiring.

Signal	Emitted	Leaves the process
Engine metrics — operations, transactions, hook errors	on every operation	no exporter is initialised
HTTP server metrics — count, in-flight, bytes, latency	per wrapped route	no exporter is initialised
Request spans	per wrapped route	no exporter is initialised
Logs	always	yes — stdout
Runtime state (queues, locks, tenant inventory)	on request	yes — admin routes
Compiled SQL for one request	on request	yes — ?stats=true
Engine debug logger (AST + SQL + timings)	library only	not wired in this service
EXPLAIN / query plan	never	does not exist

Caution

The shipped example configurations carry a telemetry: block. Nothing in this binary reads it, and this binary never initialises the OpenTelemetry pipeline. Meter and tracer lookups fall back to the global no-op providers, so every instrument and span described below records into nothing. The instruments are still registered and still cost their atomic adds; they simply have no exporter. Treat metrics and traces as unavailable until an exporter is wired into the process.

Engine metrics

The engine reports lifecycle events to one process-wide observer, built once and shared by every per-tenant engine. Ten instruments are registered. If the meter is unavailable or any registration fails, the observer degrades to a no-op for the life of the process and logs the reason at error level — engine construction never fails on telemetry.

Measures	Kind	Unit	Labels	Fires
Engine operations	counter	—	operation, entity, status	end of every query, create, update, delete, restore
Operation latency	histogram	ms	operation, entity, status	same call; covers hooks, compile, driver and result scan
Rows returned or affected	histogram	—	operation, entity	same call, skipped when the count is unknown
Transactions	counter	—	committed, status	end of every transaction
Transaction latency	histogram	ms	committed, status	begin through commit or rollback
Hook errors	counter	—	hook, phase, operation, entity	only when a hook returns an error
Migration runs	counter	—	none	library migration path only
Schema objects created	counter	—	none	library migration path only
Migration step failures	counter	—	none	library migration path only
Migration latency	histogram	ms	none	library migration path only

On the operation, latency and rows instruments, operation is one of query, create, update, delete, restore, and status is ok or error. phase is one of BeforeValidate, AfterValidate, BeforeExec, AfterExec, AfterScan. Successful hooks emit nothing, so hook-error count is a rate you can alert on directly.

Caution

The hook-error counter spells the operation label differently: Create, Read, Update, Delete or Multi — capitalised, Read where the operation instruments say query, and no restore value at all. The two label sets do not join. A dashboard that filters operation="query" across both instruments silently drops every hook error.

Nested transactions report once, at the outer call. A commit failure and a rollback both arrive with committed=false; the status label separates a clean rollback from a driver error.

The four migration instruments never fire in data.svc. They are emitted by the library’s own create-if-missing migration call, which this service does not use — its plan and apply pipeline is a separate path with no observer attached.

Note

Tenant is deliberately not a label on any engine instrument. With thousands of tenants it would multiply every series. Engine metrics tell you what the process is doing, not which tenant is doing it; use the per-request logs and the admin routes for that.

HTTP metrics

Every route registered by the service is wrapped in a per-route instrumentation middleware. Five instruments are created on first use:

Instrument	Kind	Registered unit
http.server.request_count	counter	—
http.server.active_requests	up-down counter	—
http.server.request_content_length	counter	By
http.server.response_content_length	counter	By
http.server.duration	histogram	Ms

Each carries http.method, http.scheme, user_agent.original and http.route (the route pattern, not the resolved path). When the request resolved a tenant, it also carries customer, product, env and tenant.

http.status_code is added to the attribute set only after the handler returns, so it is present on the two content-length counters and on http.server.duration, and absent from http.server.request_count.

Caution

http.server.active_requests increments before the status code exists and decrements after it is added, so the two adds carry different attribute sets and the series never nets back to zero. Read it as two separate counters, not as a gauge of in-flight requests. The duration histogram is registered with the unit string Ms; the engine’s latency histograms use ms. Normalise at the collector.

Caution

Unlike the engine instruments, the HTTP instruments are labelled with the full tenant identity. Every series is multiplied by the number of active tenants. Budget for it, or drop the four attributes at the collector.

/data/health and /data/ready are registered directly, ahead of the instrumentation wrapper, so they produce neither metrics nor spans. See Checking a tenant is servable for what the probes actually assert.

The wrapper carries a fallback that degrades to a plain passthrough — no metrics, no span — when the process resolves no service instance id. It never triggers in a booted service: the chassis registers a service-discovery client unconditionally, and when servicediscovery_url is unset that client is a stub whose service id is a non-empty placeholder. Every route is instrumented whether or not you set sid or servicediscovery_url; the placeholder id is what lands on the span as the server name.

Traces

One server span per instrumented request. The span name and the http.route attribute are both the route pattern. W3C traceparent and baggage headers on the incoming request are extracted, so the span joins a caller’s trace; a valid remote span already on the context is attached as a link. Tenant identity rides on the span as the same four attributes the metrics use, and any tags request headers are copied onto it. When the handler returns, the span records http.status_code and a span status derived from it. A second span is opened once at boot, around service start.

A panic inside a handler is recorded through the tracer by the router’s panic handler.

The engine adds nothing. It reports through metric events, not spans, so a captured trace is one span per request with no child spans for compile, hooks or the database round trip. Per-operation timing comes from the operation-latency histogram, or from ?stats=true on a single request.

Error responses carry no request id and no trace id. The response envelope is error, code and details only. Correlate a client-side failure with the logs by tenant, route and timestamp.

Logs

Logs are structured records written to stdout through a console writer, timestamped RFC 3339. Colour is applied only when stdout is a terminal, so a redirected stream or a supervisor-captured stream is clean. Every line carries service_id, datacenter and cluster, taken from the sid, datacenter and cluster configuration keys.

Key	Type	Default	Effect
log.level	string	error	Minimum severity: trace, debug, info, warn, error, fatal, panic

Caution

The default level is error, and most of what an operator needs is logged at info or warn: the boot summary, tenant cache invalidation, schema-source fallback, tenant pre-warm failures, and every one of the failures below that never reaches a caller. Run at info.

These conditions are logged and are invisible everywhere else — no probe fails, no request errors:

Condition	Consequence
A datastore init script fails	Logged; the engine still serves traffic
The materialized-refresh runner fails to install	Logged; mutations succeed and refresh tasks are never enqueued
Two schema documents declare the same entity or trait	Logged at error; the first declaration stands and the later one is ignored
schemasource: is unset, unknown, or points at a missing directory	Logged as a warning; falls back to the metadata service
A tenant listed under loadtenants: fails to warm	Logged; boot continues
The rate limiter fails to build, or a route names an undeclared profile	Logged; requests are allowed — the limiter fails open
Engine metric registration fails	Logged at error; engine metrics become no-ops
jwks.disabled: true	Logged once at error level, as a SECURITY: line — the service is accepting tokens without signature verification

Two logging knobs do nothing in this service, and both appear in shipped example configurations:

• log.loglinenumbers is read by no code. The service builds its logger with source line numbers off, unconditionally.
• realtimedebug, with the _debug_log_ query parameter or the X-Debug-Log header, switches the request context to a debug logger that this service never registers. Every log call in data.svc resolves the service logger directly, so the flag changes no output. Raise log.level instead.

Seeing the SQL

Add ?stats=true to a REST read or write and the response envelope gains a stats block containing the compiled sql, one entry per relation eager-load subquery in include_sql[], duration_ms, and rows_affected on mutations. This is the only per-request SQL visibility that works against a deployed service, and it is available to any caller who can make the request.

curl -s 'https://api.example.com/data/rest/invoice?status=open&stats=true' -H 'Authorization: Bearer <token>' -H 'X-Customer: acme' -H 'X-Product: erp' -H 'X-Env: prod' -H 'X-Tenant: eu1'

The full flag set and the exact envelope are in Request flags.

Caution

The stats block returns the statement as compiled, including the tenant’s physical schema name and its table and column names. Row-level security predicates are compiled into that statement, so the SQL also discloses the shape of the rule that filtered the caller. There is no configuration switch that strips it — the error sanitiser removes SQL from error responses only. If that disclosure is not acceptable for your product, block stats at the gateway.

The engine also carries a debug logger that reports far more: the operation, the entity, the parsed query rendered in pipe syntax, the compiled SQL, the bound arguments, every eager-load subquery, the statement count, the row count and the elapsed milliseconds. It is not wired in data.svc — there is no configuration key that turns it on. It is an option on the in-process engine, so it is reachable only from a Go service that opens the engine itself; see Embedding the engine. The standard implementation writes to stderr and prints argument count rather than argument values unless explicitly told otherwise, because bound arguments carry plaintext user input.

There is no EXPLAIN. No route, no query DSL keyword and no request flag returns a database query plan or a cost estimate, and the service issues no EXPLAIN of its own. To read a plan, take the statement from stats.sql and run it against the database yourself.

Runtime state over HTTP

These routes report what the process is currently doing for the calling tenant. All require the four CPET headers and a token.

Route	Reports	Gate
GET /data/admin/data/materialize/stats	Cross-entity refresh queue depth by status, plus cumulative runner counters. The runner block is absent when no runner exists for this tenant	admin
GET /data/admin/data/materialize/pending	Pending and failed refresh tasks with attempt counts — page size fixed at 100	admin
GET /data/admin/data/locks	Coordination locks with holder, operation kind, acquisition and expiry, is_expired and seconds_to_expiry. Accepts prefix, op_kind and include_expired	admin
GET /data/superadmin/tenant	The tenant keys this process currently holds live resources for — the closest thing to a cache inventory	superadmin
GET /data/admin/data/hashes	Nothing. The compiled-query catalogue is a stub and always returns an empty list	admin

Each /data/admin/data/… route above has a /data/superadmin/tenant/{tenant}/… twin — the same handler, re-rooted, reading the state of a named sibling tenant. The twin path drops the admin/data segment: GET /data/superadmin/tenant/{tenant}/locks. GET /data/superadmin/tenant itself has no admin equivalent, because an admin cannot list tenants it does not belong to. Migration plans, applied-step history and seed state are covered in Migrations.

What you cannot see

Not available	Detail
Query plans	No EXPLAIN emitter exists anywhere in the service
Connection pool statistics	The pool registry tracks reference counts and endpoint snapshots internally; no route exposes them
Per-tenant engine metrics	Tenant is not a label on any engine instrument, by design
A slow-query log	No latency threshold triggers a log line; latency is only in the histogram and in ?stats=true
The compiled-query cache catalogue	The route is a stub returning an empty list

The security audit stream is a further gap. Access denials and internal errors from the compiler are recorded into a tamper-evident hash chain that is held in memory, in one process-wide sink shared by every tenant engine, with no route that reads it and no eviction — it grows for the life of the process. Successful mutations are not recorded at all: the hook that emits them exists in the library but the deployable service does not register it. A durable, readable audit trail requires either per-entity history and change data capture on the entities you care about, or a Go host that wires its own append-only sink.