Metrics Inventory

Date: 2026-05-06

Scope: all Go services on the platform that emit OTLP metrics to ops-hugin

UAT context: 5 hosts running every service; ~20 GB/day arriving in Azure; ~90% of that is metrics

Audit basis: static read of source. Anything I couldn’t verify from code is marked speculation.

Summary

Five things drive the bill:

1. Default emit interval is 15s, not 60s. lib-zero/telemetry — every metric gets pushed 5,760 times per host per day. Nothing in the runtime configs overrides it.
2. Zero metric-side filtering exists in the collector. The collector exposes four processors (attribute_normalize, severity_filter, sampling, rate_limit); only attribute_normalize and rate_limit are wired into the metric path. sampling is hardcoded to logs/traces only. There is no filter / metricstransform / attributes-drop processor in the codebase at all.
3. Default rate_limit is 10,000 datapoints/sec per tenant (.collector.yaml). At 5 hosts + 1 tenant that’s the headroom for ~864M datapoints/day before anything is dropped — i.e. it never trips in UAT.
4. High-cardinality labels are baked into every chassis-served HTTP request. The chassis HTTP middleware attaches user_agent.original (unbounded), http.route (per route template), customer/product/environment/tenant (4 attrs), and http.status_code to 5 metrics per request.
5. Two services dual-write per request to chassis and a separate per-service per-handler histogram (infinity-janus, infinity-hotei) with labels including tenant, method, plugin, handler — the same dimensions chassis already covers.

The full list of high-cardinality offenders, ranked by estimated daily byte impact, is in the Top byte-volume offenders section below.

Pipeline (current state)

service (lib-zero/telemetry)
  └─ OTLP gRPC every 15s ─→ ops-hugin :7116
                                ├─ attribute_normalize  (semconv v1→v2)
                                ├─ rate_limit           (10K/sec/tenant token bucket)
                                ├─ ClickHouse exporter  (always, even when customer is routed elsewhere)
                                └─ customer_router      (per-tenant)
                                      └─ AzureMonitorExporter
                                            ├─ OTLP-proxy path (preferred)
                                            └─ Track API v2 (fallback)

Notes:

• severity_filter is commented out in .collector.yaml. Logs ship at all levels.
• sampling.log_rate=1.0 and sampling.trace_rate=1.0 (.collector.yaml) — i.e. no sampling in UAT.
• customer_router.RouteMetrics routes to whichever exporter the route names. After the explicit-enable refactor (2026-05-06), ClickHouse is just a named output — it writes only when listed as primary or mirror. UAT confirmed: the live route for tenant opus is primary: azure-opus with no mirror — Azure-only, never dual-writing. The 20 GB/day in Azure is the full ingestion volume, not half of it.

Default knobs (lib-zero/telemetry)

Knob	Default	Comment
Metric collect interval	15 s	Doc string on field says default 15s; applyDefaults confirms
Metric push timeout	30 s
Default histogram buckets (ms)	[1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000]	12 buckets. Latency-shaped — wrong for byte counters and for workflow durations that exceed 10s
Trace sampling rate (when traces enabled)	0.1
Logs level	info	UAT collector also runs at loglevel: debug (.collector.yaml)
RuntimeMetrics field default	declared true in field comment, not enforced in applyDefaults	Likely defaults to false unless callers set it. Worth verifying at runtime.

Histogram cardinality math (default buckets)

Each histogram with default buckets emits 15 datapoints per series per push (12 explicit buckets + +Inf + sum + count). At 15s interval:

• 1 series × 1 host × 1 day = 15 dp × 5,760 pushes = 86,400 datapoints/day.
• 1 series × 5 hosts × 1 day = 432,000 datapoints/day.
• A histogram with even modest cardinality (say 50 series) blows past 20M datapoints/day for one metric.

This is the dominant volume mechanic. Histograms × cardinality × interval is the lever.

Per-source inventory

A. chassis HTTP middleware (every service that serves HTTP)

Metric	Type	Unit	Notes
http.server.request_count	Counter	dimensionless	per-request
http.server.active_requests	UpDownCounter	dimensionless	+1 / −1 per request (2 dp/request)
http.server.request_content_length	Counter	By	per-request, but a histogram fits the data better
http.server.response_content_length	Counter	By	per-request, ditto
http.server.duration	Histogram	Ms	default 12 buckets — +15 dp per request per series

Labels attached:

• http.method — bounded (~7)
• http.scheme — bounded (~2)
• user_agent.original — UNBOUNDED. Browsers, curl variants, k6, internal clients each unique.
• http.route — per route template (~10–80 per service)
• kis.customer, kis.product, kis.environment, kis.tenant — 4 tenant attrs
• http.status_code — bounded (~30)

Cardinality estimate per service: routes (40) × methods (4) × statuses (5) × user_agents (50, conservative) × tenant set (1 in UAT) = 40,000 active series. For 5 metrics that’s 200K series per service. ×30 services × 5 hosts ≈ tens of millions of series.

The user_agent.original label is the dominant cardinality multiplier here. The Traefik middleware in infinity-meili explicitly avoided it for this reason — it intentionally drops http.route, calling out an “explosion of dimensions” — yet the chassis middleware reintroduces both route and user_agent.

Telemetry-V2 mode returns early when telemetry.IsV2Active() — so if v2 is on these duplicate metrics may not double-emit. Speculation: not verified whether UAT runs v2 or v1.

B. lib-guppi DAG engine (any service hosting workflows)

Metric	Type	Labels	Cardinality risk
kis.workflow.start.count	Counter	kis.tenant, kis.workflow.definition	medium — definitions per tenant
kis.workflow.complete.count	Counter	same	medium
kis.workflow.fail.count	Counter	same	medium
kis.workflow.active	UpDownCounter	same	medium
kis.workflow.duration	Histogram	kis.tenant, kis.workflow.definition	high — 15 dp × per-definition × default buckets that top out at 10s when workflows can run minutes (overflows into +Inf)
kis.workflow.node.duration	Histogram	+ kis.workflow.node.name	very high — node.name is user-defined per DAG, dozens to hundreds of unique names per definition
kis.workflow.node.retry.count	Counter	+ kis.workflow.node.name	high
kis.task.dispatch.count	Counter	+ kis.agent.id	very high — definition × node × agent
kis.task.duration	Histogram	+ kis.agent.id	very high — same explosion + 15 buckets
kis.task.success.count / kis.task.fail.count	Counter	same	high
kis.task.inflight	UpDownCounter	same	high
kis.workflow.queue.depth	UpDownCounter	kis.tenant	low
kis.workflow.queue.wait_time	Histogram	kis.tenant	medium
kis.workflow.queue.reject.count	Counter	kis.tenant	low

Also: lib-guppi’s retry manager re-registers kis.workflow.node.retry.count from a second site — verify whether this creates a duplicate instrument or reuses one. Speculation: probably reuses, but worth a runtime check.

C. lib-orca

Source	Metrics	Notes
lib-orca agent	12 observable gauges: infra.cpu_used, infra.memory_used, infra.memory_cached, infra.memory_available, data.inflow, data.outflow, infra.gpu_usage, infra.gpu_memory_used, infra.gpu_memory_total, infra.gpu_temperature, infra.disk_used, infra.disk_total	Callback-driven; no custom labels. Bounded by host. 12 series × 5 hosts = 60 active series — fine.
lib-orca orchestrator	6 metrics including kis.agent.connect.count, kis.agent.disconnect.count, kis.agent.heartbeat.count, kis.agent.state_change.count, kis.agent.backpressure.count, kis.agent.connected	Labels include kis.agent.id. Bounded by agent count (~5–50 in UAT) — manageable. State_change carries from/to states — bounded pair set.
lib-orca selectors	kis.selector.duration (Histogram), kis.selector.success.count, kis.selector.fail.count	Label: kis.tenant. Low risk.

D. lib-seshat

Metric	Type	Labels	Cardinality
data_entity_hit	Counter	namespace, entity, customer, product, environment, tenant, operation	very high — entity is per-request entity name (unbounded), namespace per-request
data_db_query_time	Counter (Float64)	same	very high
data_db_query_generation_time	Counter (Float64)	same	very high
data_annotate_exec_time	Counter (Float64)	same	very high
data_txn_success / data_txn_failure	Counter	same	very high

A per-plugin gauge in lib-seshat is a prometheus.GaugeVec with pointcut, plugin, method, type labels — set twice per pointcut method (args_parsing + execution), called from 50+ sites. Wrong primitive (gauge overwrites — distribution lost) and wrong cardinality model.

E. infinity-janus

Metric	Type	Labels	Cardinality
janus_pointcut_plugin_execution	Histogram (Int64)	customer, product, environment, tenant, plugin, handler, method, type	very high — method is the request handler method name; plugin+handler add multiplicatively

This effectively mirrors what the chassis HTTP middleware already records. Two histograms per request from one service.

F. infinity-hotei

Metric	Type	Labels	Notes
hotei_pointcut_plugin_execution	Prometheus GaugeVec	pointcut, plugin, method, type	Inconsistent stack — every other service uses OTel; only hotei imports prometheus/client_golang directly. Wrong primitive (gauge for execution time loses distribution data). High cardinality across all 4 labels.

G. cluster-thoth — NOT DEPLOYED in UAT

Metric	Type	Labels	Cardinality
snapshot_file_counter	Counter	customer, product, environment, tenant, backup, snapshot	very high (latent) — backup and snapshot are IDs
snapshot_folder_counter	Counter	same	very high (latent)
snapshot_time_counter	Counter	same	very high (latent)
snapshot_size_counter	Counter	same	very high (latent)

cluster-thoth is not currently active in the 5-machine UAT, so these metrics are emitting nothing today. The code exists and the cardinality bug is real — every new snapshot would create four new permanent series — but it doesn’t contribute to current Azure volume. Treat as a latent issue: fix when activating thoth, not before.

H. infinity-meili Traefik middleware

Metric	Type	Notes
meili.http.server.request_count	Counter	Wired only when kisaimetrics.enabled=true in middleware config. Speculation: not currently enabled in UAT — no kisaimetrics.port overrides found in any service config.
meili.http.server.active_requests	UpDownCounter	same
meili.http.server.duration	Histogram	same

Notably, the middleware explicitly drops http.route from the label set “to prevent dimension explosion.” Good — but this awareness isn’t applied in the chassis HTTP middleware that everyone else uses.

I. Services with no custom metrics found

forge-brahma, forge-forge, forge-mitra, forge-root, forge-daemonize, ops-hugin (itself), ops-munin, infinity-bff, infinity-budata, config.svc, infinity-dutah, infinity-horus, infinity-maya, infinity-naarada, infinity-r2d2, infinity-ravana, infinity-seshat (the service, distinct from lib-seshat), infinity-status, infinity-telemetry. They all still emit chassis HTTP metrics if they serve HTTP — that’s the bulk.

Top byte-volume offenders (ranked, with rough math)

The following are ordered by my best estimate of daily Azure-ingested bytes. Estimates assume 15s collect interval, 5 hosts, 1 active tenant, 50 user-agent values, 40 routes per service, OTLP encoding ~80 bytes per datapoint with attributes (rough).

#	Source	Driver	Est. dp/day	Notes
1	chassis HTTP http.server.duration	per-request histogram × 5 metrics × 5 hosts × 30 services × user-agent label	~4–8 GB/day	The single biggest contributor by far. The histogram alone is ~15× counter-volume.
2	guppi kis.task.duration	definition × node × agent × 15 buckets × 15s	1–3 GB/day at moderate workflow throughput	Default buckets cap at 10s — inputs >10s pile into +Inf, distribution useless above 10s
3	guppi kis.workflow.node.duration	definition × node × 15 buckets	0.5–2 GB/day	node.name as a label is the killer
4	seshat data_* counters/timers	entity × namespace × tenant × 4 metrics × 5 hosts	0.5–1.5 GB/day	entity is the unbounded label
5	janus janus_pointcut_plugin_execution	method × plugin × handler × tenant × 15 buckets	0.3–1 GB/day	Duplicate of chassis duration with extra dimensions
6	hotei hotei_pointcut_plugin_execution	pointcut × plugin × method × type	0.2–0.6 GB/day	Wrong primitive (gauge); double-billing with chassis
7	guppi kis.task.dispatch.count + kis.task.success/fail.count + kis.task.inflight	definition × node × agent	0.3–0.8 GB/day	One series per (def, node, agent) — agents come and go
8	chassis http.server.request_count + active_requests	same dimensions as duration	0.3–0.6 GB/day	Counters — much cheaper per dp than histograms but same cardinality
9	orca kis.agent.state_change.count	agent × from-state × to-state	0.1 GB/day	Bounded; mostly fine
10	orca agent infra gauges (12)	host-only labels	0.05 GB/day	Healthy; leave alone

Excluded: cluster-thoth is not deployed in UAT — its snapshot_*_counter metrics are latent in code but contribute zero bytes today.

Numbers are rough — within ±50% — but the ranking is robust.

Antipatterns

These are the items I’d want fixed even if Azure billing wasn’t an issue.

1. user_agent.original as a metric label — chassis HTTP middleware (lib-chassis). UA is for traces and logs, never metrics.
2. kis.workflow.node.name as a metric label — lib-guppi DAG metrics (multiple sites). User-defined identifier; no upper bound.
3. entity and namespace as metric labels in seshat — lib-seshat.
4. Snapshot/backup IDs as metric labels — cluster-thoth. Permanent series growth. Latent — thoth not deployed in UAT, but the code is wrong; fix before activation.
5. infinity-hotei uses Prometheus directly while everyone else uses OTel — infinity-hotei. Plus the gauge-as-timer primitive error.
6. Plugin execution time stored as Gauge in lib-seshat — lib-seshat. Gauge.Set() overwrites; if multiple pointcut methods fire per request only the last one survives. Should be a Histogram.
7. Default histogram buckets used for kis.workflow.duration and kis.task.duration — lib-guppi. Buckets top out at 10s; workflows and tasks routinely exceed that. Either tune the buckets per-metric or use exponential buckets.
8. 15 s CollectInterval default — lib-zero/telemetry. 60–120s is the standard for production-grade metrics; 15s is dev-mode.
9. severity_filter commented out, sampling.log_rate=1.0 — .collector.yaml. Logs ship at all levels including DEBUG when collector itself runs at loglevel: debug (.collector.yaml).
10. JSON-marshal-into-INFO inside cache hot path — lib-chassis bigcache. log.Infoln(json.MarshalToString(value)) per Get/Put.
11. Full response object logged at INFO per node completion — lib-guppi engine. log.Infoln("success response from node", response.Type.String(), response.Response).
12. Per-task INFO logging in DAG drain loop — lib-guppi DAG work queue. Fires every dispatch.
13. Leaky time.NewTicker with no .Stop() and no ctx.Done() — infinity-horus vault. Goroutine leaks on shutdown; not directly a bytes issue but bad hygiene.
14. Dual exporter write to ClickHouse + Azure — RESOLVED 2026-05-06. The original concern was that ShouldMirrorToClickHouse defaulted true for unrouted customers, creating an implicit ClickHouse mirror alongside any external primary. The customer_router has been rewritten with explicit-enable semantics: ClickHouse is now a named output (aliases clickhouse/ch/local-ch), and a route writes to it only when explicitly listed as primary or mirror. UAT inspection confirmed no dual-write was actually happening — tenant opus had primary: azure-opus only.
15. Duplicate metric registration site for kis.workflow.node.retry.count — lib-guppi DAG metrics and lib-guppi retry manager. Verify they resolve to the same instrument.

Open verification items

1. Confirm at runtime whether cfg.Metrics.RuntimeMetrics is being set true anywhere — its applyDefaults doesn’t set it. If true, Go runtime metrics (runtime.go.gc.*, runtime.go.memory.classes.* etc.) add ~30–50 series per process at 15s interval.
2. Confirm whether telemetry-V2 mode (telemetry.IsV2Active()) is on in UAT. If yes, chassis tracermiddleware is bypassed and a different middleware emits the per-request metrics — same cardinality concerns apply but source differs.
3. Confirm which customer’s route includes Mirror: "clickhouse" — RESOLVED 2026-05-06. UAT clusterconfig inspection: tenant opus has logs: { primary: azure-opus } and metrics: { primary: azure-opus }, no mirror set on either signal. No dual-write to ClickHouse was occurring. The full 20 GB/day is Azure ingestion of azure-opus-routed data only.
4. Confirm the actual UAT histogram bucket override (Metrics.HistogramBuckets). The default is the 12-bucket latency shape; if any service overrides this with more buckets, that metric’s volume scales linearly.