Skip to content
Talk to our solutions team

Collector Tuning

Date: 2026-05-06

Goal: cut UAT Azure-ingested telemetry from ~20 GB/day to ~1–2 GB/day without losing operational signal.

Companion doc: metrics inventory — full per-service metric catalog and antipattern list.

This guide is opinionated. The user has stated there are no backward-compat constraints, so it recommends the right design rather than the smallest delta.

Implementation status (2026-05-06)

Section titled “Implementation status (2026-05-06)”
LeverStatusNotes
Lever 1 — cardinality_capper (label strip)landedshipped as metric_filter.strip_attrs (cleaner naming); cardinality_capper was repurposed as a defensive series-count cap
Lever 2 — metric_filter (drop by name)landedprocessor/metric_filter.go — exact + regex deny/allow, plus strip_attrs for label removal
Lever 3 — CollectInterval 15s → 60s defaultlanded in code; pending in UATflipped in lib-zero/telemetry/config.go and lib-chassis/config/config.go (both V1 and V2 paths). UAT cluster conf.yaml still pins pushinterval: 5s — needs the YAML edit to take effect.
Lever 4 — log severity_filter + samplinglanded in code; pending in UATconfigured in .collector.yaml in the repo. UAT cluster collector/conf.yaml doesn’t yet have a processors: block — needs the YAML edit to enable.
Lever 5 — source-side primitive fixespartially landedgauge → histogram migration done for lib-seshat (86 sites) and infinity-hotei (12 sites); lib-guppi histogram bucket tuning done. Label moves (workflow.node.name, agent.id, entity, namespace, user_agent off metric labels onto spans) still pending — see source-side-fixes.md §§4-7.

Plus: customer_router was rewritten with explicit-enable semantics (no blind ClickHouse fallback). See collector-upgrade-design.md §“Routing — explicit-enable”.

Where the bytes are going

Section titled “Where the bytes are going”

Estimated split, before any cuts:

~18 GB/day metrics
   ~7 GB  chassis HTTP middleware (5 metrics × per-request × user-agent + route)
   ~5 GB  guppi DAG (workflow/task durations and counts; node.name + agent.id labels)
   ~2 GB  seshat (entity + namespace as labels)
   ~1 GB  janus + hotei (handler-level histograms, redundant with chassis)
   ~3 GB  the rest (chassis counters, orca, lib-zero runtime if on, OTLP overhead)


~2 GB/day logs
   most of it: INFO logs in DAG/queue/cache/response hot paths (see metrics-inventory §Antipatterns)

cluster-thoth is not deployed in UAT so it contributes zero to the current bill. If/when thoth is activated, the per-snapshot label issue would dominate quickly — see source-side-fixes.md §6 for the latent fix.

The 90/10 metrics/logs split your team measured matches this.

What’s actually wired today

Section titled “What’s actually wired today”

From cmd/root.go (buildProcessorPipeline) — the four processors the collector knows how to run:

ProcessorCode pathApplied to logsApplied to tracesApplied to metrics
attribute_normalizeprocessor/attribute_normalize.goyesyesyes
severity_filterprocessor/severity_filter.goyesnono
samplingprocessor/sampling.goyesyesno (commented at cmd/root.go)
rate_limitprocessor/rate_limit.goyesyesyes

This means in code, the only metric-side levers are attribute_normalize (no drops) and rate_limit (per-tenant flat token bucket). There is no filter, no metricstransform, no resource-attribute dropper, no aggregator, no temporality changer. These have to be added.

Active config in .collector.yaml:

otelcollector:
  port: 7116
  processors:
    attribute_normalize: true
    # severity_filter:           ← commented out
    #   min_level: error
    sampling:
      log_rate: 1.0              ← keep all logs
      trace_rate: 1.0            ← keep all traces
    rate_limit:
      rate: 10000                ← 10K dp/sec per tenant — never trips in UAT
      burst: 20000

Proposed end-state pipeline

Section titled “Proposed end-state pipeline”
service (15s push)  →  receiver
   →  attribute_normalize          (existing)
   →  severity_filter              (existing, turn on at WARN for UAT)
   →  cardinality_capper           (NEW — strips known unbounded labels)
   →  metric_filter                (NEW — drops a denylist of metric names)
   →  metric_aggregator            (NEW — re-aggregates per-host histograms across hosts)
   →  log_sampler                  (existing, drop to 0.1 in UAT for non-error logs)
   →  trace_sampler                (existing, drop to 0.05 in UAT)
   →  rate_limit                   (existing, lower defaults)
   →  ClickHouse exporter          (kept — local store for full fidelity)
   →  customer_router → Azure      (only routes for customers that need cloud)

Three new processors are required. They are small and idiomatic kis.ai-shaped (each is one Go file in processor/ implementing the existing MetricProcessor interface from processor/pipeline.go).

Five levers ranked by impact

Section titled “Five levers ranked by impact”

Lever 1 — Strip high-cardinality labels at the collector (biggest win, ~50% cut)

Section titled “Lever 1 — Strip high-cardinality labels at the collector (biggest win, ~50% cut)”

What’s broken: the source code in chassis, guppi, seshat (and latently in thoth, when it’s deployed) bakes unbounded values into metric labels. Fixing it at the source is the right answer long-term, but requires changes in 30+ files. The collector can strip the same labels in one place and stop the bleeding immediately.

New processor: cardinality_capper. One file in processor/cardinality_capper.go implementing the MetricProcessor interface, called from buildProcessorPipeline.

Behaviour: for every metric datapoint, drop or merge attributes from a configured deny-list. After stripping, the OTel SDK’s natural aggregation collapses datapoints that now share an identity, so the byte count drops proportionally to the cardinality reduction.

Config (drop into .collector.yaml):

otelcollector:
  processors:
    cardinality_capper:
      # Drop these attributes from all metric datapoints before export.
      drop_attrs:
        - user_agent.original          # chassis HTTP — unbounded
        - http.user_agent              # alt name same thing
        - http.route                   # if you don't need per-route cuts; otherwise see route_template_only below
        - kis.workflow.node.name       # guppi — user-defined per DAG
        - entity                       # seshat — per-request entity name
        - namespace                    # seshat — per-request namespace
        # - backup                     # thoth — defensive entry; thoth not deployed today, uncomment on activation
        # - snapshot                   # thoth — same
        - method                       # janus, hotei — handler method name (chassis already has http.method)
        - handler                      # janus — duplicate of route at handler granularity
      # If you'd rather keep http.route but ensure it's a template not a literal, add:
      # route_template_only: true      # rejects any value containing %, ?, #, or matching /\d{2,}/
      # If you want to fold tenant/customer into a single attribute (less explosion, still useful):
      collapse:
        - from: [kis.customer, kis.product, kis.environment, kis.tenant]
          to: kis.cpet
          separator: "/"

Estimated saving: chassis HTTP user-agent strip alone: ~40–60% of chassis volume → roughly 3–4 GB/day. Add the seshat entity/namespace strip and guppi node.name strip → another 2 GB/day.

Code sketch (representative, not for committing yet):

processor/cardinality_capper.go
type CardinalityCapper struct {
    drop      map[string]struct{}
    collapses []collapseRule
}
func (c *CardinalityCapper) ProcessMetrics(ctx context.Context, md pmetric.Metrics) (pmetric.Metrics, error) {
    rms := md.ResourceMetrics()
    for i := 0; i < rms.Len(); i++ {
        sms := rms.At(i).ScopeMetrics()
        for j := 0; j < sms.Len(); j++ {
            ms := sms.At(j).Metrics()
            for k := 0; k < ms.Len(); k++ {
                c.processMetric(ms.At(k))
            }
        }
    }
    return md, nil
}
// processMetric walks each datapoint's Attributes(), removes drop keys, applies collapse rules.
// After this pass, identical datapoints will need to be merged — call pmetric's natural
// aggregation by re-keying datapoints that now share attribute sets.

Wire-up in cmd/root.go:

if dropAttrs := smap.GetStringSlice(procCfg.Get("cardinality_capper.drop_attrs")); len(dropAttrs) > 0 {
    capper := processor.NewCardinalityCapper(dropAttrs, ...collapseRules...)
    pipeline.AddMetricProcessor(capper)
    count++
    log.Infoln("processor: cardinality_capper enabled (drops:", len(dropAttrs), ")")
}

Lever 2 — Drop entire metrics by name (next ~15%)

Section titled “Lever 2 — Drop entire metrics by name (next ~15%)”

Why: several metrics duplicate work. Janus and hotei pointcut histograms recapitulate what the chassis HTTP middleware already records. Some lib-guppi counters are easily reconstructed from existing histogram counts. Some haven’t been queried in months.

New processor: metric_filter. Same pattern — one file under processor/.

Config:

otelcollector:
  processors:
    metric_filter:
      mode: deny                   # deny=drop, allow=keep
      names:
        - janus_pointcut_plugin_execution    # redundant with chassis http.server.duration
        - hotei_pointcut_plugin_execution    # redundant; also wrong primitive
        - kis.task.success.count             # reconstructable from kis.task.duration count
        - kis.task.fail.count                # reconstructable from same with status attr
        - meili.http.server.*                # disabled today anyway; defensive deny
      # Or for finer granularity, name patterns:
      # patterns:
      #   - "snapshot_.*_counter"             # cluster-thoth — defensive; thoth not deployed today

Code: ~50 lines. Mirror severity_filter.go structure — one allow/deny check on metric Name() per datapoint.

Estimated saving: ~2 GB/day.

Lever 3 — Lengthen the collect interval from 15s → 60s (~25% on what remains)

Section titled “Lever 3 — Lengthen the collect interval from 15s → 60s (~25% on what remains)”

Why: 15s is dev-mode. The downstream queries (Grafana / dashboards / alerts) almost never need finer than 1-minute resolution; alerts fire on 5+ minute windows. Histograms × cardinality × interval is the volume mechanic — pulling one of three levers gives a 4× cut on everything.

Where: lib-zero/telemetry/config.go:

// Before
if c.Metrics.CollectInterval == 0 {
    c.Metrics.CollectInterval = 15 * time.Second
}


// After
if c.Metrics.CollectInterval == 0 {
    c.Metrics.CollectInterval = 60 * time.Second
}

If you can’t or won’t change the default, override per-environment in service configs (whichever YAML caller passes into telemetry.Init). 60s is recommended; 30s is the absolute floor I’d accept; 15s should never ship to UAT/prod.

Estimated saving: linear — 4× reduction in datapoints emitted by the source. For everything that survives Levers 1+2, that’s another ~3 GB/day off the post-Lever-2 number.

Lever 4 — Severity-filter logs and trace-sample (~80% of the ~2GB log+trace tail)

Section titled “Lever 4 — Severity-filter logs and trace-sample (~80% of the ~2GB log+trace tail)”

Why: the UAT collector runs at loglevel: debug (.collector.yaml), severity_filter is commented out, sampling.log_rate=1.0, sampling.trace_rate=1.0. That’s “ship everything, sample nothing.” For UAT this is a luxury you don’t need.

Config:

loglevel: info                  # was: debug
otelcollector:
  processors:
    severity_filter:
      min_level: warn           # uncomment; UAT keeps WARN+ERROR+FATAL only
    sampling:
      log_rate: 0.1             # 10% of INFO+ logs that pass severity_filter
      trace_rate: 0.05          # 5% of traces; keep error spans separately if you have an error_sampler

If you want to keep all error logs but drop INFO entirely, prefer severity_filter: error and keep log_rate: 1.0. Either is defensible; warn + 0.1 gives you noise-floor visibility.

Estimated saving: ~1.5 GB/day out of the ~2 GB log+trace tail.

Lever 5 — Fix the antipatterns at source (smaller, but compounding)

Section titled “Lever 5 — Fix the antipatterns at source (smaller, but compounding)”

These don’t show on the Azure bill until weeks of new series accumulate, but they’re the difference between a system that scales and one that explodes a year from now. Each is small.

FixFileEffect
Replace lib-seshat/metrics/plugin_time.go Gauge with Histogramlib-seshat/metrics/plugin_time.goCaptures distribution; no more last-write-wins data loss
Replace infinity-hotei/metrics/plugin_time.go Prometheus GaugeVec with OTel Histograminfinity-hotei/metrics/plugin_time.goStack consistency; correct primitive
Tune histogram buckets per-metric for kis.workflow.duration and kis.task.durationlib-guppi/dag/metrics.goUse exponential buckets up to 1h: [0.1, 0.5, 1, 5, 10, 30, 60, 300, 1800, 3600] (s). Stops +Inf overflow
Drop kis.workflow.node.name from metric attributes (move to span attribute)lib-guppi/dag/metrics.goKeeps node-level latency visible in traces; removes per-node series explosion
Drop entity, namespace from seshat metric attributes; promote to log/span fieldslib-seshat/metrics/metrics.goSame idea — wrong place for that information
Stop Gauge.Set per pointcut method in seshatlib-seshat/sem/sem_*_tx.go (~50 sites)Convert to histogram observations; the Set/Set pair becomes one Record(elapsed)
Replace per-task INFO with DEBUG, or one INFO per N taskslib-guppi/dag/engine_work_queue.goDrops a significant chunk of the log tail
Strip JSON-into-INFO from cache hot pathlib-chassis/cache/bigcache/root.goDrops cache-related log volume + per-request CPU
Strip full-response-body INFO from response handlerlib-guppi/engine/response_handler.goDrops per-node log volume
Stop the leaky ticker in fsvaultinfinity-horus/vault/fsvault.goAdd defer t.Stop() and a case <-ctx.Done() arm
Lower rate_limit.rate from 10000 to 2000 dp/sec/tenant.collector.yamlActs as a real ceiling, not a theoretical one. After Levers 1–3 the typical rate should be well under 2000/sec

ClickHouse + Azure dual-write — RESOLVED 2026-05-06

Section titled “ClickHouse + Azure dual-write — RESOLVED 2026-05-06”

The original concern is moot for UAT and addressed in code.

UAT inspection (cluster opus_uat_cluster_01, tenant opus): the live route is logs: { primary: azure-opus } and metrics: { primary: azure-opus } with no mirror set on either signal. No ClickHouse mirror was happening. The full 20 GB/day represents the Azure ingestion alone, not half of it. Earlier draft of this guide overstated the dual-write risk; correction has been propagated.

The code-level concern (default-true ShouldMirrorToClickHouse semantics) has been fixed. customer_router.go was rewritten with explicit-enable semantics:

  • ClickHouse is now a named output. Aliases clickhouse, ch, local-ch resolve to the in-process ClickHouse writer.
  • A signal writes to a backend only when the route names that backend as primary or mirror.
  • A signal with no effective route (no specific tenant route AND no routing.default) is dropped — CustomerRouter.Stats().Dropped{Logs,Traces,Metrics} exposes the volume so silent loss is visible.
  • New routing.default block in YAML supplies the fallback for unrouted customers; absence means strict blackhole.

See collector-upgrade-design.md §“Routing — explicit-enable” for design notes.

Expected post-cut volume per source

Section titled “Expected post-cut volume per source”

Targets after Levers 1, 2, 3, 4 applied. Lever 5 doesn’t appear in the table because its impact is on cardinality growth over time, not steady-state.

SourceTodayAfter capper (L1)After filter (L2)After 60s interval (L3)After log severity (L4)
chassis HTTP × 30 services~7 GB~3 GB~3 GB~750 MB~750 MB
guppi DAG~5 GB~2.5 GB~2 GB~500 MB~500 MB
lib-seshat~2 GB~600 MB~600 MB~150 MB~150 MB
janus + hotei~1 GB~700 MB~100 MB (denied)~25 MB~25 MB
other infra metrics~3 GB~2 GB~1.3 GB~325 MB~325 MB
logs + traces~2 GB~2 GB~2 GB~2 GB~400 MB
Total~20 GB~10.8 GB~9 GB~3.75 GB~2.15 GB

Numbers ±50%. The structure is the point: each lever contributes; you don’t need all four at once. cluster-thoth is excluded — not deployed today.

Order of operations I’d recommend

Section titled “Order of operations I’d recommend”
  1. Today, no code: uncomment severity_filter: warn, set sampling.log_rate: 0.1 and trace_rate: 0.05, change collector loglevel from debug → info in .collector.yaml. Lower rate_limit.rate to 2000. Restart collector. Lever 4 done. Expect ~1.5 GB/day off immediately.
  2. This week, lib-zero one-line change: flip default CollectInterval to 60s in lib-zero/telemetry/config.go. Redeploy services. Lever 3 done. Expect ~75% off everything else.
  3. Next week, write the two new processors: cardinality_capper and metric_filter under processor/, wire them in cmd/root.go (buildProcessorPipeline), populate the deny-lists in .collector.yaml. Levers 1+2 done. Expect to land at ~2 GB/day.
  4. Following sprint, source-side fixes (Lever 5): prioritized by file count: lib-seshat plugin_time, lib-guppi histogram buckets and node.name promotion to span, infinity-hotei OTel migration, the log noise items in lib-chassis and lib-guppi.
  5. Followup, separate effort: redesign customer_router to be metric-allow-list-based rather than dual-write-by-default. — DONE 2026-05-06. The router is now explicit-enable: every output (including ClickHouse) must be named. See collector-upgrade-design.md §“Routing — explicit-enable”.

Verification plan after each lever

Section titled “Verification plan after each lever”

After each step, sample 60 minutes of output and check:

  • ClickHouse otel.metrics row count (SELECT count() FROM otel.metrics WHERE TimeUnix > now() - INTERVAL 1 HOUR)
  • ClickHouse distinct attribute combinations per metric name (SELECT MetricName, uniq(Attributes) FROM otel.metrics ... GROUP BY MetricName ORDER BY 2 DESC LIMIT 20)
  • Azure ingestion size from the Azure portal billing API (or whatever you’re measuring the 20 GB at)
  • Spot-check that a known-good metric (say kis.workflow.duration for one tenant) still answers the same questions on Grafana

If any chart goes blank you’ve over-stripped — the fix is to add a label back to the capper’s keep-list.