Operations

Models are build artifacts

The single most important operational property of this block: cascade models are generated at build time, not discovered at runtime. A deployed bot’s behaviour is fixed by its artifact. Nothing retrains itself in production and quietly changes what your bot does.

The pipeline runs on every change to the bot’s repository:

bot.yaml, rules/, scripts/  ──▶ git push
        │
  1. validate definition and rules
  2. validate scripts (syntax, binding usage)
  3. generate the stage-2 dynamic model from the rule corpus
  4. train / update the stage-3 classifier
  5. fit per-stage calibration functions against labelled data
  6. run the eval suite            ◀── GATE: must meet thresholds
  7. package the artifact (config + rules + scripts + model refs + calibration)
  8. sign
  9. publish
        │
        ▼
  deploy: promote the artifact to an environment

Step 6 is a hard gate. A bot whose cascade regressed against labelled examples does not publish, which means a bad rule change fails in CI rather than in front of customers.

Because the dynamic model is generated from your rules, improving stage 1 improves stage 2 for free — adding rules makes the cheap end of the cascade smarter, not just longer.

Deploying

A bot artifact is versioned and signed. Deployment promotes an artifact to an environment; the runtime loads the definition, resolves the referenced models and calibration functions, and the cascade is live.

Rolling back is promoting the previous artifact. Since models and calibration travel with the artifact rather than living in mutable state, a rollback restores the exact prior behaviour rather than approximately restoring it.

Analytics

The cascade produces the metric that matters most here: where messages resolve.

Signal	Reading it
Resolution rate by stage	The cost curve of the whole bot
Escalation rate to stage 5	How much traffic reaches a frontier model
Rule precision	Rules firing on messages that then need escalation
Calibration drift	Predicted correctness diverging from actual outcomes
Intent distribution	Which handlers matter; which intents are missing
Turn count to resolution	Conversations going in circles

A healthy bot pushes resolution leftward over time — traffic that started at stage 5 migrates to stages 1–3 as rules and the classifier improve. If your stage-5 share is flat while traffic grows, cost grows with it and the cascade is not earning its complexity.

Watch rule precision alongside resolution rate. A rule that fires often and is frequently wrong looks like a cost win in the resolution histogram while quietly degrading answers; its learned confidence will fall, but the histogram alone will not tell you.

Compliance

PII handling, consent and retention are properties of the runtime rather than of each bot: conversations carry tenant identity end to end, and isolation between tenants is absolute — one tenant’s bots, conversations, scripts and models are invisible to another.

Model traffic flows through the AI Gateway, so bot conversations land in the same audit trail, redaction policy and retention tiers as every other model call on the platform. There is no second, bot-specific audit story to configure.

Failure modes

A stage’s model is unavailable. Messages escalate past it. The cascade degrades toward more expensive stages rather than failing — you pay more, users still get answers, and the escalation-rate metric shows it immediately.

Calibration has drifted. Thresholds stop meaning what they say, so the cascade escalates too eagerly or not eagerly enough. Drift is detected by comparing predicted against actual correctness and flagged for rebuild. It is a developer-triggered rebuild, deliberately: the platform tells you the calibration is stale rather than retraining your bot behind your back.

A channel’s credentials expire. That channel stops accepting messages; others are unaffected, since each channel’s credentials are independent vault references.

A skill or flow is unavailable. The handler fails and the conversation falls back to its escalation path rather than the whole bot going down.

What to watch

• Stage-5 share of resolutions — the cost driver, and the number to drive down
• Escalation rate trend — a step change usually means a rule or model regressed
• Calibration drift alerts — thresholds silently stop meaning what you set them to
• p99 response latency by resolution path — rag and flow are the slow paths
• Rule precision — cheap resolutions that are quietly wrong