Server Modes

For horizontal scale — load generation beyond one machine, or centralized test execution — run Testing as a control plane (orchestrator) plus workers (agents).

                    gRPC (agents dial out)
   ┌─────────┐   register + heartbeat + work   ┌──────────────┐
   │  agent  │ ───────────────────────────────▶ │ orchestrator │◀── HTTPS API (JWT)
   └─────────┘ ◀─────────────────────────────── └──────────────┘
   executes tests        test-run messages        persists runs
   streams results                                to datastore

• The orchestrator runs a gRPC server (agents register and heartbeat) and an HTTPS API for submitting runs and reading results. Results persist to the data datastore as testrun, testrunmetric, testrunlog, and scenariorun entities.
• Agents dial the orchestrator as gRPC clients (only the orchestrator needs an inbound gRPC port), execute the tests they’re handed, and stream logs/metrics back over the same channel. Each agent also serves a minimal local HTTPS endpoint: GET /ready, GET /health.

Both commands read a .test.yaml config file (-f/--config) plus the shared infra flags (--port, --sid, --config_url, service-discovery and TLS cert flags — see the CLI reference).

Orchestrator

test.svc orchestrator -f orchestrator.test.yaml

Key config:

Key	Required	Meaning
grpc	yes	gRPC server port (startup fails without it)
host / port		HTTPS API bind address
timeouts.read / timeouts.write / timeouts.readheader		server timeouts (seconds or duration strings; defaults 15s/15s/5s)
orchestrator.queue.maxdepth		work-queue depth
serviceid		service identity
loadtenants		tenants to warm-load at boot
grpccreds		gRPC credentials
localagent		optional in-process agent: id, serverurl, grpc, heartbeatduration (default 15s), infrahost, grpcaddress, httpaddress

TLS comes from the container-registered service certificates (--svccert/--svckey). Shutdown is graceful: SIGINT/SIGTERM drains agent and orchestrator work with 30s timeouts.

HTTP API

All routes are JWT-secured.

Route	Method	Purpose
/list/tests	GET	teststeps/testcases/testscenarios/testplans known for the tenant
/execute/tests	POST	fan a filtered test set out to workers
/list/test/runs	GET	run history with metrics and logs
/product/:product/run	POST / GET	the universal test runner (functional or load)
/agents	GET	registered agents
/workflow/queue	GET	queue status
/workflow/queue/:id	DELETE	cancel queued work
/ready, /health	GET	probes

POST /execute/tests body:

{
  "type": "testplan",
  "pattern": "api-smoke",
  "vars": {"baseurl": "https://api.staging.kis.ai"},
  "logresponse": false,
  "workercount": 2
}

Splits the filtered tests across workercount agents; responds {"message": "test runs started successfully", "status": {"runid": "..."}}.

POST /product/:product/run body fields:

Field	Default	Meaning
environment	default	environment name
config	—	inline map, or a filename fetched from tenant meta at tests/config/<path>
testplan	required	inline map, or filename → tests/functional/<path>
load	—	inline map or filename → tests/load/<path>; absence = functional run
loadprofile	default	profile name (loadprofiles.<name>)
agentcount	—	number of agents to fan out to

Each scenario in the plan is cloned into its own sub-plan and dispatched to a selected agent; a scenariorun record tracks each. Response: {"runid": ..., "message": "testplan sent for execution"}. GET /product/:product/run?id=<runid> returns the scenario-run rows — statuses progress created → inprogress → completed / failed.

GET /list/test/runs query params: id, filter, page (default 1), pagesize (default 10, max 100), metricfields, logfields.

Agent

test.svc agent -f agent.test.yaml

Key config:

Key	Required	Meaning
serverurl	yes	orchestrator gRPC address to dial
sid		agent instance id
grpc		agent’s own gRPC port
heartbeatduration		default 15s
infrahost, grpcaddress, httpaddress		addresses advertised on registration
tenants		tenant keys this agent serves
exclusive		reserve the agent for its tenant list
max_concurrent_tasks		concurrency cap
host / port + timeouts.*		the local /ready + /health HTTPS server
loadtenants, grpccreds		as for the orchestrator

The agent receives JSON test-run messages (run id, environment, config, testplan, load profile, tenant), executes them with the distributed writers enabled, and streams results back: metric records become testrunmetric rows, base64 log lines become testrunlog rows, and a completion/error message closes out the scenariorun.

For load runs the message carries the same loadprofile shape as local runs (see Load testing) — agents run the phases locally, so targetusers applies per agent: 4 agents × targetusers: 50 = 200 concurrent users against the target.

Operational notes

• Agents dial out — only the orchestrator’s gRPC and HTTPS ports need to be reachable.
• The orchestrator needs datastore connectivity (data host boot) to persist runs; agents do not persist anything locally.
• A known cosmetic bug: the agent logs “Using config file:” with the wrong variable — the file actually loaded is the one from -f/--config.