Testing CLI Reference

Two names, one program

kis test is the CLI. It is what you type to run tests, load test them, or compare results. Every example in these docs uses it.

test.svc is the service. It is the binary itself, and the name you use when running it as a long-lived process: an agent taking work, or an orchestrator handing work out.

kis test run  -t tests/                     # CLI: run a suite
test.svc agent --orchestrator host:8080     # service: a worker process

They are the same program. kis test <x> forwards its arguments to test.svc <x> untouched, so flags, help and behaviour are identical: there is no second implementation to drift.

The release build produces dist/test.svc; kis finds it on PATH, in ~/.kisai/bin, or beside the kis binary. If it is not installed, kis test says so and how to install it.

Command	Purpose	Doc
run (alias test)	functional test runner: the recommended way to run API tests	Getting started, Test definitions
load	load test the same suites, with phases and thresholds that gate CI	Load testing
runs	list, show and compare stored results	Outputs
agent / orchestrator	distributed execution	Server modes
tests	legacy v1 hierarchy runner; also drives load via --load	the hierarchy format (kis test tests, see the CLI Reference)
api	legacy v1 collection runner	the hierarchy format (kis test tests, see the CLI Reference)
version	print the version and build commit	—

Exit codes: 0 success, 1 on any test failure, load threshold miss, or cobra error; -1 for bare invocation with no command. tests and api exit 1 when any request or script failed, including one that continueonerror let the run continue past.

kis test run (alias: kis test test): v2 functional runner

Runs a test YAML file or a directory tree of suites. No files are written by default: results stream to the console and the process exits 0/1. See Test definitions for the YAML schema.

kis test run -t tests/                         # run a suite tree; env auto-discovered
kis test run -t tests/orders.yaml              # run one file
kis test run -t tests/ -e env.yaml -n staging  # multi-env file; pick "staging"
kis test run -t tests/ --tags smoke
kis test run -t tests/ -v url=http://localhost:8080 -v password=secret
kis test run -t tests/ --db .kis/test/results.db --junit .kis/test/junit.xml
kis test run -t tests/ --html .kis/test/report.html --artifacts .kis/test/artifacts
kis test run -t tests/ --dry-run               # validate tree + templates, run nothing
kis test run -t tests/ --orchestrator https://orch:8443   # spread the suites across agents

Flag	Short	Default	Description
--tests	-t	—	Required. Path to a test YAML file or directory.
--env	-e	—	Environment YAML/JSON file. Auto-discovered next to --tests when omitted (see below).
--name	-n	—	Environment name inside a multi-env env file.
--vars	-v	—	Inline key=value variables. Highest precedence. Repeatable.
--tags	—	—	Include tags. Commas inside one entry = AND; repeat the flag = OR.
--exclude-tags	—	—	Same syntax; matched tests are removed. Exclusion wins over inclusion.
--db	—	—	DuckDB file path. Results persist only when provided.
--junit	—	—	JUnit XML output path (CI integration).
--html	—	—	HTML report path: every test and step, what it sent and got, and the files it kept. See Outputs.
--artifacts	—	system temp	Directory that keeps the files steps produce, one subdirectory per run.
--browser	—	$TALOS_BROWSER, else obscura	Browser for the browser steps that neither they nor their suites choose: obscura, chromium, firefox or webkit. See Choosing the browser.
--dry-run	—	false	Validate the suite tree and render every step’s templates without running any step.
--forbid-only	—	false	Fail when any test sets only: true, so a focused test cannot shrink a CI run.
--no-color	—	false	Disable ANSI colors (auto-disabled when stdout isn’t a TTY).
--orchestrator	—	—	Orchestrator base URL. The run’s suites are spread across its registered agents; see Server modes.
--token	—	$TALOS_TOKEN	Bearer token for the orchestrator.
--max-agents	—	0 (all)	With --orchestrator, use at most this many agents.
--bundle-root	—	the --tests directory	With --orchestrator, the directory sent to the agents. Widen it when the suites read files outside --tests.
--timeout	—	30m	With --orchestrator, how long the agents may take. Tests not reported by then are reported errored.

Local runs, distributed runs, load iterations and distributed load shards all run a test through the same code, so a test behaves the same way in each.

Variable precedence

Low → high: suite.yaml variables: → env file → --vars.

Env file auto-discovery

When --env is omitted, the runner looks in the tests directory (or the directory containing the tests file) for these, in order, and the first hit wins:

1. env.local.yaml / env.local.yml / env.local.json
2. env.<name>.yaml / .yml / .json, only when --name is set
3. env.yaml / env.yml / env.json

The discovered path is announced on stderr. Commit env.yaml, keep env.local.yaml in .gitignore for personal overrides.

Env file shapes

Flat: a plain map of variables:

url: http://localhost:8080
password: secret

Multi-env: every top-level value is a map; keys are environment names:

default:
  url: http://localhost:8080
staging:
  url: https://api.staging.kis.ai

Selection rules: --name <key> picks that block (error lists known keys if missing); no --name picks default; a multi-env file with neither default nor --name is an error. Flat files ignore --name. Accepted extensions: .yaml, .yml, .json (extensionless parses as YAML).

kis test load: v2 load runner

Runs the same v2 suites as run, as a load workload. A missed threshold exits 1, which is what makes a load run usable as a CI gate. See Load testing.

kis test load -t tests/ --profiles load.yaml --profile stress
kis test load -t tests/ --profiles load.yaml --profile soak --db .kis/test/results.db
kis test load -t tests/ --profiles load.yaml --profile stress --orchestrator https://orch:8443

Flag	Short	Default	Description
--tests	-t	—	Required. Test YAML file or directory.
--profiles	—	—	Required. Load-profile file (profiles: key).
--profile	—	default	Profile name inside that file.
--env	-e	—	Environment file; auto-discovered beside --tests when omitted.
--name	-n	—	Environment name within a multi-env file.
--vars	-v	—	Inline key=value variables.
--tags	—	—	Include tags; comma-separated = AND, repeated = OR.
--exclude-tags	—	—	Exclude tags, same syntax.
--db	—	—	DuckDB file to persist results into.
--no-color	—	false	Disable ANSI colors.
--no-progress	—	false	Suppress the live progress line (already off when stderr is not a terminal).
--forbid-only	—	false	Fail when any test sets only: true.
--shard	—	—	Run one slice of the load, as N/M. Users and rates divide; durations do not.
--out	—	—	Write this generator’s mergeable result to a JSON file.
--orchestrator	—	—	Orchestrator base URL; the run is sharded across its registered agents.
--bundle-root	—	the --tests directory	With --orchestrator, the directory sent to the agents with the run. Widen it when the suites read files outside --tests.
--token	—	$TALOS_TOKEN	Bearer token for the orchestrator.
--max-agents	—	0	Use at most this many agents; 0 means all registered.

--shard and --orchestrator conflict and are refused together: the first runs one slice here, the second shards across agents.

kis test load merge: combine several generators

kis test load merge box1.json box2.json box3.json
kis test load merge 'results/*.json' --db .kis/test/results.db

Flag	Default	Description
--db	—	DuckDB file to persist the merged run into.
--no-color	false	Disable ANSI colors.

Counts add and distributions add bucket by bucket. Thresholds travel with the results and are judged on the merged numbers: a per-shard verdict would gate on a fraction of the load. Globs are expanded internally as well as by the shell, so the quoted form works from a Makefile.

kis test coverage: what a run proved about the service

Measures which of the service’s routes a run called and which of its statements it executed, stores both, and can gate a build. See Coverage.

kis test coverage --db results.db --routes ../orders-service
kis test coverage --db results.db --covdata ./covdata
kis test coverage --db results.db --routes ../orders-service --covdata ./covdata \
    --min-routes 80 --min-statements 60

Flag	Default	Description
--db	.kis/test/results.db	results database
--run	most recent	run to measure; accepts the shortened id runs list prints
--routes	—	service source directory, or a JSON route file
--covdata	—	GOCOVERDIR directory from go build -cover, or a coverage profile file
--min-routes	0	fail below this percent
--min-statements	0	fail below this percent
--json	false	emit the report as JSON

At least one of --routes / --covdata is required. Re-measuring a run replaces its rows rather than adding to them.

kis test monitor: resource metrics for the machine under test

Runs on the target and serves its process and host metrics; a load run polls it so memory and latency land on one timeline. See Load testing.

kis test monitor --port 8080 --collect process,host
kis test monitor --name orders-service --collect process.rss_bytes,process.cpu_pct
kis test monitor --pid 4242 --collect all --endpoint http://localhost:8080/debug/vars
kis test monitor --list
kis test monitor --collect process,host --name orders-service --otel

Flag	Default	Description
--listen	:9099	address to serve on
--pid	—	process id to measure
--name	—	measure the newest process matching this name
--port	—	measure whatever is listening on this port
--endpoint	—	expvar-shaped URL for the go group
--collect	process,host	groups (process, host, cgroup, go, all) or individual metric names
--list	false	print the metric catalogue and exit
--otel	false	print an OpenTelemetry Collector configuration that reports the same metrics, and exit

--pid/--name/--port pick the process; name and port are re-resolved every reading, so a service that restarts mid-soak keeps being measured. A load profile may narrow what it asks for but never widen it: these flags are the authority on what the host exposes.

kis test runs: inspect and compare stored runs

Reads a results database written by --db. Functional and load runs share one table, so they list and compare together. See Outputs for the schema.

kis test runs list --db .kis/test/results.db
kis test runs show <run-id> --db .kis/test/results.db
kis test runs compare <base-id> <new-id> --db .kis/test/results.db
kis test runs compare --last 2 --db .kis/test/results.db

Subcommand	Flags	Description
list	--last N, --type functional|load	Stored runs, most recent first.
show	—	One run in detail.
compare	--last 2, or two run ids	Base vs new, with the direction judged.

--db defaults to .kis/test/results.db.

kis test tests: hierarchy runner (functional + load)

Runs directory-based collections in the steps:/testcases:/scenarios:/testplans: format (see the hierarchy format (kis test tests, see the CLI Reference)). Adding --load switches the same test definitions into load mode (see Load testing).

kis test tests -p examples/v1/01-rest-basics -d examples/v1/env.json -s
kis test tests -p examples/v1/01-rest-basics -t teststep -e get-users -s
kis test tests -p examples/v1/06-load-test --load examples/v1/06-load-test/profile.yaml --profile default

Flag	Short	Default	Description
--path	-p	<forge product path>/tests	Path to the collection directory.
--patterntype	-t	all	Filter by type: testplan, testscenario, testcase, teststep.
--pattern	-e	—	Name of the test entity to execute.
--datapath	-d	—	Data file path, repeatable. .json/.yaml merge into variables; .csv fans out one run per row.
--vars	-v	—	Inline variables as key=value map.
--load	—	—	Load-profile YAML path. Presence switches to load mode.
--profile	—	default	Profile name (key under loadprofiles:).
--logresponses	-s	false	Log response bodies.
--loglinenumbers	-n	false	Log line numbers.
--loglevel	-l	panic	Log level: debug, info, warn, error, panic.
--prefix	—	test	Log file prefix.

Output: <prefix>-<RFC3339Nano timestamp>.log plus stdout, and a hierarchy summary table after the run. Any suite failure exits 1.

kis test api: collection runner

Executes Postman-style request collections (requests: YAML files). Despite the name this is a CLI runner, not a server.

kis test api -c my-collection            # <forge product path>/api/my-collection
kis test api -p ./api/my-collection      # explicit path
kis test api -c my-collection -r login   # single request by name

Flag	Short	Default	Description
--collection	-c	—	Collection name, resolved under <product>/api/<collection> or via <product>/api/collections.yaml.
--path	-p	—	Explicit collection directory (wins over --collection).
--folder	—	—	Subfolder within the collection.
--request	-r	—	Execute a single request by name.
--datapath	-d	—	Data file paths (repeatable).
--vars	-v	—	Inline variables.
--loglevel	-l	panic	Log level.
--logresponses	-s	false	Log response bodies.
--loglinenumbers	-n	false	Log line numbers.

Request failures are logged and execution continues; the command exits 1 if any failed.

test.svc agent / test.svc orchestrator: distributed mode

Server modes for distributed test execution. Both take the shared infra flag set (--config/-f, default .test.yaml, plus --port, --sid, --config_url, service-discovery and TLS cert flags). See Server modes for config keys, endpoints, and the distributed execution flow.

Flag scope

The infrastructure flags (--sid, --config_url, the service-discovery and certificate flags) belong to the root command, agent and orchestrator. The test commands take the flags in their own tables above.