Testing CLI Reference

Testing has two surfaces, and they are not equivalent.

Surface	Provides
kis test	The v2 functional runner only — kis test -t tests/ and its flags
test.svc	Every mode: test, tests, api, load, agent, orchestrator

The kis CLI exposes the v2 runner as a top-level command, so kis test -t tests/ works and takes the flags below. It does not expose the other modes: kis test load is parsed as the v2 runner with an ignored positional argument, not as the load runner. Use test.svc for anything beyond the v2 runner.

kis test -t <path> [flags]

Command	Purpose	Doc
test (alias run)	v2 functional test runner — the recommended way to run API tests	Getting started, Test definitions
tests	v1.5 hierarchy runner (steps/testcases/scenarios/testplans directories); also drives load tests via --load	Legacy tests mode, Load testing
load	legacy load runner over testplan+config+profile YAML triple	Load testing
api	Postman-style collection runner (requests: files) — a CLI runner, not a server	Legacy tests mode
agent	distributed load worker; dials the orchestrator over gRPC	Server modes
orchestrator	distributed control plane; gRPC server + HTTPS API	Server modes
version	print name, version, build commit, build timestamp	—

Exit codes: 0 success, 1 on any test failure (v2 test), load/tests-mode failures, or cobra errors; -1 for bare invocation with no command. The legacy test plan runner and api log per-item failures but still exit 0 — don’t gate CI on those two.

kis test (alias: kis test run) — v2 functional runner

Runs a test YAML file or a directory tree of suites. No files are written by default: results stream to the console and the process exits 0/1. See Test definitions for the YAML schema.

kis test -t tests/                         # run a suite tree; env auto-discovered
kis test -t tests/data.yaml              # run one file
kis test -t tests/ -e env.yaml -n staging  # multi-env file; pick "staging"
kis test -t tests/ --tags smoke
kis test -t tests/ -v url=http://localhost:8080 -v password=secret
kis test -t tests/ --db .test/results.db --junit .test/junit.xml
kis test -t tests/ --dry-run               # validate tree + templates, no HTTP calls

Flag	Short	Default	Description
--tests	-t	—	Required. Path to a test YAML file or directory.
--env	-e	—	Environment YAML/JSON file. Auto-discovered next to --tests when omitted (see below).
--name	-n	—	Environment name inside a multi-env env file.
--vars	-v	—	Inline key=value variables. Highest precedence. Repeatable.
--tags	—	—	Include tags. Commas inside one entry = AND; repeat the flag = OR.
--exclude-tags	—	—	Same syntax; matched tests are removed. Exclusion wins over inclusion.
--db	—	—	DuckDB file path. Results persist only when provided.
--junit	—	—	JUnit XML output path (CI integration).
--dry-run	—	false	Validate suite tree and template resolution without making HTTP calls.
--no-color	—	false	Disable ANSI colors (auto-disabled when stdout isn’t a TTY).
--use-guppi	—	false	Route through the guppi TestExecutionEngine instead of the in-process sequential runner.

Variable precedence

Low → high: suite.yaml variables: → env file → --vars.

Env file auto-discovery

When --env is omitted, the runner looks in the tests directory (or the directory containing the tests file) for, in order — first hit wins:

1. env.local.yaml / env.local.yml / env.local.json
2. env.<name>.yaml / .yml / .json — only when --name is set
3. env.yaml / env.yml / env.json

The discovered path is announced on stderr. Commit env.yaml, keep env.local.yaml in .gitignore for personal overrides.

Env file shapes

Flat — a plain map of variables:

url: http://localhost:8080
password: secret

Multi-env — every top-level value is a map; keys are environment names:

default:
  url: http://localhost:8080
staging:
  url: https://api.staging.kis.ai

Selection rules: --name <key> picks that block (error lists known keys if missing); no --name picks default; a multi-env file with neither default nor --name is an error. Flat files ignore --name. Accepted extensions: .yaml, .yml, .json (extensionless parses as YAML).

test.svc tests — hierarchy runner (functional + load)

Runs directory-based collections in the steps:/testcases:/scenarios:/testplans: format (see Legacy tests mode). Adding --load switches the same test definitions into load mode (see Load testing).

test.svc tests -p examples/01-rest-basics -d examples/env.json -s
test.svc tests -p examples/01-rest-basics -t teststep -e get-users -s
test.svc tests -p examples/06-load-test --load examples/06-load-test/profile.yaml --profile default

Flag	Short	Default	Description
--path	-p	<product path>/tests	Path to the collection directory.
--patterntype	-t	all	Filter by type: testplan, testscenario, testcase, teststep.
--pattern	-e	—	Name of the test entity to execute.
--datapath	-d	—	Data file path, repeatable. .json/.yaml merge into variables; .csv fans out one run per row.
--vars	-v	—	Inline variables as key=value map.
--load	—	—	Load-profile YAML path. Presence switches to load mode.
--profile	—	default	Profile name (key under loadprofiles:).
--logresponses	-s	false	Log response bodies.
--loglinenumbers	-n	false	Log line numbers.
--loglevel	-l	panic	Log level: debug, info, warn, error, panic.
--prefix	—	test	Log file prefix.

Output: <prefix>-<RFC3339Nano timestamp>.log plus stdout, and a hierarchy summary table after the run. Any suite failure exits 1.

test.svc load — legacy load runner

Drives the legacy testplan format with a config file and a load-profile file. Prefer test.svc tests --load for new load tests; this command remains for existing v1 plans.

test.svc load -i test.yaml -c config.yaml -r load.yaml -l default -e dev

Flag	Short	Default	Description
--input	-i	test.yaml	Test plan YAML.
--config	-c	config.yaml	Config YAML; environments come from its top-level environments: key.
--profiles	-r	load.yaml	Load-profile YAML (loadprofiles: key). When empty, profiles are read from the input file itself.
--loadprofile	-l	default	Profile name under loadprofiles:.
--env	-e	default	Environment name.
--output	-o	test_output	Accepted but unused. The output prefix actually comes from the environment’s output key.

Renders a live terminal dashboard while running; writes test-load-<timestamp>.log and <prefix>_summary.txt. Failures exit 1.

test.svc api — collection runner

Executes Postman-style request collections (requests: YAML files). Despite the name this is a CLI runner, not a server.

test.svc api -c my-collection            # <product path>/api/my-collection
test.svc api -p ./api/my-collection      # explicit path
test.svc api -c my-collection -r login   # single request by name

Flag	Short	Default	Description
--collection	-c	—	Collection name, resolved under <product>/api/<collection> or via <product>/api/collections.yaml.
--path	-p	—	Explicit collection directory (wins over --collection).
--folder	—	—	Subfolder within the collection.
--request	-r	—	Execute a single request by name.
--datapath	-d	—	Data file paths (repeatable).
--vars	-v	—	Inline variables.
--loglevel	-l	panic	Log level.
--logresponses	-s	false	Log response bodies.
--loglinenumbers	-n	false	Log line numbers.

Request failures are logged and execution continues (exit 0).

test.svc agent / test.svc orchestrator — distributed mode

Server modes for distributed test execution. Both take the shared infra flag set (--config/-f, default .test.yaml, plus --port, --sid, --config_url, service-discovery and TLS cert flags). See Server modes for config keys, endpoints, and the distributed execution flow.

Known quirks

• The root command registers two subcommands named test; the v2 runner wins and the legacy plan runner is unreachable under that name. Treat the legacy test flag set (-i/-c/-e/-o) as historical.
• --output/-o on load is accepted but never read; the file prefix comes from the environment map’s output key.
• The root infra flags (--sid, --config_url, …) are registered non-persistently: they apply to the bare root, agent, and orchestrator, not to the test-running subcommands.
• agent logs “Using config file:” with the wrong variable — the file actually loaded is the one passed via --config/-f.