Test Definitions

Test definitions are reusable commands that appear in the Testing menu and can be run as standalone test tasks. They let teams standardize common checks such as unit tests, linting, browser smoke tests, IBM i compile checks, or targeted test-file runs.

Use test definitions when the command should be available to every user of an environment. Use a one-off command when you only need to run something once.

Where Tests Live

Tests are stored in the environment's tests.json file in the coder setup repository:

environments/
  my-environment/
    environment.json
    tests.json

The file is a JSON object. Each key is the test name users see in the UI and CLI.

{
  "unit": {
    "description": "Run the unit test suite",
    "command": "npm test"
  },
  "lint-and-types": {
    "description": "Run linting and TypeScript checks",
    "commands": [
      "npm run lint",
      "npm run typecheck"
    ]
  }
}

If tests.json is missing, the environment simply has no named test commands.

Schema

Each test definition can include:

Field	Required	Purpose
description	No	Text shown in the Testing menu and coder test --list.
command	Yes, unless commands is present	Single shell command to run.
commands	Yes, unless command is present	Array of commands joined with &&.
parameters	No	Prompted values substituted into the command.
repos	No	CLI local-state filter. Omit for all repos, list repo names for selected repos, or use [] to skip local state.
ask_workspace	No	Lets Ask CoderFlow run this test in interactive containers: { "enabled": true, "timeout_seconds": 300 }. See Running a test from Ask.

Commands run in an ephemeral test container for the selected environment. Test tasks are visible in CoderFlow, stream output, and finish with the command's exit code.

Parameters

Parameters let users customize a test before it runs. Use ${parameter_name} placeholders in the command or commands:

{
  "single-file": {
    "description": "Run one test file",
    "command": "npm test -- ${test_file}",
    "parameters": {
      "test_file": {
        "type": "files",
        "description": "Test file",
        "required": true,
        "prompt": {
          "type": "files",
          "label": "Choose a test file",
          "path": "/workspace/app/tests",
          "pattern": "*.test.js"
        }
      }
    }
  }
}

Common parameter fields:

Field	Purpose
type	The parameter type. Use text, list, command, or files for prompted inputs. Boolean values can be authored, but the test run dialog passes them as normal substituted values.
label	Short display label.
description	Help text shown near the input.
required	Blocks the run until a value is provided.
default	Default value.
allowCustomValue	Set to false when users must pick from the prompt choices.
prompt	Prompt configuration for lists, dynamic commands, and file browsers.

The Web UI collects parameter values before submitting the test. The server validates required parameters and substitutes ${name} placeholders before running the command.

Prompt Types

Use a list prompt when choices are known ahead of time:

{
  "parameters": {
    "mode": {
      "type": "list",
      "required": true,
      "prompt": {
        "type": "list",
        "label": "Mode",
        "options": [
          { "value": "fast", "label": "Fast" },
          { "value": "coverage", "label": "Coverage" }
        ]
      }
    }
  }
}

Use a command prompt when choices should be discovered from the repository or environment:

{
  "parameters": {
    "package": {
      "type": "command",
      "required": true,
      "prompt": {
        "type": "command",
        "label": "Package",
        "command": "find packages -maxdepth 1 -mindepth 1 -type d -printf '%f\\n' | sort",
        "parser": "lines"
      }
    }
  }
}

Use a files prompt to browse files or directories:

{
  "parameters": {
    "source_member": {
      "type": "files",
      "required": true,
      "prompt": {
        "type": "files",
        "label": "Source member",
        "path": "/workspace/app/QRPGLESRC",
        "pattern": "*.{rpgle,sqlrpgle}",
        "include": "files"
      }
    }
  }
}

Dynamic prompt commands can use parsers:

Parser	Expected output
json	JSON array of strings or { "value", "label", "description" } objects.
lines	One choice per non-empty line.
git-branches	Output from git branch or git branch -r.
files	File paths, displayed by basename.

Prompt commands run in a temporary environment container. Keep them read-only and fast.

Multiple Selections

Set prompt.multiSelect to true when users can pick more than one value:

{
  "parameters": {
    "test_file": {
      "type": "files",
      "required": true,
      "prompt": {
        "type": "files",
        "label": "Test files",
        "path": "/workspace/app/tests",
        "pattern": "*.test.js",
        "multiSelect": true
      }
    }
  }
}

When a multi-select parameter has several selected values, CoderFlow expands the test into combinations and runs each substituted command in sequence inside one test task. If multiple multi-select parameters are used, CoderFlow runs the cartesian product of the selected values. The server rejects runs with more than 100 combinations.

Repository Local State

The CLI uses repos to decide which local repository changes to capture before running a test:

{
  "unit": {
    "description": "Run app unit tests",
    "command": "npm test",
    "repos": ["app"]
  },
  "env-info": {
    "description": "Print environment details without syncing local changes",
    "command": "node --version && npm --version",
    "repos": []
  }
}

Behavior:

• Omit repos to capture local state for all configured environment repositories.
• Set repos to selected repository names to capture only those repos.
• Set repos to an empty array to skip local state capture.

The Testing menu starts test tasks from the server-side environment state. The CLI local-state behavior applies to coder test.

Running From The Testing Menu

Users run configured tests from a task page:

1. Open the Testing menu.
2. Choose one or more test commands.
3. Fill in any required parameters.
4. Click Run.
5. Review the streamed output and exit code.

See Testing Menu for the runtime flow.

Running From The CLI

Use coder test --list to inspect named tests:

coder test --list --env=my-environment

Run a named test:

coder test unit --env=my-environment

Run a one-off command instead of a named definition:

coder test --cmd="npm test -- --runInBand" --env=my-environment

coder test supports --env=<environment>, --environment=<environment>, --cmd="<command>", --list, and --no-local-state.

The CLI lists and runs named test definitions, but it does not currently provide an option for passing named parameter values to tests. Run parameterized tests from the Web UI, or call the test-task API with test_parameters.

Running a test from Ask

An environment manager can let Ask CoderFlow run a saved test in the environment's existing interactive containers (the terminal and VS Code workspaces started from the dashboard or with coder start and coder shell; see Terminal Access). In the environment editor's Tests tab, select the test and turn on Allow Ask to run this test; optionally set Ask timeout (seconds). Turning the setting on also ticks Activate this version for Ask when I save. Saving writes:

{
  "unit": {
    "description": "Run the unit test suite",
    "command": "npm test",
    "ask_workspace": { "enabled": true, "timeout_seconds": 300 }
  }
}

• Absent or "enabled": false means Ask does not offer the test.
• timeout_seconds is a whole number from 1 to 600; without it the timeout is 300 seconds.
• Only tests without parameters and without ${...} placeholders can be enabled, and only when the command can be shown whole on Ask's confirmation card: at most 3,800 characters, with no hidden, control or direction-changing characters. A test whose definition contains a known secret value is not offered. Saving refuses an invalid setting with a message naming the test.
• Only environment managers (environments:manage) can change the setting. A tests.json edit through the Files tab that changes it needs the same permission. Ask itself never changes the setting.
• Activation is explicit. Ask offers a test only in the version an environment manager activated: they ticked Activate this version for Ask when I save (or turned the setting on) and saved in the environment editor. The activation is kept by the server, not in tests.json, and a test that stays unchanged stays active across later saves. Any change to the test, for example a setup repository pull, a Files tab edit (even by a manager), a change made through Ask, or an edit in the editor saved without ticking the box, leaves it inactive. CoderFlow notes the change as soon as it reads the file or reloads the configuration, and every one of those changes ends with a reload, so changing the test back to the activated version does not make it active again. The same holds when tests.json is unreadable for a while (invalid JSON, not an object, or deleted). The only change CoderFlow cannot see is one made directly on the server's disk, outside CoderFlow, and undone before CoderFlow read the file again. The Tests tab then shows Not active: changed outside the editor, or not activated yet until a manager reviews it and activates it again. Saving other settings never activates a test that was not active.
• An activation belongs to one environment instance. Deleting, renaming, importing or cloning an environment (and resetting the Essentials environment) clears its activations, and so does a configuration reload that no longer finds the environment. An environment deleted and created again under the same name, with the same tests.json, offers nothing until a manager activates its tests again.
• A change that ends an activation applies at once, even when the server cannot save it to its activation record. The server saves the record again the next time it reads it. If it restarts before that succeeds (for example after a disk error, or when it stopped while saving), it trusts none of the activations it saved before: every enabled test shows Not active until a manager activates it again. Only a disk that accepts no write at all, then recovers before such a restart, can lose the change; check the Tests tab after a storage failure.
• If the environment's name, the container or a command ID matches a saved secret value, Ask shows a Not shown card instead of the container's command list and runs nothing there. It does the same when it cannot load the saved secret values to check them.
• Ask runs the saved command exactly as written, as coder from /workspace after the environment's runtime setup, with the container's existing credentials and network access, and only after the user confirms each run. Users need containers:shell in the environment. Only Ask CoderFlow in the app runs these tests this way: external MCP clients and task agents cannot. Enable only commands you are comfortable letting anyone with shell access run this way.

See Ask CoderFlow for what users see, the limits and how runs are stopped.

Authoring Workflow

1. Open the environment's Tests tab or edit tests.json directly.
2. Add a short name, description, and command.
3. Use commands when the test has ordered steps that should stop on the first failure.
4. Add parameters for values users should choose at run time.
5. Add repos when CLI local-state capture should be limited.
6. Turn on Allow Ask to run this test if Ask CoderFlow may run it in interactive containers.
7. Save, commit, and push the setup repo change.
8. Confirm the test appears in the Testing menu and in coder test --list.