UiPath CLI - How-to: run tests from the CLI

uipath-cli

latest

false

UiPath CLI user guide

How-to: run tests from the CLI

Run a Test Manager test set from UiPath CLI as part of a CI pipeline and correctly interpret the execution verdict.

This page shows how to run a UiPath Test Manager test set from the CLI as part of a CI pipeline — and, just as importantly, how to read the verdict correctly. The uip tm tool splits a test set run into three verbs for a reason; using only one of them can produce a passing build even when tests have failed.

The golden rule: launch → wait → verify

For a test set run, no single uip tm verb exits non-zero because tests failed. Getting a pipeline to fail on a red test set run takes three commands (a package run takes one, see Run every test in an uploaded package):

Launch — uip tm testsets run. Queues the run and exits 0 as soon as Orchestrator accepts the request. Status: "Running" in the response reflects the state at launch, not the final outcome.
Block — uip tm wait. Polls until the execution reaches a terminal state (Finished or Cancelled). Exits 0 when it reaches any terminal state — because "finished" is the success signal for wait. Exits 2 on --timeout (a domain-specific reuse of the authentication-error slot, so scripts can branch on "took too long" without parsing text).
Verify — uip tm report get. Reads Data.Failed (and Passed, Skipped, PassRate). Exits 0 whenever it successfully fetches the summary, regardless of pass/fail — the script is responsible for turning Data.Failed > 0 into a non-zero exit.

A launch+wait shortcut does exist for test sets — uip tm testsets run --wait launches and blocks in one call (--timeout/--poll-interval tune the poll) — but it still isn't launch+wait+verify: a Finished run exits 0 even when tests inside it failed (only a Cancelled run exits 1). You still need the separate verify step to catch a red run. Always write the three steps (or the two-step run --wait plus the same report get verify at the end).

Run every test in an uploaded package

When the tests ship in a package already uploaded to Orchestrator, uip tm run --type package launches, waits, and verifies in one call. It runs every test case in the package, with no test set to author, writes a JUnit or UiPath report to --result-path, and exits 1 when a test fails or reports no result, no test runs, or the run is cancelled, so the pipeline fails on a red run without a separate verify step.

uip tm run --type package \
  --project-key DEMO \
  --package-name MyTests \
  --package-version 1.0 \
  --folder-path Shared \
  --result-path ./results/junit.xml
uip tm run --type package \
  --project-key DEMO \
  --package-name MyTests \
  --package-version 1.0 \
  --folder-path Shared \
  --result-path ./results/junit.xml

Exit code	Meaning
`0`	Every test passed.
`1`	A test failed or reported no result, the run was cancelled, no test ran, or a service call failed. A missing login or no access to the project also exits `1`, before the run starts.
`2`	Login or permission problem after the run starts.
`3`	Bad option or input.
`4`	The run exceeded `--timeout`.

Minimum viable snippet

#!/usr/bin/env bash
set -euo pipefail

# 1. Launch
EXECUTION_ID=$(uip tm testsets run \
  --test-set-key DEMO:10 \
  --output-filter "ExecutionId" \
  --output plain)

# 2. Block (default timeout: 30 min; adjust with --timeout)
if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --timeout 1800; then
  code=$?
  if [ "$code" -eq 2 ]; then
    echo "test run did not finish within 30 minutes" >&2
    exit 2
  fi
  echo "wait failed (exit $code)" >&2
  exit "$code"
fi

# 3. Verify
FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"
#!/usr/bin/env bash
set -euo pipefail

# 1. Launch
EXECUTION_ID=$(uip tm testsets run \
  --test-set-key DEMO:10 \
  --output-filter "ExecutionId" \
  --output plain)

# 2. Block (default timeout: 30 min; adjust with --timeout)
if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --timeout 1800; then
  code=$?
  if [ "$code" -eq 2 ]; then
    echo "test run did not finish within 30 minutes" >&2
    exit 2
  fi
  echo "wait failed (exit $code)" >&2
  exit "$code"
fi

# 3. Verify
FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"

--output plain returns the bare value for a scalar filter result; this is the safest way to capture a field into a shell variable without parsing JSON. See Scripting patterns — reading values out of the envelope.

Picking inputs to the commands

uip tm testsets run needs the test set key, not its UUID. The key has the form <ProjectKey>:<Number> — for example, DEMO:10. Two ways to find it:

# From the Test Manager UI — it is the "Key" column in the test set list.

# Or from the CLI:
uip tm testsets list --project-key DEMO --filter smoke \
  --output-filter "[*].{key:TestSetKey, name:Name}"
# From the Test Manager UI — it is the "Key" column in the test set list.

# Or from the CLI:
uip tm testsets list --project-key DEMO --filter smoke \
  --output-filter "[*].{key:TestSetKey, name:Name}"

list takes the project as --project-key or --project-id. run derives the project from the test set key's prefix (DEMO:10 → project DEMO).

If you need the numeric UUID for uip tm executions list --test-set-id, that is the Id field on the same list output — not the TestSetKey.

Picking which cases run

uip tm testsets run --execution-type controls which test cases inside the set run:

automated (default) — only automated test cases.
manual — only manual test cases.
mixed — both.
none — no type filter.

For CI, keep the default (automated) — manual cases require a human to mark pass/fail and will sit in InProgress until --timeout.

Passing parameter overrides

If the test set exposes parameters (for example, a target URL or an account ID), override them at launch with --input-path:

cat > ./params.json <<'EOF'
[
  {"name": "TargetUrl", "type": "String", "value": "https://staging.example.com"},
  {"name": "AccountId", "type": "String", "value": "acme-staging"}
]
EOF

uip tm testsets run --test-set-key DEMO:10 --input-path ./params.json
cat > ./params.json <<'EOF'
[
  {"name": "TargetUrl", "type": "String", "value": "https://staging.example.com"},
  {"name": "AccountId", "type": "String", "value": "acme-staging"}
]
EOF

uip tm testsets run --test-set-key DEMO:10 --input-path ./params.json

Overrides are matched against the test set's current parameter definitions by name (case-insensitive) and, when present, type. If the server reports no parameter definitions, the inputs are sent as-is. The overrides are saved on the test set, so later runs without --input-path reuse them.

Reading the verdict in detail

uip tm report get is the summary reader. Default output gives you the scorecard; pass --query (a jq-style filter) to extract a subset in one call:

# Pretty scorecard
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO

# Script-friendly — JSON with just the counts
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO \
  --query '{total: .TotalTests, passed: .Passed, failed: .Failed}'
# Pretty scorecard
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO

# Script-friendly — JSON with just the counts
uip tm report get --execution-id "$EXECUTION_ID" --project-key DEMO \
  --query '{total: .TotalTests, passed: .Passed, failed: .Failed}'

The Data envelope has these fields (full schema in the report get reference):

TotalTests, Passed, Failed, Skipped, PassRate ("80%"), Duration (HH:MM:SS).
FailedTests — one entry per failed test case, with TestCaseName and an Error string (either the log's info field, or a concatenated list of failed assertion messages).

When you need a JUnit-XML file that your CI's test dashboard can ingest, use uip tm result download instead of (or alongside) report get.

Surfacing failed cases in CI logs

Turn the FailedTests array into a human-readable line per failure before exiting the pipeline:

if [ "$FAILED" -gt 0 ]; then
  uip tm report get \
    --execution-id "$EXECUTION_ID" \
    --project-key DEMO \
    --output-filter "FailedTests[*].[TestCaseName, Error]" \
    --output plain >&2
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi
if [ "$FAILED" -gt 0 ]; then
  uip tm report get \
    --execution-id "$EXECUTION_ID" \
    --project-key DEMO \
    --output-filter "FailedTests[*].[TestCaseName, Error]" \
    --output plain >&2
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

For deeper failure analysis — assertion-by-assertion detail on a single failed case — use uip tm testcaselog list-assertions on the log IDs returned by uip tm executions testcaselogs list --only-failed.

Retrying flaky cases

If a run has failures that might be flaky (timing, environment), retry only the failed cases in place:

uip tm executions retry \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO
uip tm executions retry \
  --execution-id "$EXECUTION_ID" \
  --project-key DEMO

uip tm executions retry reuses the same execution ID — no new one is created. If there are no failures to retry, it prints an informational message and exits 0 (intentionally not an error).

After retry, loop back through wait + report get on the same execution ID.

Common pitfalls

Parsing testsets run output to decide pass/fail. The Status field at launch is typically Running — it means the run was queued, not that every test passed. Always call wait and then report get.
Forgetting the --timeout on wait. Default is 1800 seconds (30 minutes). Pass 0 to wait indefinitely (rarely what you want in CI); pass a larger number for long suites.
Treating exit 2 from wait as an auth failure. On wait specifically, 2 means timeout, not AuthenticationError — see uip tm wait exit codes.
Using testsets verbs to create tests. Test cases and their links to Orchestrator automations come from uip tm testcases; testsets only groups them.

Full CI-ready example

A bash snippet for a pipeline step — fails the build when any test fails, times out, or errors:

#!/usr/bin/env bash
set -euo pipefail

TEST_SET_KEY="${TEST_SET_KEY:?TEST_SET_KEY is required}"
PROJECT_KEY="${PROJECT_KEY:?PROJECT_KEY is required}"
TIMEOUT="${TEST_TIMEOUT:-1800}"

EXECUTION_ID=$(uip tm testsets run \
  --test-set-key "$TEST_SET_KEY" \
  --output-filter "ExecutionId" \
  --output plain)

echo "started execution $EXECUTION_ID" >&2

if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --timeout "$TIMEOUT"; then
  code=$?
  case "$code" in
    2) echo "execution $EXECUTION_ID did not finish within ${TIMEOUT}s" >&2; exit 2 ;;
    *) echo "wait failed with exit $code" >&2; exit "$code" ;;
  esac
fi

# Get the scorecard and the names of any failed cases in one shot
uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY"

FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"
#!/usr/bin/env bash
set -euo pipefail

TEST_SET_KEY="${TEST_SET_KEY:?TEST_SET_KEY is required}"
PROJECT_KEY="${PROJECT_KEY:?PROJECT_KEY is required}"
TIMEOUT="${TEST_TIMEOUT:-1800}"

EXECUTION_ID=$(uip tm testsets run \
  --test-set-key "$TEST_SET_KEY" \
  --output-filter "ExecutionId" \
  --output plain)

echo "started execution $EXECUTION_ID" >&2

if ! uip tm wait \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --timeout "$TIMEOUT"; then
  code=$?
  case "$code" in
    2) echo "execution $EXECUTION_ID did not finish within ${TIMEOUT}s" >&2; exit 2 ;;
    *) echo "wait failed with exit $code" >&2; exit "$code" ;;
  esac
fi

# Get the scorecard and the names of any failed cases in one shot
uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY"

FAILED=$(uip tm report get \
  --execution-id "$EXECUTION_ID" \
  --project-key "$PROJECT_KEY" \
  --output-filter "Failed" \
  --output plain)

if [ "$FAILED" -gt 0 ]; then
  echo "$FAILED test case(s) failed" >&2
  exit 1
fi

echo "all tests passed"

Run this script after your Solution deploy step in any of the CI recipes. The only env-specific config is TEST_SET_KEY / PROJECT_KEY — set them per environment in the pipeline, and the same script promotes cleanly across dev/stage/prod.

How-to: run tests from the CLI

The golden rule: launch → wait → verify

Run every test in an uploaded package

Minimum viable snippet

Picking inputs to the commands

Picking which cases run

Passing parameter overrides

Reading the verdict in detail

Surfacing failed cases in CI logs

Retrying flaky cases

Common pitfalls

Full CI-ready example

See also

Was this page helpful?

The golden rule: launch → wait → verify​

Run every test in an uploaded package​

Minimum viable snippet​

Picking inputs to the commands​

Picking which cases run​

Passing parameter overrides​

Reading the verdict in detail​

Surfacing failed cases in CI logs​

Retrying flaky cases​

Common pitfalls​

Full CI-ready example​

See also​

Was this page helpful?

The golden rule: launch → wait → verify

Run every test in an uploaded package

Minimum viable snippet

Picking inputs to the commands

Picking which cases run

Passing parameter overrides

Reading the verdict in detail

Surfacing failed cases in CI logs

Retrying flaky cases

Common pitfalls

Full CI-ready example

See also