NVIDIA/SkillEvaluator

---

title: "Installation" description: "Install SkillEvaluator with uv, from source, or in Docker, and choose per-tier extras to keep the footprint small."

Start with the base package for deterministic quality checks, then add only the extras needed for your next command. No API key is needed to install or to run deterministic Tier 1 checks. LLM-backed features use a configured provider (Providers & Credentials); live Tier 3 evaluation adds an agent runtime and a sandbox.

For a complete first run against a bundled skill, use the keyless Quickstart, or add the llm extra for the NVIDIA Build route. Those routes use a source checkout and uv run; the standalone install below is convenient when you already have your own skill directory.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Requirements

• uv for the recommended install paths (plain pip works too — see the source tab below).
• Python 3.12, 3.13, or 3.14. uv tool install, uv run, and uv sync provision a supported interpreter automatically when passed --python 3.13 (or --python 3.14); only the plain-pip path needs one on PATH.
• Git for installs from the repository and for the bundled-example checkout.
• Docker, only for the container install and Tier 3 Docker-mode evaluation.
• No API key is needed for deterministic Tier 1 checks. Full default scanner coverage requires the security extra; Tier 1 provisions SkillSpector and Gitleaks itself, and Semgrep is optional (see System tools). LLM-backed checks and Tiers 2–3 use a configured provider — see Providers & Credentials.

Install

<Tabs> <Tab title="uv tool (recommended)" language="uv">

Install the base package as a standalone tool — uv provisions Python 3.13 for you:

uv tool install --python 3.13 "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator quality-check ./my-skill

Replace ./my-skill with a directory containing a SKILL.md. This quality check needs no API key, Docker, or security scanner. Full static scanner coverage has separate requirements under System tools.

Add the llm extra when you want to make your first rubric-evaluation call:

uv tool install --python 3.13 --reinstall "skillevaluator[llm] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

Then follow the NVIDIA Build setup, using skillevaluator rubric-eval ./my-skill for a standalone installation. Provider clients install with the extra; an API key is still required for the model call.

For other features, select their extras or use the optional evaluation bundle:

uv tool install --python 3.13 --reinstall "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
uv tool install --python 3.13 --reinstall "skillevaluator[tier2,tier3] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
uv tool install --python 3.13 --reinstall "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

These are alternatives: include all the extras you need in the chosen command. all includes tier2, tier3, and security; it excludes the optional dashboard extra and external tools.

To pick up the latest commit later, rerun the install with --reinstall, keeping your chosen extras. Choose the command matching your install:

uv tool install --python 3.13 --reinstall "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"
uv tool install --python 3.13 --reinstall "skillevaluator[llm] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

Then confirm the command resolves:

skillevaluator --version

</Tab> <Tab title="From source" language="source">

The Quickstart uses uv run --python 3.13 for the base quality check and adds --extra llm for the NVIDIA Build rubric call. This installs only the dependencies needed for each route.

For development or an editable environment with every Python extra, clone the repository and let uv create the environment from the committed lockfile:

git clone https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv sync --python 3.13 --all-extras

Or with plain pip. This path needs Python 3.12, 3.13, or 3.14 on PATH. Create and activate a virtual environment before installing, including on Homebrew-managed Python. If your system python3 is newer than 3.14, use python3.13 -m venv .venv instead, or prefer the uv sync path above, which provisions a supported Python automatically:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all]"

Editable extras work the same way as the one-liner:

python -m pip install -e .
python -m pip install -e ".[security]"
python -m pip install -e ".[all,dev]"

Then confirm the command resolves inside the environment:

uv run skillevaluator --version   # uv sync path
skillevaluator --version          # inside the activated pip venv

</Tab> <Tab title="Docker" language="docker">

The repository ships a Dockerfile that installs the evaluation bundle ([all], excluding the optional dashboard). Run the following from a repository checkout to build the image locally and run a deterministic validation against the included sample skill:

docker build -t skillevaluator:local .
docker run --rm skillevaluator:local --version
docker run --rm \
  -v "$PWD/tests/fixtures/skills/simple:/workspace/skills/simple:ro" \
  skillevaluator:local validate /workspace/skills/simple --no-llm --tiers 1

The image runs as the unprivileged skillevaluator user (UID 10001). Its home and /app/reports directory are writable, so default reports can be created without root. Input bind mounts can remain read-only.

To save reports to a host directory on Linux, create that directory first and run with its owner's UID and GID. Docker's --user override preserves access to host-owned output paths without changing their permissions:

mkdir -p reports
docker run --rm \
  --user "$(id -u):$(id -g)" \
  -e HOME=/tmp \
  -v "$PWD/tests/fixtures/skills/simple:/workspace/skills/simple:ro" \
  -v "$PWD/reports:/output" \
  skillevaluator:local validate /workspace/skills/simple --no-llm --tiers 1 -o /output

The UID override also uses /tmp as a writable home for per-user caches. Provider credentials and access to external sandbox services still need to be configured for the selected runtime user.

The image does not bundle the external Semgrep, SkillSpector, or Gitleaks executables, so their required scanner evidence is incomplete and validate exits non-zero — the other checks still run and the report is still produced (see Troubleshooting). Tier 3 still requires credentials for the selected provider and agent at runtime — pass them into the container as environment variables when you get there.

</Tab> </Tabs>

Choosing extras

The base package completes the schema, PII, license, quality, Unicode, script-lint, and hygiene checks on its own. quality-check needs no scanner setup. When you run the full static suite, scanner-backed checks report INCOMPLETE until the security extra and the required external scanner executables are installed. Running without an LLM or API key does not remove those scanner requirements. Everything else lives behind an extra:

Harbor is the open-source agent evaluation framework that the tier3 extra installs for live task execution and environment backends.

| Extra | What it unlocks | | --- | --- | | llm | Shared LLM and embedding clients used by LLM-backed checks, Tier 2, and Tier 3 | | tier2 | Intra-skill deduplication and local-catalog inter-skill similarity; includes llm | | tier3 | Docker, local, and cloud live-agent evaluation through Harbor (pins harbor==0.24.0 and requires python-dotenv>=1.2.0); includes llm | | security | Bandit and pip-audit | | all | tier2, tier3, and security in one bundle | | dashboard | Local Streamlit tables for comparing saved reports and retained runs (skillevaluator dashboard); not included in all | | dev | Build, test, coverage, and formatting tools for contributors |

<Tip> `[all]` installs the evaluator extras (`tier2`, `tier3`, `security`); add `dashboard` with `[all,dashboard]`. External scanner tools, Docker, agent CLIs, and the Harbor extras for managed sandbox backends (such as `harbor[modal]==0.24.0`) stay separate no matter which extra you pick — see [Agents & Sandboxes](agents-and-sandboxes.mdx#where-trials-run). </Tip>

System tools

<Note> Tier 1 provisions the external security scanners in their own environments, so their dependency stacks do not constrain the SkillEvaluator distribution:

• SkillSpector: when no executable is found, Tier 1 downloads the 2.13.3 release wheel once into the SkillEvaluator runtime folder, verifies its SHA-256 checksum, and runs it with uv tool run. The first run needs uv and network access; later runs use the uv cache.
• Semgrep: optional. When it is installed, Code Risk Analysis adds its rules to Bandit; when it is not, the check notes it was skipped. Set SKILLEVALUATOR_REQUIRE_SEMGREP=1 to report a missing Semgrep as INCOMPLETE (the Tier 1 CI image does).
• Gitleaks: on first use, SkillEvaluator downloads the pinned 8.30.0 release into its runtime folder and verifies its SHA-256 checksum. There is no prompt, so CI runs install it too. Linux and macOS only.

Installed executables take precedence. To use your own, put them on PATH or set SKILLEVALUATOR_SEMGREP_PATH and SKILLEVALUATOR_SKILLSPECTOR_PATH:

uv tool install semgrep
uv tool install git+https://github.com/NVIDIA/SkillSpector.git@v2.13.3
brew install gitleaks          # or a binary from https://github.com/gitleaks/gitleaks/releases

Docker is required only for Tier 3 Docker-mode evaluation. Agent CLIs and their credentials are user-supplied — see Agents & Sandboxes.

skillevaluator health-check --tier 1 reports the path and version of each installed scanner and exits non-zero when one is missing, misconfigured, or older than the supported release. Add --checks security,code-integrity,dependency to include pip-audit. </Note>

Verify your install

Two commands tell you whether the install landed:

skillevaluator --version
skillevaluator health-check

--version confirms the CLI is on PATH — that is the whole smoke test for a Tier 1-only install. health-check goes one step further: a quick readiness check of the CLI and the selected live-eval backend (the agent follows the configured provider; the environment defaults to docker), useful once you plan to run Tier 3. On an install without the tier3 extra or a configured provider it lists the missing pieces and exits non-zero — expected for a Tier 1-only setup, not a broken install. For a deeper diagnosis, doctor covers the same ground; add doctor --verify-models to make a live credential-bearing catalog request for each resolved agent model. A public or otherwise non-authoritative result is reported as a warning because catalog access alone does not prove inference access. See the CLI Reference for both commands.

Troubleshooting

<AccordionGroup> <Accordion title="The shell can't find skillevaluator after installing"> uv installs tools to `~/.local/bin`, which may not be on PATH yet. Run `uv tool update-shell` and open a new terminal. </Accordion> <Accordion title="Validation reports INCOMPLETE and exits non-zero"> A required scanner could not run. Check that uv is on `PATH` and GitHub and PyPI are reachable (SkillSpector), or that GitHub is reachable for the Gitleaks download, or install the scanner named in the report. Missing required evidence leaves validation `INCOMPLETE` and non-zero. To run without that scanner, pass `--skip` with the check named in the fix list (e.g. `--skip secrets`); the check is then reported as skipped and the run cannot certify Tier 1. </Accordion> </AccordionGroup>

Next steps

<CardGroup cols={2}> <Card title="Quickstart" icon="rocket" href="quickstart.mdx"> Check a bundled skill without an LLM, or use NVIDIA Build for a rubric review. </Card> <Card title="Providers & Credentials" icon="shield" href="configuration.mdx"> Wire up an LLM provider for Tier 2, Tier 3, and LLM-backed checks. </Card> </CardGroup>