JRich Wakeword Trainer

This download trains English multi-word LiveKit WakeWord classifiers on your NVIDIA GPU. Download the current Linux or Windows/WSL2 archive from the JR-Gateway Wakewords page or from the public trainer releases. The ML environment runs in a pinned Docker image; the host launcher uses only Python's standard library. No microphone audio, API credential, Docker socket, or network service enters the training container.

Host requirements

Linux with Python 3.11 or newer;
Docker Engine accessible to your user;
an NVIDIA GPU, compatible driver, and NVIDIA Container Toolkit;
32 GB RAM and eight CPU cores recommended; and
at least 40 GiB free for the first experiment. The reusable LiveKit assets currently occupy about 17 GB.

Windows is supported through WSL2, not as a native Windows CUDA application. Use the .zip download and run start-windows.ps1; it copies the small launcher into your Ubuntu WSL2 filesystem and starts the same Linux wizard. Install the current NVIDIA Windows driver, enable WSL2, and make Docker with NVIDIA GPU access available inside WSL2; do not install a second Linux NVIDIA display driver in WSL.

An exclusive GPU with at least 12 GB VRAM is the friendly recommendation. The measured default uses Piper batch 10 and can coexist more safely on a 24 GB RTX 3090, but other GPU processes still reduce the available margin.

Start the trainer

On Linux, extract the .tar.gz and run:

./start.sh

On Windows, extract the .zip, right-click start-windows.ps1, and choose Run with PowerShell. The script verifies WSL2 is available, copies the launcher into your WSL home, and starts it there.

The launcher automatically pulls this exact public runtime when it is absent:

git.jimandkrista.com/jr-public/jrich-wakeword-training@sha256:cef5b1e769a6160bbde69f67df76f0b865f02c27794a3da096398c18cc9a3c2f

It verifies the registry manifest digest, resolves it to an immutable local image ID, and only then creates the request. No Gitea account or registry login is required. To build the same image locally instead, run:

From the extracted archive:

scripts/build-wakeword-training-image jr-wakeword-training:0.2.1-jrich-2

The build is the only dependency-installing step. It verifies the reviewed dependency hashes and bundles LiveKit WakeWord 0.2.1, PyTorch/TorchAudio 2.7.1 CUDA 12.6, ONNX export, and offline pronunciation data.

Interactive training

scripts/jrich-wakeword-trainer

The wizard asks for spoken phrases, near misses, and labels, shows the complete resolved request, checks Docker, the selected GPU, and free disk, then downloads/cache-populates the LiveKit assets. Actual training runs with --network none. Progress is streamed to the terminal and every attempt is retained under ./jrich-wakeword-runs/runs/.

On a shared GPU, explicitly acknowledge the coexistence trade-off:

scripts/jrich-wakeword-trainer --allow-active-gpu-processes

The runner never stops other GPU processes. Without that option it fails closed when another compute process is active.

The setup cache is inventoried by relative path, byte size, and SHA-256 after setup. The final content-addressed source lock—not its location—is part of the immutable training request, and the offline container verifies the complete inventory again before it starts a LiveKit stage. Moving an unchanged cache preserves its identity; changing, adding, or removing a file does not.

Automated and overrideable use

Every prompt has a flag equivalent. This produces a resolved plan without Docker or training when the immutable image ID and GPU UUID are supplied:

scripts/jrich-wakeword-trainer \
  --non-interactive --plan \
  --phrase "Hey Jay Rich" \
  --phrase "Yo Jay Rich" \
  --near-miss "Jay Rich" \
  --model-name jrich --family-slug jrich \
  --image-ref sha256:<local-image-id> \
  --gpu-uuid GPU-<uuid>

Use --pipeline-config overrides.json for a validated deep override of built-in fields. Precedence is measured built-in profile, then the JSON override. Unknown keys fail rather than being silently ignored. Useful overrides include:

{
  "tts_batch_size": 4,
  "model": {"model_size": "small"},
  "steps": 30000,
  "threshold": 0.62
}

The generic default threshold is "auto", which records LiveKit's held-out calibrated operating point. A manual threshold override should be justified by frozen per-phrase and negative evaluation, not microphone observations used repeatedly for tuning.

Use --skip-setup after the versioned cache is populated. Use --cache-dir and --work-root to put large data on another disk.

Use --setup-only with at least one --phrase to populate the cache without starting training. It prints the canonical source-lock digest required by the API worker.

To revisit a completed local experiment, pass --reuse-from <run-id> with the same phrases, sources, image, and model name, then increase sample counts and/or adjust training settings in the override file. This creates a new immutable attempt, copies only the prior generated corpus, rebuilds augmentation/features, and trains from scratch. It is deliberately called corpus reuse—not exact checkpoint resume or PT warm start.

A failed, canceled, or timed-out attempt may also be a reuse source when its generation stage finished and its per-file stage hashes still match. An identical terminal request can be retried as a new immutable attempt with --retry-of <run-id>. Inspect and control a foreground/background attempt from another terminal with:

scripts/jrich-wakeword-trainer --status <run-id>
scripts/jrich-wakeword-trainer --logs <run-id>
scripts/jrich-wakeword-trainer --cancel <run-id>

Cancellation writes an attempt-scoped marker; the owning runner observes it and stops only its own deterministically named container. Failed, canceled, and timed-out attempts return a nonzero process exit code, which is also the contract a future API worker can trust.

Results and privacy

A successful Local Training Experiment ends in completed, not in a catalog publication state. Its attempt directory contains:

exports/<release>-public.tar.gz: ONNX, manifest, evaluation, license/notices, and checksums for clients;
output/*.pt, feature arrays, generated/augmented audio, and metrics: private retraining material;
training-result.json: hashes and a compact private-artifact inventory;
resolved request/configuration, logs, resource evidence, stage evidence, and checksums.

The final PT contains model weights, not a complete optimizer/RNG checkpoint. It supports safe re-export and a future explicit warm start, but not exact interrupted-training resume. Do not share the private run directory unless you intend to share its generated/training data and provenance.

The future authenticated training API will submit the same versioned Wakeword Training Request to the same runner. API ownership, quota, leases, cancellation authorization, object storage, and publication remain server-side concerns rather than alternate ML behavior.

7.0 KiB Raw Permalink Blame History