jrich-wakeword-trainer/README.md

# JRich Wakeword Trainer

This download trains English multi-word LiveKit WakeWord classifiers on your NVIDIA GPU. Download
the current Linux or Windows/WSL2 archive from the JR-Gateway **Wakewords** page or from the
[public trainer releases](https://git.jimandkrista.com/jr-public/jrich-wakeword-trainer/releases).
The ML environment runs in a pinned Docker image; the host launcher uses only Python's standard
library.
No microphone audio, API credential, Docker socket, or network service enters the training
container.

## Host requirements

- Linux with Python 3.11 or newer;
- Docker Engine accessible to your user;
- an NVIDIA GPU, compatible driver, and NVIDIA Container Toolkit;
- 32 GB RAM and eight CPU cores recommended; and
- at least 40 GiB free for the first experiment. The reusable LiveKit assets currently occupy
  about 17 GB.

Windows is supported through WSL2, not as a native Windows CUDA application. Use the `.zip`
download and run `start-windows.ps1`; it copies the small launcher into your Ubuntu WSL2 filesystem
and starts the same Linux wizard. Install the current NVIDIA Windows driver, enable WSL2, and make
Docker with NVIDIA GPU access available inside WSL2; do not install a second Linux NVIDIA display
driver in WSL.

An exclusive GPU with at least 12 GB VRAM is the friendly recommendation. The measured default uses
Piper batch 10 and can coexist more safely on a 24 GB RTX 3090, but other GPU processes still reduce
the available margin.

## Start the trainer

On Linux, extract the `.tar.gz` and run:

```bash
./start.sh
```

On Windows, extract the `.zip`, right-click `start-windows.ps1`, and choose **Run with PowerShell**.
The script verifies WSL2 is available, copies the launcher into your WSL home, and starts it there.

The launcher automatically pulls this exact public runtime when it is absent:

```text
git.jimandkrista.com/jr-public/jrich-wakeword-training@sha256:cef5b1e769a6160bbde69f67df76f0b865f02c27794a3da096398c18cc9a3c2f
```

It verifies the registry manifest digest, resolves it to an immutable local image ID, and only then
creates the request. No Gitea account or registry login is required. To build the same image locally
instead, run:

From the extracted archive:

```bash
scripts/build-wakeword-training-image jr-wakeword-training:0.2.1-jrich-3
```

The build is the only dependency-installing step. It verifies the reviewed dependency hashes and
bundles LiveKit WakeWord 0.2.1, PyTorch/TorchAudio 2.7.1 CUDA 12.6, ONNX export, and offline
pronunciation data.

## Interactive training

```bash
scripts/jrich-wakeword-trainer
```

The wizard asks for spoken phrases, near misses, and labels, shows the complete resolved request, checks Docker,
the selected GPU, and free disk, then downloads/cache-populates the LiveKit assets. Actual training
runs with `--network none`. Progress is streamed to the terminal and every attempt is retained under
`./jrich-wakeword-runs/runs/`.

On a shared GPU, explicitly acknowledge the coexistence trade-off:

```bash
scripts/jrich-wakeword-trainer --allow-active-gpu-processes
```

The runner never stops other GPU processes. Without that option it fails closed when another
compute process is active.

The setup cache is inventoried by relative path, byte size, and SHA-256 after setup. The final
content-addressed source lock—not its location—is part of the immutable training request, and the
offline container verifies the complete inventory again before it starts a LiveKit stage. Moving an
unchanged cache preserves its identity; changing, adding, or removing a file does not.

## Automated and overrideable use

Every prompt has a flag equivalent. This produces a resolved plan without Docker or training when
the immutable image ID and GPU UUID are supplied:

```bash
scripts/jrich-wakeword-trainer \
  --non-interactive --plan \
  --phrase "Hey Jay Rich" \
  --phrase "Yo Jay Rich" \
  --near-miss "Jay Rich" \
  --model-name jrich --family-slug jrich \
  --image-ref sha256:<local-image-id> \
  --gpu-uuid GPU-<uuid>
```

Use `--pipeline-config overrides.json` for a validated deep override of built-in fields. Precedence
is measured built-in profile, then the JSON override. Unknown keys fail rather than being silently
ignored. Useful overrides include:

```json
{
  "tts_batch_size": 4,
  "model": {"model_size": "small"},
  "steps": 30000,
  "threshold": 0.62
}
```

The generic default threshold is `"auto"`, which records LiveKit's held-out calibrated operating
point. A manual threshold override should be justified by frozen per-phrase and negative evaluation,
not microphone observations used repeatedly for tuning.

Use `--skip-setup` after the versioned cache is populated. Use `--cache-dir` and `--work-root` to put
large data on another disk.

Use `--setup-only` with at least one `--phrase` to populate the cache without starting training.
It prints the canonical source-lock digest required by the API worker.

To revisit a completed local experiment, pass `--reuse-from <run-id>` with the same phrases,
sources, image, and model name, then increase sample counts and/or adjust training settings in the
override file. This creates a new immutable attempt, copies only the prior generated corpus, rebuilds
augmentation/features, and trains from scratch. It is deliberately called corpus reuse—not exact
checkpoint resume or PT warm start.

A failed, canceled, or timed-out attempt may also be a reuse source when its generation stage
finished and its per-file stage hashes still match. An identical terminal request can be retried as
a new immutable attempt with `--retry-of <run-id>`. Inspect and control a foreground/background
attempt from another terminal with:

```bash
scripts/jrich-wakeword-trainer --status <run-id>
scripts/jrich-wakeword-trainer --logs <run-id>
scripts/jrich-wakeword-trainer --cancel <run-id>
```

Cancellation writes an attempt-scoped marker; the owning runner observes it and stops only its own
deterministically named container. Failed, canceled, and timed-out attempts return a nonzero process
exit code, which is also the contract a future API worker can trust.

## Results and privacy

A successful Local Training Experiment ends in `completed`, not in a catalog publication state. Its
attempt directory contains:

- `exports/<release>-public.tar.gz`: ONNX, manifest, evaluation, license/notices, and checksums for clients;
- `output/*.pt`, feature arrays, generated/augmented audio, and metrics: private retraining material;
- `training-result.json`: hashes and a compact private-artifact inventory;
- resolved request/configuration, logs, resource evidence, stage evidence, and checksums.

The JR-Gateway result buttons expose four different artifacts:

- **Client bundle (recommended for another project):** the deployment-friendly ONNX model,
  threshold/config manifest, evaluation report, checksums, license, and third-party notices.
- **ONNX:** only the model used by a client for wakeword detection.
- **PT:** the PyTorch model weights, mainly for developers and future conversion.
- **Retraining bundle:** the large private archive containing PT, generated and augmented training
  data, feature arrays, configuration, provenance, and integrity records. Keep it only if you may
  retrain, increase the data scale, or recover the training environment later. Clients do not need
  it.

The final PT contains model weights, not a complete optimizer/RNG checkpoint. It supports safe
re-export and a future explicit warm start, but not exact interrupted-training resume. Do not share
the private run directory unless you intend to share its generated/training data and provenance.

The future authenticated training API will submit the same versioned Wakeword Training Request to
the same runner. API ownership, quota, leases, cancellation authorization, object storage, and
publication remain server-side concerns rather than alternate ML behavior.