163 lines
7.0 KiB
Markdown
163 lines
7.0 KiB
Markdown
# JRich Wakeword Trainer
|
|
|
|
This download trains English multi-word LiveKit WakeWord classifiers on your NVIDIA GPU. Download
|
|
the current Linux or Windows/WSL2 archive from the JR-Gateway **Wakewords** page or from the
|
|
[public trainer releases](https://git.jimandkrista.com/jr-public/jrich-wakeword-trainer/releases).
|
|
The ML environment runs in a pinned Docker image; the host launcher uses only Python's standard
|
|
library.
|
|
No microphone audio, API credential, Docker socket, or network service enters the training
|
|
container.
|
|
|
|
## Host requirements
|
|
|
|
- Linux with Python 3.11 or newer;
|
|
- Docker Engine accessible to your user;
|
|
- an NVIDIA GPU, compatible driver, and NVIDIA Container Toolkit;
|
|
- 32 GB RAM and eight CPU cores recommended; and
|
|
- at least 40 GiB free for the first experiment. The reusable LiveKit assets currently occupy
|
|
about 17 GB.
|
|
|
|
Windows is supported through WSL2, not as a native Windows CUDA application. Use the `.zip`
|
|
download and run `start-windows.ps1`; it copies the small launcher into your Ubuntu WSL2 filesystem
|
|
and starts the same Linux wizard. Install the current NVIDIA Windows driver, enable WSL2, and make
|
|
Docker with NVIDIA GPU access available inside WSL2; do not install a second Linux NVIDIA display
|
|
driver in WSL.
|
|
|
|
An exclusive GPU with at least 12 GB VRAM is the friendly recommendation. The measured default uses
|
|
Piper batch 10 and can coexist more safely on a 24 GB RTX 3090, but other GPU processes still reduce
|
|
the available margin.
|
|
|
|
## Start the trainer
|
|
|
|
On Linux, extract the `.tar.gz` and run:
|
|
|
|
```bash
|
|
./start.sh
|
|
```
|
|
|
|
On Windows, extract the `.zip`, right-click `start-windows.ps1`, and choose **Run with PowerShell**.
|
|
The script verifies WSL2 is available, copies the launcher into your WSL home, and starts it there.
|
|
|
|
The launcher automatically pulls this exact public runtime when it is absent:
|
|
|
|
```text
|
|
git.jimandkrista.com/jr-public/jrich-wakeword-training@sha256:cef5b1e769a6160bbde69f67df76f0b865f02c27794a3da096398c18cc9a3c2f
|
|
```
|
|
|
|
It verifies the registry manifest digest, resolves it to an immutable local image ID, and only then
|
|
creates the request. No Gitea account or registry login is required. To build the same image locally
|
|
instead, run:
|
|
|
|
From the extracted archive:
|
|
|
|
```bash
|
|
scripts/build-wakeword-training-image jr-wakeword-training:0.2.1-jrich-2
|
|
```
|
|
|
|
The build is the only dependency-installing step. It verifies the reviewed dependency hashes and
|
|
bundles LiveKit WakeWord 0.2.1, PyTorch/TorchAudio 2.7.1 CUDA 12.6, ONNX export, and offline
|
|
pronunciation data.
|
|
|
|
## Interactive training
|
|
|
|
```bash
|
|
scripts/jrich-wakeword-trainer
|
|
```
|
|
|
|
The wizard asks for spoken phrases, near misses, and labels, shows the complete resolved request, checks Docker,
|
|
the selected GPU, and free disk, then downloads/cache-populates the LiveKit assets. Actual training
|
|
runs with `--network none`. Progress is streamed to the terminal and every attempt is retained under
|
|
`./jrich-wakeword-runs/runs/`.
|
|
|
|
On a shared GPU, explicitly acknowledge the coexistence trade-off:
|
|
|
|
```bash
|
|
scripts/jrich-wakeword-trainer --allow-active-gpu-processes
|
|
```
|
|
|
|
The runner never stops other GPU processes. Without that option it fails closed when another
|
|
compute process is active.
|
|
|
|
The setup cache is inventoried by relative path, byte size, and SHA-256 after setup. The final
|
|
content-addressed source lock—not its location—is part of the immutable training request, and the
|
|
offline container verifies the complete inventory again before it starts a LiveKit stage. Moving an
|
|
unchanged cache preserves its identity; changing, adding, or removing a file does not.
|
|
|
|
## Automated and overrideable use
|
|
|
|
Every prompt has a flag equivalent. This produces a resolved plan without Docker or training when
|
|
the immutable image ID and GPU UUID are supplied:
|
|
|
|
```bash
|
|
scripts/jrich-wakeword-trainer \
|
|
--non-interactive --plan \
|
|
--phrase "Hey Jay Rich" \
|
|
--phrase "Yo Jay Rich" \
|
|
--near-miss "Jay Rich" \
|
|
--model-name jrich --family-slug jrich \
|
|
--image-ref sha256:<local-image-id> \
|
|
--gpu-uuid GPU-<uuid>
|
|
```
|
|
|
|
Use `--pipeline-config overrides.json` for a validated deep override of built-in fields. Precedence
|
|
is measured built-in profile, then the JSON override. Unknown keys fail rather than being silently
|
|
ignored. Useful overrides include:
|
|
|
|
```json
|
|
{
|
|
"tts_batch_size": 4,
|
|
"model": {"model_size": "small"},
|
|
"steps": 30000,
|
|
"threshold": 0.62
|
|
}
|
|
```
|
|
|
|
The generic default threshold is `"auto"`, which records LiveKit's held-out calibrated operating
|
|
point. A manual threshold override should be justified by frozen per-phrase and negative evaluation,
|
|
not microphone observations used repeatedly for tuning.
|
|
|
|
Use `--skip-setup` after the versioned cache is populated. Use `--cache-dir` and `--work-root` to put
|
|
large data on another disk.
|
|
|
|
Use `--setup-only` with at least one `--phrase` to populate the cache without starting training.
|
|
It prints the canonical source-lock digest required by the API worker.
|
|
|
|
To revisit a completed local experiment, pass `--reuse-from <run-id>` with the same phrases,
|
|
sources, image, and model name, then increase sample counts and/or adjust training settings in the
|
|
override file. This creates a new immutable attempt, copies only the prior generated corpus, rebuilds
|
|
augmentation/features, and trains from scratch. It is deliberately called corpus reuse—not exact
|
|
checkpoint resume or PT warm start.
|
|
|
|
A failed, canceled, or timed-out attempt may also be a reuse source when its generation stage
|
|
finished and its per-file stage hashes still match. An identical terminal request can be retried as
|
|
a new immutable attempt with `--retry-of <run-id>`. Inspect and control a foreground/background
|
|
attempt from another terminal with:
|
|
|
|
```bash
|
|
scripts/jrich-wakeword-trainer --status <run-id>
|
|
scripts/jrich-wakeword-trainer --logs <run-id>
|
|
scripts/jrich-wakeword-trainer --cancel <run-id>
|
|
```
|
|
|
|
Cancellation writes an attempt-scoped marker; the owning runner observes it and stops only its own
|
|
deterministically named container. Failed, canceled, and timed-out attempts return a nonzero process
|
|
exit code, which is also the contract a future API worker can trust.
|
|
|
|
## Results and privacy
|
|
|
|
A successful Local Training Experiment ends in `completed`, not in a catalog publication state. Its
|
|
attempt directory contains:
|
|
|
|
- `exports/<release>-public.tar.gz`: ONNX, manifest, evaluation, license/notices, and checksums for clients;
|
|
- `output/*.pt`, feature arrays, generated/augmented audio, and metrics: private retraining material;
|
|
- `training-result.json`: hashes and a compact private-artifact inventory;
|
|
- resolved request/configuration, logs, resource evidence, stage evidence, and checksums.
|
|
|
|
The final PT contains model weights, not a complete optimizer/RNG checkpoint. It supports safe
|
|
re-export and a future explicit warm start, but not exact interrupted-training resume. Do not share
|
|
the private run directory unless you intend to share its generated/training data and provenance.
|
|
|
|
The future authenticated training API will submit the same versioned Wakeword Training Request to
|
|
the same runner. API ownership, quota, leases, cancellation authorization, object storage, and
|
|
publication remain server-side concerns rather than alternate ML behavior.
|