jr-public/jrich-wakeword-trainer
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Resource evidence is inaccurate and declared requirements are ~7x over-stated; local runs emit placeholder "evidence" into the shipped client manifest #2

New Issue
Closed
opened 2026-08-14 20:33:11 -05:00 by westtexasfish · 3 comments
westtexasfish commented 2026-08-14 20:33:11 -05:00
Owner
Copy Link
Copy Source

Follow-up to #1. That issue covers the blockers that stop the trainer running at all. These four are correctness/accuracy problems in what the runner records, found by measuring a completed full-scale run rather than by hitting a failure. None of them block a run.

Evidence base: one full local experiment at the shipped default profile (50,000 steps, 10,000 samples, medium conv_attention, Piper batch 10, three phrase variants), status: completed / stage: sealed / exit 0, 1h41m wall clock, 5,598 resource samples at 1 s. RTX 3090, driver 591.86.

1. peak_vram_mib / mean_vram_mib are whole-GPU, not run-scoped

ResourceSampler queries nvidia-smi --query-gpu=...,memory.used,..., which reports total device memory in use by every process on the card, then records it into the per-run evidence file as if it were the run's consumption:

"peak_vram_mib": max(numeric["vram_used_mib"], default=0),
"mean_vram_mib": ...

On this host the very first sample — taken before the pipeline allocated anything — already reads 4,187 MiB, which is the Windows desktop compositor:

samples      : 5598
first (base) : 4187.0 MiB
min          : 3876.0 MiB
max          : 5623.0 MiB
peak - min   : 1747.0 MiB   <-- actually attributable to the run

So the recorded mean_vram_mib: 4264 is almost entirely other people's memory, and peak_vram_mib: 5623 overstates the run's peak by roughly 3.2x.

This matters because resource-summary.json is a provenance artifact retained per attempt. Two identical runs on differently-loaded machines record materially different "evidence", and the numbers cannot be compared across hosts or used to size hardware.

Suggested: sample --query-compute-apps=pid,used_gpu_memory filtered to the container's processes; or capture a pre-stage baseline and record the delta; or rename the fields to make the whole-device scope explicit (device_peak_vram_mib) and add the baseline alongside.

Note this is also why --allow-active-gpu-processes is effectively mandatory on any machine with a display attached: under WSL2, --query-compute-apps returns empty (compute apps live on the Windows side), so the runner cannot see what is actually resident.

2. disk_growth_bytes excludes the export bundles — understates by 45%

The sampler stops before finalize writes exports/, so the recorded growth misses both tarballs:

recorded disk_growth_bytes :   6,392,329,725   (6.39 GB)
actual run dir on disk     :  11,606,857,613  (11.61 GB)
exports/ total             :   5,128,285,637   (5.13 GB)
recorded + exports         :  11,520,615,362  (11.52 GB)
unrecorded                 :   5,214,527,888   (5.21 GB)

recorded + exports reconciles to actual within rounding, confirming the cause. A user planning capacity from disk_growth_bytes will under-provision by the size of the private retraining bundle, which is the single largest artifact the run produces.

Suggested: re-stat the run directory after finalize and record the post-seal total, or add a separate exports_bytes field.

3. recommended_gpu_vram_bytes is ~7x the measured requirement

RELEASE.json declares:

"recommended_gpu_vram_bytes": 12884901888   // 12 GiB

and the README states "An exclusive GPU with at least 12 GB VRAM is the friendly recommendation", plus a caution that the default "can coexist more safely on a 24 GB RTX 3090, but other GPU processes still reduce the available margin".

Measured at that exact default profile, the run added ~1.7 GB. Even the whole-device peak, with a desktop compositor and browser resident, was 5.6 GB of 24 GB. Peak utilisation 84%, peak power 237 W, peak temp 69 C.

The 12 GiB recommendation appears to be inherited assumption rather than measurement, and it will turn away users with 4-6 GB cards who could run the default profile comfortably. The "coexist more safely on a 24 GB 3090" framing likewise reads as far more cautious than the data supports.

Two smaller requirement notes from the same run:

  • recommended_cpu_cores: 8 — the pipeline averaged 1471% CPU (~14.7 cores) on a 32-core host. It will still run on 8, just slower; worth saying so rather than implying 8 is the comfortable figure.
  • minimum_free_bytes: 40 GiB held up fine: 17 GB cache + 11.6 GB run = ~28.6 GB for a first experiment. Reasonable margin, no change needed.

VRAM is advisory only — nothing gates on it — so this is a documentation/accuracy fix, not a functional one.

4. finalize-run.py writes hardcoded placeholders into the shipped client manifest

For any non-commercial run kind, _evaluation() fabricates the evidence block:

result = {
    "schema_version": 1,
    "phrase_duration_evidence": {phrase: [1.0] for phrase in [primary_phrase, *variants]},
    "release_negative_hours": 0.0,
    "release_false_activations": 0,
    "speaker_macro_recall": 0.0,
    "p95_detection_latency_ms": 0.0,
    "quarantined": run_kind == "quarantined_smoke",
}

These land verbatim in manifest.json inside <release>-public.tar.gz, the bundle the README describes as "recommended for another project":

"phrase_duration_evidence": {
    "Hey Elminster": [1.0],
    "Yo Elminster":  [1.0],
    "Oy Elminster":  [1.0]
},
"speaker_macro_recall": 0.0,
"p95_detection_latency_ms": 0.0,
"release_negative_hours": 0.0

Nothing in the emitted values distinguishes not measured from measured as zero. phrase_duration_evidence is the sharpest case: 1.0 is not a null or a sentinel, it is a plausible-looking duration in a field named "evidence", and in the commercial path the same field is validated against a real 0 < x <= 1.8 constraint. A downstream consumer diffing a local manifest against a commercial one sees the same schema with the same value types.

There is a "promotion": {"eligible": false, "status": "local_only"} marker, so a careful reader can tell — but that puts the burden on the consumer to know that one flag invalidates four other fields.

Suggested: emit null, or omit the keys entirely, for non-commercial run kinds. Absent is unambiguous; 1.0 is not.

Not a bug — recorded for completeness

While investigating I checked two things that turned out to be correct behaviour, in case they come up again:

  • threshold: "auto" works as documented. elminster_eval.json reports threshold: 0.5 / recall 0.91 / fpph 0.54, but the shipped manifest.json records threshold: 0.68 — the calibrated optimum at recall 0.878 / fpph 0.196, which meets the preset's target_fp_per_hour: 0.2. Both raw and calibrated figures are retained under upstream_metrics for audit. Correct on all counts.
  • The nltk cmudict download error under --network none is cosmetic. cmudict is baked into the image (123,455 entries, readable to an unprivileged uid via NLTK_DATA=/usr/local/share/nltk_data). Something calls nltk.download() defensively, it fails offline, and the bundled copy is used. Silencing it would remove a scary-looking line from every run log, but nothing is broken.

Reference run

run_id            : 6e082ef30f0541c783d5517901a8d9d6
status/stage      : completed / sealed          exit 0
wall clock        : 1:41:35
recall @ 0.68     : 87.8%      fpph 0.196      AUT 0.00153
validation        : 6,000 positive / 36,684 negative over 20.38 h
max_rss           : 18.24 GB
peak GPU util     : 84%        peak power 237 W      peak temp 69 C

Run with the #1 launcher patch applied and an image whose /opt/jr-wakeword is world-readable, so the numbers above reflect the fixed configuration.

Follow-up to #1. That issue covers the blockers that stop the trainer running at all. These four are correctness/accuracy problems in what the runner *records*, found by measuring a completed full-scale run rather than by hitting a failure. None of them block a run. **Evidence base:** one full local experiment at the shipped default profile (50,000 steps, 10,000 samples, medium `conv_attention`, Piper batch 10, three phrase variants), `status: completed` / `stage: sealed` / exit 0, 1h41m wall clock, 5,598 resource samples at 1 s. RTX 3090, driver 591.86. --- ## 1. `peak_vram_mib` / `mean_vram_mib` are whole-GPU, not run-scoped `ResourceSampler` queries `nvidia-smi --query-gpu=...,memory.used,...`, which reports **total device memory in use by every process on the card**, then records it into the per-run evidence file as if it were the run's consumption: ```python "peak_vram_mib": max(numeric["vram_used_mib"], default=0), "mean_vram_mib": ... ``` On this host the very first sample — taken before the pipeline allocated anything — already reads 4,187 MiB, which is the Windows desktop compositor: ``` samples : 5598 first (base) : 4187.0 MiB min : 3876.0 MiB max : 5623.0 MiB peak - min : 1747.0 MiB <-- actually attributable to the run ``` So the recorded `mean_vram_mib: 4264` is almost entirely *other people's memory*, and `peak_vram_mib: 5623` overstates the run's peak by roughly 3.2x. This matters because `resource-summary.json` is a provenance artifact retained per attempt. Two identical runs on differently-loaded machines record materially different "evidence", and the numbers cannot be compared across hosts or used to size hardware. **Suggested:** sample `--query-compute-apps=pid,used_gpu_memory` filtered to the container's processes; or capture a pre-stage baseline and record the delta; or rename the fields to make the whole-device scope explicit (`device_peak_vram_mib`) and add the baseline alongside. Note this is also why `--allow-active-gpu-processes` is effectively mandatory on any machine with a display attached: under WSL2, `--query-compute-apps` returns empty (compute apps live on the Windows side), so the runner cannot see what is actually resident. ## 2. `disk_growth_bytes` excludes the export bundles — understates by 45% The sampler stops before finalize writes `exports/`, so the recorded growth misses both tarballs: ``` recorded disk_growth_bytes : 6,392,329,725 (6.39 GB) actual run dir on disk : 11,606,857,613 (11.61 GB) exports/ total : 5,128,285,637 (5.13 GB) recorded + exports : 11,520,615,362 (11.52 GB) unrecorded : 5,214,527,888 (5.21 GB) ``` `recorded + exports` reconciles to actual within rounding, confirming the cause. A user planning capacity from `disk_growth_bytes` will under-provision by the size of the private retraining bundle, which is the single largest artifact the run produces. **Suggested:** re-stat the run directory after finalize and record the post-seal total, or add a separate `exports_bytes` field. ## 3. `recommended_gpu_vram_bytes` is ~7x the measured requirement `RELEASE.json` declares: ```json "recommended_gpu_vram_bytes": 12884901888 // 12 GiB ``` and the README states *"An exclusive GPU with at least 12 GB VRAM is the friendly recommendation"*, plus a caution that the default *"can coexist more safely on a 24 GB RTX 3090, but other GPU processes still reduce the available margin"*. Measured at that exact default profile, the run added **~1.7 GB**. Even the whole-device peak, with a desktop compositor and browser resident, was 5.6 GB of 24 GB. Peak utilisation 84%, peak power 237 W, peak temp 69 C. The 12 GiB recommendation appears to be inherited assumption rather than measurement, and it will turn away users with 4-6 GB cards who could run the default profile comfortably. The "coexist more safely on a 24 GB 3090" framing likewise reads as far more cautious than the data supports. Two smaller requirement notes from the same run: - `recommended_cpu_cores: 8` — the pipeline averaged **1471% CPU (~14.7 cores)** on a 32-core host. It will still run on 8, just slower; worth saying so rather than implying 8 is the comfortable figure. - `minimum_free_bytes: 40 GiB` held up fine: 17 GB cache + 11.6 GB run = ~28.6 GB for a first experiment. Reasonable margin, no change needed. VRAM is advisory only — nothing gates on it — so this is a documentation/accuracy fix, not a functional one. ## 4. `finalize-run.py` writes hardcoded placeholders into the shipped client manifest For any non-commercial run kind, `_evaluation()` fabricates the evidence block: ```python result = { "schema_version": 1, "phrase_duration_evidence": {phrase: [1.0] for phrase in [primary_phrase, *variants]}, "release_negative_hours": 0.0, "release_false_activations": 0, "speaker_macro_recall": 0.0, "p95_detection_latency_ms": 0.0, "quarantined": run_kind == "quarantined_smoke", } ``` These land verbatim in `manifest.json` inside `<release>-public.tar.gz`, the bundle the README describes as *"recommended for another project"*: ```json "phrase_duration_evidence": { "Hey Elminster": [1.0], "Yo Elminster": [1.0], "Oy Elminster": [1.0] }, "speaker_macro_recall": 0.0, "p95_detection_latency_ms": 0.0, "release_negative_hours": 0.0 ``` Nothing in the emitted values distinguishes *not measured* from *measured as zero*. `phrase_duration_evidence` is the sharpest case: `1.0` is not a null or a sentinel, it is a plausible-looking duration in a field named "evidence", and in the commercial path the same field is validated against a real `0 < x <= 1.8` constraint. A downstream consumer diffing a local manifest against a commercial one sees the same schema with the same value types. There is a `"promotion": {"eligible": false, "status": "local_only"}` marker, so a careful reader can tell — but that puts the burden on the consumer to know that one flag invalidates four other fields. **Suggested:** emit `null`, or omit the keys entirely, for non-commercial run kinds. Absent is unambiguous; `1.0` is not. --- ## Not a bug — recorded for completeness While investigating I checked two things that turned out to be correct behaviour, in case they come up again: - **`threshold: "auto"` works as documented.** `elminster_eval.json` reports `threshold: 0.5 / recall 0.91 / fpph 0.54`, but the shipped `manifest.json` records `threshold: 0.68` — the calibrated optimum at `recall 0.878 / fpph 0.196`, which meets the preset's `target_fp_per_hour: 0.2`. Both raw and calibrated figures are retained under `upstream_metrics` for audit. Correct on all counts. - **The `nltk cmudict` download error under `--network none` is cosmetic.** cmudict is baked into the image (123,455 entries, readable to an unprivileged uid via `NLTK_DATA=/usr/local/share/nltk_data`). Something calls `nltk.download()` defensively, it fails offline, and the bundled copy is used. Silencing it would remove a scary-looking line from every run log, but nothing is broken. ## Reference run ``` run_id : 6e082ef30f0541c783d5517901a8d9d6 status/stage : completed / sealed exit 0 wall clock : 1:41:35 recall @ 0.68 : 87.8% fpph 0.196 AUT 0.00153 validation : 6,000 positive / 36,684 negative over 20.38 h max_rss : 18.24 GB peak GPU util : 84% peak power 237 W peak temp 69 C ``` Run with the #1 launcher patch applied and an image whose `/opt/jr-wakeword` is world-readable, so the numbers above reflect the fixed configuration.
westtexasfish commented 2026-08-15 02:33:10 -05:00
Author
Owner
Copy Link
Copy Source

Fixed in gateway PR #188.

The matching public trainer release is available now:

  • v0.2.1-jrich-5
  • image: git.jimandkrista.com/jr-public/jrich-wakeword-training@sha256:e5854a420060c897d192cd6bb3d85fba079a521587ebc1f627691058fa34fd8a

Implemented:

  • explicit whole-device GPU scope, pre-execution baseline, device deltas, and null run-scoped usage where attribution is unavailable;
  • sealed disk remeasurement after both exports, including logical/allocated/export fields and explicit exclusions;
  • not_measured plus nulls for local/quarantined evaluation fields;
  • evidence-backed 8 GB GPU, 8-core floor/~16-core recommendation, 32 GB RAM, and 40 GiB free guidance.

The exact public tar.gz/ZIP assets have passed checksum, metadata, mode, import, and anonymous-download verification. I am leaving this issue open until PR #188 merges and deploys.

Fixed in gateway PR [#188](https://git.jimandkrista.com/ai/jrich-gateway/pulls/188). The matching public trainer release is available now: - [v0.2.1-jrich-5](https://git.jimandkrista.com/jr-public/jrich-wakeword-trainer/releases/tag/v0.2.1-jrich-5) - image: `git.jimandkrista.com/jr-public/jrich-wakeword-training@sha256:e5854a420060c897d192cd6bb3d85fba079a521587ebc1f627691058fa34fd8a` Implemented: - explicit whole-device GPU scope, pre-execution baseline, device deltas, and null run-scoped usage where attribution is unavailable; - sealed disk remeasurement after both exports, including logical/allocated/export fields and explicit exclusions; - `not_measured` plus nulls for local/quarantined evaluation fields; - evidence-backed 8 GB GPU, 8-core floor/~16-core recommendation, 32 GB RAM, and 40 GiB free guidance. The exact public tar.gz/ZIP assets have passed checksum, metadata, mode, import, and anonymous-download verification. I am leaving this issue open until PR #188 merges and deploys.
westtexasfish commented 2026-08-15 03:10:57 -05:00
Author
Owner
Copy Link
Copy Source

PR #188 is now fully green after SQLite, Postgres, gateway-image, web-image, and Playwright e2e gates. The initial CI run caught and fixed the quarantine-ingest compatibility edge; unmeasured quarantine evidence is accepted only when explicitly statused/reasoned/all-null, while unmeasured commercial candidates remain rejected. Awaiting the required human merge/deployment before closing this issue.

PR #188 is now fully green after SQLite, Postgres, gateway-image, web-image, and Playwright e2e gates. The initial CI run caught and fixed the quarantine-ingest compatibility edge; unmeasured quarantine evidence is accepted only when explicitly statused/reasoned/all-null, while unmeasured commercial candidates remain rejected. Awaiting the required human merge/deployment before closing this issue.
westtexasfish commented 2026-08-15 09:38:32 -05:00
Author
Owner
Copy Link
Copy Source

Resolved and deployed via ai/jrich-gateway PR #188 (merge commit 8a02aa4d73384d4d69d0bf8f339c9e75250dd9da). The branch was refreshed from master before merge; SQLite, Postgres, web-build, gateway/web image build, and e2e all passed on the refreshed PR and again on master. Live deployment completed successfully at Alembic head 0019 and was tagged live-20260815-2. Direct verification against the live gateway returned HTTP 200 and confirmed runner 0.2.1-jrich-5 with the pinned public image digest sha256:e5854a420060c897d192cd6bb3d85fba079a521587ebc1f627691058fa34fd8a.

Resolved and deployed via `ai/jrich-gateway` PR #188 (merge commit `8a02aa4d73384d4d69d0bf8f339c9e75250dd9da`). The branch was refreshed from master before merge; SQLite, Postgres, web-build, gateway/web image build, and e2e all passed on the refreshed PR and again on master. Live deployment completed successfully at Alembic head `0019` and was tagged `live-20260815-2`. Direct verification against the live gateway returned HTTP 200 and confirmed runner `0.2.1-jrich-5` with the pinned public image digest `sha256:e5854a420060c897d192cd6bb3d85fba079a521587ebc1f627691058fa34fd8a`.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
claude-agent (Claude-Agent) westtexasfish (westtexasfish)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: jr-public/jrich-wakeword-trainer#2