# A1 result: pitch detection is useful to investigate, but fails the chosen reliability gate

2026-09-25 · Ari · Technical evidence, not customer or learning validation.

The frozen shipped detector returned an estimate within 50 cents on **75.3% of annotated voiced frames**. It missed the predeclared 85% gate, and 17 of 40 clips missed the 70% per-clip gate. The unvoiced false-alarm rate of 6.0% passed its 10% gate. Do not describe the detector as validated for real-world singing or use a conditional success rate as the headline.

| Measurement | Observed numerator / denominator | Result | Predeclared gate |
|---|---:|---:|---|
| Returned estimate within 50 cents of labelled pitch | 8,761 / 11,640 voiced frames | 75.27% | At least 85%: failed |
| Every clip reaches 70% voiced correctness | 23 / 40 clips | 17 failed; range 47.3–97.8% | All clips: failed |
| Returned estimate on an unvoiced reference frame | 354 / 5,897 unvoiced frames | 6.00% | At most 10%: passed |
| Returned estimate on a voiced reference frame | 9,032 / 11,640 | 77.59% | Diagnostic |
| No returned estimate on a voiced reference frame | 2,608 / 11,640 | 22.41% | Counted as primary failures |
| Returned voiced estimate within 50 cents, conditional on detection | 8,761 / 9,032 | 97.00% | Not overall accuracy |
| Returned estimate within 25 cents | 8,182 / 11,640 | 70.29% | Sensitivity check only |
| Octave errors among returned voiced estimates | 55 / 9,032 | 0.61% | Never folded into a correct answer |

A 50-cent tolerance is half a semitone. These denominators refer to overlapping windows, not independent people or attempts. Equal-weight clip mean for the primary measure is 75.01%, close to the pooled result. Counts were independently recalculated from `frames.csv` in Python and match the Bun runner.

## What failed

Among 2,608 missed voiced frames, 1,522 failed the periodicity check, 579 were below the existing amplitude floor, 445 fell below the worker's clarity gate, and 62 triggered the clipping check. These are implementation rejection categories, not diagnoses of the singer. Of 9,032 returned voiced estimates, 271 were farther than 50 cents from the annotation, including 55 octave errors. Four reference frames were outside the supported 75–900 Hz range; that does not explain most failures.

Clip 34 had the lowest primary result (47.3%) and 19 octave errors. Clip 13 reached 49.4% with 62 clipped voiced windows. Quiet recordings explain some gaps but are not the only failure. Keep all clips in the scorecard; do not improve the score by discarding them.

## Method and scope

All 40 vocadito v3 clips, 29 released singer IDs and 817.13 seconds were evaluated at original amplitude and 44.1 kHz. Complete 4096-sample windows, 2048-sample hops and center timestamps reproduce the open-mode estimator input and >=0.9 clarity gate; all centers align to the dataset's 256-sample annotation grid. There are 17,537 evaluation frames. Source revision: `0627f339f9ab435a77ccad09280874fec1ea6886`; detector SHA256 `c37aa3236522afcb5cbbb7d4f1baa05e4c8a0bc7f68f93beac4145fb6c2fcfa0`.

Median detector execution on this container was 1.54 ms and the 95th percentile 2.58 ms. This excludes capture, browser scheduling, rendering and device audio latency. It is not a live-latency claim. Browser noise processing, backing-track interference, physical mobile devices and observed customer behavior were not tested by this benchmark.

Dataset credit: Rachel M. Bittner, Katherine Pasalo, Juan José Bosch, Gabriel Meseguer-Brocal and David Rubinstein, *vocadito* (2021), [pinned v3 dataset](https://zenodo.org/records/5578807), [technical report](https://arxiv.org/abs/2110.05580), [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Human annotators corrected machine-proposed F0 labels; these are not physiological ground truth. The corpus has no Korean/Japanese language clips and cannot establish target-market validity. Dataset singers are not our customers or research recruits. No recording or lyric is redistributed in the product.

## Decision

1. Keep the early-prototype claim and explicitly explain that a missing or wrong trace is possible. Never prompt people to push their voice to satisfy the graph. Open audio still cannot judge the intended song melody.
2. Update the existing localized practice guide with the failed gates and reproducible aggregate data. Test whether an honest explanation helps people interpret the tool; publishing it does not establish demand.
3. Do not tune against this entire dataset and then present the same recordings as an unseen validation set. It is now diagnostic. Any replacement estimator or threshold change needs a separate fixed evaluation set and real device checks.
4. Do not add a paywall or scale an improvement claim. The next commercial evidence remains voluntary usefulness, comparison with actual alternatives, repeat practice and later real payment.

Reproduction: obtain and checksum the original dataset as described in `protocol.md`; use the pinned company source tree and Bun; run `bun research/pitch-benchmark/run.ts /path/to/vocadito /path/to/output`. The runner writes all per-frame predictions, per-clip metrics and summary. `audit.json` records the independent recount and source/archive hashes. No result was selected from multiple tuned runs.