Caption accuracy on gameplay audio: how to measure it

How word error rate is computed

Word error rate (WER) is the number of word-level edits needed to turn a tool’s captions into a correct transcript, divided by the number of words in that transcript. Microsoft’s speech documentation calls it the industry standard for measuring model accuracy and gives the same formula:

WER = (substitutions + deletions + insertions) / words in the reference

Lower is better, and it can exceed 100% if a tool adds a great many words. The edits are found by alignment, not by comparing word position. NIST’s sclite scoring tool uses a dynamic-programming alignment that minimises a Levenshtein distance, weighting correct words, insertions, deletions and substitutions at 0, 3, 3 and 4. That weighting matters because different alignments can produce different error counts, so a comparison should use one scorer throughout. The BenchmarkSTT tutorial notes that its default strict variant gives slightly higher WER than Levenshtein distance. Two figures from two scorers should not be compared.

Normalise both sides before counting

Normalisation removes formatting differences so that only recognition errors are counted. Speechmatics’ benchmarking guide says this must happen before WER is calculated, and shows one pair of transcripts going from 41% to 8% once formatting is removed. sclite maps input to a single case by default, unless told otherwise. Write your rules down first, and apply the same rules to the reference and to every tool. These proposed rules adapt the general idea to game talk:

Normalisation is not perfect, as Speechmatics notes, so spot-check where the remaining errors fall. Punctuation and capitalisation still affect how readable captions are, even though WER cannot see them.

Why gameplay audio is a harder case

Gameplay audio puts speech in competition with effects, music and team voice chat. Clean-speech accuracy figures do not automatically carry over, for reasons we can state without having measured them:

• gunfire, explosions and music overlap with speech
• several people may speak at once on comms
• game terms, character names and streamer slang may not be in a general vocabulary
• excited speech is often shouted or clipped

The research on voice chat points the same way. A 2025 paper on game-oriented speech recognition describes general systems struggling with short phrases, rapid speech, jargon and noise. That paper concerns team voice communication and not stream captions, so treat it as context only. As reasoning, not a finding: overall WER may hide differences between tools, so count game-term errors as well. If two tools have a similar overall WER but different game-term error rates, the second figure is the more useful one for gaming content.

Writing a reference transcript for game audio

The reference has to be verbatim, and game audio forces some decisions. BenchmarkSTT’s tutorial warns that subtitles are edited and include descriptions of sounds, so they are not suitable for an absolute WER. Do not use another tool’s captions as your reference.

Decide these before typing:

• Whose words count. A sensible default is that the streamer and teammates are counted, and in-game dialogue is marked and excluded.
• Unclear words. Bracket them and exclude them from scoring. A guess adds errors to every tool equally.
• Overlaps. Transcribe the main voice, and note the overlap so a separate overlap figure can be reported.
• Text-to-speech. Donation voices are excluded, and noted.

Use footage you own or have permission to use.

Run a two-minute version

A short sample can show whether a tool is clearly unusable on your audio.

1. Pick three one-minute stretches from one VOD: one with quiet commentary, one with heavy game audio and one with several voices.
2. Type out what was actually said, then check it by listening again.
3. List the game terms in those minutes: characters, weapons, maps, abilities and slang.
4. Export each tool’s captions before any editing, ideally as a subtitle file. If captions are burned in only, copy them from the screen.
5. Apply your normalisation rules to everything, then count errors. The free sm-metrics command in Speechmatics’ guide and NIST’s sclite both do the alignment for you.
6. Note separately any line that changes the meaning of what was said.

Worked calculation

The arithmetic is short once the transcripts are aligned. Illustration only, using an invented line and invented totals:

Reference: push b site they are one shot behind the smoke (10 words)

Caption, after normalising: push b sight they are one shot behind smoke now

WER = (1 + 1 + 1) / 10 = 30%.

Over a whole sample, with invented totals for an unnamed tool: 1,420 reference words, 61 substitutions, 38 deletions and 14 insertions give (61 + 38 + 14) / 1,420 = 8.0%. For game terms alone, 16 errors across 96 reference terms give 16 / 96 = 16.7%. We count substitutions and deletions for that figure, because an inserted word cannot be attributed to a term.

The split between error types is useful diagnostically. Microsoft’s guidance links many deletions to weak audio signal, insertions to noise and crosstalk, and substitutions to too few domain-specific terms in training. Read in a gaming context, as our reasoning: deletions suggest speech buried under game audio, insertions suggest comms or game sound captioned as speech, and substitutions suggest missing game vocabulary.

BenchmarkSTT also defines a Bag of Entities Error Rate, which compares how many times each listed term appears in the reference and in the captions. It ignores position, so it is a quick way to check a game-term list.

Proposed caption bands

These bands are proposals. Fix them before you score anything. As of 8 November 2026, Microsoft describes a WER of 5-10% as good quality and 20% as acceptable for its custom speech models, with 30% or more signalling poor quality. That guidance was not written for game audio, so treat it as a starting point only.

Gameplay caption failures to log

The examples here are invented, to show the kinds of error worth counting.

• Term replaced by a common word: flank captioned as thank, or a champion name turned into an everyday word.
• Words invented over noise: text appearing during gunfire when nobody spoke.
• Merged voices: a teammate’s callout run into the streamer’s sentence.
• Game dialogue captioned as commentary: a character’s line presented as if the streamer said it.
• Condensed speech: a tool that shortens what was said. This raises deletions, and it may or may not suit your style.

Sample size, bias and ties

Small word counts cannot separate close tools. Speechmatics recommends at least 10,000 words, or around 1 hour of continuous speech, per language, and says close engines might not differ significantly on 5,000. With the invented 1,420-word sample above, that is 10,000 / 1,420, or about seven samples.

On one sample of that size, the standard error of an 8% WER is about 0.7 points, so the difference between two tools has a 95% margin of about 2 points, assuming errors are independent. They are not, because errors cluster in noisy stretches. As a proposed rule, treat gaps under 3 points on a single sample as a tie, and pool samples before trusting anything smaller. Game-term figures rest on far fewer words and need wider allowance still.

Choice of audio is the main bias. Testing only the noisiest minute exaggerates errors, while a calm stretch flatters every tool, so report each kind of minute separately. Microphone quality, accent and game-audio mix all shift results, and Speechmatics also notes that speaker characteristics can affect recognition. Results from one streamer may not carry over to another.

What to record for each tool

For each tool and sample type, record three figures: WER on captions as exported, normalised WER and game-term error rate. Log the vendor settings and any custom vocabulary, and export each tool’s captions before any manual editing so they can be compared with the reference.

OpenAI’s Whisper is open-source speech recognition that can run locally, with model sizes from tiny to large and a turbo option. Its README lists the size and approximate memory each model needs. Adding one Whisper model to your comparison gives the paid tools a free benchmark, so that a claimed advantage has something to beat. Keep the reference word count beside each figure.

What stays unknown without a test: how any particular tool scores on gameplay audio, and whether the proposed bands suit your own microphone and game mix.

What to do with your numbers

Use the pattern of errors to decide, not the single percentage.

• Low WER, high game-term errors: look for custom vocabulary or a saved word list, and keep your own correction list.
• High deletions: check your microphone level against game audio before blaming the tool.
• No better than a free local model: captions are not a reason to pay for that tool.
• All similar: caption style and editing speed should decide it. The best AI captions for gaming Shorts page compares what vendors document.

Caption placement is a separate question from accuracy. Other guides are listed in the tests section.