Skip to content

fix: join whisper subword tokens using leading-space word boundaries - #9

Merged
zeke merged 1 commit into
mainfrom
fix/wts-word-boundaries
Oct 7, 2026
Merged

zeke merged 1 commit into
mainfrom
fix/wts-word-boundaries

Conversation

@zeke

@zeke zeke commented Oct 6, 2026

Copy link
Copy Markdown
Owner

This PR fixes split words in captions like "Cloud fl are" and "Wr angle r".

whisper.cpp marks the start of each word with a literal space before the token in the .wts output. parse_wts() stripped that space, so every subword token became its own word unless someone had added it to COMPOUND_MERGES. That list kept growing and never covered everything, including words split across two whisper segments (replace Wr / angler.).

parse_wts() now joins tokens without a leading space onto the previous word. This also covers contractions and trailing punctuation, so the separate punctuation and contraction regexes are gone. COMPOUND_MERGES stays as a fallback for words whisper actually heard as two words (Cloud flare).

I checked this against 25 clips from a recent video and got no split words. Tests cover the subword, cross-segment, and fallback cases. The README and AGENTS.md are updated.

parse_wts stripped the leading space that whisper.cpp puts before word-initial tokens, so subword tokens became separate words ("Cloud fl are", "Wr angle r") unless listed in COMPOUND_MERGES. Use that space to decide word boundaries, including words split across whisper segments. COMPOUND_MERGES stays as a fallback for words whisper heard as two words.
@zeke
zeke merged commit 6694ff0 into main Oct 7, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant