How does audio language model hold speaker identity throughout an utterance.
the first hypothesis that comes to mind is that TTS models may have attention sink at speaker token or may have high attention score there and the model keeps looking at speaker token. but looking at attention maps gives you a different story.
we studied mid checkpoint of silk-oss an open source TTS model by rumik.ai which extends base LLM to emit neural audio codec tokens. we find that the model doesn’t use speaker tags as attention sink for speaker consistency throughout the utterance, instead the actual attention sink is <BOS> token taking 62.7% of the mass meanwhile the speaker token gets 0.6% and colon after speaker token gets 0.53% on average.
speaker token and colon gets comparatively high attention score when generating the first audio token as compared attention scores when generating the remaining audio tokens.
| Token | Attention Scores at the first audio token | Attention Scores after 32 audio tokens | Attention scores after 64 audio tokens | Ratio |
|---|---|---|---|---|
| BOS | 0.7334 | 0.6955 | 0.6059 | -1.2x |
| name | 0.0214 | 0.0108 | 0.0029 | -7.4x |
| colon | 0.0307 | 0.0143 | 0.0065 | -4.8x |
| sentence | 0.0920 | 0.0186 | 0.0046 | -20.1x |
| prior audio | NA | 0.1769 | 0.3383 | +1.9x |
we also ran a knockout test where we zeroed out the value (V) vector in every layer for chosen prompt positions, without touching the attention weights. so that the model still attends to that token but retrieves nothing. and measured speaker similarity using ECAPA-TDNN and WER of the transcript generated by whisper-large-v3
| Token at which V vector is zeroed | Speaker Identity | Word Error Rate |
|---|---|---|
| nothing(baseline) | 1.00 | 0.00 |
| the colon | 0.607 | 0.00 |
| speaker name | 0.486 | 0.107 |
| <BOS> token | 0.028 | 0.964 |
the speaker identity scores go down by 39.3% when value at colon is zeroed and when speaker tokens are zeroed the speaker identity scores goes down by 51.4%. zeroing the BOS token destroys both the speaker identity and intelligibility since most of the internal calculation happens at attention sinks.
but this explains how speaker identity gets introduced and not how it is sustained.
this makes it clear that speaker tokens doesn’t hold up the consistency over the time. that makes us think, what else does?
the prompt starts the voice and audio keeps it
we found a clue in attention scores throughout the audio token generation… mass on BOS falls while the mass on the model’s own prior audio climbs from 0.18 to 0.34. maybe… the speaker identity is not held by the prompt at all, but the model listening to itself.
to prove this we took a simple approach of using only a sliced prefix from model’s own output. we took audio tokens sample from model’s output but with different speaker and attached it after prompt naming another speaker.
the mechanism is auto-regressive self-conditioning with a short handover. the prompt decides the voice and within few seconds the audio stream becomes responsible for it, after that speaker prompt is close to vestigial. swapping it for a different speaker still does not change the voice 92% of the time.
we also have more exciting insights about the model and will be posting soon how we used these insights to increase performance of our model.