Seonglae ChoText Prompt: Role assignment (e.g., insurance consultant), Voice Sample: Voice cloning → Combined into a Hybrid Prompt that enables real-time conversational AI to control both "role" and "voice" simultaneously. Also extends the existing Full-Duplex-Bench
Native full-duplex S2S is based on Moshi. A single autoregressive Transformer models multiple streams simultaneously (multi-stream) as one joint sequence. The reason "speaking while listening" works = separation of observed tokens vs sampled tokens. At each time step, the joint sequence contains both user/agent tokens, but user tokens are observed (input) while agent tokens are sampled (generated by the model). This enables operation without turn segmentation.
- User audio stream
- Agent audio stream
- Agent text stream (predicts/uses time-aligned text in Inner Monologue style)
However, logically it's multi-stream, but implementation-wise it's a single AR stream.
model
pdf