Seonglae ChoText Prompt: Role assignment (e.g., insurance consultant), Voice Sample: Voice cloning → Combined into a Hybrid Prompt that enables real-time conversational AI to control both "role" and "voice" simultaneously. Also extends the existing Full-Duplex-Bench
Native full-duplex S2S is based on Moshi. A single autoregressive Transformer models multiple streams simultaneously (multi-stream) as one joint sequence. The reason "speaking while listening" works = separation of observed tokens vs sampled tokens. At each time step, the joint sequence contains both user/agent tokens, but user tokens are observed (input) while agent tokens are sampled (generated by the model). This enables operation without turn segmentation.
- User audio stream
- Agent audio stream
- Agent text stream (predicts/uses time-aligned text in Inner Monologue style)
However, logically it's multi-stream, but implementation-wise it's a single AR stream.
NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice
We introduce PersonaPlex, a full-duplex conversational AI model that enables natural conversations with customizable voices and roles. PersonaPlex handles interruptions and backchannels while maintaining any chosen persona, outperforming existing systems on conversational dynamics and task adherence.
https://research.nvidia.com/labs/adlr/personaplex/
model
nvidia/personaplex-7b-v1 · Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
https://huggingface.co/nvidia/personaplex-7b-v1
pdf
research.nvidia.com
https://research.nvidia.com/labs/adlr/files/personaplex/personaplex_preprint.pdf