Research Scientist Intern - Audio-Visual Conversation Scene Generation

The Meta Reality Labs Research Team brings together a world-class team of researchers, developers, and engineers to create the future of virtual and augmented reality, which together will become as universal and essential as smartphones and personal computers are today.

Reality Labs Research Audio builds the audio-visual perception that future smart glasses will rely on. That includes understanding who the wearer is talking to, who is speaking, and how a conversation is organized. To train and evaluate these models at scale, we need realistic conversation data that comes with exact ground truth. This internship explores generative models that produce such data.

The project

Generate realistic video with audio of multi-person conversations. Inputs can be text alone, text plus one image of a group, or text plus a reference image for each person. Beyond looking real, the scenes should show natural social interaction: turn-taking, gaze, head orientation, listener reactions and group formations. Each scene should also be controllable: the 3D position of each person, how people are grouped, and the position and pose of the egocentric camera. That control is what lets the output serve as accurately labelled training data for audio-visual models, e.g. predicting the glasses wearer's intent or detecting conversation groups.