Human-Centered Multimedia: making remote togetherness possible

Context and intention

Of course, there is a catch to capturing and transmitting highly realistic representations in 3D in real-time. “Managing finite resources is a core issue of computer science,” Cesar says. “We want to provide the best quality of experience, adapted to any technological conditions.” So, no waiting around for faster processors and 6G mobile communication but rather an optimisation based on human-centric principles and methodologies.

“I do a lot of communication with my hands,” Cesar says. “In most situations, only my face and hands need to be represented life-like. The wrinkles in my shirt can be ignored, as can my feet. Unless I’m dancing at a wedding of course.” Whereas the human brain is primed for ignoring any non-essential information based on a specific context, computers are not. One of the core challenges of his research therefore is to develop algorithms that are context aware, that can decide which information is important to send and which information can be safely ignored. “To optimise what we send or not, we need to understand what is happening, what you are looking at, where you are moving to, even the emotional state of the people communicating.”

Domain experts

Cesar has made it a personal mission to put human-computer interaction at the core of computer science. “We run a lot of experiments with users, for example on how to represent interpersonal distances in virtual reality,” he says. “For our experiments we collaborate with domain experts, such as sociologists and psychologists, but also people from opera and theatre.”

He also fosters collaborations in the Netherlands and Europe regarding the technical aspects of volumetric video conferencing. “Our own focus is primarily on the networking aspects, on the communication itself, on the understanding of humans and on the optimisation of all the signals related to that,” he says. “We collaborate with experts in multi-sensory experiences and, for actual and efficient 3D-rendering, with experts in computer graphics.”

More personal than a director’s cut

To see where this research is going, it helps to provide some examples of what Cesar has already achieved in some of his many projects. For one, he simulated a chance passer-by having a volumetric video consult with a doctor regarding an injured cyclist. It involved a mobile phone camera with depth perception and all data was transmitted using the public mobile 5G network. “The quality of the video was very low,” he says. “It served as a proof of concept.” 

He also transmitted multiple feeds from a soccer match to a pub full of supporters. “The idea behind transmitting multiple streams, rather than just the director’s cut, is that it allows personalisation of what to ‘broadcast’ on the receiver side, based on personal preferences.”