Creating a machine learning agent to ease interactions between participants in virtual and augmented reality environments —such as the metaverse— in order to help them achieve their goals and address conflict situations. This is the main objective of GuestXR, the European project under the coordination of professor Mel Slater, from the Faculty of Psychology and the Institute of Neurosciences of the UB (UBNeuro).

As part of the GuestXR project (A Machine Learning Agent for Social Harmony in eXtended Reality), coordinated by the Eurecat technological centre, a machine learning system called guest will be designed, and it will work as an agent able to examine the individual and group behaviour of the participants by drawing on existing models from the neuroscience and social psychology standpoint. Moreover, Deep Reinforcement Learning will train the Guest to learn over time —both in simulations and real meetings— and it will be more efficient in social interactions.

In these lines, the agent “will take part in the conversation in different ways, performing a range of multisensory actions through music or changes in space, among others”, notes the director of Eurecat Audiovisual Technologies, Adan Garriga.

“Each meeting in virtual and augmented reality has a purpose, be it explicit (for instance, reaching an agreement on a topic) or implicit (such as enjoyment). The basic idea of the guest is to help the group of people to accomplish this purpose. This involves an exciting research with a strong interdisciplinary element, which opens new paths, and hopefully, contributes to the success of virtual and augmented reality”, notes Professor Mel Slater.

In order to modulate this social interaction, the guest will carry out a series of multisensory actions, “through, for instance, visual and auditory features to create specific states of mind and stimulate a relaxed environment when it identifies a conflict”, highlights researcher Umut Saying, from Eurecat Audiovisual Technologies.

This project “is designed to help offering solutions to the existing challenges in online environments of social interaction such as online harassment, for which there isn’t a specific European legislation”, says the coordinator of the Eurecat GuestXR project, Aurora Sesé.

The intervention of the machine learning system will initially be tested in conflict resolution situations, interactions with participants with hearing difficulties and contexts that lead to polarized debates such as climate change.

Two open calls to include other innovative use cases to test the effectiveness of the system are expected. One case will be aimed at society in general —including companies and associations, among others— while the other will aimed at the field of arts.

The GuestXR technology will be carried out under the Ethics by design approach, which entails considering the potential ethical problems derived from the use of artificial intelligence (respect for human autonomy, prevention of harm, fairness, etc.).

The GuestXR consortium, coordinated by the UB and the Eurecat technology centre, is formed by eight organizations from six countries with multidisciplinary teams of the field of the expanded reality, machine learning, artificial intelligence, social psychology, neuroscience of emotions, multisensory integration, ethics of research and technology transfer. Among the collaborators are the University of Maastricht (the Netherlands), the University of Warsaw (Poland), the University of Reichman (Israel), the National Institute for Research in Digital Science and Technology (INRIA, France), and the companies Virtual Bodyworks and G.tec medical engineering GMBH.