InsidePETA"EyeToEye":VirtualRealityLive-ActingExperience

After winning an FWA of the day for PETA "Eye to Eye," our VR Experience developed in collaboration with Kolle Rebbe, the FWA team asked us to share some insights with their readers and users. Here we go...

What did you aim to achieve with the project?

In collaboration with Kolle Rebbe and PETA, we wanted to create a uniquely emotional virtual reality experience that allows users to engage in real-time dialogue with a virtual animal and see the world from its perspective.

Our idea was to develop an unprecedented and comprehensive application. The goal was not only to raise awareness about the suffering of animals but also to put people in the shoes of their counterparts - to evoke a deeper emotional feeling. As the world's largest animal rights organization, PETA wanted to offer a new, striking, and compelling way to reach people who have had little or no engagement with animal rights - and to create an experience that allows animals to communicate eye-to-eye with humans.

What technical challenges did you encounter and how did you solve them?

The biggest challenge was the hardware setup combined with flawless character design in the application, which combines excellent image quality and real-time interaction: A motion capture setup transfers the voice, facial expressions, and gestures of an actor to the modeled character - a rabbit - which in turn mirrors the actor's gestures. The actor's voice is transmitted into the virtual world and triggers the dialogue with the respective user. Applying the anatomy and gestures of a human to a convincing 3D object in real-time was quite a challenge - a mix of proven tracking software and a custom hardware setup with constant improvements was our way. Last but not least: The hair work and level of detail of our virtual rabbit are something we are truly proud of. Approximately 500 hours of 3D art and creative engineering were needed for this component.

What did it mean for you to win an FWA?

The final release of the application was quite intense. So it was really great for us to receive feedback right at the start. While we were still working on the mobile VR version, the event setup received amazing feedback: On average, people spent 12 minutes in the experience. At the same time, winning an FWA after just one voting day was the cherry on our vegan cake. It pushed us forward in the final project phase and helped to draw global attention to this experience. What more could you ask for...

Tools Used

An essential part of the setup is stable and performant motion capture. For body tracking, we used Perception Neuron by Noitom. Face tracking is performed by FaceWare Live. For the complex modeling of characters and environment, we primarily used Maya, ZBrush, 3ds Max, and TopoGun. Texturing was done with Mari, Substance Painter, Photoshop, and Knald. The grooming of the main character was done with Peregrine Labs Yeti. Our team used FMOD as the sound engine, while sound editing was done with Nuendo. Last but not least, the application was created in the 3D Engine Unreal with NVIDIA HairWorks and VRWorks as additional components.

Three Hot Facts:

1. Our experiment led hundreds of people to rethink their actions towards animals. It also affected our colleagues: Some of them decided to become vegetarians, including our sound engineer, who reviewed around 200 hours of video and audio files on animal abuse content to deliver realistic SFX results.
2. After the release, many VIP supporters of PETA tried the application, including Germany's biggest "Let's Play" influencer UNGE. Besides becoming a trend on YouTube, the application videos garnered several million views and received great feedback from media worldwide. Fun fact: Among the test subjects was a German actor named Sky DuMont. As we always say: Sky is not the limit.
3. The experience was launched at re:publica '18 in Berlin and presented at many other occasions such as festivals and trade shows. We hope that in the long run, we have helped to draw attention to an important issue in a truly new and deeply connecting way.