Description
VDungeon is an online virtual tabletop RPG and chat room, where the players' faces are displayed as virtual avatars. Game masters can host rooms that their players can join over the internet. Our application uses your camera and OpenCV to detect and track facial features, then uses this information to animate a simplified 3D head model with your own movements and expressions. VDungeon also has several features to support tabletop roleplaying game mechanics, such as a board editor, interactive props and simulated dice rolling.
Goals and Motivation
We want to provide a more engaging way to play Roleplaying Games (RPGs) at a distance.
The current pandemic has forced the majority of us to stay at home, so we wanted to create something that could bring people together online. We thought of the many people that started approaching RPGs online and what could be the difficulties with it. Clearly, communication is the crucial part in this kind of game, and most of it is non-verbal so showing your face with a webcam helps a lot.
But what if I don't feel confortable with showing my face to, possibly, strangers?
VDungeon solves this problem by providing a virtual avatar that will match your facial expression. This also allows to fully immerse into the story, by being able to see what your teammates look like, and even letting you see the world from the eyes of your character!
Challenges
The biggest challenge was networking, as none of us was familiar with making a multiplayer game, it ended up being more complex than expected. In order to avoid implementing the netcode from a low level we based our application on a high level API, which made the development faster, but also required a well defined structure of the project, so modifying the structure of the project at a second stage took us quite some time. At the beginning we struggled creating face animations that could be read by OpenCV. We were able to circumvent this obstacle by using blandshaps on a simpler model. Finally, managing a project without ever being able to meet in person was not trivial. Communication plays a crucial role to avoid clashes of individual works.
The Team
At the start of the project I fucesed mostly on the design of the software: defining classes and distributing the work across the team. During the development of VDungeon I implemented the networking features that allow multiple player to connect together and interact with each other, sharing the same environment. Part of my time was also spent into the design of some of graphics, such as different characters, that didn’t made to the current version of the project.
Show contributions
My focus in the beginning of the project was on learning character animation, this unfortunatly didn’t end up being used in the final version as we decided to switch our approach to using a simple head as the character instead of a fullbody model. After this switch, I created the initial head model with blendshapes along with the interface used to animate them.
My primary contributions during the first half of the project were the implementation of the grid system through which the enviroment operates, including positioning and collision rules for props and walls within the grid, as well as pathfinding. At a later point, I also implemented a set of interactions which allows the user to edit the environment by adding, removing and moving props around. During the second half of the project, I focused mostly on creating 3D models, textures and interactions for environmental props and structures.
In the early stage of this project, I mainly contributed to the control of the game, including the handle of multiple and dynamic cameras, drag-and-move control on the game board, as well as the first person control on both PC and mobile devices through gyroscope or swiping. In the later stage of the project, I focused more on the UI design and implementation, including the set-up of the main menu scene, the connection to the network manager, and the UI animation throughout the game. I also implemented the particle effects for fire and included simple global illumination.
Show contributions
My main contribution to the project is the implementation of the face tracking system and building the mapping between face tracking data and the 3D model. I also build the server-client connection between Python and Unity in order to communicate data from face tracking in Python to Unity. During the later stage of the project, I focus on implementing Kalman filter and Median filter to reduce noise in image processing and stabilize the model animation. And I created a dice system with crafted model and smooth rolling interaction, which is a typical game mechanic in a tabletop game.