French international football concluded the story of FIFA 18 by raising the trophy in the final. Meanwhile, a team of researchers from Google, University of Washington and Facebook produced a visionary look of what an AR football match could look like in future.
An end-to-end system is organized to create a 3D reconstruction of a real football match, which can be seen through an AR device like HoloLens. The team trained their convolutional neural network (CNN) by giving ample time to virtual player data, which they captured from EA’s FIFA video games. This data has helped them to use a single monocular YouTube video and grow 2D/3D hybrid output.
Some issues which the team is facing in a ‘perfect’ 3D reconstruction include: 2D textures for players projection, the jittery positioning of individual players and tracking of football is not visible. The other main problem is the occlusion effect because of single monocular shots. The texture of the players vanishes from view because player actions are hidden from the camera. On the other side, this AR experience is quite amazing as it can be displayed on your coffee table.
“There are numerous challenges in the monocular reconstruction of a soccer game. We must estimate the camera pose relative to the field, detect and track each of the players, re-construct their body shapes and poses, and render the combined reconstruction,”
The Research Team
Live broadcast of the matches is not possible. To watch the full match on HoloLens a real-time reconstruction method is required. Compression of data with efficient techniques and delivery to your AR headset also needs a proper procedure. The system represents a lower level video with current technique as it relies on the standard footage. Although the system is based on 4K video issues like motion blur, chromatic aberration, and compression artifacts still occur.
For the best possible outcome, the stadium must be set with multiple cameras, specifically for AR capture. It is a building block to get live 3D sports in AR. We have no idea where this story will end but one thing is super exciting that nothing is better than having a whole live football stadium on your table top.