BSD XR Blog
Creating 3D Assets from Volumetric Video
Share this article
Creating 3D Assets from Volumetric Video
Data Science Students at Regis University are developing a novel process for creating 3D models from volumetric video
By Kevin Carbotte
Bit Space Development, Link to Learn , and Regis University formed a partnership that places Data Sciences students in real-world projects developing new processes and pipelines for 3D object tracking. This partnership is meant to challenge the limits of these students’ skills while making a meaningful contribution to technology development.
Work Experience for School Credits
As part of their final assignments, Taylor Ramsey, Kathy Xiong, and Sanjay Maharjan, three students in the Data Sciences program at Regis University’s Anderson College of Business and Computing in Denver, Colorado, recently helped build the first stages of a pipeline that creates 3D models of objects captured from volumetric video files. The pipeline they developed will eventually be used for professional augmented reality training experiences.
Bit Space Development broke the project into several components that would be completed by multiple groups of students over an extended period. The first group accomplished the first two stages, including building a solution for saving the video stream and depth data from a Microsoft Azure Kinect sensor as image files and converting the images into a usable 3D point cloud.
Advanced Technology
The project that these Regis Data Sciences students were assigned relies on cutting edge technology to work. We’re using volumetric video captured with a Microsoft Azure Kinect sensor to create the point cloud data. The Azure Kinect has a depth sensor and stereoscopic camera, but it does not automatically generate a 3D scan. The default output is a standard MKV video file alongside an additional set of depth data files. The work that the students did enables the creation of 3D assets in the standard .obj file format.
The process includes using the Azure Kinect to capture video of an object or person, either while performing an action or maintaining a stationary pose. It requires a scan of all sides of the subject, which can be accomplished by rotating the Kinect sensor around the subject or using an array of sensors to capture all sides at once.
A Multi-Stage Process
The students created a transformation script that breaks the video into individual images for each frame and then feeds them into a tool called Visual SFM , which converts the images into a collection of 3D PLY files. The files are subsequently combined to build a 3D model of the scanned subject.
The next stage takes the PLY files and feeds them into
Mesh Lab
to build a usable 3D model, which can be manipulated in 3D modelling software or imported into virtual environments and used in VR or AR educational and entertainment experiences. This is not yet an automated process within the pipeline and must be done manually.
A Challenging Task
Ramsey, Xiong, and Maharjan came into this project with limited experience with the tools they had to use, including the programming language, 3D point clouds, and the Azure Kinect technology that they had to work with. Using resources such as Microsoft’s Azure Kinect SDK, they adapted to their task and achieved their goals.
The students each said that this project challenged them in ways that they weren’t expecting, and it gave them a very different perspective of their field of work from the classroom education that led them there. Ramsey said he “learned to be comfortable with being uncomfortable” as he pushed through every challenge. Maharjan discovered that “there are a lot of ways of solving a single problem, but you need to find the solution that satisfies your requirements.” Xiong’s background is in biomedical research, so she felt she was prepared for scanning body parts and challenged by the project’s coding side.
More Work Ahead
The end goal of this project is to take a standard capture of a human body and feed it into a tensor flow database, the output of which would create a 3D reconstruction of a human body. That information will help create a machine learning algorithm for human object tracking for future research and commercial projects.
The partnership with Regis University is an ongoing project that will have groups of students cycling in for their work-experience terms. The next group of students will pick up where Ramsey, Xiong, and Maharjan left. We thank them for their efforts and look forward to working with the next group of students next year.
Click to share this article
Other posts from our blog
Transform your business with immersive technologies
Schedule a consultation to see how BSD can help your organization unlock new avenues of engagement.