ETH Zurich researchers and Disney Research have produced a new algorithm that allows non-invasive reconstruction of the teeth and gums from digital photos.

In recent years, IT experts have developed sophisticated image-based digital reconstruction methods for modelling the human face. These methods capture highly detailed images of the face or specific areas of the face - such as hair, eyes or eyelids - and create a digital model of them. In the past, however, very little attention has been given to the mouth cavity in general, and the teeth in particular. Image-based approaches capable of modelling teeth in a very natural way are still in the early stages of development. Accurate rendering of teeth, however, is crucial for the realistic display of facial expressions. In the past, researchers have relied on handmade plaster casts of teeth for this purpose.

A group of scientists led by Chenglei Wu, a post-doctorate at the Computer Graphics Lab of ETH Zurich has collaborated with the Max-Planck Institute in Saarbrücken and Disney Research on the development of new software which allows the non-invasive reconstruction of the teeth and gums. The computer scientists presented this new software at SIGGRAPH Asia 2016 in Macao, China.

Digital photos instead of plaster casts

All that is required to produce such a detailed reconstruction is some digital photos showing a person’s face. The software is capable of creating a complete reconstruction of the teeth and its roots even if they are partly or fully occluded.

On top of that, Wu and his colleagues have even managed to produce reconstructions from short video clips of a person’s face recorded with a smartphone. Furthermore, the new approach can be seamlessly integrated into existing photogrammetric multi-camera reconstruction setups for entire faces.

The scientists first have to teach the programme the basic features of tooth positions, shapes and rows by feeding it with high-resolution 3D scans of oral cavities. The software then uses these data to ‘learn’ how to recognise sets of teeth, assess the shape of the teeth and synthesise the missing elements not discernible on the images, along with the positioning of the teeth. The programme also takes into account any discolouration of the teeth and matches the gum pigmentation. The end result: an extremely natural-looking digital reconstruction of both teeth and gums. ‘This approach opens up new avenues for individual high-quality tooth reconstruction’, says the project coordinator, Thabo Beeler from Disney Research.

Inspired dentists

The new method could also be applied to other areas such as the entertainment industry, where it can naturally extend traditional photogrammetric face-capture systems using multiple cameras. It can be used in medical dentistry as well. Dentists are looking for the simplest method possible to produce the most natural looking model of patients’ teeth, so as to give the patient a visual impression of how replacement teeth would look, for example. ‘We have received very positive feedback from dentists’, Beeler confirms. ‘This shows this method is not only suitable for modelling digital characters in films or games, but also for practical applications in the real world.’

To produce digital three-dimensional models of the teeth, dentists currently use special scanners for capturing images of the patient’s mouth cavity and teeth. Both procedures are cumbersome, and the scanner is also expensive and unpleasant for patients, who have to keep their mouth wide open for the entire scanning procedure. Thanks to the ETH software, digital teeth reconstructions will one day become commonplace, without the need for expensive specialised equipment.

Reference

Wu C, Bradley D, Garrido P, Zollhöfer M, Theobalt C, Gross M, Beeler T. Model-based Teeth Reconstruction. ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH Asia 2016: Volume 35 Issue 6, November 2016, doi: 10.1145/2980179.2980233