A textbook comes to life: Augmented reality for dental specialists

While textbooks are useful and contain valuable knowledge, their high information density often makes them inaccessible and can be off-putting. Have you ever imagined engaging with the contents of your textbook in a playful and interactive way? The AR Osteoclast app makes this possible. Game developer Jascha Grübel, from the ETH Game Technology Center (GTC), and Bernd Stadlinger, a member of the ETH AI Center and Chief of Service at the Clinic for Cranio-Maxillofacial and Oral Surgery at the Center for Dental Medicine in Zurich, designed the app to give dentists a better understanding of bone remodelling processes.

If students point the camera of their tablet or smartphone at a certain image in their Cell-to-Cell Communication textbook, they end up on the bone surface next to a blood vessel. In this microcosm, magnified by a factor of 600, players control otherwise automatic biological and biochemical processes. Each of the game’s six chapters is introduced by a voice explaining the ins and outs of bone resorption as well as the students’ role in the game (see video).

Every year, 10 percent of our skeleton undergoes bone remodelling without us noticing. Cells known as osteoclasts break down bone tissue, while osteoblasts synthesise new bone. This is crucial knowledge for dentists, since osteoclasts play an important role not only in tooth exfoliation but also in the integration of dental implants into the natural bone structure. On top of that, they protect the jaw bones against bacterial infection and support tissue repair after a tooth extraction.

Turning a mind map into a game – with mathematics

Before getting started, the game’s main developer Grübel had to come to grips with the subject matter: “I watched instructional videos and mind mapped all the cells involved as well as their functions, actions and interactions.” Since these bodily processes are automatic, Grübel had to figure out which ones to assign to the players. “For instance, when the player taps on the screen, the osteoclast releases messenger substances to attract stem cells, which then turn into osteoblasts and repair the bone,” Grübel explains. On occasion, the players receive symbolic aids: in chapter three they control two taps – one for hydrochloric acid and another for enzymes – enabling them to dissolve the bone with the right amount of each substance. Grübel builds these game mechanisms on a game development platform called Unity. “It’s important for a game for time to pass in the computer as well and to determine the next step in the game,” Grübel says. “Unity predetermines the background for that, such as the laws of physics that should apply in the game.” Because, at the end of the day, all game mechanisms are mathematical computations, Grübel explains.

The triangle of play, science and education

Finding the right balance between play, science and education is the main challenge, according to the GTC’s Managing Director Fabio Zünd: “A game must be fun and a source of motivation. This means we have to simplify the biological processes without distorting them.” To this end, the game developers entered into an intense exchange of ideas with dental specialists, researchers and the publishers. “Certain moves reminded the researchers of outdated knowledge. So we had to work out together how best to represent the current state of knowledge,” Grübel reports.

Another challenge was to ensure that the app would work on as many devices as possible: “At the GTC, we usually deal with research prototypes – so they might only run on one device, but they can do crazy things,” Zünd says. AR Osteoclasts, by contrast, is an app store product intended to reach as many people as possible. In fact, the developers had to restrict the technology in AR Osteoclasts to ensure that the app would work on older devices as well. The digital artist Violaine Fayolle has compressed the detailed 3D model of the bone on whose surface the player is located to prevent devices from running hot when displaying the game. The 3D model was created by medical film producer interActive Systems. The company participated in the development of the app alongside the publishers

Which is better, playing or reading?

A study is currently underway to determine if the app improves students’ motivation and learning performance. However, the two computer scientists are both convinced that games have great educational potential. “Games can be a vehicle for information, but it’s difficult to know what to communicate,” Grübel says. “We still need more results from the learning sciences.” In striking a balance between play and knowledge transfer, the app may not have turned out as playful as Grübel had hoped; however, considering the short development time, he is pleased with the result. “Turning the contents of a book into an interactive experience is new,” he says. But the developer will get a chance to play around some more after all: together with Bernd Stadlinger, who published Cell-to-Cell Communication, he plans to take the project one step further. “We would like to realise the game in virtual reality, which offers more opportunities for interaction and allows us to study how people and the computer interact,” Zünd says.