LARG  (Laparoscopic Augmented Reality Games)

 

This page describes LARG  (Laparoscopic Augmented Reality Games) – an approach that extends laparoscopic box-trainers with Augmented Reality (AR) and Gamification techniques.  This researches aim at increasing training motivation and provide user assessment. Our proposal introduces an extra camera, and uses tracking algorithms in order to monitor movements of the manipulated objects. With this approach, we are able to observe the user performance during the training activity (e.g., time to finish the activity, amount of errors). 3D animations are inserted in the environment with AR in order to improve the user immersion. Scores, sound effects, and ranks are included in the gamified activities. We designed and developed two activities for navigation skills training.

 Check this video to see the current version of the box-trainer.

Early Version

The first prototype of LARG simulator consisted of a plastic box, two cameras and two applications: one responsible for making the motion detection and the other one for rendering the augmented reality. The object tracking software was made using the Aforge.Net library, while the projection of AR was made with the NyARToolkit library, which is an adapted version of the ARToolkit Library. The game logic was developed by using the Unity 3D Game Engine.

In this first version, we developed a game, inspired by the Genius game, where a sequence of steps were presented with AR (the disks were highlighted), and the learner should repeat them. If the student correctly repeat the sequence, then the sequence were increased with a new step to be followed. This prototype was completed, and user tests were performed. We found problems related to system calibration and the occlusion of the AR objects. These problems withdrew the sense of immersion. These results are the basis for the development of a new physical architecture and indicated the need for modifying the simulator software modules. We submited these results to a Portuguese Workshop in SBGAMES 2013 [2].

Current Version (Paper submitted to CBIE 2015)

The current version of LARG is composed by a gamification software, a glass fiber box mimicking an human abdomen, two cameras, conventional laparoscopic instruments (grasping forceps), and game scenes of the training activities. The two cameras, and game scenes are placed inside the box, while the instruments are introduced through the openings as depicted in the Figure below. A scene, which consists of a rigid plastic structure and rubber made gadgets, is placed in a previously established position.  The tracking camera is fixed in the ceiling of the box, and its images are used to track changes in the scene, such as moving gadgets, and tools position. The view camera captures images of the scene that are merged with AR projections to be displayed to the user.

This version was developed with Open CV and Ogre as software modules. The oclusion problem was solved with the use physical objects and scenes. The Figure below presents the PEG Transfer Task.

Last User Evaluation

We  implemented an initial user evaluation conducted with a group of eighteen users.  They were requested to answer a questionnaire, in which they could provide classified data about their background, the simulator in general, and their impression about the two activities (PEG Transfer and Follow-the-Path). Before using the simulator, they answered a first set of questions about their background on laparoscopic training and professional profile, informing their residency or internship level, their previous experience towards MIS (Minimal Invasive Surgery) training using box-trainers, AR or VR simulators, and whether they have attended to a course or training that used any computer-aided medical procedure. The second set of questions intended to provide data about each activity independently, thus inquiring about the gamification techniques employed, the design, the difficulty level, whether the user noticed improvements in his skills throughout the activities, and their motivation for training and learning using the tool. Finally, users could issue suggestions and comments for future releases.

The questionnaire is available at this link. The results obtained in the questionnaire can be seen in the following figure. More details of this user evaluation were submitted to SBIE 2015 [1]

---

 

Publications

1- Full Paper in CBIE 2015 describes the LARG architecture and its first user evaluation.

Silva, L. F. M., Valerio, J., Viana, W., dos Santos, A. D., Trinta, F., Melo Filho, A. A., & Melo, A. (2015, October). Avaliação do uso de realidade aumentada e gamificação para o treinamento de habilidades em laparoscopia. In Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação-SBIE) (Vol. 26, No. 1, p. 627).

2- First version of the project LARG published on SBGames 2013.

SANTOS, A. D. ; VIANA, W. ; MELO JR., A. ; Filho, A. A. M. ; SOUZA JUNIOR, H. C. ; MENDONCA JUNIOR, G. . Inserção de técnicas de Gamificação e Realidade Aumentada para auxílio no ensino de Medicina. In: XII Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames 2013) – Workshop, 2013, São Paulo. Proceedings do XII Simpósio Brasileiro de Jogos e Entretenimento Digital (SBGames 2013), 2013. v. 1. p. 41-44.

Projects Participants

Research Leaders

• Windson Viana de Carvalho - Prof. Adjunto II – Instituto UFC Virtual
• Antônio Aldo Melo Filho – Prof. Adjunto II – Depto. Cirurgia – Faculdade de Medicina – UFC

Researchers

• Fernando Antônio Mota Trinta - Prof. Adjunto II – Computer Department
• Alysson Diniz dos Santos – Prof. Assistente I – Instituto UFC Virtual
• Antônio José Melo Leite Junior – Prof. Assistente II – Instituto UFC Virtual
• Julian Alberto Valério – Msc Student - Computer Department
• Luis Fernando Silva – PHD Student - Computer Department