Khresmoi: Multimodal Multilingual Medical Information Search

24th International Conference of the European Federation for Medical Informatics Quality of Life through Quality of Information – J. Mantas et al. (Eds.) MIE2012 / CD / Village Presentation Khresmoi – multimodal multilingual medical information search1 Niraj ASWANIi, Thomas BECKERSg, Erich BIRNGRUBERk, Célia BOYERj, Andreas BURNERk, Jakub BYSTROŇh, Khalid CHOUKRId, Sarah CRUCHETj, Hamish CUNNINGHAMi, Jan DĚDEKh, Ljiljana DOLAMICj, René DONNERk, Sebastian DUNGSg, Ivan EGGELb, Antonio FONCUBIERTAb, Norbert FUHRg, Adam FUNKi, Alba GARCIA SECO DE HERRERAb, Arnaud GAUDINATb, Georgi GEORGIEVe, Julien GOBEILLb, Lorraine GOEURIOTf, Paz GOMEZc, Mark GREENWOODi, Manfred GSCHWANDTNERl, Allan HANBURYa,2, Jan HAJIČh, Jaroslava HLAVÁČOVÁh, Markus HOLZERk, Gareth JONESf, Blanca JORDANc, Matthias JORDANg, Klemens KADERKk, Franz KAINBERGERk, Liadh KELLYf, Sascha KRIEWELg, Marlene KRITZl, Georg LANGSk, Nolan LAWSONj, Dimitrios MARKONISb, Iván MARTINEZc, Vassil MOMTCHEVe, Alexandre MASSELOTj, Hélène MAZOd, Henning MÜLLERb, Pavel PECINAh, Konstantin PENTCHEVe, Deyan PEYCHEVe, Natalia PLETNEVAj, Diana POTTECHERc, Angus ROBERTSi, Patrick RUCH b, Matthias SAMWALDa, Priscille SCHNELLERd, Veronika STEFANOVa, Miguel Angel TINTEc, Zdeňka UREŠOVÁh, Alejandro VARGASj and Dina VISHNYAKOVAb a Vienna University of Technology, Austria b University of Applied Sciences Western Switzerland c ATOS Origin, Spain d Evaluations and Language Resources Distribution Agency, France e Ontotext, Bulgaria f Dublin City University, Ireland g University of Duisburg-Essen, Germany h Charles University Prague, Czech Republic i The University of Sheffield, United Kingdom j Health on the Net Foundation, Switzerland k Medical University of Vienna, Austria l Association of Physicians in Vienna, Austria Abstract. Khresmoi is a European Integrated Project developing a multilingual multimodal search and access system for medical and health information and documents. It addresses the challenges of searching through huge amounts of medical data, including general medical information available on the internet, as well as radiology data in hospital archives. It is developing novel semantic search and visual search techniques for the medical domain. At the MIE Village of the Future, Khresmoi proposes to have two interactive demonstrations of the system 1 The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement n° 257528 (Khresmoi). This submission is endorsed by the EFMI WG on Natural Language Understanding. 2 Corresponding Author. E-mail: hanbury@ifs.tuwien.ac.at under development, as well as an overview oral presentation and potentially some poster presentations. The Khresmoi project The Khresmoi project [1] is developing a multilingual multimodal search and access system for medical and health information and documents. It addresses the challenges of searching through huge amounts of medical data, including general medical information available on the internet, as well as radiology data in hospital archives. The latter includes Magnetic Resonance (MR) and Computed Tomography (CT) images. The system will allow text querying in several languages, in combination with image queries. It will return translated document summaries linked to the original documents. Semi-automated estimation of the trustworthiness and readability of documents is also being developed. It has three main end user groups: members of the general public, physicians and radiologists (a group of physicians for which image search is of immense importance). An outline of the Khresmoi concept is shown in Figure 1. Figure 1. Outline of the Khresmoi System. The project consortium is made up of 12 partners from 9 European countries (see author list). Khresmoi started on the 1st of September 2010 and runs for four years. Khresmoi achievements The current search practices of the end user groups, as well as their requirements and wishes for future health and medical information search systems were elicited. To achieve this aim, three surveys were done. The first was an online survey aimed at members of the general public [2]. The survey of the radiologists was done on a smaller scale [3], but included initial experiments on using eye tracking to determine the parts of an image on which a radiologist concentrates [4]. The final survey was aimed at medical doctors, and was also performed online [5]. At the heart of the Khresmoi search engine are a number of open source components, extended by the work done in Khresmoi. New components based on research results obtained in Khresmoi will also be included. Work on extending the existing components to meet the needs of the project has begun. This includes the addition of the capability to rank search results to Mimír [6], a search engine capable of indexing and searching over text, annotations, semantic schemas (ontologies), and semantic meta-data (instance data); as well as the development of a browser-based user interface and image display capabilities for ezDL [7], a framework for interactive search applications. Furthermore, basic research has led to innovative techniques for anatomical structure identification and localisation, as well as visual-based anatomy retrieval and anomaly retrieval in 3D radiology images. The annotation of the medical documents is done by the GATE software [8][8]. Manual annotation to correct the automated annotations and hence allow the system to learn to improve the annotations is being done. All knowledge is stored in the OWLIM repository [9]. Way of presentation The proposed way to present the results of the Khresmoi project is through an overview presentation, as well as two interactive demonstrations set up at a “booth”. The first demonstration will be of the system for searching by lay-people and physicians. This demonstration focuses on the multilingual and collaborative aspects of the search process. The second demonstration will focus on the image search technology for searching though large 3D radiology images based on their visual features. A video showing an early version of this demonstrator is available [10]. MIE participants will have the opportunity to experiment with the demonstration systems and to give feedback on the systems and suggestions for improving them. Posters going into detail on some of the Khresmoi results, such as the survey results, can also be displayed for discussion. References [1] khresmoi.eu (last accessed: 4 April 2012) [2] Pletneva N, Vargas A, Boyer C. Requirements document for the general public health search. Khresmoi Deliverable D8.1.1, khresmoi.eu/assets/Deliverables/WP8/KhresmoiD811.pdf (last accessed: 4 April 2012) [3] Müller H. Report on image use behavior and requirements. Khresmoi Deliverable D9.1, khresmoi.eu/assets/Deliverables/WP9/KhresmoiD91.pdf (last accessed: 4 April 2012) [4] Dungs S, Holzer M, Markonis D, Langs G, Kriewel S, Müller H, Using eye tracking on a clinical viewing station to help investigate image search requirements of radiologists. 2011. In: RSNA Proceedings [5] Gschwandtner M, Kritz M, Boyer C, Requirements document for health professional search, Khresmoi Deliverable D8.1.2, khresmoi.eu/assets/Deliverables/WP8/KhresmoiD812.pdf (last accessed: 4 April 2012) [6] gate.ac.uk/mimir (last accessed: 4 April 2012) [7] ezdl.de (last accessed: 4 April 2012) [8] gate.ac.uk (last accessed: 4 April 2012) [9] www.ontotext.com/owlim (last accessed: 4 April 2012) [10] www.youtube.com/watch?feature=youtu.be&hl=en-GB&v=yl6dBdrX0yg (last accessed: 4 April 2012)