KMeD: Knowledge-based Image Retrieval with Spatial and Temporal Constructs

Wesley W. Chu*, Alfonso Cárdenas*, Ricky Taira**

*Computer Science Department, University of California, Los Angeles

**University of Washington

Contact Information

Wesley W. Chu, P.I.

Computer Science Department
University of California, Los Angeles
Los Angeles, CA 90095
Phone: (310) 825-2047
Fax: (310) 825-7578
Email: / Alfonso F. Cárdenas, Co-P.I.
Computer Science Department
University of California, Los Angeles
Los Angeles, CA 90095
Phone: (310) 825-7550
Fax: (310) 825-7578
Email: / Ricky Taira, Co-P.I.
Children’s Hospital
University of Washington
Seattle, WA
Phone: (206) 528-2744
Fax: (206) 528-2730
Email:

WWW Page:

List of Supported Students and Staff

Graduate Students (past and current): Alexander A. T. Bui, Jason M. Borja, Christine Chih, Mylin Lam, Chih-Cheng Hsu, Sang-Hyun Park, and Roderick Y. Son

Keywords

Content-based retrievalMedical image database system

Knowledge-based query processingMultimedia database system

Spatial temporal query languageIconic query language

Project Award Information

  • Award Number: IRI-9619345
  • Duration: 08/15/1997 – 07/31/2000
  • Title: Knowledge-based Image Retrieval with Spatial and Temporal Constructs

Project Summary

A knowledge-based approach to retrieve medical images by feature and content with spatial and temporal constructs is being developed. Selected objects of interest in a medical image (e.g. x-ray, MR image) are segmented, and contours are generated from these objects. Features (e.g. shape, size, texture) and content

(e.g. spatial relationships among objects) are extracted and stored in a feature and content database. Knowledge about image features can be expressed as a hierarchical structure called a Type Abstraction Hierarchy (TAH). TAHs can be generated automatically by clustering algorithms based on feature values in the databases and hence are scalable to large collections of image features. Further, since TAHs are generated based on user classes and applications, they are context- and user-sensitive.

A knowledge-based semantic image model that provides a mechanism for accessing and processing spatial, evolutionary, and temporal indexing techniques is also being developed. A stream is used to represent chronological sets of medical data. A stream-based model is used to increase flexibility in data representation and ease data. A temporal indexing scheme with continuity has been developed using time-warping distance, suffix tree, and categorization to search for similar sequences of images. A knowledge-based spatial temporal query language (KSTL) will be developed which supports operators for approximate matching via feature and content, conceptual terms, and temporal logic predicates. Further, a visual query language is has also been proposed and is being implemented that will accept point-click-and-drag iconic input on the screen that maps into KSTL. The proposed system has been implemented in a testbed to illustrate the timeline visualization, which provides a timeline based presentation of information, consolidates patient information from multiple databases, and incorporates textual, numerical, and imaging data into a unified view. evaluate the functionality of the proposed tasks. The results from this research are applicable to other multimedia information systems as well.

Goals, Objectives, and Targeted Activities

  • Incorporating patient specific knowledge based image segmentation techniques enabling the extraction and detection of anatomical contours and anomalous regions.
  • Developing a temporal sequence indexing technique for multidimensional spatial-temporal data.
  • Developing on multimedia stream data modeling to support the concept of substream and aggregated streams from medical images.

Our research is focused on the retrieval of medical images by feature and content represented by spatial and temporal constructs. We have developed a comprehensive data model methodology and indexing technique for sequence databases for knowledge-based query processing. We are implementing a subset of the visual query language MQuery and of the underlying KSTL language, and a knowledge-based query processing technique in the Knowledge-based Medical Image Databases (KMeD) testbed. We are collaborating and planning with projects at the School of Medicine proof of concept of the applicability and attractiveness of our advances in the actual medical domain.

One important aspect of our content-based image retrieval challenge is to perform automatic image segmentation, so as to then capture this knowledge in our database. More specifically, we want to generate the contour of an object (e.g. tumor) in an image. We will continue collaborating with other colleagues focusing on automated segmentation techniques for our applications. We are now testing segmentation advances to incorporate into our testbed.

Indication of Success

We have progressed as we projected. This progress has earned publication/presentation of our work in several journals, proceedings, and conferences. We have attracted the interest of actual medical users to the point of working closely together through this grant and in larger grants from NIH. This has also led to obtaining the largerecent proposals to NSF and NIH Program Project grant in which our project advances are an important element to build on.

Project Impact

Our data modeling and query processing technology is being transferred to the on-going NIH Program Project Grant at the Radiology Department in the UCLA School of Medicine for application in thoracic oncology and other medical domains, as well as the medical digital library project.. Our project and other NIH grants continue with much synergism, with visible impact towards the goal of a paperless hospital..

We have supported three graduate research assistantships. The project has led to four Master degrees, one three Doctoral degrees and two on-going doctoral students; threefour undergraduate students have also participated on the project via special studies courses.

GPRA Outcome Goals

  1. Discoveries at and across the frontier of science and engineering.

– Techniques for content-based retrieval of medical images.
We have developed content-based techniques to retrieve multimedia medical images such as x-rays, MRI's, etc. Since image features such as tumor sizes and tumor locations cannot be matched exactly, knowledge-based approximation techniques are developed to provide approximate matching. Ranking of the matched images is also provided according to specified measures.
2. Connections between discoveries and their use in service to society.

– Develop clinical image workstation software with content-based retrieval techniques.
Together with the staff at the UCLA School of Medicine, we have applied our advances and developed clinical workstation software for thoracic oncology.
Currently, the workstation software is beingis being tested for future other clinical applications.

 Developing Medical Digital Libraries. We are currently working with the staff at the UCLA School of medicine to transfer our content based retrieval technology to develop a medical digital library and linking relevant documents with patient records on profile.

Project References
  • W. W. Chu, A. F. Cárdenas, and R. K. Taira. KMeD: A Knowledge-based Multimedia Medical Distributed Database System. Information Systems, 20(2): 75-96, 1995.
  • C. C. Hsu, W. W. Chu, and R. K. Taira. A Knowledge-based Approach for Retrieving Images by Content. IEEE Transactions on Knowledge and Data Engineering (TKDE), 8(4): 522-531, 1996.
  • A. A. A. T. Bui, D. R. Aberle, M. F. NcNitt-Gray, J. G. Goldin, L. E. Greaser, and A. F. Cárdenas. User Interface Design for an Integrated Multimedia Medical Information System in Oncology. Radiology 205(P): 745, 1997.

W. W. Chu, A. F. Cárdenas, R. K. Taira, A. A. T. Bui, D. B. Johnson, et al. KMeD: The Knowledge-based Multimedia Medical Distributed Database System. Radiology 205(P): 745, 1997. InfoRad Exhibit RSNA 1997.

  • J. D. N. Dionisio, A. F. Cárdenas, R. B. Lufkin, A. DeSalles, K. L. Black, R. K. Taira, and W. W. Chu. A Multimedia Database System for Thermal Ablation Therapy of Brain Tumors. Journal of Digital Imaging. 10(1):21-26, February 1997.
  • A. A. A. T. Bui, D. R. Aberle, M. F. McNitt-Gray, A. F. Cárdenas, and J. G. Goldin. The Evolution of an Integrated Timeline for Oncology Patient Care. J of American Medical Informatics Association (JAMIA) Annual Proceedings 1998:165-169.
  • W. W. Chu, C. C. Hsu, I. T. Ieong, and R. K. Taira, Content-Based Image Retrieval Using Metadata and Relaxation Techniques. Managing Multimedia Data: Using Metadata to Integrate and Apply Digital Data, edited by Wolfgang Klas and Amit Sheth, McGraw Hill, 1998.
  • W. W. Chu, C. C. Hsu, A. F. Cárdenas, and R. K. Taira. Knowledge-Based Image Retrieval with Spatial and Temporal Constructs. IEEE Transactions on Knowledge and Data Engineering, 1998.
  • J. D. N. Dionisio and A. F. Cárdenas. A Unified Multimedia Data Model for Representing Multimedia, Timelines and Simulation Data. IEEE Transactions on Knowledge and Data Engineering, 1998.
  • J. D. N. Dionisio and A. F. Cárdenas. Advances in Database Query Languages. Book Chapter in Advances in Biomedical Image Databases, S. Wong (editor), Kluwer Academic Press, 1998.

G. Giuffrida, L.Cooper, and W. W. Chu. A Scalable Bottom-Up Data Mining Algorithm for Relational Databases. Tenth International Conference on Scientific and Statistical Database Management (SSDBM), Capri, Italy, 1998.

  • J. A. Goldman, W. W. Chu, D. S. Parker, and R.M. Goldman, Term Domain Distribution Analysis: a Data Mining Tool for Text Databases Methods of Information in Medicine, 1999.
  • D. Johnson and W. W. Chu. Domain Specific Document Retrieval Using n-word Combination Index terms. International Conference on Fusion, San Jose, CA, June 1999.
  • S. H. Park, D. W. Lee, and Wesley W. Chu. Fast Retrieval of Similar Subsequences in Long Sequence Databases, The Third IEEE International Knowledge and Data Engineering Exchange Workshop (KDEX'99), Chicago, IL, November 1999.

W. W. Chu, D. Johnson, and H. Kangarloo. A Medical Digital Library to Support Scenario and User-Tailored Information Retrieval. To appear in IEEE Transactions on Information Technology in Biomedicine, 2000.

 S. H. Park, W. W. Chu, J. H. Yoon, and C. C. Hsu. Efficient Searches for Similar Subsequences of Different Lengths in Sequence Databases. The 16th IEEE International Conference on Data Engineering (ICDE2000), San Diego, CA, February 2000.

  • G. Giuffrida, W. W. Chu, D. M. Hanssens. NOAH: An Algorithm for Mining Classification Rules from Datasets with Large Attribute Space. Proc. 12th Int'l Conf. on Extending Database (EDBT), Konstanz, Germany, March 2000.
  • W. W. Chu, W.W., David . Johnson, HooshangH. Kangarloo. A Medical Digital Library to Support Scenario and User-Tailor Information Retrieval. IEEE Transactions on Information Technology in Biomedicine, Vol.4, No.2, June 2000.
  • SanghyunS. H. Park, Sang-WookS. W. Kim and WesleyW. W Chu. An Index-Based Approach for Similarity Search Supporting Time Warping in Large Sequence Databases. Proc. 17th IEEE Int’l Conf. On Data Engineering (ICDE), Heidelberg, Germany, 2001.
  • W. W. Chu, W.W., , Sang-HyunS. H. Park, Discovering and Matching Elastic Rules from Sequence Databases. Fundementa Informaticae, 2001.

G. Giuffrida, W. W. Chu, D. M. Hanssens. NOAH: An Algorithm for Mining Classification Rules from Datasets with Large Attribute Space. Proc. 12th Int'l Conf. on Extending Database (EDBT), Konstanz, Germany, March 2000.

Area Background

Database systems, knowledge-based systems, multimedia systems, medical systems

Area References

H. K. Huang and R. K. Taira. Infrastructure Design of a Picture Archiving and Communication System. American Journal of Roentgenology, 158:742-749, 1992.

S. Wong (editor), Advances in Biomedical Image Databases, Kluwer Academic Press, 1998.

Potential Related Projects

There are several content-based image retrieval projects (e.g. Faloutsstos, Ghafoor and Yu) in the IDM that could be leveraged on each other’s research work.