The psychology of multimedia databases

Multimedia information retrieval in digital libraries is a difficult task for computers in general. Humans on the other hand are experts in perception, concept representation, knowledge organization and memory retrieval. Cognitive psychology and science describe how cognition works in humans, but can offer valuable clues to information retrieval researchers as well. Cognitive psychologists view the human mind as a general-purpose symbol-processing system that interacts with the world. A multimedia information retrieval system can also be regarded as a symbol-processing system that interacts with the environment. Its underlying information retrieval model can be seen as a cognitive framework that describes how the various aspects of cognition are related to each other. In this paper we describe the design and implementation of a combined text/image retrieval system (as an example of a multimedia retrieval system) that is inspired by cognitive theories such as Paivio's dual coding theory and Marr's theory of perception. User interaction and an automatically created thesaurus that maps text concepts and internal image concept representations, generated by various feature extraction algorithms, improve the query formulation process of the image retrieval system. Unlike most "multimedia databases" found in literature, this image retrieval system uses the the functionality provided by an extensible multimedia DBMS that itself is part of an open distributed environment.