In this article we claim that the linguistic-centered view within hypermedia systems needs refinement through a semiotic-based approach before real interoperation between media can be achieved. We discuss the problems of visual signification for images and video in dynamic systems, in which users can access visual material in a non-linear fashion. We describe how semiotics can help overcome such problems, by allowing descriptions of the material on both denotative and connotative levels. Finally we propose an architecture for a dynamic semiotic-aware hypermedia system.

Taylor&Francis
New Review of Hypermedia and Multimedia
Human-Centered Data Analytics

Nack, F., & Hardman, L. (2001). Denotative and Connotative Semantics in Hypermedia: Proposal for a Semiotic-Aware Architecture. New Review of Hypermedia and Multimedia, 7, 7–37.