The Modalities of Media: A Model for Understanding Intermedial Relations

Summary

Ellestrom proposes a systematic model for understanding what media are and how they relate to each other. He argues that all media can be described in terms of four modalities: (1) the material modality – the physical or virtual interface through which the medium manifests (e.g., paper, screen, sound waves, LEDs); (2) the sensorial modality – the sense or senses through which the medium is perceived (visual, auditory, tactile, etc.); (3) the spatiotemporal modality – the medium’s relationship to space and time (static vs temporal, 2D vs 3D, sequential vs simultaneous); (4) the semiotic modality – the kinds of meaning-making the medium employs (iconic, indexical, symbolic/conventional). Every medium involves all four modalities, but media differ in which modes within each modality they activate. Ellestrom also distinguishes three levels of media: basic media (defined purely by their modal configuration), qualified media (basic media as recognised by historical and social convention – e.g., “cinema,” “literature”), and technical media (the physical devices that realise media – e.g., a projector, a book, an ESP32). Intermediality, in this framework, is “a bridge between medial differences that is founded on medial similarities” – media can be compared and their border crossings analysed by examining which modalities they share and where they differ.

Key Contributions

• Provides a rigorous, four-part framework for analysing any medium: material, sensorial, spatiotemporal, semiotic
• Distinguishes basic media (abstract modal configurations), qualified media (culturally recognised forms), and technical media (physical devices) – three analytically separate levels
• Bridges the gap between intermedial studies (focused on media borders and crossings) and multimodal studies (focused on how multiple semiotic modes combine within a text)
• Defines intermediality through modalities: media are intermedial when they share some modal characteristics and differ in others – the sharing enables the crossing
• Moves beyond purely theoretical typologies (like Rajewsky’s) to an operational framework that can be applied to any medium, including new and experimental ones

Methods

• Theoretical model construction drawing on semiotics (Peirce), media theory, and multimodal discourse analysis
• Systematic taxonomy: defines each modality, its possible modes, and how they combine
• Comparative analysis of existing media (literature, music, cinema, theatre, painting) using the four-modality model to demonstrate its explanatory power
• Engagement with prior intermediality frameworks (Rajewsky, Bolter & Grusin, Wolf) to show how the modality model subsumes and extends them
• Integration of Kantian epistemology (space and time as a priori intuitions informing the spatiotemporal modality)

Connections to Active Projects

AutoTonnetz: Ellestrom’s model is directly applicable as an analytical framework for describing AutoTonnetz as a medium.

• Material modality: ESP32-S3 microcontroller, WS2812B addressable LEDs, MIDI synthesiser (downstream), PCB substrate, 3D-printed housing. The technical medium is the physical AutoTonnetz unit.
• Sensorial modality: Visual (LED colour and pattern) and auditory (MIDI-synthesised sound). Potentially tactile (physical interaction with the unit). This bimodal sensorial configuration is the defining characteristic of the instrument.
• Spatiotemporal modality: Temporal with fixed sequentiality (cellular automata evolve forward in discrete time steps). Spatial in the arrangement of the hexagonal Tonnetz grid (2D but mapping a toroidal mathematical space). The spatial layout of LEDs is meaningful – adjacent LEDs represent harmonically related pitches.
• Semiotic modality: Iconic (LED colours may iconically represent pitch register – low=red, high=blue). Conventional (the Tonnetz geometry encodes music-theoretic conventions – fifths, thirds, triads). Indexical (LED activation indexes the current CA state – it points to “what is happening now”).

This analysis helps articulate what AutoTonnetz IS at a theoretical level, which is essential for the OrganisedSound paper’s literature review.

Organised Sound: The four-modality model provides the theoretical vocabulary for the perception study’s framework section. When raters evaluate coherence, they are assessing whether the sensorial modality (what they hear and see) forms a unified percept – essentially whether the auditory and visual modes within the sensorial modality are experienced as belonging to the same medium or as separate media that happen to co-occur. The spatiotemporal modality matters because synchrony is primarily a spatiotemporal alignment between the auditory and visual streams. The semiotic modality matters because if the LED patterns carry readable meaning (e.g., recognisable triadic shapes), this may enhance perceived coherence by adding a semiotic bridge between the visual and auditory content. Ellestrom’s framework helps the paper move beyond vague claims about “audiovisual unity” to precise descriptions of which modalities are aligned and which are not.

Suggested Reading

• Rajewsky (2005) – Intermediality, Intertextuality and Remediation (the typological approach Ellestrom extends)
• Kress & van Leeuwen (2001) – Multimodal Discourse (the multimodal studies tradition Ellestrom bridges to)
• Peirce (1931-58) – Collected Papers (the semiotic framework underlying the semiotic modality)
• Bolter & Grusin (1999) – Remediation (the remediation concept viewed through the modality lens)
• Ellestrom (2014) – Media Transformation (Ellestrom’s own later extension of this model)