Multimodality as an Approach to Model Digital Publications

Niels-Oliver Walkowski

Comunicación breve
Publicación y diffusion digital

In my presentation I will introduce a new approach to research on digital publications, especially in the context of e-Science and Digital Humanities. The content of this presentation reflects an ongoing research project which aims at defining an OAI-ORE based model for multimodal digital publications.

Problem: The State of Digital Publication Forms

Research is discursive practice. This claim is applicable in two ways: first, research is a social activity and second, knowledge is a phenomena which resides and is created in discourse. Publications are objects that used to support well both aspects because they partly suspended space and time in communication. Globally, this quality of publications improved the conditions for knowledge dissemination. Structurally, it improved the conditions for the creation of more complex meaning and reliable knowledge. Publications take discourse out of the flow of space and time and re-materialize it into an object which moves across space and time instead. However, materiality dramatically changed shape in digital world and there are activities allover to change the shape of publications likewise.

Approaches like Enhanced Publications (Sierman, Schmidt, and Ludwig 2009; Holl 2012), Scientific Publication Packages (Hunter 2008) or Semantic Publications (David Shotton 2009; D. Shotton et al. 2009; Bourne et al. 2012), Nano Publications (Mons and Velterop 2009; Groth, Gibson, and Velterop 2010) among many others propose to make publications modular, formally annotated, distributed, multi-media, including data and software, a living entity, automatically reproducible and much more. In contrast to these abundant activities which started at least in 1995 with the ACM Publishing Plan (Denning and Rous 1995) its success seems quiet limited. This is especially true for the field of the Humanities. The PDF is still the most common digital publishing strategy (David Shotton 2009). Where publications are more sophisticated its richness is bound to the portal where it is produced or presented, like it is the case for Scalar (ANVC), myExperiment (“MyExperiment”) or Authorea (“Authorea”).

Digital publications are often ideologically bound to certain domains and to the End of Theory debate (Anderson 2008). Contrasting examples from the field of DH (ANVC; “Journal of Digital Humanities”) explore new possibilities of multimedia writing but do not offer a generalizable model. Hence, new publications lack sustainability in terms of concepts and preservation. Accordingly, there are many models and few tools for their production. Their potentials remain unexploited and the reality of digital discourse that takes place is rarely objectifiable.

Approach: Multimodal Analysis and Systemic-Functional-Grammar

The benefit of publications is twofold as I claimed at the beginning. However, digital publications tend to focus on the communication and dissemination aspect of publications and neglect or take for granted aspects of meaning making and knowledge – the ideological background. Naturally, these aspects are of special importance for the field of Humanities which study and communicate in distinct types of meaning material. It is this part which suggest changes in the way knowledge and thus publications function in society. The question of digital publications is not how to do old things better due to digital technology. The question is how to do things good in a changing world of digital technologies. Thus, what is missing in the debate around digital publications is a theoretical framework which is capable to contribute both to the sociocultural re-evaluation of knowledge and publications as well as to the formalization and implementations of digital publications. In this respect I propose the field of and Multimodal Analysis.

Multimodal Analysis is a research field which grew out of Social Semiotics, a branch in linguistics initiated by Michael Halliday (1978; 1985). One of the main claims of Social Semiotics stresses that signs are socially constructed and thus that there is no categorical realm of signs, for example language. Multimodal Analysis is following this claim by investigating semiotic and grammatical structures in linguistic and non-linguistic resources like: visuals, audios, colour, space, rhythm and so forth. Furthermore, Multimodal Analysis investigates how these resources relate to each other grammatically. In doing so Multimodal Analysis offers sophisticated insights on structural and semantic aspects of discourse beyond language. Additionally, the historical orientation in a social approach to language makes it possible to evaluate how the configuration of signs and grammar as a condition for meaning and knowledge change over space and time.

By stretching Multimodal Analysis it is possible to make two claims:

  1. computation is one but not the only theme in and for digital publications as many existing models suggest. Instead, the omnipresence of computation provokes a proliferation of strategies (interfaces) to create knowledge;
  2. the dominant function of publications in a digital world will shift from dissemination to the creation of complex knowledge which is hardly organisable outside of such a framework like publications.

Benefit: How Multimodal Analysis improves the Conditions for Digital Publications

The design of digital publications may benefit from Multimodal Analysis in several ways: Its grammatical and functional contents are very useful to derive semantics for a formal model of digital publications. The sociocultural perspective is suitable to approach digital publications in a sustainable way. This will promise to improve conditions for long term preservation and for authoring tools. Multimodal Analysis and the realization of multimodal digital publications will help to advance digital literacies, thus the capacities to write and read multimodal texts in a publicly understandable that means rule oriented way. Finally, it will support the adoption of digital technologies and computational methods in the Humanities because they are contextualized in a way which is integrative and not exclusive.

In the presentation I will give a short overview about examples of digital publications for the purpose to expose its common problems. I will substantiate the claim that multimodal analysis is a promising approach to frame modeling tasks for digital publications. Finally, I will indicate how certain high-level concepts of multimodal analysis can be transformed to fit the needs of modeling digital publications. Publications which are capable to include data, code and media of varying type and to organize them in distinct ways.


Anderson, Chris. 2008. “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete.” Wired Magazine, no. 16.07.

ANVC. “Scalar.”


Bourne, P. E., T. W. Clark, R. Dale, A. de Waard, I. Herman, E. H. Hovy, and D. Shotton. 2012. “Improving The Future of Research Communications and E-Scholarship.” 

Denning, Peter J., and Bernard Rous. 1995. “The ACM Electronic Publishing Plan.” Communications of the ACM 38 (4): 97–109. DOI: 10.1145/205323.205348.

Groth, P., A. Gibson, and J. Velterop. 2010. “The Anatomy of a Nanopublication.” Information Services and Use 30 (1): 51–56.

Halliday, Michael. 1978. Language as Social Semiotic: The Social Interpretation of Language and Meaning. Baltimore: University Park Press.

———. 1985. “Part A.” In Language, Context, and Text: Aspects of Language in a Social-Semiotic Perspective, by Michael Halliday and Ruqaiya Hasan, 1–49. Geelong, Victoria: Deakin University Press.

Holl, Andras. 2012. “Information Bulletin on Variable Stars - Rich Content and Novel Services for an Enhanced Publication.” D-Lib Magazine 18 (5/6). DOI: 10.1045/may2012-holl.

Hunter, J. 2008. “Scientific Publication Packages–A Selective Approach to the Communication and Archival of Scientific Output.” International Journal of Digital Curation 1 (1): 33–52.

“Journal of Digital Humanities.”

Mons, B., and J. Velterop. 2009. “Nano-Publication in the E-Science Era.” In Workshop on Semantic Web Applications in Scientific Discourse (SWASD 2009).


Shotton, D., K. Portwin, G. Klyne, and A. Miles. 2009. “Adventures in Semantic Publishing: Exemplar Semantic Enhancement of a Research Article.”

Shotton, David. 2009. “Semantic Publishing: The Coming Revolution in Scientific Journal Publishing.” Learned Publishing 22 (2): 85–94.

Sierman, Barbara, Birgit Schmidt, and Jens Ludwig. 2009. Enhanced Publications: Linking Publications and Research Data in Digital Repositories. Surf / EU-Driver. Amsterdam: Amsterdam University Press.