Transmediation is the transfer and exchange of communicative content between different kinds of media, such as written text, moving images, gestures, songs, and photographs. Transmediation is a cross-cultural and cross-historical phenomenon, common in all kinds of communication: holiday snapshots are described by speech, interviews are written down, novels are made into movies, news reports are translated into sign language, and scientific diagrams are explained with the aid of gestures. As transmediation cannot be achieved without transformation of the communicative content, the understanding of its complex mechanisms appears to be a highly important matter with far-reaching significance for communication at large. Even though the phenomenon is well known, its intricate nature has not been researched properly.
The two main theoretical fields of the research group, multimodality and intermediality, are well suited for generating new knowledge about transmediation. Multimodality is the study of the multi-faceted characteristics of media products (which may, for example, be both visual and auditory or both verbal and pictorial). Intermediality is the study of different media types and their historical and theoretical interrelations. One of our strengths is that we have developed a conceptual framework that goes beyond the obvious (and not always illuminating) categories, such as text and image, and instead makes more subtle and precise distinctions concerning materialities, sensory qualities, and so forth, which allows us to analyze multimodal media properties in a refined way.
The central questions of the research program are these: In which ways are communicative content transformed when it is transferred among different kinds of media? Which media-specific factors can be isolated as determinant for these transformations? How does context change the outcome of the transmediation? The theoretical aim is to explore the different multimodal characteristics of media in order to better understand how communicative content can or cannot be transferred across media borders. The empirical aim is to make systematic analyses of transmediation in a selection of actual communicative settings and to systematically compare the outcome of the analyses of these different empirical areas, observing in particular the essential similarities and differences in media traits and contextual factors.
One of the most pertinent differences to be found in the empirical material is that transmediation, however problematic, is often unavoidable and sometimes highly desirable. Whereas in many situations, such as trials, it is vital for both individuals and society at large to be able to transfer communicative content across media borders as accurately as possible, in other situations, such as aesthetic creation, the aim of transmediation is rather to broaden the understanding of certain occurrences. The transformative aspects of transmediation may thus be perceived as both limitations and possibilities.
We demarcate the field of investigation in two crucial ways. First, four spheres of empirical investigation have been chosen for the program, representing a selection of vital parts of human communication: (1) representations of the Anthropocene, (2) science education, (3) communication involving sensorially impaired persons, and (4) trials for criminal cases. These areas are clearly demarcated and yet diverse enough to guarantee a broad scope of communicative aims and functions. Second, we have chosen to focus our investigations of these empirical areas on two kinds of communicative content. We will investigate how both narratives and truth claims can be transferred among dissimilar media types. We believe that these two kinds of communicative content are important to explore because of their human and societal relevance.
Our primary research method is to systematically compare the results from the empirical investigations to each other with the aim of better understanding how certain communicative content can or cannot be transmediated in dissimilar empirical settings because of the diverse multimodal traits of different media and their different contexts. These comparisons of how narratives and truth claims are transmediated by different media types in a carefully selected range of communicative settings will make it possible to model the theoretical understanding of transmediation with greater accuracy.
An oral testimony may be written down. Yet the reader of the text will only capture parts of the essence of what was initially communicated through voice qualities, gestures, and facial expressions. Similarly, a blind person who visits an art museum may often experience substantial parts of an exhibition with the aid of audio descriptions, tactile representations of visual objects, and explanations written in Braille. Yet this is not the same as seeing the exhibition. Why? Because all media have their limitations. It is equally correct, however, that all media have their possibilities. Science teachers who cannot verbally explain the nature of an atom may use a visual diagram. By the same token, the implications of climate changes described in written articles are better grasped by many people if possible future scenarios are dramatized in a motion picture.
Different kinds of media are thus partly similar and partly dissimilar. Media, according to our definition, are those entities that make communication among humans possible: speech, gestures, blogs, news, advertisements, literature, music, film, and so forth. Although these entities are always material in some way, whether they are corporeal or external to the human body, their communicative abilities are determined by their potential to create a mental response. Media are cultural products that cannot be fully understood unless both their specific physical traits and their capacity to affect cognition are considered.
One of the principal capabilities of the interaction between media products and the human mind is that the outcome of one communicative act can be produced again through another kind of medium. A photograph in the newspaper may be described by spoken words; a musical score may be performed by a musician; the oral statements of a witness may be written down; a theatrical play may be adapted to a movie; the gist of a scientific account may be rendered into a visual diagram. We call these processes of transfer and transformation transmediation (Elleström, Media Transformation, Palgrave Macmillan 2014). In other words, transmediation is at hand when one tries to recreate what one takes to be vital content of one communicative process through a communicative process of a different kind. This extraordinarily complex process of transfer and sometimes exchange of communicative content is the focal point of the research program.
We argue that there are several important reasons for trying to find ways of investigating the basic mechanisms of transmediation. Often it is vital for both individuals and society at large to be able to transfer communicative content over media borders as accurately as possible. Education, for instance, involves a broad range of media from speech and gestures to written texts and images of various kinds. Although good teaching is not characterized by simple transmission of fixed knowledge entities from teacher to pupil, it is certainly undermined when vital communicative content is distorted in transmediation. By the same token, news reports in television and newspapers and on the Internet are characterized by multifaceted transmediations that we expect to affect the distribution of realities as little as possible. When trying to establish the guilt or innocence of persons who are prosecuted of criminal acts, the inescapable biases of transmediation in interrogations and legal proceedings are a major problem; yet it is clearly necessary to use a variety of media types –transcriptions of oral statements, audiovisual recordings, and photographs – in the legal context. Another example of communicative contexts where accuracy is demanded is facilities for persons with sensory impairments. Building a society with equal rights involves trying to make information and entertainment accessible to those with, for instance, visual or hearing impairments. The only way to get around these obstacles is to transmediate.
In other circumstances, the function of transmediation is to broaden the comprehension of certain occurrences. Different kinds of media have dissimilar affordances, and transmediation may be a method of finding the best and most natural way of representing things that escape simple explanations. Written or oral language is not necessarily always the best tool for research and teaching; it must often be supplemented, for example, with diagrams and images. The transfer of communicative content among different kinds of media may thus enhance rather than distort communication. This aspect of transmediation is perhaps most clearly seen in artistic activities, where communicative content, such as stories, themes, and motifs, is constantly transmediated among different media types as written literature, dance, movies, music, performance, and photography. In such a context, where the aim is to offer an accumulation of multifaceted representations of the world and human experiences, the reshaping nature of transmediation is generally an asset rather than an obstacle.
The different affordances of dissimilar media types are a fundamental condition for human communication in general. Yet, transmediation remains largely unexplored. The questions to be tackled in the research program concern vital communicative issues: In which ways are communicative content transformed when transferred among different kinds of media? Which media-specific factors can be isolated as determinant for these transformations? To which extent is it possible to draw reliable conclusions (or even to make predictions) regarding the communicative capacity of different kinds of media? In line with preliminary results, these questions can be condensed into one: In which ways do the generally multimodal basic qualities of different kinds of media set the limits for and create possibilities for transfer of communicative content among dissimilar media types?
This fundamental question must be combined with a complementary approach: How does context affect the outcome of transmediation? Because context – culture, interpretive communities, and ideologies – is virtually limitless, such a blend of “internal” and “external” conditions for communication may create insurmountable barriers to transmediation. Therefore, the question of context will be delimited to what is relevant for the specific studies to be performed within the program.
The research program will provide knowledge of essential media properties that, together with contextual factors, give a more complete understanding of transmediation than has been possible hitherto. The interconnected aims of the research program can be summed up as theoretical, empirical, and practical.
Our theoretical aim is to explore the different multimodal characteristics of media in order to better understand how communicative content can or cannot be transferred across media borders. The goal is to develop, refine, and integrate communicative, multimodal, and intermedial concepts and terms and ultimately to form theoretical models that can help us better analyze and, to a certain extent, predict what happens when communicative content is transmediated.
Our empirical aim is to make systematic analyses of transmediation in a selection of actual communicative settings. The analyses will focus both on basic material, spatiotemporal, sensorial, and semiotic media properties and on contextual factors. We will compare the outcome of the analyses of these different empirical areas, observing in particular the essential similarities and differences in media traits and contextual factors. Our goal is to gain more precise knowledge of problems and possibilities of transmediation in our selection of specific communicative situations that differ widely in character and purpose but nevertheless are central for our human societies. Additionally, we want to detect crucial common denominators in the diverse processes of transmediation, which is expected to give substantial feedback to our analytical tools and help us to develop the theoretical models.
Our practical aim is to pinpoint ways of applying the knowledge we gain in order to optimize transmediation, if possible, with the goal of facilitating communication based on transmediation (meaning, for instance, better education, public service, and legal procedures). We acknowledge that the research program can only expect to reach this ambitious goal in part. Nevertheless, the research program may well lead to deeper knowledge of how different kinds of media can be used either to increase the degree of reliable communication or to increase the degree of creative potential in communication.
Because transmediation is a vast area, we delimit the field of investigation in two crucial ways. First, four spheres of empirical investigation have been chosen for the program, representing a selection of vital parts of human communication dealing with global environmental changes, schooling, civil rights, and justice: (1) representations of the Anthropocene, (2) science education (3) communication involving sensorially impaired persons, and (4) trials for criminal cases. These four areas are clearly demarcated and yet diverse enough to guarantee a broad scope of communicative aims and functions. Investigating a variety of widely differing empirical fields makes it possible to highlight both the fundamental relevance of different multimodal media constellations, which are more or less common for the different areas, and the importance of contextual factors, which differ widely.
Second, we have chosen to focus our investigations of these empirical areas on two kinds of communicative content. We understand this communicative content as transmedial: it can actually be transferred among different types of media to a certain extent. We will thus investigate how both narratives and truth claims can be transferred among dissimilar media types. We believe that these two kinds of communicative content, having considerable magnitude, are important to explore because of their human and societal relevance. Narratives, understood in a broad sense as represented events that are sequentially related to each other in a meaningful way, are present not only in language but in most areas of human communication. The notion of truth claims is circumscribed in order to catch the crucial idea that narratives and other representations can claim to correspond to actual states of the world, as far as those can be established.
Methodologically, our research will proceed through three steps. Starting with theoretical tools designed for scrutinizing transmediation, the first step is to analyze crucial processes of transmediation in the selected empirical fields, emphasizing the ways the communicative content is transformed. The second step is to compare the results from the empirical investigations with each other in order to better understand how certain communicative content can or cannot be transmediated in dissimilar empirical settings, depending on both the diverse multimodal traits of media and their different contexts. The selection of a limited amount of communicative content to focus on is central for obtaining a manageable and relevant comparative element in the research program; it guarantees that on the one hand, the empirical material can be very varied and representative for important sectors of the society, and that on the other, the comparison can be strict. The third methodological step is to broaden the results of analyses and comparisons of analyses in ways that allow us to deduce more wide-ranging mechanisms of and conditions for transmediation.
This kind of empirical research can lay the groundwork for better theorizing. We may return from time to time to step one and analyze more complex processes of transmediation with the aid of refined theoretical tools that will allow us to perform more sophisticated comparisons in step two, leading to even greater theoretical progress. Our three steps are thus to be understood as parts of a research spiral rather than a set of clearly linear advances. Such a spiral makes it possible to realize both rich cross-fertilization among investigations of a range of selected empirical areas and grounds for more useful theories. This is achievable only within the well-controlled frames of a major research program.
The research results will be published almost exclusively in English or other major languages and a substantial part of the publications will be co-authored by researchers from different disciplines. We aim to reach a broad audience through articles in highly ranked international peer-reviewed journals. We will furthermore act as guest editors of thematic issues of journals and issue more specialized monographs and edited books on selected subjects.
Linnæus University Centre for Intermedial and Multimodal Studies
The institutional framework for the research program is provided by the Linnæus University Centre for Intermedial and Multimodal Studies (IMS), one of the university’s strategic centers of excellence. IMS is led by Lars Elleström, the principal investigator of the proposed program, and a majority of the IMS members are part of the program, which is complemented by a handful of researchers from other universities with specific skills required for the projects of the program (Linköping University, Umeå University, and the Norwegian University of Science and Technology, Trondheim). We continuously host international guest researchers and professors to support our research (most of them are chosen from our expert panel, consisting of some 30 prominent international scholars). Two of these will act as consultants specifically for the program: professors Thomas Leitch (University of Delaware) and Maria Nikolajeva (Cambridge University).
Our activities have attracted considerable attention in the academic world. In the recent publication Handbook of Intermediality, three successful centers for intermediality research in the world are specifically mentioned, one of them being IMS (Rippl 2015: 2). This reputation includes producing high-quality research, which is acknowledged in another recent publication emphasizing “the discipline-defining work in intermedial studies [- - -] of the group around Lars Elleström” (Harrow 2013: 4, 13).