Issues in the Multimodal Annotation of Video Games
This week IMS seminar is a guest presentation “Issues in the Multimodal Annotation of Video Games” by Professor Dusan Stamenkovic (Södertörn University), organized by the research group Multimodal Communication at Örebro University.
About the seminar
Drawing on a set of previous multimodal studies of video games (e.g., Bateman et al. 2017; Hawreliak 2018; Stamenković & Wildfeuer 2021; Wildfeuer & Stamenković 2020), this presentation explores how decisions in multimodal annotation can affect the analysis of video games. As complex meaning-making systems, video games blend different modes to make meaning, which can, in turn, make their analysis very demanding. The game genre can also shape the annotation process, as different genres emphasize distinct forms of expression. By systematically categorizing semiotic modes and annotating meaning[1]making elements in canvases and sub-canvases, researchers can examine how different elements contribute to gameplay and narrative, as well as making meaning in general. The concept of canvases and sub-canvases becomes particularly relevant here, as game environments serve as overarching multimodal spaces, the main one being the most dominant game canvas, while individual interface layers and embedded elements function as sub-canvases within this dynamic system. The presentation will involve several examples of canvases from different video games. The main canvas serves as the primary space where gameplay unfolds and encompasses camera techniques, character design, environmental elements, interactive objects, in-world text, game-integrated elements, and cutscenes. Sub-canvases, such as user interface (UI) elements, provide essential gameplay feedback through health bars, mini-maps, inventory screens, and mission logs. Along with these, we can also identify parallel canvases, which extend the experience through auditory elements like diegetic and non-diegetic sounds and haptic feedback. By refining multimodal annotation techniques, researchers can gain a better insight into how games construct meaning, influence player behaviour, and evolve across genres and platforms, which improves our understanding of video games as a dynamic form of communication and expression.