A Scientific Foundation of Simulation Games for the Analysis and Design of Complex Systems.

Background. The use of simulation games for complex systems analysis and design has been acknowledged about 50 years ago. However, articles do not combine all salient factors for successful simulation games, and often stem from a clear view of one particular field of science only. With combining multiple disciplines, connect analysis and design as well as research and practice, we provide deep insights in design and use of simulation games. Aim. This article analyzes the design and evaluation process of a variety of game-based projects and activities, using existing scientific concepts and approaches, in order to establish games as a valid research tool. Our focus lies on the approach towards the use of games as design instrument; using them as an intervention in a larger, complex context, in order to design this context. With our contribution, we aim at providing insights and recommendations on the design and use of games as valid research tools, the limitations of this use, possible pitfalls, but also best practices. Method. We carried out a literature review of related work to identify the most important scientific concepts related to our approach of game design. Further use of combined quantitative and qualitative case study analyses highlights the design process and results of our own game studies. Results. The analyses yielded a consolidated conceptualization of simulation games as research instruments in complex systems analysis and design. The results also include methods for the evaluation of simulation games, additional evaluation methods, and limitations to use simulation games as research instruments. Conclusions. We propose guidelines for using simulation games as research instruments that may be of value to practitioners and scientists alike. Recommendation. We recommend practitioners and scientists to apply the guidelines presented here in their efforts to analyze and design complex systems.

Introduction

Due to increasing dynamics, interconnectedness and the role of technology in today’s societies, more components in people’s lives and work are getting more complex (Heylighen, 2002). Societies have grown in complexity, because of the existence of different functional systems and their interconnections (Dalpiaz, Giorgini, & Mylopoulos, 2013; Davis, Challenger, Jayewardene, & Clegg, 2014; Von Bertalanffy, 1968, 1975). Such (complex) systems are inherently bound to concepts such as discussed by sociologist Niklas Luhmann, referring to the term social system, like self-organization, adaptation, feedback mechanisms and emergence (Luhmann, 1995). Actors shape social systems with their horizontal and vertical interactions and communications (Klabbers, 2018). A complex system can very broadly be defined as “one in which there are multiple interactions between many different elements of the system” (Ridolfi, Mooij, & Corpino, 2012, p. 39). Due to the characteristics as emergence, self-organization and adaptation (Holland, 1995), complex systems are difficult both to analyze and to design.

Our research with and on games focuses on the design, use and evaluation of simulation games for complex systems analysis and design. The nature of systems is leading for our understanding of games as a means, such as introduced by Klabbers (2018). We see gaming as a powerful approach towards the understanding of highly integrated, large-scale systems with many actors dealing with deep uncertainties. Simulation games or simulation gaming already has a long history and proven value in education, research and policy making (Duke & Geurts, 2004; Klabbers, 2006, 2009). The last decades many new game types have emerged, such as computer games, video games, mobile games and new applications and application fields such as Virtual and Augmented Reality (VR/AR) together with new terminology like persuasive games, applied games, games for health and many more entering the field. Therefore, it is important to define simulation games as used in this article. In our research, simulation games can be described as experimental, rule-based, interactive environments, where players learn by taking actions and by experiencing their effects through feedback mechanisms that are deliberately built into and around the game (Mayer, 2009 p. 825). In the language of sociologist Anthony Giddens, games represent a duality of structure as players of games constitute the emerging social organization, while at the same time the game is the very medium of its constitution (Giddens, 1993; Klabbers, 2018).

For the purpose of using games as a means for complex systems analysis and design, we not only need to understand how such games have to be used, but also how they should be designed. Numerous design principles for the development of simulation games exist, e.g. Duke and Geurts (2004), or Harteveld, Guimarães, Mayer, and Bidarra (2010). However, design principles for understanding complex systems in e.g. complex, simulated environments have been rather underexplored. Even for traditional board games, boundaries of game design, implementation, and evaluation are not yet described in a structured way. Further, we need to understand how simulation games work and how they have to be embedded for the analyses of complexity. This includes an understanding of how simulation games support the design process of complex systems. In order to frame our games, we use a couple of theoretical concepts as lenses to evaluate the contribution of simulation games, such as complex adaptive systems (Holland, 1995), situational awareness (Endsley, 1995), and socio-technical systems (De Bruijn & Herder, 2009). The purpose of this article is to analyze existing research projects based on simulation games for complex systems design, in order to develop recommendations for researchers and practitioners in this field. The recommendations will especially contribute to embed simulation games in scientific foundations. Furthermore, we pay attention to and strengthen the use of validated research instruments to improve the knowledge of simulation games for analyzing and designing complex systems.

We will discuss our approach based on recent research projects in various domains such as safety and security, transport and logistics, and health, to illustrate how simulation games can be used as instruments for analysis and design of complex systems. Our interest in these societal domains aims to support actors dealing with complex systems and problems. With this approach, we are able to connect communities of observers (researchers) with the communities of practice (in different domains). Coming from a strong design science oriented background and science understanding, we aim at influencing thought and action (Klabbers, 2009). In this sense, simulation games are a tool clearly focused on creating situational awareness for and insights in complex systems in order to enable problem solving.

Purpose of the Article

In this article, we address the following research question:

How can the use of scientific concepts and scientific foundations strengthen the design, use and effectiveness of simulation games for the purpose of analyzing and designing complex systems beyond the use of games as isolated research tools?

The main research question can be decomposed to sub questions at three levels of inquiry, as proposed by Klabbers (2018):

Philosophical level: How do we view the phenomenon of simulation games? What knowledge is involved in the design, use, and evaluation of games?

Scientific level: How is knowledge used and constructed in game-related activities (by designers/developers, players, facilitators, and researchers) beyond the fun and engaging element? How can our simulation game experience contribute to the development of the scientific foundations?

Practical (application) level: What evidence is there to support the viability of simulation games as useful research instruments for the analysis and design of complex systems?

The outline of the paper is as follows. In the next section, we further elaborate on our view of simulation games for complex systems, which is going beyond games as an engaging activity. We show that simulation games can be used as a scientific tool to contribute to scientific questions as well as solving societal issues. After that we introduce the scientific toolbox for designing, using and evaluating games for complex systems. This toolbox is filled with design philosophies and methodologies, scientific concepts and validated instruments. In the case chapters, several examples from the fields of transport and logistics, health and security are introduced. It is explained how the toolbox is used and improved. Finally, general conclusions will be drawn about the experiences and contributions to game science.

Simulation Games for Complex Systems Analysis and Design

This section refers to the philosophical level of our approach, namely the underlying concepts leading to our conceptualization of simulation games: How do we view the phenomenon of simulation games? What knowledge is involved in the design, use, and evaluation of games?

Simulation games and complex systems have been connected for a long time already. In 1974, Duke already described the value of gaming for communication between multiple people and the role of simulating behavior of systems consisting of many interrelated elements (Duke, 1974; Duke & Geurts, 2004; Klabbers, 2006). Communication, cooperation and participation are basic skills needed in todays’ complex working environments. Training activities can be used to introduce newcomers to tasks and working environments for their day-to-day work. Alternatively, they can be designed to support employees in adapting their work behavior to new situations, which is critical in modern, complex work environments. Games foster communication between players, which makes them a valuable method for supporting social learning processes (Van der Meer & Geurts, 1995).

Further, simulation games are representations of real-world complex systems, which can take into account the network complexity of multiple stakeholders with a variety of values, means and resources in a simulated physical or virtual reality (Bekebrede, 2010; Klabbers, 2006). In other words, simulation games represent social systems, and are interrelated with the social system around them (Klabbers, 2006, 2018). Although there is a boundary that separates the game from the real world, the so-called magic circle (Klabbers, 2006), this boundary is open to transfers between game and reality. This concept illustrates one more similarity of games and social systems, as conceptualized by Luhmann (1995). In his work, Luhmann describes the systems of modern society as closed, self-referential ones, which means that they constitute themselves solely out of their own elements, with no means for direct interaction to other systems. Yet, other systems can influence a system in the form of an irritation. In games, such transfers occur through the player’s background that influences gameplay, and through the process of acquiring knowledge and skills from the gameplay that can be useful in a specific real context (Copier, 2007). Simulation games support knowledge transfer from the virtual to the real world, particularly when they are designed with so-called there-reality, which involves high-level, physical fidelity (Chalmers & Debattista, 2009). This capacity of games opens the opportunity to analyze the behavior of individual actors within a social network and in relation with changes in the physical environment. Consequently, the variety of simulation games in relation to complex systems is enormous, varying from teaching about complexity to understanding system behavior, testing relations and rules between system elements or as an experiential environment to explore a variety of system designs (Bekebrede, 2010; Klabbers, 2006; Mayer, 2009).

Games are active media requiring constant physical input by the gamer: action, doing, pressing buttons, controlling, and so on (Aarseth, 1997). Because of this, a game must be realistic in doing and in action. Since the primary phenomenological reality of games is that of action (rather than that of perceiving as in media like cinema or television), the gamer has a more intimate relationship with the medium itself, and therefore with the deployment of realism (Galloway, 2004). As Klabbers (2018) discusses, the player is not only a subject, but also on agent or actor, who demonstrates purposeful behavior, based on a certain set of skills and knowledge.

These characteristics of simulation games; the ability to foster multilogue communication (Duke, 1974), a (realistic) representation of the social and physical system, and the need for active involvement, provide an instrument going beyond learning and engagement. These characteristics give room for active experimentation with the complexity around us. In comparison to pure computer simulations, games can provide insights and knowledge for understanding the complexity of systems as they directly take the human factor of social complexity into account. As Klabbers (2018) states, mathematical models (and simulations) potentially provide a misleading feeling of realism about the social world. Simulations also include the risk of providing a naïve, numeric view on a highly dynamic and uncertain social reality. Greenblat and Duke (1975) on the other hand argue that the design of a simulation game itself is a systematic translation of understandings into an operating model. The subsequent examination of such model through observation of play can lead to a refining of theoretical formulations and consequently to a higher level of social scientific understanding. Simulation games thus have the capacity of being scientifically valid, but still require a scientific foundation of design and evaluation.

Games have a strong potential for being used as research instrument, as they are engaging in nature, can increase participation, often come as computed environments that automatically allow for data collection, and enable manipulation and control through a researcher (Deterding et al., 2015). Games enable researchers to replicate the same experimental set-up many times, without the need of change in the experimental design or environment. Games can be designed using a participatory design approach, including the target group already from the beginning of the cycle of designing and using games as research instruments (Lukosch, van Ruijven, & Verbraeck, 2012). Despite these obvious benefits of simulation games, no best practices or established frameworks for the use of games as research tool exist yet. Limitations of the use of games as research instruments are also related to the large number of variables, that are hardly to be controlled in a dynamic situation of game play, and the limited possibilities to play a game many times (Geurts & Vennix, 1989).

On the philosophical level of game science, we can thus conclude that simulation games can be used as research tools, to support the analysis and design of complex systems. They are especially useful in representing dynamic systems and situations, including the social or human factor. Moreover, simulation games represent a tool for learning and discussing complexity and system behavior.

The Scientific Toolbox of Simulation Gaming for Complex Systems Design

After this brief introduction to our perspective on and conceptualization of simulation games, we now go one step further into the direction towards the definition of games as valid research instrument. We describe which approaches we use for their design, in order to being able to develop valid research instruments. We also show which approaches we use for the evaluation of the games and their role both for researchers as well for players. We address the scientific as well as the practical level of gaming research with this chapter, as introduced by Klabbers (2018):

Scientific level: How is knowledge used and constructed in game-related activities (by designers/developers, players, facilitators, and researchers)? How can our simulation game experience contribute to the development of the scientific foundations?

Practical (application) level: What evidence is there to support the viability of simulation games as useful research instruments for the analysis and design of complex systems?

When simulation games are used for the analysis and design of complex systems, it is important that the foundation of the games is solid and that the games are valid in one or more ways (Peters, Vissers, & Heijne, 1998). Therefore, as well as for the design-in-the-small (design of the game as artifact) as for the design-in-the-large (design of the complex systems with the help of the game) (Klabbers, 2003), scientific and proven methods and instruments need to be used. This increases the validity of the game as well as the repeatability of the research.

For the research projects described in this article, a design research approach is applied. The objective of design research is to create and evaluate new artefacts intended to solve identified problems (Hevner, March, Park, & Ram, 2004). The design cycle of the artefact is embedded in a highly structured cycle and a relevance cycle, which means that the design uses the latest scientific insights and knowledge from the application field for validation and verification, while the final design and use of the artefact contributes to new scientific knowledge and a better understanding and recommendation for the real-world problem.

Design Process of Simulation Games as Research Instrument

In simulation games, participants create a future state of a system they want to study and become an active member of the design and the analytical process of this system (Geurts, Duke, & Vermeulen, 2007). In this sense, design is understood as the construction of knowledge about the world, and about complex systems. Design further means to conceive and to implement courses of action aimed at changing existing dysfunctional situations into preferred ones (Simon, 1969). A design process of course can also lead to systems that do not yet exist (Nonaka, 1995).

A game can support actors in changing existing systems as well as developing a new system, as it enables players to deploy new structures, collaborations, interactions, and so forth. In our perspective, design is one step in a larger research project, starting from a thorough systems analysis up to an implementation of new courses of action in a real system again (Kriz, 2003). First, research in the broad sense of exploration is thus needed. Games allow for deep insights in existing systems, are able to make players aware of certain challenges and problems (H. Lukosch, Groen, Kurapati, Klemke, & Verbraeck, 2016) and allow insights in player behavior (van den Hoogen et al., 2008). For the design of games as research instruments, different types of knowledge are needed. We can differentiate between.

Contextual knowledge: which means an understanding of the real-world system, the challenges and dilemmas

Scientific knowledge: theoretical frameworks to link such systems, challenges and dilemmas with theoretical concepts as complex adaptive systems, situational awareness, or social innovation (e.g. policy, education, psychology)

Game design knowledge (and experiences): to design the games (including knowledge about (new) technologies and applications: to develop alternative game interfaces

Knowledge about simulations: to develop models and simulations with high as well as low fidelity to inform the game design and enrich the feedback of the games

To end up with a valid research instrument embracing all types of knowledge, we have to follow a structured way of designing the game itself, having the purpose of the artifact in mind. In our work, we follow three main design approaches, which we use in a combined way in our practical work.

Triadic Game Design Philosophy

Any reference to reality, the meaning and the play elements of the game should be balanced in ways that enhance the learning effects (Harteveld, 2011). The reality element of a game represents its model of reality and should, according to Harteveld (2011), be tighter in games with serious purposes, for example simulation games for learning such as the TOPSIM game, a business simulation that is widely used in higher education to teach management skills. The ‘meaning’ of a game is connected to the creation of value through the game. It is related to the purpose of the game, or the goals the game needs to achieve, the strategy used for achieving this purpose, concrete operations that have to be taken in order to achieve the purpose, and the context in which the value creation takes place. Thus, a game designer has to define each step in a game, starting from a high-level aim up to each action a player can take within the game. Aside from its relationship to reality and meaning, a game represents a medium or a tool with specific characteristics and elements that immerse people in an interactive, fictitious scenario. In designing a good, playable and effective simulation game, all three elements are equally important (Harteveld, 2011).

9 Steps of Designing Simulation Games

One of the first researchers describing simulation game design for policy making contexts was Duke (1981). In his work, Duke describes 9 steps of game design for complex systems with several sub-steps that should be followed up in an incremental way. Following this game design process, a designer starts in step 1 with formulating the specifications of the game, including its requirements and determining factors. Usually, this is being done together with the client and/or the target group. The second step of this game design addresses the systems or problem analysis in systems terms. As described earlier, we aim at starting our game design process with a thorough systems analysis in order to end up with a valid game design for research. In this step, the system is described following fundamentals of systems engineering (see e.g. Sage & Rouse, 2009). These first two steps show similarities to the ‘reality’ aspect of Harteveld’s TGD philosophy, while the third addresses the play element. Here, a game designer has to make certain decisions on which elements and problems from reality go into the game design (Duke, 1981). After answering the ‘what’ question in this step, step 4 relates to the question of ‘how’ to capture the system component in the game design. Duke proposes several mechanics for doing so (see Duke, 1981, pp. 367-373). Step 5 of Duke’s game process defines the game elements exemplified in step 4 in much more detail, while in step 6 the representation of each game element is chosen from a given repertoire of the game designer. Just now, in step 7, the actual game is designed in an iterative manner of designing and testing. In this step, it is important that all actors involved in the process of game design, client, target group, and designers, must be aware of the fact that this very step may take a while based on the interconnectedness of designing and testing. Step 8 refers back to the requirements formulated in step 1 of the process, the requirements. These are evaluated in this step. Finally, step 9 describes the implementation of the game in the field, and refers to the training of facilitators, the design of game workshops, and the implementation in an actual system.

Following this fundamental work of Duke, Duke and Geurts’ (2004) handbook on ‘Policy Games for Strategic Management’ also serves as one of our (more extensive) guidelines towards the design of games for complex systems analysis and design. As it also covers more general ideas on this topic, we don’t want to summarize its content here, while we acknowledge the work as publication with significant importance in the field.

A Practical Guideline Towards Simulation Game Design

A more recent approach towards game design follows the same traditions as the two described above, but is clearly influenced by and meant for practical application: the handbook on simulation game design, edited by Peters and van de Westelaken (2014). In this handbook, called a Concise introduction to the game design process, Peters and van de Westelaken describe several recommendations when designing games, and propose a design process that illustrates on the one hand the translation of real world problems into the game, steps that we already saw in Harteveld’s (2011) as well as Duke’s (1981) approach. On the other hand, they clearly include the process of debriefing in the design process, as feedback mechanism of the experiences during game play to the real world. Peters and van Westelaken propose 4 phases with 10 design steps in total. The first phase relates to the design specifications, which is very similar to the first step of Duke’s game design method (Duke, 1981). The second phase addresses the systems analysis, which aims at identifying the core system’s elements and their relation to each other. Phase 3 of their approach is then dedicated to the design process itself, comprising out of the steps of selecting the system components, developing a matrix of the components and their corresponding game elements, the choice for a game format, and a conceptual design of the game on paper. The fourth and last phase describes the game construction and transfer to the client, which is again similar to step 7 and 8 of Duke’s framework (Duke, 1981). In this sense, the approach of Peters and van Westelaken replenishes Duke’s (1981) approach towards game design with a phase of debriefing.

A Combined Toolbox of Game Design for Complex Systems

We find the three above briefly described approaches of Duke (1981), Harteveld (2011), as well as and Peters and van Westelaken (2014), equally useful and important for our work on the design and use of simulation games for complex systems analysis and design. For most of our games, a debriefing phase is a crucial element of the whole design process. Structured ways of designing these phase as described by Kriz (2010) help us to enable players to reflect on what has happened during game play, what they have learned, and how to translate these lessons to the real system. We can thus conclude that our own game design approach is mainly based on above-mentioned three approaches, while we always aim at including further knowledge from complementary scientific fields, as well as new approaches towards game design. For example, in recent research projects, the game design lenses as introduced by Schell (2008) are applied. Our first own game design process can thus be seen as an iterative process of (1) analyzing the (elements of the) real system, (2) defining the overall aim of the game, (3) deciding which elements of reality underpin the real-world problem as well as the aim of the game, (4) defining how these elements should be represented within the game, (5) and how the player can use and act upon these elements within the game, (6) testing and redefining the game design, (7) designing a game session including debriefing and additional research instruments in order to provide rich feedback both to the players as well as the researchers. It should be noted that these steps do not necessarily represent a strict sequence of actions, but that some steps are made in parallel, or are redefined in a later state of the game design. In addition, we always aim at including the stakeholders, the problem owners and/or target group of the game envisioned, in the whole process as early as possible. We are aware that on the one hand, we need validated and proven approaches towards game design when aiming at conceptualizing simulation games as valid research tool, on the other hand, we make use of concepts describing the locus of our research, the complex systems itself.

Complementary Scientific Concepts to approach the Analysis and Design of Complex Systems

Our work focuses on the design and use of games for complex systems design and analysis. Hence, in the following we introduce our understanding of complex systems, and what aspects of these systems are core to our research, like (situational) awareness of actors or their participation in and engagement with complex systems, in the context of simulation games.

Complex Systems

Complex systems consist of distinct stakeholders that individually and jointly act and react on the input from their environment. Social or complex systems always consist of multiple actors, which actions are related to one another (Maturana, 1980; Weick, 1979). Social systems enact their environment that again consists of systems (Luhmann, 1995) and systems of systems (Checkland, 1981). Therefore, these stakeholders are adaptive, and the systems itself are denoted as complex adaptive systems (CAS) (Holland, 1995). Due to their adaptive character, systems are dynamic and show emergent behavior as well as self-organization. Additionally, many systems in modern societies are defined as socio-technical systems (systems of actors who are supported by any kind of technology) and often signified as complex, due to their involvement of physical-technical elements and networks of interdependent actors (De Bruijn & Herder, 2009). Varying goals, incomplete, uncertain and/or ambiguous information, problems with communication and coordination are characteristics that might be encountered in complex systems (Rouse, Cannon-Bowers, & Salas, 1992). In summary, the number of actors involved, the use of technology, the interrelatedness of the elements of complex systems, and their dynamics make them hard to understand and manage.

Research on complex systems is diverse; complexity theory is not a unified body of theory yet (Ridolfi, Mooij, & Corpino, 2012; Thrift, 1999; Walby, 2007). Complexity notions have developed across a range of disciplines, from natural and social sciences, to ecology and mathematics (Capra, 1997; Maturana & Varela 1980; Waldrop, 1992). Research on complex systems focuses on different units of analysis, e.g. individual, group and organizational/institutional (in social sciences) respectively micro, meso and macro level (in economics). Main challenges, which are present on all three levels, are how to cope with the complexity and decision making in systems, where multiple agents are involved and that are largely technology-based nowadays, and how to ensure the performance of individuals, teams and in turn, of organizations.

Given these characteristics of complex systems, we aim at designing games as design artifacts that address the complexity of systems on individual, team, and system level. We want to make actors aware of the complexity and related challenges, but moreover want to enable them to develop courses of action of how to participate in complex systems, and how to engage with them in daily (work) life. Thus, we further need to conceptualize the foci of our research, which are (situational) awareness, engagement, and participation.

Situational Awareness

One of the aspects that is related to the skills needed for individuals, teams and systems to deal with increasing complexity, is the concept of situational awareness (SA) (Endsley, 1995). SA is seen as critical for successful collaboration (Stanton et al., 2006) and system performance. SA refers to the understanding of others as context for own activities (Dourish & Bellotti, 1992). The application domains of SA currently range from large-system operations to everyday affairs like driving. SA provides dynamic orientation to the situation, the opportunity to reflect not only on the past, present and future, but also on the potential features of the situation. The dynamic reflection contains ‘logical-conceptual, imaginative, conscious and unconscious components which enables individuals to develop mental models of external events’ (Bedny & Meister, 1999). The most widely used definition for individual situational awareness is ‘the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future’ (Endsley, 1995). SA helps to generate an analysis of the complex and dynamic (work) environment as well as being critical for good decision-making under time pressure, and thus enhances decision quality.

Policies for Participation and Engagement

As we no longer live in a world that is shaped by what Max Weber would call Traditionalism (Weber, 1978), where roles and expectations of members of a society are relatively clear, we have to seek for new ways of policy design and participation in society and its systems. Policy making for complex systems has become increasingly important. It also changed its nature: Policy making is no longer seen as a purely top-down process but rather as a negotiation among many interacting policy systems (Bovaird, 2007). This negotiation process requires the active participation of actors in a complex world, or the emergence of participatory systems. Actors in a participatory system behave strategically to achieve their individual goals (Rezaee, Oey, Nevejan, & Brazier, 2015). In simulation games, we can represent rational choices and model behavior, which we can again test. As result of such ‘a conscious endeavor to reproduce the central characteristics of a system in order to understand, experiment with and/or predict the behaviour of that system’ (Duke & Geurts, 2004), we can feed back the results of simulation gaming sessions to the actors, enabling them to better engage with and participate in complex systems.

We have now introduced the various topics that represent the main focus of our research. As stated by Duke (1981), we have to evaluate whether the games we define fit their purpose as research tool for complex systems analysis and design. But first, we briefly introduce additional instruments in our scientific practice, in order to enrich the pure use of simulation games in complex systems.

Frameworks to Analyze Simulation Games as Artifacts for Research

To evaluate the games as design-in-the-small artifacts itself, we mainly use two complementary frameworks. On the one hand, Klabbers’ (2006) concept of the relationship between complex systems and games is very helpful in understanding the role of games for complex systems design (cf. Figure 1). In his conceptualization, resources represent the technical sub-systems of socio-technical systems. Rules describe the relationships between these technical systems, but more importantly, between the actors involved in the systems, and between actors and resources. When designing simulation games for complex systems analysis and design, it is thus crucial to address all three aspects: the actor layer, the technological layer, and the relationship between actors and between actors and technology (Klabbers, 2018). Especially defining the relationships can lead to interesting game mechanisms, as forms and rules of communication, information sharing, or power relationships can be displayed in games. Games can be used to highlight or research these relationships, and can motivate players to change them as well as representing the consequences of such actions (and changes) on the system as a whole.

Figure 1.

Klabbers’ model to represent socio-technical systems with simulation games (Klabbers, 2006).

On the other hand, Meijer’s (2009) framework of using games for complex systems design is leading in our own work. This framework starts from the clear conceptualization of simulation games as also introduced by Klabbers (2006, 2009), and is also based on the distinction between rules, roles, and resources. In addition, Meijer includes the input and output variables of a simulation game session in his own concept. Furthermore, in addition to the elements of rules, roles, and resources, Meijer adds objectives as well as constraints to his conceptualization of a game itself, and the load and situation to the game session. The experience of the participants changes during the session, which can be measured based on qualitative and quantitative data. Thus, Meijer’s model explicitly mentions that a game session creates two different types of outcomes, namely the direct outcome for the player as of a changed experience, and the data that can be used by researchers to better understand and/or design complex systems.

These two major frameworks of simulation games for complex systems allow for measuring the usefulness of the games as research instruments itself. Furthermore, we would like to know whether the games we design and use fulfill the purpose they are designed for. Again, we make use of established frameworks for the evaluation of the effects of simulation games in our studies.

Concepts Towards the Evaluation of Simulation Games and their Effects

The work of Kriz and Hense (2006) provides us with a theory-based evaluation framework for simulation games, in which they try to overcome the bridge between pure academic research and practical experience, provided by simulation games. Again, their work refers to Klabbers’ framework (2006) of the use of a design-in-the-small approach (games) for design-in-the-large processes (complex systems). Where Meijer (2009) focus lay on the input and especially the different types of output derived from a gaming session, Kriz and Hense (2006) also include the dynamics during the game play itself in their framework. They further distinguish between short-term and long-term outcomes of the game session, whereas especially the long-term outcomes are interesting for research purposes.

Another high-level approach, following the design science philosophy is proposed by Kurapati (2017). In her work on the use of games for development of situational awareness, Kurapati adopts the design science framework as well as above mentioned game design concepts of simulation games, resulting in an approach shown in Figure 2.

Figure 2.

Kurapati’s combined approach towards research with simulation games (Kurapati, 2017).

In this approach, several earlier concepts of simulation games for complex systems are combined. Elements that are relevant for our conceptualization of simulation games and complex systems are used to develop a new approach, which has already been applied in several research projects, as we will briefly introduce in a following section. First, we want to illustrate additional research instruments we use in our simulation gaming studies.

Complementary Research Instruments

We are convinced that the use of games as the only research instrument is too limited to explore complex systems. Furthermore, for the support of actors in this context to develop awareness, improve decision-making processes, engage with and participate in complex systems, a richer feedback than only from the game itself is needed. Hence the use of complementary research instruments is seen as an integral part of the game design cycle. Amongst proven instruments that we use for our research are the following main means:

Interviews with the target group or experts to learn about the system under study and identify the real challenges behind the first mentioned problems. A design team consisting of game professionals as well as content-wise experts is formed to have direct access to the problem owners or target groups

Questionnaires (developed by ourselves based on existing questionnaires or validated ones such as the game experience questionnaire (Ijsselsteijn, de Kort, & Poels, 2013)

Observations (in-game, debriefing)

Structured briefing and debriefing

Additional tests (e.g. personality traits, multi-tasking)

Modelling (Kurapati, Lukosch, Eckert, Corsi, and Verbraeck, 2017)

In the following section, we will highlight how we select and apply both the scientific frameworks as well as the instruments in our research projects. We do this to draw conclusions towards the use of simulation games as rigorous research instruments in complex systems design on the one hand. On the other hand we aim at providing a set of recommendations to researchers who also want to bridge the gap between academic knowledge and practical experience of simulation games.

Illustrating and Reflecting the Scientific Toolbox

Over a period of 15 years, the gaming research group at Delft University of Technology designed a large amount of simulation games for the analysis and design of complex systems. Besides the design and application of simulation games in research projects, an enormous effort is put in the scientific foundation of the design and use of these games. As a consequence, new frameworks and instruments are developed and validated. In this section, the experiences of several gaming projects are presented. In the description of the projects, the focus lies on the scientific methods and instruments used. Detailed descriptions of the games, the target groups and effects can be found in a variety of publications about the specific projects.

Case Studies in the Transportation and Logistics Domain

Digital game: SIMPORT-MV2

One of the first computer games that has been developed at TU Delft, was the game SIMMV2 (2004) followed by SIMPORT-MV2 (2007). This game’s goal is to support the strategic decisions around the extension of the port area in Rotterdam with the Maasvlakte 2, hence the name MV2. SIMPORT-MV2 is a multiplayer game. A group of 5 to 6 participants form the executive board of directors of the Port Authority, divided in different functions. The game starts with a discussion about a building, a commercial strategy, and an allocation plan. After this first round, the participants have to execute their strategy in three rounds, representing three times 10 simulated years. The players use a digital environment that calculates the consequences of the decisions and visualizes them for the players. During the game, the participants see the port area in development and which parcels clients use. They also get feedback about the financial situation and the satisfaction of the clients.

Development of SIMPORT-MV2

The first objective of SIMPORT-MV2 (see Figure 3) was to support the strategic decision-making process of the Port Authority Rotterdam. In the first place, this meant that the design of the game needed to be highly valid and scientifically supported. Therefore, the design approach of Duke and Geurts (2004) was used as a starting point for the design. This approach has been very useful as an important part of the design is getting a good understanding of the system. However, this approach lacks the design of a computer game and the way to validate the computer simulation. An adapted form of the approach has been applied. More details about the design steps and the development of the game can be found in Bekebrede (2010).

Figure 3.

Screenshot of SIMPORT-MV2.

The advantage of following this approach is that from the start of the development the real-world situation is central. In this design process, several interviews were held with professionals of the Port Authority with varied perspectives of the design of the Port area. After each interview, we developed and adapted the so-called schematic, a visual of the conceptual model of the system. This schematic was the starting point of the next interview. Each time an interview was held, the schematic was validated and missing elements were added to get a better picture of the whole system. Already in this state, the real Port Authority of Rotterdam was surprised by the clarifying possibilities of the port development project.

In the design of the game, validated methods were used. Verification and validating techniques, such as extreme situations and sensitivity analysis, from the field of computer simulations were used to check and improve the game (see Bekebrede, 2010) for detailed information about the design and development process). In this way, other designers were enabled to follow each step and could repeat the design process. This not only represented a valuable step within the game improvement process, but also enables researchers to use the tool, and to validate the results of the study, an important aspect of a rigorous research process. However, there is one element, which is hard or maybe impossible to make scientifically valid and which became clear already during this very early game research project. This aspect is related to the creativity of the designer(s). At a certain moment in the process, the design team gets the idea for a concept/a type of game. Although many choices can be rationalized, this part cannot.

Evaluation of SIMPORT-MV2

An important research question related to MV2 was, whether the game supported the policy making process in the actual system. For answering this question scientific models were used. As background for the evaluation, the theory-based evaluation framework of Kriz and Hense (2006) was adapted and applied. The adaptation consisted of adding elements related to computer game design. In this research, we used surveys, interviews, focus groups and observations, the discussion in the debriefing and direct game output for data collection. Such combination of methods provided us with many insights about how participants played the game and what they learned. A shortcoming in our instruments was the use of self-developed surveys and not using existing validated surveys. At that moment, they were hardly available, which required new surveys. We made use of existing surveys in the field of gaming, simulation and e-learning to develop a set of relevant questions.

Although the game was intended for the Port Authority, several (higher) educational institutes embedded the game in their education. Consequently, the number of participants in the game and the amount of collected data was large (> 800 players, and 60 game sessions) and gave possibilities for an in-depth analysis of relations between variables. As games are relatively open environments, as participants have the freedom to make their own narrative in the game and every participant has a different background and brings unique values and norms into the game.

There is an inherent tension between the development of a game and simulating a Complex Adaptive System. On the one hand, Complex Adaptive Systems are open systems in which a small change in one element can have significant consequences for the system behavior as a whole. On the other hand, the development of simulations in general - and games specifically - is often driven by setting clear boundaries that are just large enough to be able to take relevant elements into account. The boundaries have to be carefully chosen, and these define the validity of the model. A serious game therefore does not provide certainty about the future dynamics of the system or promise that the strategy of the winning team will give the best results in reality.

Digital games: The YARD CRANE SCHEDULER games

YARD CRANE SCHEDULER (YCS1 and YCS2) games were developed as research instruments to assess the role of situational awareness (SA) on decision making and performance and the factors affecting SA among individual actors, teams and groups in the operational planning of intermodal transport operations in container terminals.

Two so-called microgames, small games that focus on a specific learning objective (H. Lukosch, Groen, et al., 2016), YARD CRANE SCHEDULER1 (YCS1) and YARD CRANE SCHEDULER2 (YCS2) were developed and hosted in an online portal known as Whitebox to conduct SA research in individual and group settings.

YCS1 is a single player game that requires participants to align several planning processes in a container terminal, under time pressure and varying workloads (cf. Figure 4 left). YCS2 is a multi-player game that is an extension of YCS1. YCS2 is played by 4 players who take the different planning roles in a container terminal with the task to align the planning processes under time pressure and varying workload (cf. Figure 4 right).

Figure 4.

YCS1 (left) and YCS2 (right) screenshots.

Both games were designed to sufficiently represent the various interdependencies and complexities of container terminal operations. They also should make the players experience time pressure while making critical decisions to represent the complex work environment of operational planning in container terminals. The participants needed to have some background on container terminals since naturalistic decision making in socio technical systems requires basic subject matter knowledge (Klein, 2008). The participant group that fit this criterion were professionals and students in the field of supply chain, logistics and transportation, an umbrella field that comprises container terminal operations as well. Our games should be relatively simple to play, that do not require special computer gaming skills to play because we would like to measure SA related performance and not game playing proficiency.

We built the key components of the game- roles, scenarios, operating procedures and accounting system based on the principles of the Triadic Game Design (TGD) (Harteveld, 2011). The first step of the iterative design process began with a so-called game storm, a brainstorming session together with three researchers, four subject matter experts from the container terminal business and two professional game designers to discuss the research objectives, game requirements and to define the first version of the key game components using the TGD philosophy.

We tested the games with 200 participants in the field of logistics, supply chain and transportation in Netherlands, Germany, and the United States. Then the game was redesigned to accommodate the suggestions and shortcomings encountered during the testing and validation sessions. After four iterations, we reached a playable single player version of YCS1 and YCS2. We conducted two major studies using the YCS games and the results are discussed in detail in H. Lukosch, Groen, et al. (2016; H. Lukosch, Kurapati, Groen, & Verbraeck, 2016) as well as (Kurapati et al., 2017).

YCS1 was used to study the role of individual SA and factors such as personality type, multi-tasking ability and individual differences on the decision making and performance of individual actors in container terminal operations. It also serves as an indirect SA performance measure. YCS1 is currently being used at a leading container terminal as a tool for employee training and selection.

We used YCS2 to study the effect of Shared SA and Distributed SA and related team factors on the team performance in container terminal planning operations. Our research design was quasi-experimental since the sample population was not randomly drawn. The experimental session began with a short briefing lecture explaining operations in container terminals and the various planning roles involved to manage these operations. All the participants were provided with laptops (cf. Figure 5). They were directed to play the online game YCS1. YCS1 acted as a tutorial to the players before they play YCS2, so that all the participants were on an equal footing to play YCS2. Each team of 4 players had to play the YCS2 game at varying configurations of SA- SA, Shared SA, and Distributed SA. The YCS2 game had different scenarios for the different levels of game play with the same level of complexity in order to control for learning bias.

Figure 5.

YCS2 game session.

Validation and Evaluation of the YCS games

The YARD CRANE SCHEDULER games were designed, developed and evaluated together with a professional game design company and a consultancy for the container terminal industry. The research objective was quite clear already in the first game storming session because the experts from the consultancy substantiated it with the real-world problems in the container terminal regarding lack of awareness and a holistic picture of integrated planning operations among the planning departments in container terminals. The first prototype of the YCS1 game was developed within two months after a few follow up meetings regarding the processes and roles in the container terminals. The first prototype was already quite satisfactory in terms of reality, meaning and play from experts’ perspective. We tested the prototype with students and experts in above described settings.

The following versions of the game were made to incorporate the suggestions from expert validation and testing to reduce technical glitches, improve visualization and improve the scoring system. The final game was tested for contrast, psychological, face and predictive validity. Given the single player nature of the game, balancing the game and designing the scoring system to capture the various game metrics was relatively straightforward. The most challenging aspect of the game development was the design of the data collection and storage mechanism to perform controlled studies. The game developer designed an online portal, the Whitebox, exclusively for our research project where a participant can login with credentials provided by the researcher. The platform could host several games, and can incorporate surveys, briefing and debriefing materials. The researcher also had control over the visibility of the games, scoring and widgets in the portal. The data from a gaming session is saved in an online database that could be downloaded by the researcher in the form of a spreadsheet. The design and creation of a robust data collection platform is essential for games that are developed for research.

As both games were designed to influence the situational awareness of players for interdependent planning tasks, we chose for a validated instrument to evaluate the effects of the game in relation to this intended goal. The SART questionnaire (Salmon et al., 2009), a self-rating technique to measure situational awareness was chosen as this instrument does not interrupt the process of the game, but can be used before and after a game session. Additionally, we used a personality test as well as the scores of the game itself to test for influences of personality traits, and to measure the performance of the players. In the YCS3 sessions, we also made use of video recording in order to being able to observe and analyze the communication behavior of the players.

Physical game: DISRUPTION GAME

As the performance of complex socio-technical systems may depend on the coordinated work of multiple individuals, that have responsibility for different subsets of goals, different access to information and different situation perspectives (Roth, Multer, & Raslear, 2006). The organizational and environmental complexity of these systems combined by human error and external factors give rise to disruptions that negatively affect the performance of these systems (Wilson, Burke, Priest, & Salas, 2005). In the context of container terminals, unplanned and unanticipated events that affect the normal flow of goods and operations in supply and transport networks are termed as disruptions (Svensson, 2000).

The disruption board game was developed to understand the decision making of actors involved in operational planning in container terminals under such disruptive situations. We used the DISRUPTION GAME to study the effect of SA on the operational resilience of teams and groups within complex systems when faced with unexpected disruptions. The target participants of this game are academic researchers, students and professionals in the transportation, logistics and supply chain domain. Teams or groups that function from the same location have several characteristics including a shared physical environment, presence of non-verbal communication, and social cohesion (Magerkurth, Memisoglu, Engelke, & Streitz, 2004). We decided on a tabletop board game format because it can represent these characteristics.

Tabletop board games can provide rich social situations because players usually sit together around a table and can look at each other to interpret mimics and gestures, which may help them understand each other’s actions even without verbal communication (Magerkurth et al., 2004). The social cohesion is also prominent since they may laugh together or confront each other, in relation to the decisions made during the game play (Magerkurth et al., 2004). Tabletop games usually require a game master, who steers the gaming with a dynamic form of storytelling, making the game reactive to the decisions of the players in a fun manner (Tychsen, Hitchens, Brolund, & Kavakli, 2005). The main objective of the board game is to represent dynamic situations in a container terminal. As introduced, socio-technical systems are characterized by uncertainty, vulnerability and dynamism. We represent these characteristics in the game by scenarios in the game that disrupt the normal flow of operations in a container terminal in an unexpected manner. The decision making of the players who assume the roles of planning personnel that have to manage the disruptions at varying SA configurations are studied using this game.

We started the design of the board game using knowledge from literature study and brainstorming sessions with three subject matter experts in the container terminal business. Based on this, we synthesized the various disruptions that can affect container terminal operations. Then we had to translate the management of these challenges under different SA configurations to contextualize the game play. The development of the game took over 8 months since it was an iterative process with design, evaluation and validation cycles (Kurapati, Lukosch, Verbraeck, & Brazier, 2015).

The disruption management board game was tested in several academic and professional settings. The researchers from the technical university in the Netherlands were used mostly for prototype testing. Five professionals and 60 American university students were used to test and validate the final version of the board game. The issues faced by these participants were observed and the game design was further updated twice based on the feedback from these sessions.

A simulation game is not a stand-alone instrument. In order to deliver the full potential of the game, it is usually presented to the participants in the form of a game session. For each of the test sessions, participants were gathered around a table in a spacious room, as shown in Figure 6. The room was prepared in advance for the game play, by prearranging the required game objects. Depending on the size of the group one or more game facilitators orchestrated the game play. The game facilitator was given a game manual that describes their role and the method of orchestration.

Figure 6.

A game session with the DISRUPTION GAME.

Every game session of the DISRUPTION GAME began with a briefing, usually lasting 20 to 25 min. The various intermodal operations, terminal processes, roles in the container terminal, and the equipment used were described in the introduction.

Validation and Evaluation of the DISRUPTION GAME

The research objective of the disruption management board game was to study the effect of various SA modes (individual, shared and distributed) on the decision-making and performance of socio technical systems under dynamic situations. In our context the dynamic situations are disruptions to the operational processes of a container terminal that is an example of a complex socio-technical system. We had to shelve two prototypes before finalizing the game that is described in this chapter. For the first prototype we defined the roles in the game based on literature review and consultations with experts in the container terminal business. We searched newspapers and consulted scientific and technical journals for real world disruptions related to container terminals operations. The early prototypes were not very fun to play because the processes were extremely slow and complicated in the first gameplay. In the final prototype, we further reduced the complexity and modelled the interdependency between terminal processes in terms of scores, i.e., the action of one player can affect the score of another and vice versa. To add an additional action card and increase the ’play’ element of the game we added a ’Joker’ option. We also introduced objects like dice, communication tokens to create a ’fun’ environment. We also introduced individual and joint game boards and simplified the scoring systems instead of the spreadsheet. Pawns were placed on the game boards and the game master could change the scores easily using the pawns based on the actions of the players.

When we tested the final version of the game, we found that the participants had a very positive game experience. They understood the research objective very well after playing one or two rounds of the game and enjoyed playing the game. Therefore, we finalized the third prototype to be our research instrument to study the role of SA in decision-making under dynamic circumstances in container terminal operations. Although the game was well received by both students and professionals in the supply chain, logistics and transportation business, it has a few limitations. The game master or the game facilitator has a critical role in the game play. He/she is the ’game engine’. This means a game facilitator should have a thorough understanding of the game and needs to be always present to steer the complex game processes. The evolution of the scores during the game is not traceable. The game master has to rely on memory or observer notes for debriefing. The board game requires all players to be present in a single room. While this fosters a common game experience, it imposes an unrealistic situation, as in reality, especially in complex systems, the different persons could be distributed across the container terminal. The board game also presents scalability and portability issues. It is difficult to collect a large amount of data using the board game. Also modifications are hard to implement in the board game version. Despite these limitations, of which some due to the nature of being a board game, we were able to use it as a research instrument in our studies (see for more details e.g. Kurapati, 2017).

Case Studies From the Safety & Security Domain

Digital Game: CHARLIE PAPA

Public safety work is a domain where following pre-defined routines, teamwork and communication is important, as it heavily depends on well working information processing. The main objectives of a simulation game developed for this sector are to train police officers and private security guards situational awareness skills – how to recognize potentially dangerous objects and behaviors – as well as team communication and collaboration. Skills enhancement should improve the ability to intervene, both in advance of and during hazardous situations, as well as to manage a situation once an event has taken place. In our research context, the project was meant to explore the role of situational awareness and communication in the complex system of security work. The simulation game CHARLIE PAPA provides a realistic 3D-environment of parts of the city of The Hague, as shown in Figure 7. This environment represents the reality element of the game and the strong relation to the actual workplace of the target group. In the game, teams of police agents have to identify and classify dangerous objects and behaviors.

Figure 7.

Screenshots of the CHARLIE PAPA game.

Three prototypes of different game scenarios were developed and tested with the target group to ascertain if this type of game could be helpful for professional learning in reconnaissance work. Our research aimed at exploring the influence of the level of realism (or ‘fidelity’) of the game environment on the gameplay (or usability of the game) and how this type of game can help individuals and teams enhance their situational awareness and communication skills – thus how to balance the elements of reality, meaning and play of the game. Forty-three police officers and employees from a single private security company participated in the test sessions. The participants were asked to fill in pre- and post-test questionnaires to surface their expectations of the game as well as their previous game-play experience. The game session itself was videotaped for after-action review. The after-action review included a de-briefing session in which participants were interviewed about their experiences of the game and its capacity as a tool for learning situational awareness. Situational awareness in this sense does not only mean viewing what’s going on around someone, but also comprehending meaning from a given situation, interpreting the situation and being able to predict how the situation may evolve and anticipate the consequences of one’s own actions and the actions of others (Endsley, 1995).

The game has been tested several times with security experts in an experimental setting. The test sessions included a stand-alone walk-through of the players through the interactive 3D-scenario in single or team player mode, and a debriefing phase. The debriefing phase was lead by a trainer the same way as this happens in real-world working situations. When played without a trainer, the players would act as peer group for each other and conduct the de-briefing as peer-review. The debriefing phase at the end of the game gave players an opportunity to receive feedback and reflect on their performance (Kriz, 2003).

The game provided participant players immediate feedback through its interface; it showed the player a score related to his or her actions in the game and the number of potentially dangerous objects he or she identifies and classifies. This scoring encouraged players to reflect on their own abilities, skills, knowledge and actions.

Validation and Evaluation of CHARLIE PAPA

The game has been played and evaluated with several experts from the field (see for details H. K. Lukosch, van Ruijven, and Verbraeck (2012). The case study illustrates that games can be a purposeful tool that support learning processes in complex systems. However, to be useful for learning, games have to be designed and implemented very carefully. From the evaluation of the game, five major insights could be drawn. First, in circumstances where a high level of physical fidelity and visual realism is required to represent the real system, there is a risk that even minimal mistakes within the game-world results in a player’s resistance towards the game as a meaningful tool. Thus, the element of reality can both serve as motivator as well as de-motivator. Secondly, a debriefing phase in a simulation game is a powerful way to raise awareness of actions and consequences. Debriefing allows players to reflect on their actions and thus enable transfer from the game world to the real world. Thirdly, simulation games are useful as learning tools for professionals with different levels of experience. For novices, games provide orientation and introduction to a work domain. For more experienced workers, games function as a basis for discussion and exchange of experiences. This insight can be transferred to other games used in complex systems design, where we have to deal with a number of stakeholders with possible different backgrounds, knowledge and expertise. Fourthly, the game elements of competition and challenge can motivate professionals who are new to gaming as well as experienced games. Finally, we can conclude that simulation games provide a cost-effective learning solution in dynamic systems and situations where traditional training is expensive, dangerous or inflexible in relation to time, space and participants. In this case, the game was not used as tool for systems analysis, but served as a way to improve, thus design, vital skills needed in a complex system, and the lessons learned from it can also be pertained to other application cases.

Digital-Physical Augmented Reality Game: THE AR BOX GAME

In a project with the Dutch Police, we investigated in how far Augmented Reality (AR) affects communication, information sharing, and as precondition for decision-making, also situational awareness of operational teams in the safety and security domain. A series of experiments with professionals from the safety and security domain that were using realistic scenarios showed that AR can be used to remotely support professionals by providing additional information as well as that AR can be used to exchange context-relevant and local information between different teams (S. Lukosch, Lukosch, Datcu, & Cidota, 2015).

In order to further test the AR technology and to understand the impact of remote support on coordination and communication of distributed teams, we designed THE AR BOX GAME. THE AR BOX GAME is a collaborative game that mimics characteristics from a complex real scenario but can also be played by non-professionals to allow for studies involving more participants. For the game, we chose a scenario related to crime scene investigation. The game needs to be played by three players. One player has the role of a remote police agent, while the other two have the role of police agents investigating a crime scene.

At the crime scene, there are several boxes which can contain pieces of evidence (cf. Figure 8 left). During the game, these pieces of evidence are resembled by blue, red, green, yellow and white Lego blocks. One of the local players, has the skill to collect the evidence in form of blue and red Lego blocks. The other player can collect the green and yellow Lego blocks. None of the players can collect a white Lego block, as a white Lego block resembles dangerous and possibly explosive material.

Figure 8.

The game setup for the local players (left) and the remote player (right).

The players can arrive at the crime scene at the same time and jointly investigate the crime scene or they can do so sequentially. In their task of collecting all non-dangerous pieces of evidence, the police agents are supported by a remote colleague who knows which boxes contain dangerous material and which do not. For this purpose, the local agents share their view with remote colleague (cf. Figure 8 right).

The overall goal of the game is to collect all evidence without opening one box containing a dangerous piece of evidence. Once all pieces of evidence were collected, the local players receive instructions from the remote player on how to combine the individual pieces of evidence, i.e. build a tower of Lego blocks showing a specific colour pattern.

During the game play, the players can exchange information in AR (for a detailed description of the underlying technology refer to Datcu, Lukosch, Lukosch, and Cidota (2015). Only the remote player can see whether a box is dangerous or safe (cf. Figure 9 left). Local and remote player though can annotate boxes in AR to, e.g., indicate the colour of the block in the box or mark which boxes already were emptied. The information the players leave behind on the boxes is left in place allowing them to share information and coordinate their work (cf. Figure 9 left). In addition to AR, the players can talk to each other during the whole game play. Again, it is up to the players how they use the communication channel to coordinate their activities.

Figure 9.

View of the remote (left) and local player (right) showing information in AR.

Validation and Evaluation of THE AR BOX GAME

The game design was tested with professionals from the field to evaluate the realism of the game as well as give first insights on the use of remote support in AR on coordination and communication. From earlier experiments in a realistic training site, we saw that a realistic situation was too complex to study a novel technology in a rigorous way (see for more details e.g. Datcu, Cidota, Lukosch, and Lukosch (2014, 2015). To decrease the complexity of the experimental setting, we invented THE AR BOX GAME. The outcomes showed that the game was able to represent the situation of distributed work as in the real system. The story of the explosive hidden in one of the boxes was strong enough to make the players feel as if they were in a real, serious situation. One participant remarked after they failed one round and opened the wrong box that he was shocked as in reality; his colleague would have been injured based on his wrong information. That provoked a discussion of the (ethical) limitations of the technology and its general use in dangerous situations. The game thus fostered a secondary goal. Not only have we been able to conduct a usability test of the AR technology, but we were also able to activate a discussion on policies and restrictions of its use. It further showed that games combining the digital and physical world have the potential of provoking strong experiences. It is an open question in how far further technological support for collaboration in AR such as envisioned by S. G. Lukosch, Billinghurst, Alem, and Kiyokawa (2015) will improve such capabilities of AR games.

Findings, Discussion and Recommendations

In our work, we make use of a bundle of accepted frameworks of game design and evaluation, together with self-developed and validated research instruments. We aim at understanding and designing complex systems with games as tools, and developed a toolbox of processes and instruments that help us in our work. As the discussed case studies illustrate, we are able to address different problems in different domains with them. While the early development of the digital game SIMPORTMV2 mainly focused on decision-making processes within a large, inter-organizational infrastructure project, the digital games YCS 1 and 2 as well as the physical DISRUPTION GAME focused on the awareness of actors within a system to improve their performance. While in earlier studies, we had to rely on additional instruments that were informed by scientific concepts, but not (yet) validated, in later studies, we were able to apply proven instruments and concepts, such as a SA questionnaire. In the safety and security domain, we experimented with different levels of fidelity of the games, from a highly realistic VR environment, to a realistic training setting, to a simplified experimental process making use of paper boxes and Lego blocks. Throughout our work, we were able to not only formulate recommendations for the design of complex systems, but we were also able to provide tools for their design. We understood these tools better and are now able to summarize some of our findings with regard to the design, use and evaluation of simulation games for complex systems.

Design Phase

When designing games for a complex problem or system, it is important and useful to include the problem owner and or target group of the game as early as possible, to align their needs and expectations with the actual game design (e.g. described in H. K. Lukosch, van Ruijven, and Verbraeck (2012). Additionally, many of our projects show that the design phase alone already contributes to a better understanding of the problem itself and is thus very useful in the design-in-the-large process. The use of accepted concepts and frameworks makes it easier for experts to accept the game and to validate the results later on. For the use of games as research tools, it is advisable to use approaches from the design research method (e.g. Hevner et al. (2004), as those provide tools to relate the scientific dimension to the practical implementation of the artifact. In the design phase, we see that addressing the aspect of reality might be the most difficult step. A game designer has to understand – and if appropriate, use - the language of the experts, has to be able to raise the right questions, and has to take the right decisions on which elements of reality have to go into the game, and which have to be left out. A game designer has to step into the role of content-matter expert, and to rely on experts from the field. It is valuable when a designer of simulation games has tools at hand that help him or her in analyzing and formalizing the ‘real’ system, understood as the referent system, such as actor or network analysis. Still, the process of transferring the referent system into a simplified version is a very delicate one, and depends a lot on the experience, creativity and openness of the game designer. Only when the first steps of game design are carried out with the required diligence, the game can represent a valid research instrument of a complex system or problem can be developed. In other domains, patterns have successfully been used to capture design knowledge and help designers to focus on the core functional attributes of an application (S. Lukosch & Schümmer, 2004). It is an open question though on whether patterns can achieve the same for the design of games.

When addressing the meaning element of a simulation game, it is a bit easier, as a game designer can rely on existing pedagogical or psychological concepts and approaches. For example, we made use of the 4 components instructional design model, developed by Van Merriënboer and Kirschner (2012) for the design of a training game for mechanics mechatronics (see for more detail H. Lukosch, Van Bussel, and Meijer (2012). The last step, the definition of game elements that support the engaging experience of players is a difficult one again. In the design of the game elements, a developer of a simulation game has to carefully balance the three aspects of reality, meaning, and play, in order to provide a valid, meaningful and yet motivating experience. The last is again very much depending on the creativity of the game designer, and a sensitivity for finding and keeping a working balance between the three aspects. We know a lot about the effect and design of certain game elements, especially through the entertainment game market, but also through works like the proposition of game lenses of Schell (2008). Still, it is very difficult to come up with specific guidelines for the first and the last step of the game design process. Additionally, the size and composition of the design team is crucial for the success of the whole project. From our work, we see how valuable a trans-disciplinary group of content-matter experts, practitioners, scientists, and game designers can be. Yet, when too many ‘experts’ are involved, it is a huge challenge to combine different perspectives and expectations in one game design.

Game Execution Phase

The game execution phase might be the phase of using games for complex systems analysis and design, which is most difficult to control. Dynamics of game play are hard to predict and control. The reality of the game session (Kriz, 2003) is a very unique one, depending on situational and external factors, such as motivation, experience and expectations of players, their content-wise knowledge and game-play skills, and their social skills. Additionally, we sometimes see that people’s motivation and openness towards the game play can be very different, which of course can influence the effect and outcomes of the simulation game, especially when used as research instrument. Too much engagement can be evenly disturbing, as the situation might not be taken seriously enough to draw valid conclusions. Too low engagement could lead to wrong actions and decisions, influencing the data derived from the game.

In our work we see that a lot of physical game sessions are very dependent on the abilities and the person of the facilitator. Not only does she or he need to have strong competences in facilitating game sessions, he or she has to have the objective of the game also in mind – be it data collection on a new technological system, or the development of new measures for a new infrastructure project. As our games are still representations of complex systems, many of the actions and relations in the game are complex, too, and in physical games, a lot of these actions have to be carried out by the facilitator in order to allow for sufficient control of the game session.

Evaluation Phase

As simulation games are a means to achieve a goal, like developing a better understanding of a certain complex issue, or to increase the awareness of actors for interdependencies in complex systems, it is crucial to select the right ways of evaluating the games. This has two sides. First, we want to know about the quality of the instrument itself. Expert validations can for example help to evaluate the right level of realism of the game, while test sessions with a prototype and the target group can help to improve the game elements. Moreover, we want to evaluate the effect of games. In our studies, we developed a toolbox of instruments and approaches, such as questionnaires, observation forms, structured debriefing, and additional instruments to control for certain variables that might influence both the game play performance as the results of a gaming session. Especially in the evaluation phase, we see vital differences in the use of digital and physical games. While digital games already provide us with rich data, and enable us to follow up each decision and action of a player in combination with the performance of a player. It is hard to record every single action in a physical game environment. Digital data can immediately be used for further processing, e.g. in more detailed models of the same system, to enrich the feedback of the game session. A digital platform as the Whitebox, where YCS 1 and 2 are embedded in, is very useful, as it can create feedback for both learner and researcher at the same time. Furthermore, online games can be distributed to a large group of players, leading to a rich set of data to analyze. As physical game sessions require a huge amount of manpower when objective analysis is aimed at, it is much harder to derive quantitative data from them. On the other hand, they allow for direct interaction of the players, which might make the transfer back from the game to the real system a bit easier than having to translate the amount digital data back into the field. Big data sets always need the processing (and translation) of experts, while the immediate experience of a physical game is a pure situation for the players, without any additional translation needed.

Limitations

Above-mentioned challenges in the three phases of design, execution and evaluation of game sessions, can also become limitations. If a game designer is not able to understand the problem, if a facilitator is not able to control the game play, and if data cannot be collected or processed, the value of the simulation game is restricted.

In transferring the results of a game play session back to reality, we still see the biggest limitation or challenge of the use of simulation games for complex system analysis and design. As games can only cover certain aspects of reality, and always represent a pre-defined, designed situation, it is an important question to make sure that the results derived from the session are not only valid for the game session itself, but also for the real system.

Another limitation is the insufficient discussion of ethical questions related to the use of games for complex systems analysis and design. As a game requires the active participation of players, and sometimes also manipulates a player’s behavior in order to reach a certain goal, a number of ethical questions are involved in using games as research instrument. How do we for example guarantee the privacy of the players’ data? To whom do belong the data collected in a game? To the player, the facilitator, or the researcher? How can we make sure that data is collected and stored in a way that guarantees for the privacy? As games always allow for interaction, and always represent an intervention, this instrument can never be seen as pure research instrument alone. Intended as well as unintended effects of simulation games should be considered when using games for qualitative research and should be addressed in the debriefing. Games can also be used for the assessment of the players’ competencies, skills, and knowledge. Is it ethical to confront someone with an artificial, yet realistic situation, and to observe his or her behavior in order to explore and to evaluate someone’s competencies, skills and knowledge?

Conclusions and Future Research

In our contribution, we aimed at answering the question

How can the use of scientific concepts and scientific foundations strengthen the design, use and effectiveness of simulation games for the purpose of analyzing and designing complex systems beyond the use of games as isolated research tools?

Based on the approaches we use, and brief descriptions of case studies of our group, conducted throughout the last 15 years, we were able to draw some conclusions for simulation games for complex systems analysis and design. Despite all efforts of the scientific community, we still see that the field of (simulation) gaming is scattered due to the variety of concepts used and the views and approaches of different schools. Consequently, designers and researchers are not aware of the variety of scientific methods available. In our endeavor to define simulation games as valid research instrument, we have to agree on a certain conceptualization of the games, on valid approaches towards game design, a structured and well thought-through use of the games as instruments amongst others, and the grounded analysis and thorough interpretation of the data gathered.

In our work, we see that proven concepts and foundations improve game design, and the use of simulation games, as well as from the perspective of form and content of games. Still, creativity represents a blind spot in simulation game design. That refers to the creativity and skills of the game designer, to those of the player and facilitator, as well as to the skills of the researcher to use games and to select the right concepts and additional instruments to end up with valid answers to a defined problem. From a scientific and philosophical standpoint, ethical issues have to be addressed in future research on games for complex systems, together with better abilities of data collection. Privacy issues play a more important role than ever.

The use of scientific concepts, methods and instruments as proposed here is needed to design and develop a game, which has a valid foundation in theory and (complex) practice. This makes the development of games - as research instruments repeatable for other researchers and assures a good validation of the design and use. Our aim for future studies is to validate the outcomes as well we the instruments of our research. For example, we want to be able to pre-define the right level of a game’s fidelity, in order to represent the referent system and to enable learning transfer. The same holds for the evaluation of games. As comparison between games is still difficult and the number of respondents is limited in most cases, the need for validated instruments becomes more important. Many of such instruments might be available already in other fields, such as psychology, pedagogy, social sciences, or information sciences. Pursuing the interdisciplinary nature of game science is thus crucial to elevate this science to at least the trans-disciplinary level of highly structured design and research.