The impact of working memory and the "process of process modelling" on model quality: Investigating experienced versus inexperienced modellers.

A process model (PM) represents the graphical depiction of a business process, for instance, the entire process from online ordering a book until the parcel is delivered to the customer. Knowledge about relevant factors for creating PMs of high quality is lacking. The present study investigated the role of cognitive processes as well as modelling processes in creating a PM in experienced and inexperienced modellers. Specifically, two working memory (WM) functions (holding and processing of information and relational integration) and three process of process modelling phases (comprehension, modelling, and reconciliation) were related to PM quality. Our results show that the WM function of relational integration was positively related to PM quality in both modelling groups. The ratio of comprehension phases was negatively related to PM quality in inexperienced modellers and the ratio of reconciliation phases was positively related to PM quality in experienced modellers. Our research reveals central cognitive mechanisms in process modelling and has potential practical implications for the development of modelling software and teaching the craft of process modelling.

The processes of ordering shoes from an online store, taking out a loan, booking a flight, or renting a car can be depicted in formalized graphical models called process models (PM; Fig. 1A). PMs are most often applied to organize, optimize, and communicate a company´s business processes. They facilitate organizational processes and support the development of different types of information systems including process-aware information systems, service-oriented architectures, and web services123. PMs can be found in a variety of different domains including insurance companies, bank institutes, and hospitals.

Figure 1

(A) Depicted is an example of a (goal state) PM. (B) Illustrated are two modelling phase diagrams. (B1,B3) show that based on the same textual process description different creation processes are revealed. In (B2,B4) the x-axis represents time and the y-axis the number of elements in the model. The diagram depicts the number of C-, M-, and R-phases and their length of time. In (B1) a model is created in a straight–forward series of modelling interactions. The model of (B1) starts with adding a start (grey circle) event, followed by adding an activity element A. In turn, in (B3) two activities A and C are created, which are connected to the start and end event in a sequence (black circle). Following this, the modeller realizes that an activity B should be mutually exclusive to activity C. Thus, the modeller has to remove parts of the PM to add the missing model element. Consequently, several model elements are placed in the modelling area. Finally, the PM is laid out to complete the PM (B was taken from Pinggera et al.22) (C). Schematic illustration of the CEP modelling environment. The figure shows the textbox for the process description (C1), the modelling area including parts of a PM (C2), and the modelling toolbox with various operators (C3).

Various well known factors affect the ease with which a modeller will understand a PM, e.g., modelling expertise and process knowledge45, cognitive abilities, learning style and learning strategy6, the modelling notation7, as well as model characteristics8. While PM comprehension is well understood, only few studies focused on PM creation910. It has been demonstrated that complex PMs tend to contain more errors11, and that characteristics of the modelling task influence PM quality11. In addition, the importance of modelling knowledge is emphasized45 and first results point towards the impact of the process of creating a PM on its quality10. Nonetheless, a more detailed understanding of factors influencing PM quality, and therefore the creation of PMs, is in demand.

The creation of PMs can be characterized as a design activity within the field of problem solving121314. A variety of challenges are imposed, including the construction of a mental model of the domain as well as the externalization of the mental model by mapping it to the modelling elements provided by the modelling notation using a modelling tool. Literature suggests that cognitive abilities such as working memory (WM) are crucial for creating PMs of high quality, for design activities, and problem solving in general1516.

The aim of the present study was to obtain a better understanding of the factors leading to PMs of high quality by examining the role of working memory in experienced and inexperienced modellers. As a primary contribution, our research reveals central cognitive mechanisms in process modelling, and has potential practical implications for the development of modelling software and process modelling teaching. In addition, the paper aims to explore the roles of specific phases in the process of process modelling (PPM) while PMs are created. In the following we provide theoretical backgrounds on: (i) process modelling and PPM phases, (ii) PM quality, (iii) WM, and (iv) experience. This is followed by the description of our hypotheses, their statistical testing and discussion. Finally, a breakdown of the methodological approach is presented.

Theoretical Backgrounds

Process modelling as a cognitive design activity and PPM phases

Process modelling can be embedded in the PM lifecycle, which involves elicitation and formalization phases17. In elicitation phases, information is generated by domain experts, forming the foundation for formalization phases, in which process modellers create a PM18. The formalization of a PM can be described as a cognitively demanding problem solving process13192021 where modellers try to transform an initial state into a goal state. For instance, a textual and/or oral description of a process in a specific domain is translated into a formal PM, based on the application of specific operators (e.g. two activities such as “evaluate” and “decide” can be associated with the “AND” gateway; Fig. 1) and their restrictions (only use the “AND” gateway when activities are executed in parallel). The challenges imposed on the modeller have been assigned to three specific phases in the PPM by Pinggera et al.22. According to Pinggera et al.22 the PPM can be divided into three phases: comprehension, modelling, and reconciliation. Comprehension (C)-phases imply the understanding of the problem, mapping this understanding to specific modelling constructs of the modelling notation, and validating the (partial) mental and/or externalized model. Modelling (M)-phases define the externalization of the mental model as a PM, i.e. the creation of the PM through interaction with the modelling environment. Reconciliation (R)-phases involve reorganizing activities (e.g. renaming activities) and the usage of secondary notations23 (e.g. laying out of activities). The study by Claes et al.10 points towards the importance of the modelling process for the creation of high-quality models. The role of PPM phases for the creation of high-quality PMs is, however, not yet sufficiently understood and will be investigated as part of this study.

PM quality

PM quality is used as dependent construct in the present study and assesses the extent to which the final PM represents the requirements of the domain in a syntactically and semantically correct way. The final PM (representing the end state a particular modeller reached) results from a sequence of operator applications. To assess its quality, at least in laboratory settings, this final model can be compared with a model representing the goal state, i.e., the ‘ideal solution’ for this process. To operationalize the difference of the final PM and the goal state model we use two quality measures, graph edit distance (GED) and activity correctness (AC). GED quantifies the transformation steps (i.e. operator applications) required to convert the current PM into the ‘ideal solution’ of the PM2425. In turn, AC measures whether activities are identified and represented in a semantically correct way, i.e. whether all activities of the goal state model are included in the final PM as well.

Working memory

One of the most important cognitive systems for creating a PM is WM. WM defines a system, which retains a limited amount of information in a temporarily accessible state and makes it available for processing262728. For instance, when a PM is created, relevant information of the process to be modelled has to be activated, selected, and maintained and existing knowledge about different PM patterns, solution strategies, and software applications have to be implemented. Interindividual differences in WM capacity limitations have been related to a host of higher-order cognitive processes such as language comprehension29, logic learning30, fluid intelligence31, and the integration of pre-existing domain knowledge32 - cognitive abilities of high importance in creating PMs. There is an ongoing debate about the nature of interindividual differences in WM capacity and therefore different explanations exist. For instance, for Engle and colleagues3334 WM capacity is best defined as sustained attention in the face of interference or distraction. For Unsworth and Engle35 WM capacity refers to active information maintenance in primary memory (i.e. within the focus of attention) and a controlled search and retrieval from secondary memory (i.e. outside the focus of attention, in long-term memory). For Oberauer and colleagues363738 limits arise from interferences between information element bindings (e.g. words or sentences and their relation to each other).

There are a variety of different tasks for measuring WM capacity. In the present study we focussed on measuring two WM functions, (i) concurrent holding and processing and (ii) relational integration. Holding and processing was examined with the operation span (OSPAN) task. This task measures the ability to hold a limited amount of information outside the focus of attention (i.e. letters), while other information is simultaneously processed (i.e. solving arithmetic equations)3940. Holding and processing represents a WM function that has most often been related to higher order cognition293031. Relational integration measures the ability to build new relations between elements. This ability was examined with the spatial short-term memory (SSTM) task in which a sequence of dots have to be integrated into a pattern and maintained for a later recall4142. Oberauer et al.3743 found that both WM functions are highly correlated but separable processes and dissociable from a third construct, the so called supervision, which is responsible for attentional focus shifting between task sets44.

Within the process modelling literature there is clear awareness of the WM construct, its functions, and possible roles in process modelling1923,45–47. However, most often the central importance of WM in process modelling was primarily theoretically implemented. Only a few studies tested the role of WM in PM creation14 or related concepts (e.g. learning styles) in understanding PMs48. For instance, a direct evidence for a possible role of WM in process modelling was empirically tested by Sachse et al.14. Taking into account that WM consists of a set of processes, Sachse et al.14 related three different WM functions to various measures of PM quality. They found that WM updating was negatively related to syntactic errors, and relational integration was negatively related to semantic errors and positively related to GED.

Experience

A host of studies show that deliberate practice (training) improves performance in various domains such as chess, music, memory, informatics, and sport49. Practice can lead to cognitive and behavioural processes of much more accuracy, effectiveness, and ease50. One view about the theoretical explanation of this effect is that through practice an increasing number of information units (domain knowledge specific chunks) are generated and heavily interconnected in long-term memory, leading to a much more effective information processing despite limited WM capacity resources51. In other words, limited WM capacity resources can be circumvented by an increasing body of domain knowledge that is stored in long-term memory32495051. For instance, in the domain of sport knowledge, Fincher-Kiefer, Post, Greene, and Voss52 found that participants high in baseball knowledge outperformed participants low in baseball knowledge on a reading span task (which measures WM capacity for verbal information) when sentences were baseball-related. No differences were found when sentences were domain knowledge free. However, not only the amount of knowledge changes performance. Recent studies show that WM capacity explains additional variance in expert performance. For instance, Hambrick and Engle32 found that WM capacity contributed to memory performance even at high levels of knowledge in the game of baseball. In sum, practice together with WM capacity limitations play a relevant role for the development and interindividual differences in expert performance. In the present study we investigated two groups with various amounts of experience in the creation of a PM. Even though empirical evidence exists that modelling experience, education, and knowledge on process modelling are relevant factors for the understandability of PMs45, no study to date investigated the role of cognitive (WM) and PPM factors within and between modellers of various experience levels in the creation (vs. comprehension45) of PMs. More specifically, our study investigated relevant WM factors in addition to relevant PPM phases in the earliest phase of expertise development.

The Present Study

In line with the theoretical background discussed above, we argue that the quality of PMs depends on the modeller’s process modelling experience, his/her cognitive abilities (i.e., WM), as well as the process that is followed in terms of PPM phases. However, the relations between specific WM functions, PPM phases in creating a PM, and the role of modelling experience is poorly understood. Therefore, we investigated two WM functions (holding and processing with the OSPAN task, and relational integration with the SSTM task42) and three PPM phases (C-, M-, and R-phases22) in relation to the quality of a PM, in inexperienced and experienced modellers.

Existing studies show that modelling experience is a relevant factor for PM comprehension45. In addition, studies show that experience in a certain domain can improve domain specific task performance49. We hypothesized that:

H1: Experienced modellers create PMs of higher quality compared to inexperienced modellers.

There are theoretical considerations of WM as a relevant construct for creating PMs1923454647. What remains unaddressed is the question whether there are differences depending on modelling experience. In extension to Sachse et al.14, the present study investigated the WM functions of concurrent holding and processing of information, and relational integration in relation to PM quality, in experienced and inexperienced modellers. Based on the view that the applied WM tasks are domain knowledge free measures, and existing studies show that WM is related to task performance within inexperienced1437 and experienced participants53 we hypothesized that:

H2: Both WM functions are positively related to PM quality in inexperienced and experienced modellers.

The roles of different PPM phases1922 for the creation of high-quality PMs is not yet sufficiently understood, leading to difficulties in formulating strong hypotheses. In the present study, the ratios of the number of C-, M-, and R-phases and total processing times (in the following rC-, rM-, and rR-phases) were related to PM quality for the first time. We assumed that unfamiliar and complex tasks are cognitively demanding and time consuming, especially for inexperienced modellers. Higher rC-phases can therefore be an indicator for problems within the modelling process, for instance, difficulties in text understanding, application of operators, or creating a mental model based on conceptual information. The same can be supposed to be the case for rM-phases. A higher number of rM-phases, i.e. higher numbers of adding and deleting operations can be an indicator for problems during the externalization of the PM. Assuming an ideal modelling process without the modeller facing any problems, a modelling process with just a single rC- and a single rM-phase would be the result. In a first step a goal state model is mentally created within a single rC-phase before the PM is externalized in a second step, within a single rM-phase. Based on this view we hypothesized the following:

H3: rC-phases are negatively related to PM quality, particularly in inexperienced modellers.

H4: rM-phases are negatively related to PM quality, particularly in inexperienced modellers.

Regarding the rR-phases we assumed that reorganizing and restructuring model elements can be viewed as important steps in creating readable and understandable PMs. This should lead to a higher chance of finding contained errors before the creation of the PM is finished and facilitates the creation-process itself by lowering the demands for the correct positioning and integration of new elements within the PM. Consequently, PMs of higher quality should be created. We hypothesized that:

H5: rR-phases are positively related to PM quality in inexperienced and experienced modellers.

Results

Descriptive statistics in addition to the results of the group specific analyses based on t-tests can be found in Table 1.

Table 1

Descriptive and inference statistics of PM quality, total processing time, WM functions, and PPM phases for inexperienced and experienced modellers.

		Quality	Time	OSPAN	SSTM	C-phases	M-phases	R-phases	rC-phases	rM-phases	rR-phases
Inexperienced	M	0.80	29.66	0.61	0.84	18.21	21.04	12.04	0.62	0.70	0.41
	SD	0.10	9.73	0.06	0.06	5.74	5.11	6.62	0.08	0.13	0.18
Experienced	M	0.82	23.61	0.60	0.88	14.72	16.00	10.48	0.62	0.73	0.45
	SD	0.10	8.13	0.07	0.04	5.85	4.76	5.74	0.11	0.11	0.20
Group differences	t	−0.47	−2.55	0.84	−2.82	2.27	3.78	0.95	0.05	−0.73	−0.89
	df	57	55	50	53	55	53	55	55	56	57
	p	0.637	0.014	0.407	0.007	0.027	0.001	0.348	0.958	0.471	0.379
	η2	0.004	0.106	0.014	0.131	0.084	0.212	0.016	0.001	0.001	0.014

Note. M: mean; SD: standard deviation; Time: total processing time in minutes; OSPAN: operation span; SSTM: spatial short-term memory; C-phases: number of comprehension phases; M-phases: number of modelling phases; R-phases: number of reconciliation phases; rC-phases: ratio of comprehension phases; rM: ratio of modelling phases; rR: ratio of reconciliation phases. rC-, rM-, and rR-phases scores are based on the C-, M-, and R-phases divided by individuals´ total processing time in minutes. Bold: significant results, p < 0.05.

H1. T-test analysis revealed no significant differences regarding PM quality between inexperienced and experienced modellers.

H2. T-test analyses showed that inexperienced and experienced modellers did not significantly differ in ability to hold and simultaneously process information, measured with the OSPAN task. However, experienced modellers were significantly better in integrating information elements and their relations, measured with the SSTM task. In a multiple regression analysis, where quality was regressed on the two WM functions we found that only relational integration (SSTM) was significantly related to quality, in both groups (inexperienced modellers: R = 0.227, F(2,21) = 4.56, p = 0.024; f = 0.479; OSPAN: β = −0.027, p = 0.889; SSTM: β = 0.570, p = 0.007; experienced modellers: R = 0.488, F(2,23) = 10.01, p = 0.001; f = 0.953; OSPAN: β = 0.272, p = 0.121; SSTM: β = 0.551, p = 0.004). The corresponding data distribution of the two WM functions with quality can be found in Fig. 2.

Figure 2

Depicted are the plots of the relations between PM quality (x-axis), OSPAN, SSTM, rC-, rM-, and rR-phases (y-axis) for inexperienced and experienced modellers.

OSPAN: operation span; SSTM: spatial short-term memory; rC-phases: ratio of comprehension phases; rM-phases: ratio of modelling phases; rR-phases: ratio of reconciliation phases.

H3–H5. First, we analysed group differences regarding the total processing time and number of C-, M-, and R-phases before we focussed on group differences regarding rC-, rM-, and rR-phases and their relations to PM quality. T-test analyses revealed that inexperienced modellers needed significantly more time for creating the PM compared to experienced modellers. Regarding the number of C-, M-, and R-phases we found, that inexperienced modellers showed significantly more C- and M-phases compared to experienced modellers. Both modeller groups did not significantly differ regarding the R-phases. Further, no significant differences in rC-, rM-, and rR-phases were found. In a multiple regression analysis, where quality was regressed on the three r-phases we found, that rC-phases were significantly related to PM quality in inexperienced modellers (R = 0.451, F(3, 25) = 6.03, p = 0.004; f = 0.821; rC-phases: β = −0.646, p = 0.011; rM-phases: β = −0.035, p = 0.890; rR-phases: β = −0.112, p = 0.574). In experienced modellers rR-phases were significantly related to quality (R = 0.326, F(3, 27) = 3.88, p = 0.022; f = 0.484; rC-phases: β = 0.244, p = 0.248; rM-phases: β = −0.415, p = 0.065; rR-phases: β = 0.612, p = 0.003). Corresponding data distributions of the three PPM phases with PM quality can be found in Fig. 2.

To summarize, our results revealed that higher abilities in relational integration and a lower number of comprehension phases per time increased PM quality for inexperienced modellers. In experienced modellers, higher abilities in relational integration and a higher number of reconciliation phases per time increase PM quality. Finally, we compared the beta weights of the regression analyses between inexperienced and experienced modellers. We found no significant differences in the beta weights between the two modeller groups (OSPAN: p = 0.208; SSTM: p = 0.268; rM-phases: p = 0.127; rR-phases: p = 0.056), except for rC-phases (p: = 0.010). These results indicate that only the relation between the rC-phases and PM quality was significantly stronger for inexperienced modellers compared to experienced modellers.

Discussion

The present study investigated the role of different WM functions and PPM phases in process modelling. Our results indicate that relational integration is strongly related to PM quality in inexperienced and experienced modellers. Additionally, lower rC-phases in inexperienced modellers and higher rR-phases in experienced modellers seem to be an indicator for PMs of higher quality.

H1 – PM quality differences

In our study, inexperienced and experienced modellers created PMs of the same quality, a finding that contradicts H1. There are at least three explanations for this discrepancy. First, we set no time limit for creating the PM. Compared to experienced modellers, inexperienced modellers needed significantly more time for creating the PM. We assume that processing time can support mental processes like cognitive control (e.g. examining whether a specific activity was set or a PM solution fits with the text content), problem solving processes (e.g. mentally creating one or more solution alternatives), and even can reduce stress induced error behaviour (e.g. no time pressure for thinking, modelling, reorganizing, and laying out the PM; see Arnsten54). Second, task complexity might have been too low, i.e. the number of notational elements required to create the solution model was rather small. This explanation was supported by the relatively high quality scores of .82 and .80 for experienced and inexperienced modellers, respectively. Third, we investigated expertise in its earliest phase. That is, even though not expected, our intensive training session for the inexperienced modellers might have led to a steep learning progress resulting in similar modelling performance between the two modeller groups (at least under our modelling conditions; but see studies into the power and exponential functions of learning5556). In this context it is even more interesting to see that relational integration in both groups and different PPM phases are related to similar PM quality outputs. In other words, differences between the two modeller groups appeared in the PPM phases, whereby relational integration was a common predictor for PM quality in both groups.

H2 - Working memory and PM quality

The significant relation between relational integration and PM quality is in line with existing studies which show that relational integration represents a central predictor for reasoning ability and fluid intelligence3743. High abilities in holding the relations of verbal information and modelling elements (e.g. operators) in an active state seem to reduce modelling errors and increase the probability of representing the model in a semantically and syntactically correct manner57. The relevance of relational integration in process modelling seems to be valid for inexperienced and experienced modellers. This finding is interesting, because one could argue that training should reduce the dependency of our cognitive system on WM51. This is, among other things, because training/experience leads to accumulation of process modelling knowledge which is organized and stored in chunks in long-term memory. These knowledge chunks are easily accessible by only a few cues. As a result, high amounts of process modelling knowledge are at hand and ready for a relatively automatic application48. However, there are at least two reasons for our results. First, our experienced modellers cannot be considered as modelling experts. Regarding this, WM or specific WM functions should play a role, because WM reveals its predictive power especially in new situations, contexts, and tasks31585960. Second, even in experienced modellers WM represents a central predictor for task performance. Existing studies show that training and the application of mental strategies can circumvent existing short-term memory capacity limitations and reduce the predictive power of IQ61. However, contrary to these findings a recent meta-analysis indicates that deliberate practice has a significant impact only in specific domains like games, music, and sports, and is less important in contexts like education and professions62. For Macnamara et al.62 WM and IQ represent strong candidates for explaining variance in performance that deliberate practice cannot explain53. Our study supports this view and extends it assuming that specific WM functions may be obtained while the predictive power of others is weakened by training depending on the task context (e.g. with and without the possibility of information externalization).

Finally, we did not find a relation between the function of holding and processing and quality, even though this WM function is, like relational integration, strongly related to a host of higher-order cognitive abilities (e.g. problem solving, text comprehension, intelligence29303132). One possible explanation for this finding is that WM capacity is often defined as ability to actively maintain information outside the focus of attention, while simultaneously other information is processed within the focus of attention33. In our context, the textual process description, modelling operators, and the created model were visible throughout the whole processing time. Therefore, information inside and outside the focus of attention could be continuously refreshed, possibly making this specific WM function less relevant2627.

H3 to H5 - PPM phases and PM quality

Our results indicate that low amounts of rC-phases, i.e. phases in which no operator applications took place, increased the quality of a PM. We reason that high WM capacity modellers, presumably modellers with higher abilities in relational integration create PMs of higher quality with less rC-phases. Training of the specific cognitive abilities like relational integration and/or pre-existing process modelling knowledge, which helps to circumvent existing WM capacity limitations51, can serve as arguments why experienced modellers showed significantly less rC-phases compared to inexperienced modellers.

Regarding reconciliation phases, we proposed that renaming and layout activities increased the understandability and subsequently the quality of the PM. This view is supported by our results. We assume that reconciliation can be understood as an external thinking strategy. After externalization of model information, visual organization of model elements can facilitate the problem solving process through, for instance, validation and/or reinterpretation processes57. For example, the reorganization of existing model elements can help to create subsequent model elements more effectively and accurately, reducing superfluous modelling steps and cognitive load. However, it is important to note that we did not classify our participants into various types of modellers. For instance, we did not differentiate between modellers (i) creating correct models from the beginning without the need for renaming and layout activities, (ii) continuously incorporating R-phases in process modelling, and (iii) applying R-phases not until the end. Additionally, even though higher rR-phases led to PMs of higher quality, one cannot exclude that different modelling styles can be equally effective, for instance, modellers creating high quality PMs without the need for a higher number of R-phases. Applying such a modelling behaviour would require a precise depiction of the PM in mind and an accurate translation of the mind’s model into an externalized model.

Our results with respect to rC- and rR-phases can explain why rM-phases did not seem to be related to PM quality in both groups. We assume that PM quality heavily depends on the amount of effort spent on the comprehending the textual process description and the effectiveness of visually and semantically organizing the PM in order to correctly transform the task description into a high quality PM. The simple act of moving an already mentally chosen element from the operator toolbox into the modelling area seemed not to be of relevance in this process. However, it should be noted that these findings do not rule out a possible role of modelling phases in creating PMs of high quality. The impact of the modelling phases on PM quality could be revealed by investigating their relations with specific quality measures. For instance, it is conceivable that there is a relation between modelling phases and syntactic quality63.

In this context, the question arises how WM is related to the three PPM phases, highlighting a central role of WM in the creation process of a PM. Based on the outline above one can hypothesize that WM is negatively related to rC-, rM-, and positively related to rR-phases because WM plays a central role in integrating, creating and restructuring information elements. Testing this, we found that experienced modellers with lower relational integration abilities showed more rC-phases (R = 0.319, F(2, 26) = 5.61, p = 0.010; f = 0.468; OSPAN: β = −0.095, p = 0.602; SSTM: β = −0.524, p = 0.008). This finding is in line with existing studies which show negative relations between WM and, for instance, reading comprehension and processing time6465, problem solving31, and encoding of new information59. Additionally, we found a significant regression model for the rR-phases in the experienced modellers group (R = 0.221, F(2, 26) = 3.40, p = 0.050; f = 0.284; OSPAN: β = −0.265, p = 0.182; SSTM: β = −0.305, p = 0.126). Interpretation of this result is not straightforward. On the one hand, we showed that high rC phases lead to PMs of higher quality. On the other hand, our findings indicate that high WM capacity individuals had less rC-phases. This result confirms the assumption that rR-phases play a role as cognitive strategy to circumvent WM limitations, and therefore decrease cognitive load.

Finally, our findings indicate that inexperienced and experienced modellers differed in their ability to integrate information elements and their relations. Whether this finding is a training and/or population specific effect cannot be solved with our study. It is a matter of ongoing debate whether WM and its functions are trainable and transferable to other cognitive abilities like fluid intelligence and/or problem solving6667.

Limitations

A possible limitation of our study was that we investigated process modelling under laboratory settings. A methodological advantage of this approach is the possibility to control for interacting factors, e.g., task instructions, task complexity, working conditions (e.g. use of external memory aids like sketches), and experience. However, a disadvantage is that laboratory experiments lack ecological validity. This means that in contrast to laboratory settings, modellers in applied settings integrate relevant information from various sources (e.g. a conversation, piece of paper) that is presented in a less well-structured way. Additionally, future research has to confirm our results and delineate them from performance in expert modellers (e.g. based on a minimum number of created PMs). This has to be investigated not only for additional measures of PM quality (e.g. syntactic errors) but also for the PPM. Finally, when interindividual modelling differences are investigated additional factors, beside practice and cognitive abilities, have to be considered (e.g., personality factors such as “passion” or achievement motivation; handling performance anxiety and “choking” under pressure; executive functions such as memory updating and inhibition of irrelevant information; and genetic variances)53.

Implications

Potential implications of our results can be seen in the construction of modelling environments in which the modeller together with an intelligent modelling software agent creates PMs of higher quality, much more effectively. For instance, an agent could detect the individual modelling behaviour and dynamically organize and support specific tool features to optimally structure modelling sessions especially in early PPM phases68. At the same time this agent can flexibly shape the modelling environment in a way that it adapts automatically to the cognitive (WM) capacity constraints of a modeller through, for instance, WM unloading opportunities in certain PPM phases (e.g. C-phases). Further, our results can have an impact on teaching the craft of modelling. For instance, students creating PMs can be monitored on the fly, immediately identifying problems during the creation process (i.e. in which phase) in order to provide individual support.

Methods

Ethics statement

The present research was conducted with approval by the Institutional Review Board of the University of Innsbruck (Faculty of Psychology and Sport Science). All applied methods were carried out in accordance with the approved guidelines. Participants gave written informed consent.

Participants

Sixty students from the University of Innsbruck took part in the experiment. Experienced modellers (22 males, 8 females, mean age = 26.76 years, SD = 2.90) represented computer science students familiar in process modelling (through process modelling lectures and/or the composition of a bachelor or master thesis on process modelling). Inexperienced modellers (16 males, 14 females, mean age = 22 years, SD = 4.58) represented psychology students without process modelling experience, but trained in process modelling notation and program handling for the first time.

Materials

Modelling task

Participants of both groups were required to model a lending process based on a textual process description. Participants of both groups were tested in single experimental settings and worked on the same process modelling task (for the full task text: http://bpm.q-e.at/WorkingMemory). No time limit was set. The modelling task was implemented in Cheetah Experimental Platform (CEP)69. For a representative quality score we computed a mean global quality score based on two measures, graph edit distance (GED) and activity correctness (AC).

GED quantifies the transformations required to convert one PM to another PM using a series of elementary transformations2425. More specifically, node substitutions, i.e. replacing one node by another, node insertions/deletions and edge insertions/deletions are used24. By associating each elementary transformation with a cost function, the cost of transforming one PM to another can be calculated. In our context, we calculated the cost of transforming the PM created by a modeller to a model representing the goal state. For this, we conducted the following steps. First, we manually constructed a mapping of the activities in the PM to the activities of the goal state model. This is necessary because participants differ in the way they label the activities. Second, the GED was calculated using the algorithm described in Dijkman et al.24 to obtain the elementary transformations for each PM. Third, the graph edit similarity was calculated to obtain a normalized result between 0.0 and 1.0, where 1.0 constitutes an exact match of the two PMs, i.e. the PM and the goal state model (Fig. 1A).

Regarding AC we applied a grading scheme measuring which activities described in the textual process description were correctly identified and represented in the PM. Therefore, we compared the activities depicted in the end model of each participant with the goal state model (Fig. 1A). In order to obtain the overall quality of the PM, each correct activity was scored by one point. The overall AC score of the PM was defined as the sum of the correct modelling interactions. For the purpose of normalizing the AC score and making it comparable with the GED score we divided the AC sum score by the maximum number of correct activities. Therefore, we obtained an AC score between 0.0 and 1.0, where 1.0 indicated that all activities were present. To reduce measure specific variance we calculated a general quality score. The general quality score was defined as the mean of the normalized GED and AC scores.

The number of C-, M-, and R-phases were automatically identified based on an algorithm described in Pinggera et al.22. For further analysis we calculated a ratio C-, M-, and R score. For this, we divided the number of C-, M-, and R- phases by the individuals’ total processing time (rC-, rM-, and rR-phases).

Working memory

We measured the WM function of holding and processing with the OSPAN task, and relation integration with the SSTM task. Both WM tasks were taken from the WM test battery of Lewandowsky, Oberauer, Yang, and Ecker40. The tasks ran on MATLAB 7.10.0.499 (R2010a) with the Psytoolbox Version 37071.

OSPAN task

Participants saw an alternating sequence of arithmetic equations (e.g. 4 + 3 = 7) and consonants that had to be remembered in the correct order for later serial recall. In each trial the list length (i.e. the number of equations-letter alternations) varied between four to eight (3 trials per list length). There were 15 experimental trials and 3 practice trials. Participants’ operation span (OSPAN) score was computed based on the proportion of items recalled correctly. A detailed description of the task can be found in Lewandowsky et al.42.

SSTM task

Participants saw a 10 × 10 grid in which various numbers of dots were presented one by one. Participants were required not to recall the absolute spatial position of the dots, but their spatial relations. The set size (number of dots per trial) was between two to six dots. The task consisted of 30 experimental trials, 6 at each set size. Experimental trials were preceded by two practice trials. Participants´ spatial short-term memory (SSTM) score was computed according to the distance between the learning dots and the generated dots. If the distance was 0 cells, 2 points were gained. If the distance was 1 cell, 1 point was gained. If the distance was further than 1 cell, 0 points were gained. The total score in the SSTM task represented the sum of all scores on all trials divided by the total number of possible points. A detailed description of the task can be found in Lewandowsky et al.42.

Procedure

Inexperienced modellers

Participants were first informed that the experiment consists of two sessions, (i) a process modelling training and (ii) a testing session. In the first part of the training session participants were prompted to conduct two working memory tasks, OSPAN and SSTM. WM tasks were conducted one by one always after a short task specific introduction through the examiner. After a break of 15 minutes participants were trained in CEP69 (see Fig. 1C for the CEP modelling environment). The training consisted of three parts. First, participants got an example-based introduction to process modelling (what is understood by process modelling, “XOR” and “AND” gateways etc.). Second, participants were induced to the CEP environment (the examiner guided through CEP with a simple example PM participants had to create, including layout guidelines and examples for PMs of low quality). Third, participants were required to model a process modelling task themselves. An ideal PM solution for this task was presented at the end of the training session and all open questions were answered. Training took about 90 minutes in a computer laboratory with 25 laptops. Training group size was between 5 and 15 participants. After all participants had finished CEP training, they were prompted to indicate one possible time point for the process modelling testing session. All participants were tested within seven days after training. The process modelling testing session consisted of two parts, a familiarization and a testing phase. Preliminary to the testing phase participants of the inexperienced modellers group were given the possibility to freely use CEP for several minutes to get familiar with the program before they called the examiner that they are now ready for modelling the main experimental task. The examiner started CEP and the participant began working on the modelling task.

Experienced modellers

Participants were directly invited to the single testing sessions. Testing sessions were fully identical to the inexperienced modellers group, except that experienced modellers were first tested in the modelling session before they conducted the two WM tasks (OSPAN and SSTM) after a break of 15 min.

Statistical analyses

For statistical analysis the alpha level was set at .05, testing two-tailed. We first analysed group differences regarding PM quality, the different WM functions, and PPM phases based on t-tests. This was followed by multiple regression analyses where we regressed quality on the two WM functions, and quality on the three PPM phases. Variations in the degrees of freedom were based on missing values and/or due to software malfunction. Outlier analyses were based on boxplot diagrams, calculation of the leverage values and studentized deleted values7273.

Additional Information

How to cite this article: Martini, M. et al. The impact of working memory and the “process of process modelling” on model quality: Investigating experienced versus inexperienced modellers. Sci. Rep. 6, 25561; doi: 10.1038/srep25561 (2016).