Effectiveness of game development-based learning for acquiring programming skills in lower secondary education in Croatia.

A European initiative CODING4GIRLS (C4G) promotes the acquisition of programming skills through a game development process with the aim of preparing young learners, especially girls, to enter computer science careers and raising awareness of the relationship between ICT and the real world. Using the C4G game development-based learning methodology, students develop serious games for which they need to learn specific programming concepts. This paper presents the results of a study with a mixed-gender group of both boys and girls (N = 773) carried out with the aim of examining the effectiveness of the C4G development-based learning approach in lower secondary education in Croatia. In-service and pre-service teachers organized learning activities for students based on the C4G learning scenarios, which include the development of games in the programming language Snap! with topics that are interesting for both boys and girls and which involve solving real-world problems. The results showed that students accepted the C4G methodology and were motivated to learn how to program by developing games for solving real-world problems. Teachers and experts consider this approach as a relevant and effective method for achieving learning objectives related to programming, applicable and suitable for lower secondary students (11-15 year olds).

Introduction

Introducing programming concepts and enhancing the development of programming skills in an appropriate manner is a challenge at all levels of education. Therefore, various innovative methodologies, including game-based learning (GBL), are used in teaching (Topalli and Cagiltay 2018), (Shabalina et al. 2017), (Tuparova 2019). To encourage learning programming from an early age, specialized visual tools and programming languages such as Scratch (Scratch 2020), Snap! (Snap! 2020), and Alice (Alice 2020) have been developed. Using these tools, students create programs by manipulating graphical elements (blocks) instead of writing text commands and can easily create interactive stories and games, which positively affects their motivation and engagement in learning activities involving programming (Lau 2018), (Holenko Dlab et al. 2019), (Sáez-López et al. 2016).

Using these tools, teachers and researchers try to motivate students for the acquisition of programming skills but also to encourage some of them to choose computer science and ICT for their future careers. Teaching programming in primary and lower secondary education is considered as one of the major factors for promoting continuing education in these fields among students (Wong et al. 2016). Despite advances in the use of methods and tools for teaching programming, there is still a lack of skilled ICT workers as well as a gap between the skills that students acquire through formal education and skills needed by industry (European Commission 2010). Another global problem is the gender gap in computer science education and careers. There is only 17% of females among ICT specialists and 34% of females among STEM graduates (European Commission 2019). Also, there is only 18% of females among university graduates in computer science in the US (Cunningham et al. 2015). Therefore, interventions to address this problem at the global level are needed to ensure sustainable economic growth. One approach is to try to make computer science, STEM, and fields related to the ICT attractive to boys and girls using digital educational games (De Carvalho et al. 2020), (Carmichael 2008). This approach should be used already from primary and lower secondary education since at that age many students lose interest in computer science and STEM (Chavatzia 2017).

When it comes to acquiring programming skills, students can learn programming concepts and constructs through playing games (Shabalina et al. 2017), (Kazimoglu et al. 2012). By visualizing algorithms or facing students-players with various challenges, such games encourage good programming practices (Giannakoulas and Xinogalos 2018). Many games for learning programming are available online (Franković et al. 2019). Examples include puzzle-based games like LightBot (LightBot 2020) where the player needs to guide a robot around a 3D maze using the minimum number of commands and Dragon Dash (Tynker 2020) where the player should lead the dragon to the treasure as well as adventure games like Run Marco! (Allcancode 2020) and Bee (Code.org 2020) where the player leads the main character through the given path using block-based programming. Games like LightBot and Bee are appropriate for very young students since commands are represented with symbols for movement and other actions so students do not have to know to read. Besides for learning basic concepts like sequence, loops, variables, and conditionals, some of these games can also be used for learning more advanced concepts like abstraction, function, and reuse (López et al. 2016), (Wu and Wang 2012).

Besides playing educational games, young students can be encouraged to develop games and in that way master the programming concepts (Weintrop and Wilensky 2015). Such game development-based learning (GDBL) approach assumes that students create their own games using block-type programming tools approaches (Rugelj and Lapina 2019). The complexity of the games should be aligned with the learning outcomes, i.e. programming concepts that students need to master during the course. The simplest type of GDBL activity is that students modify an existing game by adding some components. The more advanced type is to create a simple game from scratch to practice one or more programming concepts, while in the most advanced type (final exercise), students are expected to develop a game by applying all the concepts from the course (Wu and Wang 2012), (Shabalina et al. 2017). There are many research studies that confirmed the effectiveness of the GDBL approach for acquiring programming skills among university students (Wu and Wang 2012). The GDBL approach has also proven effective in upper secondary education, for example in Greece where students were supposed to follow the instructions and create games using Scratch in order of learn basic programming concepts (Seralidou and Douligeris 2020), but studies in primary and lower secondary schools are focused on GBL approach and explore the effectiveness of playing games for learning programming.

Due to the lack of textbooks in their countries that offer practical GDBL activities and exercises for students, teachers often need to produce assignments and instructions for students by themselves (Seralidou and Douligeris 2020). Therefore, there is a need to create collections of materials for learning programming using the GDBL approach that will be publicly available to teachers, which will affect the wider application of this approach and more research studies. GDBL scenarios or educational sheets for learning programming can be inspired by real-world situations and promote critical thinking, which is an integral part of programming (Biagi and Loi 2012; European Commission 2010). Student engagement in game-based activities can depend on gender. Girls prefer puzzles, fantasy, and role-playing games as well as games that include storyline, exploration, challenges, and problem-solving tasks (Alserri et al. 2018), Hosein 2019), but are not fond of direct competition (Alserri et al. 2018). A European initiative that introduces an innovative methodology for building programming skills through game development-based learning (GDBL) is CODING4GIRLS (C4G) (De Carvalho et al. 2020). Using C4G learning scenarios, young students are encouraged to acquire and apply programming knowledge in developing games that address real-world problems with topics interesting for both girls and boys.

This paper presents the results of a study that explores the effectiveness of the GDBL approach for acquiring programming skills in lower secondary education in Croatia. The results contribute to the promotion of the GBDL approach to teaching programming and further understanding of the elements of GDBL approach that influence the positive attitudes of students and teachers regarding the effectiveness and appropriateness of the approach for building programming skills.

CODING4GIRLS Project

CODING4GIRLS (Coding4Girls web site 2018) is an ongoing project funded by the Erasmus+ Programme of the European Union under the Key Action 2: Cooperation for innovation and the exchange of good practices (Action Type: Strategic Partnerships for school education).

The project CODING4GIRLS aims to address the gap between male and female participation in computer science education and careers by a introducing methodological learning framework that would make this field attractive to young students (De Carvalho et al. 2020), (Hoić-Božić et al. 2020). Computer science and ICT are traditionally male-dominated areas, both in the European Union (EU) and beyond, so the main goal of the project is to attract girls by raising their awareness of the possibilities for professional and personal growth that computer science offers. The idea of the project is also to promote equality of gender and prepare both, boys and girls, to enter computer science careers by building programming skills.

The project started in September 2018 and lasts until December 2020. The project coordinator is the University of Ljubljana (Slovenia) and partners are Virtual Campus (Portugal), University of Thessaly (Greece), University of Rijeka, Department of Informatics (Croatia), EU-Track (Italy), Governorship of Istanbul, European Union and Foreign Affairs Department (Turkey), and South-West University “Neofit Rilski” (Bulgaria). Members of the project team are experts in the field of programming, methodology of programming, e-learning, learning with the help of educational games and computer game development, chosen to ensure the multidisciplinary approach to the project.

The project involves the development of the three main intellectual outputs:

Methodological Learning Framework,

Promoting the Development of Programming Skills among Girls through Serious Games,

Instructional Support Content.

C4G aims to introduce an innovative Methodological Learning Framework for building programming skills through the game-based learning (GBL) and game development-based learning (GDBL). The framework enables learners to apply the newly developed programming knowledge in wider learning contexts as well as to build transversal competencies related to programming (e.g. analytical and critical thinking). Innovative interventions are introduced concerning factors that cause girls not to choose computer science and ICT to be a field of their future occupation such as insufficient level of acquired skills and their misperceptions regarding the benefits and suitability of computer science careers (Franković et al. 2019).

The development of entrepreneurial competences related to the design thinking approach is also encouraged, as students are asked to devise solutions that meet the needs of the community and solve real-world problems in an innovative way. Design thinking methodology also encourages students to think entrepreneurially about how computer science and ICT can solve real-world problems (Wrigley and Straker 2017). Long-term involvement in the fields of computer science and ICT is also promoted, pointing to their importance in addressing issues of wider social interest.

Using the Methodological Learning Framework, C4G promotes the development of programming skills. Students are encouraged to develop game-based solutions that address real-world problems in a block-based visual programming environment Snap!. These are simpler problems suitable for children, but they can serve as an example of how programming can be used to solve problems from real life. The framework enables students to start solving easier problems, gradually introducing them to more difficult ones, to encourage continued participation and interest (Repenning et al. 2010). Students are also presented with semi-finished games that they should complete with the implementation of smaller modules so that it all makes a meaningful whole. In that way, girls and boys are provided with skills and knowledge to design and develop their own games taking a constructive approach to learning.

In addition to GDBL, the development of programming skills is also promoted by GBL since students are encouraged to play a video game. To achieve this, a 3D single-player first person adventure game was developed in Unity. The game includes various logical games attractive to girls (e.g. puzzles, mazes, and other problem-solving mini-games) that are combined with the challenges in Snap!. In addition, mini-games are used to entertain the students but also to introduce new programming concepts more easily. Each chapter of the game includes one GDBL challenge in which students can apply newly acquired knowledge by programming game in Snap!. The challenge (the game) includes a real-world problem in line with design thinking methodology and it is presented at the beginning of the chapter. Students have the opportunity to discuss and share ideas regarding the possible solutions and after the brainstorming, they start solving the challenge in Snap!. Although the Scratch programming language is often used to develop programming skills in elementary schools, Snap! was chosen for the C4G methodology because it is suitable not only for beginners but also for advanced learners, as it incorporates higher-level programming concepts (e.g., class lists and procedures, nested functions). An additional value for the C4G methodology was the fact that the user interface of Snap! could be embedded in the 3D game, unlike the Scratch user interface.

The C4G project also aims to provide educators with instructions on how to integrate the proposed methodologies and tools into their teaching practice. Therefore, teachers are provided with the instructional support content that includes learning sheets, video tutorials, and user guide on the C4G serious game approach for building programming skills among girls through design thinking approaches.

Research methodology

The study described in this paper aims to explore the effectiveness of the C4G GDBL development-based approach for learning programming in lower secondary education in Croatia. For this purpose, the implementation of learning activities based on the C4G approach was carried out in elementary schools in Croatia. The research methodology was aligned with the CODING4GIRLS validation methodology that is based on a descriptive case study research (Yin 2003).

The specific research questions are:

How do students’ habits regarding the use of digital devices, the Internet and video-games, motivation to learn programming, and perceived level of their programming skills differ by gender?

How does motivation for learning programming differ by gender?

To what extent are students familiar with basic programming concepts?

What is the difference in the students’ perceptions regarding the level of their programming skills before and after C4G GDBL activities?

Do students consider the C4G methodology effective and appropriate for building programming skills and why?

Do teachers and external experts consider the C4G methodology relevant, effective, and appropriate for building programming skills among students aged 11–15 and why?

In order to answer the research question, quantitative and qualitative methods were used.

Context and participants

In Croatia, primary and lower secondary education is organized as a single structure system and delivered by the network of elementary schools. It is compulsory, starting at the age of 7 and lasting 8 years.

According to the “National Curriculum for the Subject of Computer Science/Informatics” for primary schools and secondary schools – gymnasiums (MZO 2018) which was implemented in schools starting from the 2018/2019 school year, learning objectives related to the development of programming skills are realized in several grades within the courses with the unified name Informatics. Informatics is a compulsory subject for 5th and 6th grade students, and optional for 7th and 8th grade students. There are four domains in which the learning outcomes are realized: e-Society, Digital Literacy and Communication, Computational Thinking and Programming, and Information and Digital Technology.

For the CODING4GIRLS project, the domain Computational Thinking and Programming is the most relevant (although there is a strong link between learning outcomes from all four domains). According to the curriculum, solving a particular problem by creating a computer program involves not only standard program development procedures, but also innovation, entrepreneurship, and taking the initiative in designing and developing new models and products using computer technology. Programming develops self-confidence, perseverance and precision in correcting mistakes, the ability to communicate and work together to achieve a specific goal.

Participants of the study were in-service teachers of informatics (NT = 8) from 7 elementary schools in Rijeka, Croatia together with their students (NS = 773). All the teachers have years of experience in teaching informatics and they are mentors in informatics for pre-school teachers – students of informatics from University of Rijeka (UNIRI) during the teaching practice in informatics. Pre-service teachers - students of informatics (NST = 35) also participated in the study. They were at the time in the final year of their study and therefore obliged to attend a teaching practice in informatics. Depending on the number of classes and students, each in-service teacher (mentor) was assigned with 3 to 6 pre-service teachers (students of informatics) who helped to prepare and conduct the C4G activities under the mentor’s supervision.

The C4G approach for building programming skills was applied in 38 mixed-gender classes and involved a total of 773 students aged from 11 to 15 years (5th to 8th grade of elementary school). Table 1 shows number of students – participants of the study by age/grade. The number of students in the classes ranged from 13 to 28 while the average number of students in one class was 20.

Table 1

Number of students by age/grade

Years of age	Grade	Classes	Number of students
11–12	5th	9	191
12–13	6th	14	308
13–14	7th	8	146
14–15	8th	7	128
Total		38	773

Three external experts (NE = 3) were also included in the study based on their expertise. Two of them are university professors and researchers, one in the field of teacher education and the other in the fields of computer science and teaching programming. Third expert is a teacher advisor for informatics who works in an elementary school.

Data collection tools and analysis

During the study, the data collection tools provided by the C4G validation strategy (CODING4GIRLS 2020) were used.

The students were asked to complete two questionnaires: the preliminary questionnaire (S1) and the follow-up questionnaire (S2). In S1, students were profiled in relation to the use of digital devices, the Internet and video games (using the questions in Table 3), familiarity with programming concepts (students had to tick all the concepts they were familiar with in the list shown in Table 5) and about what motivates them to learn programming (students could choose from the answers shown in Table 6 or write their own answers). The quantitative data were analysed statistically, while for the qualitative data the percentage of respondents who chose a particular answer was calculated (overall and by grade level and/or gender).

Table 3

The use of digital devices, the Internet and video-games by gender

Question		N	Min	Max	Mean	SD
1.1.1. For how long have you been using computers, tablets or other digital devices (in years)?	Boys	283	1	12	6.237	2.055
	Girls	286	0	14	5.767	1.882
	Total	569	0	14	6.001	1.982
1.1.2. How many hours per week do you use a computer, tablet or other digital device?	Boys	283	1	168	19.834	19.637
	Girls	281	0	168	16.196	19.419
	Total	564	0	168	18.021	19.596
1.1.3. How many hours per week do you use the Internet?	Boys	282	1	140	21.603	21.315
	Girls	277	0	168	17.643	19.502
	Total	559	0	168	19.611	20.517
4. How many hours per week do you play video games?	Boys	282	0	62	11.261	10.803
	Girls	284	0	35	2.905	4.922
	Total	565	0	62	7.075	9.367

Table 5

Familiarity with the programming concepts by grade and gender

Concept		5th grade	6th grade	7th grade	8th grade	Total
Loops	Boys	39.71%	64.86%	68.42%	72.34%	60.78%
	Girls	39.19%	71.07%	71.15%	66.67%	62.24%
	Total	39.44%	68.10%	69.72%	69.77%	61.51%
Conditionals	Boys	26.47%	45.95%	61.40%	65.96%	47.70%
	Girls	25.68%	47.93%	63.46%	58.97%	46.50%
	Total	26.06%	46.98%	62.39%	62.79%	47.10%
Variables	Boys	16.18%	49.55%	73.68%	78.72%	51.24%
	Girls	13.51%	45.45%	78.85%	71.79%	46.85%
	Total	14.79%	47.41%	76.15%	75.58%	49.03%
Statements (sounds, movement, looks, drawing)	Boys	63.24%	73.87%	70.18%	78.72%	71.38%
	Girls	62.16%	75.21%	82.69%	74.36%	73.08%
	Total	62.68%	74.57%	76.15%	76.74%	72.23%
Operators	Boys	11.76%	11.71%	17.54%	27.66%	15.55%
	Girls	9.46%	7.44%	9.62%	15.38%	9.44%
	Total	10.56%	9.48%	13.76%	22.09%	12.48%
Events	Boys	26.47%	45.05%	45.61%	31.91%	38.52%
	Girls	22.97%	40.50%	48.08%	30.77%	36.01%
	Total	24.65%	42.67%	46.79%	31.40%	37.26%
Parallelism	Boys	0.00%	1.80%	0.00%	6.38%	1.77%
	Girls	2.70%	2.48%	0.00%	7.69%	2.80%
	Total	1.41%	2.16%	0.00%	6.98%	2.28%

Table 6

Motivation for learning programming by grade and gender

Statement		5th grade	6th grade	7th grade	8th grade	Total
I’m not motivated	Boys	17.65%	13.51%	24.56%	17.02%	17.31%
	Girls	17.57%	8.26%	19.23%	28.21%	15.38%
	Total	17.61%	10.78%	22.02%	22.09%	16.34%
I want to succeed in the programming class	Boys	42.65%	53.15%	49.12%	55.32%	50.18%
	Girls	62.16%	80.99%	67.31%	61.54%	70.98%
	Total	52.82%	67.67%	57.80%	58.14%	60.63%
I want to show other students I can program	Boys	8.82%	11.71%	10.53%	12.77%	10.95%
	Girls	9.46%	9.09%	7.69%	2.56%	8.04%
	Total	9.15%	10.34%	9.17%	8.14%	9.49%
I want to follow a career in programming	Boys	29.41%	20.72%	22.81%	19.15%	22.97%
	Girls	4.05%	7.44%	1.92%	5.13%	5.24%
	Total	16.20%	13.79%	12.84%	12.79%	14.06%
I enjoy solving logic problems and puzzles	Boys	23.53%	30.63%	15.79%	21.28%	24.38%
	Girls	20.27%	22.31%	19.23%	10.26%	19.58%
	Total	21.83%	26.29%	17.43%	16.28%	21.97%

In S2, student satisfaction with the C4G methodology and the organization of implementation was assessed using the list of statements with a 5-point Likert scale (1 – strongly disagree, 5 – strongly agree) shown in Table 8. Students’ responses were statistically analysed overall and by gender.

Table 8

Number of students who solved follow-up questionnaire S2 by gender and grades

	5th grade	6th grade	7th grade	8th grade	Total
Boys	72	78	31	35	225
Girls	67	79	30	23	199
Total	139	166	61	58	442
Response rate	72.77%	53.9%	41.78%	45.31%	54.85%

In both questionnaires, S1 and S2, students were asked to self-assess their current level of programming skills on the following scale: 0 - I have never coded or programmed before, 1 - I am a novice programmer (just have basic ideas), 2 - I can code simple programs, 3 - I am fluent in programming (can create a full program), 4 - I can design a solution of a problem in the form of a program. The answers to this question were paired based on the code that the students entered in both questionnaires, and a Wilcoxon’s Signed Rank test for paired samples was used to determine the difference between the students’ self-assessed initial and final level of programming skills. Rank-biserial correlation was used to calculate the effect size.

In addition to the questionnaires, the teachers collected the students’ comments using an open-ended questions (S3). The aim of using this instrument was to provide a platform for collecting richer comments and opinions. Using the questions from the form S3, the teachers interviewed the students about the overall organization of the implementation, their perception of the acquired knowledge, their perception of the relevance and effectiveness of game development-based learning, and their perception of the fun achieved. Students were also able to report on any learning difficulties or problems encountered during the activities and express their views on how the C4G methodology could be improved.

To collect observations and comments from in-service and pre-service teachers, two forms with open-ended questions were used. Using the form T1, teachers reported on student participation and involvement, as well as learning difficulties and problems. The form T2 was used to collect their comments regarding the achievement of the learning objectives, the relevance, effectiveness and acceptance of the proposed methodology by the students and the general organization of the implementation.

Eternal experts were also asked to give their qualitative opinion on the achievement of the learning objectives by the students, the relevance, effectiveness and acceptance by the students of the proposed methodology and the general organization of the implementation activities. Form E with open-ended questions was used to collect the experts’ comments.

Questionnaires S1 and S2 were prepared using Google Forms, while the data collection tools S3, T1, T2 and E were prepared as Word documents for teachers and experts to write observations and comments in. All data collection instruments were available in Croatian language.

Study design and procedure

Introductory workshops were organized at UNIRI for in-service and pre-service teachers as well as external experts, where project members presented information about the C4G project and the approach to building programming skills. Details of the study design and the procedure in Croatian schools were agreed. Teachers and experts were also provided with all necessary content and tools.

The activities for building up programming skills using the C4G approach were organized as part of the regular teaching of subject Informatics in elementary schools, which at that time, due to the pandemic COVID -19, was conducted in online learning environments. The implementation activities took place between March and June 2020 and were carried out in the 5th and 6th grade, where the subject Informatics is compulsory, and in the 7th and 8th grade of elementary where the subject is optional. Teachers provided the students with all learning materials in virtual classrooms (created with Microsoft Teams, Edmodo or similar tools). Younger students required more guidance from teachers while older students were more independent.

The sequence of implementation activities is shown in Fig. 1. At the beginning of the study, the students were familiarized with the C4G project and the approach to learning programming. They were asked to answer the preliminary questionnaire (S1).

Fig. 1

Project activities during the implementation

To enable the students to learn programming concepts, the teachers first introduced the concept to the students. The students were able to practice the concept through exercises, and then they were expected to create a serious game containing the learned programming concept based on the C4G learning scenario. During the sessions, the teachers provided guidance and help to the students with the given tasks. At the end of each session, students could present their games and experiences to peers and participate in a debriefing session in the virtual classroom. In the last session, students answered the follow-up questionnaire (S2) about their perceptions and views on the C4G learning approach. They had to write the anonymized code they had received from the teacher in order to ensure a comparison of the results regarding the self-assessment of their programming skills. The teachers collected the qualitative opinions and comments from the students through a group discussion in virtual classrooms using S3.

Teachers also reported on student response and progress in building programming skills with the C4G approach (using T1) and their own views on the relevance and effectiveness of the CODING4GIRLS approach to building programming skills (using T2).

In order to further improve the CODING4GIRLS game development-based learning approach and to ensure that it meets the needs of learners and teachers, their views on the relevance and effectiveness of the approach were collected from external experts (using E). The experts were given access to project documentation and to the results and content produced during implementation in the schools.

Materials

During the study, learning scenarios and instructions for students were used, which were developed by the project consortium (CODING4GIRLS 2020). They follow the active C4G learning approach based on game development. The learning scenarios provide a concise description of all learning activities that need to be implemented to develop the proposed serious game and help teachers to integrate them into their teaching practice.

Each learning scenario includes information about overall educational objective, covered concepts, specific learning objectives and expected learning outcomes, step-by-step description for game development, assessment methods, questions for initiating discussion among learners in the context of class collaboration, and tools and resources for the teacher and students. Prepared set consist of 21 learning scenarios that cover the basic programming concepts included in the learning objectives for lower secondary education in Croatia (loops, conditionals, variables, statements, operators, events, parallelism). Each learning scenario allows students to learn and apply one or more programming concepts by creating a game that addresses a real-world problem. To motivate girls to learn programming even more, the themes of the real-world problems are chosen to be attractive to girls (e.g. picking up trash in the park, buying food for a picnic, animals in a shelter, recycling). Teachers can decide whether to use all learning scenarios in the suggested order or to select a subset according to their preferences and needs. Screenshots from the games that can be created using learning scenarios are shown in Fig. 2.

Fig. 2

Games developed in Snap!: a – Cleaning the park, b – Recycling, c – Cats in shelter

All C4G learning scenarios were translated into Croatian language prior to implementation in schools. The in-service teachers selected a subset of 6–8 learning scenarios for implementation in a specific class based on the students’ prior knowledge and learning outcomes. To adapt the approach to online learning, the in-service teachers also prepared additional learning materials with instructions and tasks for C4G activities to support the independent work of the students. Depending on the age of the students, the instructions were more or less detailed.

Video tutorials were also recorded and published on YouTube (Fig. 3) to make it easier for students to work independently in an online environment. For example, sets of videos were prepared to introduce students to the tool or to help them with the development of the projects included in the learning scenarios.

Fig. 3

A set of video tutorials for students - Learning scenario “Picking up the trash”

Results

Results of questionnaires for students

Before the C4G implementation activities, students were asked to fill the preliminary questionnaire (S1) and after the C4G implementation activities, they were asked to fill the final questionnaire (S2). In both questionnaires, students have entered the code that was used to pair the self-assessed levels of programming skill.

Preliminary questionnaire

A total of 569 students (73.61% of students who participated in C4G activities) solved the preliminary questionnaire about the use of digital devices and the perceived level of programming. The mean age of students was 12.31 years (SD = 1.024). Table 2 shows the number of students who solved S1 by gender and grade. The number of girls and boys who responded is approximately equal.

Table 2

Number of students who solved preliminary questionnaire S1 by gender and grade

	5th grade	6th grade	7th grade	8th grade	Total
Boys	68	111	57	47	283
Girls	74	121	52	39	286
Total	142	232	109	86	569
Response rate	74.35%	75.32%	74.66%	67.72%	73.61%

Table 3 shows a descriptive statistical analysis of participants’ responses to the questions related to the use of digital devices, the Internet, and video-games. The comparison of the overall average results by gender (Fig. 4) shows that boys and girls have been using digital devices for the same length of time. However, on a weekly basis, boys use digital devices and the Internet more and spend significantly more time playing games.

Fig. 4

The use of digital devices, the internet and video-games – comparison by gender

The average values show that students spend a little more time on the Internet than using digital devices. It can be assumed that such numbers are the results of misconceptions. For example, some students may think that using their smartphone to access the Internet does not count. It is worth noting the values of the standard deviation, which for some questions indicates larger deviances from the average values. Analysis of individual responses shows that several students reported very high values (e.g., 100 h per week for the use of the Internet) which affected these results.

The participants (N = 569, 283 boys, 286 girls) self-assessed the level of their programming skills on the scale from 0 - I have never coded or programmed before to 4 - I can design a solution of a problem in the form of a program. Most of the students stated for themselves that they are at level 1 - novice programmers (33.39%) or on level 2 – can code simple programs (32.51%). If we compare these results by gender (Fig. 5), it can be seen that the boys prevail among the students that self-assess their level of programming with levels 3 and 4.

Fig. 5

Self-assessment of programming skills – comparison by gender

The analysis by grades (Table 4) shows that the largest number of students who have never coded (level 0) is from the 5th grade, as expected. In 6th, 7th, and 8th grade most of the students stated that they can code simple programs (level 2).

Table 4

Self-assessment of programming skills by grade and gender

Level of programming skills		5th grade	6th grade	7th grade	8th grade	Total
0 - I have never coded or programmed before	Boys	19.12%	6.31%	10.53%	2.13%	9.54%
	Girls	20.27%	2.48%	3.85%	2.56%	7.34%
	Total	19.72%	4.31%	7.34%	2.33%	8.44%
1 - I am a novice programmer (just have basic ideas)	Boys	41.18%	26.13%	26.32%	36.17%	31.45%
	Girls	45.95%	32.23%	30.77%	30.77%	35.31%
	Total	43.66%	29.31%	28.44%	33.72%	33.39%
2 - I can code simple programs	Boys	23.53%	32.43%	35.09%	25.53%	29.68%
	Girls	20.27%	38.02%	44.23%	43.59%	35.31%
	Total	21.83%	35.34%	39.45%	33.72%	32.51%
3 - I am fluent in programming (can create a full program)	Boys	13.24%	25.23%	24.56%	31.91%	23.32%
	Girls	9.46%	23.14%	15.38%	17.95%	17.48%
	Total	11.27%	24.14%	20.18%	25.58%	20.39%
4 - I can design a solution of a problem in the form of a program	Boys	2.94%	9.91%	3.51%	4.26%	6.01%
	Girls	4.05%	4.13%	5.77%	5.13%	4.56%
	Total	3.52%	6.90%	4.59%	4.65%	5.27%

In the preliminary questionnaire, the participants also stated which programming concepts are they familiar with. The results (Table 5) show that students are most familiar with the statements (72.23%) and loops (61.51%) while they are the least familiar with the operators (12.48%) and parallelism (2.28%). According to the results, there is no big difference in the familiarity of programming concepts between the genders (Fig. 6). The largest difference in percentages can be observed for the concept operators. Comparison of the results by grade and gender shows that fifth graders are at least familiar with all the concepts which was expected.

Fig. 6

Familiarity with the programming concepts – comparison by gender

For some of the concepts (loops, statements), the results for 6th, 7th, and 8th grade students do not differ, while for some (more advanced) concepts (conditionals, variables, operators) there is an increase in the level of familiarity. As in the case of overall results, there are no big gender differences in the familiarity with the programming concepts except concept operators.

Results regarding students’ responses about what motivates them to learn to program (students could choose one or more responses) are shown in Fig. 7. The result show that most of the students are motivated by success in the programming class (60.63%). Comparison by gender (Table 6) shows that this factor motivates girls (70.98%) to a greater extent than boys (50.18%). Also, remarkably more boys want to follow a career in programming.

Fig. 7

Motivation for learning programming – Comparison by gender

Comparison by grade shows that sixth graders are the most motivated, especially by wanting to succeed in the programming class. They also enjoy solving logic problems and puzzles more than students from other grades (particularly boys).

Besides choosing among the offered responses, students had the opportunity to write everything else that motivates them to learn to program. Table 7 shows the frequency of students’ responses for 4 categories with examples. Most of the students wrote that they are motivated by the possibility to create their own games (16 students) and to have fun while programming (12 students). Some of them (6 students) are motivated by the desire to acquire new knowledge and skills while one student pointed out that his uncle (programmer) motivates him.

Table 7

Motivation for learning programming (students’ free responses)

Category	Frequency	Examples
Creating games	16	“I want to learn how to make my own game.”
		“I’m interested in making programs or games.”
Having fun	12	“It’s fun.”
		“I have fun while programming.”
New knowledge and skills	6	“I want to acquire new skills.”
		“I want to learn something new.”
Role models	1	“I’m motivated by my uncle who is a programmer.”

Follow-up questionnaire

A total of 424 students (54.85%) solved the follow-up questionnaire. The mean age of students was 12.25 years (SD = 1047). Table 8 shows the number of students who solved S2 by gender and grades. The number of girls and boys who responded is approximately equal in all grades except in the 8th grade where the questionnaire was solved by a larger number of boys.

In the follow-up questionnaire, students expressed their attitudes regarding the programming and coding activities and their implementation according the C4G methodology using the 5-point Likert scale (1 – strongly disagree, 5 – strongly agree). According to the results (Table 9), both boys and girls felt engaged in this way of learning and think that conducted activities were relevant for learning programming. They understood the presented concepts and had fun during conducted activities. Boys enjoyed programming to a slightly greater extent and think that things they have learned will be relevant for their future.

Table 9

Satisfaction with C4G learning methodology

Statement		1	2	3	4	5	AVG	SD
I found programming challenging.	Boys	9.78%	25.33%	31.12%	24.44%	9.33%	2.982	1.126
	Girls	4.02%	23.12%	33.16%	31.16%	8.54%	3.171	1.011
	Total	7.08%	24.29%	32.08%	27.59%	8.96%	3.071	1.076
I found programming motivating.	Boys	2.67%	9.78%	18.22%	46.67%	22.66%	3.769	.995
	Girls	6.03%	13.57%	21.61%	41.2%	17.59%	3.508	1.114
	Total	2.24%	11.56%	19.81%	44.11%	20.28%	3.646	1.06
I found programming easy.	Boys	12.44%	28.44%	29.78%	20.44%	8.9%	2.849	1.151
	Girls	14.07%	30.65%	32.67%	18.59%	4.02%	2.678	1.057
	Total	13.21%	29.48%	31.13%	19.58%	6.6%	2.769	1.11
I enjoyed programming.	Boys	3.55%	7.56%	19.56%	28.89%	40.44%	3.951	1.107
	Girls	6.02%	11.06%	21.61%	33.67%	27.64%	3.658	1.169
	Total	4.72%	9.2%	20.52%	31.13%	34.43%	3.814	1.145
I understood most of programming concepts.	Boys	3.11%	7.56%	13.78%	48.88%	26.67%	3.884	.989
	Girls	4.53%	8.04%	20.6%	38.19%	28.64%	3.784	1.086
	Total	3.77%	7.78%	16.98%	43.88%	27.59%	3.837	1.036
Learning this way is fun.	Boys	4%	8.89%	15.11%	38.67%	33.33%	3.884	1.092
	Girls	4.02%	9.05%	21.61%	29.65%	35.68%	3.839	1.13
	Total	4%	8.96%	18.16%	34.42%	34.42%	3.863	1.109
I felt engaged with this way of learning.	Boys	1.33%	6.67%	7.56%	35.11%	49.33%	4.244	.949
	Girls	2.51%	5.02%	9.55%	29.15%	53.77%	4.266	.997
	Total	1.88%	5.9%	8.49%	32.31%	51.42%	4.255	.971
The activities were relevant to learn.	Boys	1.33%	4.89%	7.56%	40.89%	45.33%	4.24	.889
	Girls	3.01%	3.52%	13.57%	36.68%	43.22%	4.136	.983
	Total	2.12%	4.25%	10.37%	38.92%	44.34%	4.191	.935
At any time, it was clear what I had to do.	Boys	3.56%	11.56%	19.56%	35.11%	30.22%	3.769	1.11
	Girls	5.03%	16.58%	29.14%	28.14%	21.11%	3.437	1.144
	Total	4.25%	13.92%	24.06%	31.84%	25.94%	3.613	1.137
What I learned will be relevant for my future.	Boys	3.11%	4.89%	16.89%	36%	39.11%	4.031	1.019
	Girls	5.03%	7.03%	22.11%	37.69%	28.14%	3.769	1.09
	Total	4%	5.9%	19.34%	36.79%	33.96%	3.908	1.06

The participants again self-assessed the level of their programming skills on the scale from 0 - I have never coded or programmed before to 4 - I can design a solution of a problem in the form of a program. A total of 347 students or 44.89% solved the preliminary and the follow-up questionnaire (179 boys, 168 girls) so their self-assessment results were compared. Table 10 shows data on the difference between the self-assessed initial level and the self-assessed final level of programming skill. Over 40% of students stated that they have progressed, most of them for 1 level (29.39%). Almost 5% of students self-assessed their programming skills level higher before participating in C4G activities. The analysis showed that most of them were students who initially self-assessed their skills with level 3 or 4. It can be assumed that these students were not able to self-assess their programming skills realistically before the C4G activities. Comparison by gender shows that slightly fewer girls remained at the same level of programming skills than boys (the difference is 0). Also, slightly more girls progressed by one level than boys (the difference is 1). For other values (differences), the results are approximately the same.

Table 10

The difference between the self-assessed levels of programming skills

	Difference
	−2	−1	0	1	2	3	4
Boys	0.56%	3.91%	55.31%	25.70%	11.73%	2.23%	0.56%
Girls	0.60%	4.76%	48.81%	33.33%	9.52%	2.98%	0%
Total	0.58%	4.32%	52.16%	29.39%	10.66%	2.59%	0.29%

A Wilcoxon’s Signed Rank test for paired samples showed that students self-assessed their programming skills significantly higher after the C4G activities compared to self-assessment before the C4G activities (Table 11). The results of rank-biserial correlation (rB), which are considered as an effect size, show a large effect size, overall and by gender.

Table 11

Comparison of self-assessment of programming skills

		Descriptive statistics					Wilcoxon’s signed rank test results
		N	Min	Max	Mean	SD	W	p	Effect size (rB)
Boys	S1	181	0	4	1.878	1.068	3112.5	< .001	.829
	S2	181	1	4	2.403	.982
Girls	S1	166	0	5	1.825	1.015	3168	< .001	.818
	S2	166	1	4	2.380	.938
Total	S1	347	0	4	1.850	1.034	12.702	< .001	.833
	S2	347	1	4	2.392	.960

Students’ comments

In their comments after the implementation activities, the students stated that they are very satisfied with this kind of learning, that they think they have learned a lot and that they feel like real developers because they have developed their games. During the implementation of C4G approach, they could hardly wait for each new task and project. They were very satisfied with the material they received. This way of learning was a lot of fun and they can hardly wait for further tasks like this.

Some of the students’ comments include: “I had a great time developing these games”, “The instructions were clear and detailed and I think all the teachers really tried to prepared it all.”, “It was not very difficult for me because everything was written and I just had to read the instructions more carefully and do it.” “The ‘Food project’ was quite difficult but the other students gave me good hints.”, “For me it was much easier to solve a task when I have half-baked solution.”, “I’ve learned that it’s all simple if you think about it a little better.”.

Teachers’ observations and comments

After the implementation activities, in-service teachers and students – future teachers (pre-service teachers) were asked to express their qualitative opinion on the C4G methodology and the implementation process using forms T1 and T2.

Teachers’ observations

Using the form T1, in-service teachers (NT = 8) and pre-service teachers (NST = 35) reported on student participation and engagement, as well as learning difficulties and problems. They all reported that students were interested and actively participating in the activities. Most of the students successfully completed all tasks with the help of the prepared materials. According to the teachers, video tutorials were particularly useful for them.

Students collaborated in virtual classrooms and were happy to answer each other’s questions about the problems they encountered. However, the teachers felt that due to the specific nature of online learning, the students were not able to collaborate to the extent that collaboration could be achieved in classrooms.

Due to technical limitations, some students were slower in completing the tasks. Students working on tablet computers had problems with Snap! because the tool was not working properly and fast enough. Students had problems adding costumes to sprites and with occasional freezing its interface. The students who created projects on desktop computers or laptops had no problems. The teachers suggested the Scratch programming language as a replacement because there is a mobile application that works well on tablets.

Teachers’ comments

Using the form T2, in-service teachers (NT =8) and pre-service teachers (NST = 35) reported their comments the C4G methodology. The teachers’ opinion of the teachers is that game development-based learning is fun for students. They are motivated to solve tasks (problems) which makes this way of learning effective for learning to program. All teachers stated that the learning objectives were fully achieved by the students during the C4G activities.

According to the teachers, the students fully accepted the C4G methodology. They found the project topics very interesting and most students enjoyed using their newly acquired programming skills to create games. The C4G projects had a positive effect on their desire and motivation to eliminate all mistakes. The students persisted until they reached the set goal - a game they could play.

Since the C4G activities were conducted online, the insight into the fun achieved by the students could not be complete, but teachers explained that from the communication with the students, it is clear that the students had fun and were looking forward to gaining new knowledge with the C4G methodology.

In terms of the overall organization of the implementation, teachers reported that it was fully aligned to the teaching needs (achieving the results associated with programming). The materials produced for the implementation of C4G activities are comprehensive and clear. During the implementation, the logistical support provided by the project team members was efficient and available at all times.

The teachers also agree that the C4G methodology is applicable in future work. Learning scenarios with projects to be developed in Snap! are appropriate to the age of the students. Teachers also believe that a game development-based learning can also stimulate the students’ creative potential, which can lead to some new ideas and game extensions or modifications (e.g. students can add characters, change the rules of the game, etc.). Teachers who have participated in the implementation activities plan to use them in their future work.

Experts’ comments

Following the implementation activities, external experts were also asked to give their qualitative opinions on the C4G methodology and implementation process using the form E.

All three experts agreed that the C4G methodology is very suitable for developing basic programming skills in students aged 11–15 years. The Snap! programming interface allows students to create interactive stories and games that they enjoy and are stimulated by. The advantage of coding with blocks is that students of this age do not need to learn the programming syntax, as this negatively affects the students’ motivation to learn programming (they usually forget the syntax very quickly).

They emphasized that the C4G approach assumes solving real-world problems, which is interesting for students, especially for girls. In programming classes mathematical tasks are very common (e.g. calculating the range, the area of a square, rectangle or currency conversion and the like), which further demotivates students to learn programming. According to the experts, projects in C4G learning scenarios are well-designed and enable learning goals to be achieved. The projects stimulate students’ creativity and allow them to learn in a playful way by creating games. The topics included in the projects are interesting for the girls and motivate them to solve the given problem using newly acquired programming skills. At the same time the topics are also interesting for boys.

According to experts, the C4G approach is effective and applicable for teaching programming lessons for both girls and boys.

Discussion

Regarding the RQ 1, the results of the preliminary questionnaire (S1) showed that students of both genders use the Internet and digital devices for a few hours a day on average and have 5 to 6 years’ experience of using them. This indicates that students had the appropriate digital skills required for the implementation of the C4G approach. The large number of hours that some students spend per week using digital devices and the Internet is partly due to the online teaching that was conducted during this study, as students were expected to study teaching materials, create assignments and communicate with teachers in virtual classrooms. The results also showed that students play video games and that boys spend significantly more time playing them than girls. This suggests that game-based activities, such as those designed under the C4G approach, should be tailored to girls’ interests in order to keep girls motivated.

The results of the preliminary questionnaire also showed that younger students, especially in the 6th grade, are more motivated to learn programming than students in the 7th and 8th grades. Factors that motivate students for learning programming differ by gender (RQ 2). In general, students are mostly motivated by success in the programming class, girls to a greater extent than boys. Students (especially in the 6th grade) are motivated by the fact that they enjoy solving problems and puzzles and that they enjoy developing their own games. This confirms that the C4G methodology, which involves developing games to solve real-world problems, is suitable for the target group of students. According to the results, boys (especially fifth graders) are further motivated by the desire to become programmers, while girls are very little motivated by this factor. This shows that girls are less interested in pursuing a career in programming, so that they need to be motivated by approaches such as C4G.

In the initial self-assessment of their programming skills, most students indicated that they are novice programmers (level 1) or that they can develop simple programs (level 2). This is confirmed by the results regarding familiarity with programming concepts (RQ 3), which showed that most students are only familiar with basic concepts such as instructions and loops. Some students attend computer science for several years, but do not feel that they have developed programming skills. The reason could be that they find programming difficult and are not interested in learning.

After the implementation activities, teachers reported that the activities carried out enabled students to achieve learning outcomes and have fun at the same time. Teachers believe that developing games is a very effective way for students to learn programming concepts and they plan to continue using the C4G methodology in the future. They observed that the C4G approach with real-world problems encouraged creativity and problem solving and that students were motivated to complete the project (their own game) to the end. The external experts who participated in the study agreed with these observations and supported the application of the game development-based learning approach using visual programming tools to learn programming. They emphasized the good selection of topics of the projects contained in the learning scenarios that are interesting for girls and encouraged them, as well as boys, to apply their programming skills.

Regarding the RQ 4, students self-assessed their programming skills after the C4G activities as significantly higher compared to the self-assessment before the C4G activities - there is a large effect range in the overall results and in the results grouped by gender. These results confirm the effectiveness of the GDBL activities carried out for learning programming.

To sum up the results associated with the RQ 5 and RQ 6, the students’ attitudes towards the C4G methodology expressed in the follow-up questionnaire showed students’ satisfaction. They think that they have acquired programming skills and gave positive qualitative feedback on the GDBL approach. Students from 5th to 8th grade consider the approach to be effective and appropriate for learning programming. They were motivated to create their own games, especially since the games include solving real-world problems - this made programming fun for them. They have successfully developed games thanks to good instructions and enjoyed the process. Validation of the C4G GDBL approach is also supported by teachers’ and experts’ comments. They consider the methodology relevant, effective and appropriate for building programming skills among students aged 11–15. They also think that learning scenarios are well-designed and enable the achievement of learning outcomes. According to teachers and experts, the GDBL approach motivated students to finalize their projects (games) and then test them by playing. They emphasized solving real-world problems within the games as an advantage of the approach. They think the topics included in the Snap! projects are very interesting to boys but especially to girls and that this certainly affects the relevance, effectiveness, and appropriateness of the approach.

Conclusions and future work

The study presented in this paper examined the effectiveness of game development-based learning for the acquisition of programming skills in lower secondary education in Croatia based on the C4G methodology. The C4G methodology promotes building programming skills using learning scenarios for the development of serious games in Snap!. The aim of using GDBL approach was used to prepare young learners, especially girls, for a career in computer science and to raise awareness of the relationship between ICT and the real world. The results showed that the C4G methodology is suitable for students aged 11 to 15 years old and enables the achievement of learning outcomes related to programming in an effective and fun way. Furthermore, the approach has proven to be applicable in an online environment. For this purpose, materials produced within the C4G project were used, as well as additional instructions produced by teachers to facilitate students’ independent work.

Applied C4G learning scenarios included solving real-world problems that cover topics interesting to both boys and girls. These characteristics, according to the comments of teachers and students, proved to be important for the effectiveness of this approach. In addition to the possibility of creating their own games, students and teachers assessed the inclusion of real-world problems and interesting topics as motivating for learning. For the successful application of the GDBL approach, it is also important that the instructions for the activities are well defined and that students have all the necessary materials, but also a chance to get help from colleagues or teachers when they need it.

Additional studies will be conducted in our future research to further confirm the effectiveness of the proposed methodology by assessing the extent to which students achieved the learning outcomes. The future studies will also include the combination of GBL and GDBL approaches for learning programming. Within the project Digital Games (Digital Games web site 2020), the effectiveness of the combination of the C4G GDBL learning scenarios with playing the C4G 3D game in lower secondary education in Croatia will be explored. C4G Methodological Learning Framework will be broadened to Pedagogical-Technological Framework that will include additional learning scenarios for encouraging students to master programming concepts.