Validity and reliability of the South African health promoting schools monitoring questionnaire.

Health promoting schools, as conceptualised by the World Health Organisation, have been developed in many countries to facilitate the health-education link. In 1994, the concept of health promoting schools was introduced in South Africa. In the process of becoming a health promoting school, it is important for schools to monitor and evaluate changes and developments taking place. The Health Promoting Schools (HPS) Monitoring Questionnaire was developed to obtain opinions of students about their school as a health promoting school. It comprises 138 questions in seven sections: socio-demographic information; General health promotion programmes; health related Skills and knowledge; Policies; Environment; Community-school links; and support Services. This paper reports on the reliability and face validity of the HPS Monitoring Questionnaire. Seven experts reviewed the questionnaire and agreed that it has satisfactory face validity. A test-retest reliability study was conducted with 83 students in three high schools in Cape Town, South Africa. The kappa-coefficients demonstrate mostly fair (κ-scores between 0.21 and 0.4) to moderate (κ-scores between 0.41 and 0.6) agreement between test-retest General and Environment items; poor (κ-scores up to 0.2) agreement between Skills and Community test-retest items, fair agreement between Policies items, and for most of the questions focussing on Services a fair agreement was found. The study is a first effort at providing a tool that may be used to monitor and evaluate students' opinions about changes in health promoting schools. Although the HPS Monitoring Questionnaire has face validity, the results of the reliability testing were inconclusive. Further research is warranted.

INTRODUCTION

In South Africa, the development of a healthy school environment has been identified as an important strategy, not only to promote the health and wellbeing of the school community, including students (learners), teachers, parents, and community members, but also to achieve educational goals (Department of Basic Education, 2010). Health promoting schools, as conceptualised by the World Health Organisation Expert Committee on Comprehensive School Health Education and Promotion (1997), have been developed in many countries to facilitate this health-education link (Burgher ). In 1997, at the World Health Organisation’s Fourth International Conference on Health Promotion, the Jakarta Declaration on Leading Health Promotion into the 21st Century highlighted key strategies to promote health in the next millennium, including a comprehensive approach to health development within particular settings such as schools, with the participation of people who have access to education and information, and who are empowered (World Health Organisation, 1997). Health promoting schools incorporate the action areas described in the Ottawa Charter (World Health Organisation, 1986), including the development of healthy school policies, healthy school physical and psychosocial environments, healthy skills, healthy links with the community and appropriate support services. According to Burgher , p.1), a health promoting school ‘… aims at achieving healthy lifestyles for the total school population by developing supportive environments conducive to the promotion of health. It offers opportunities for, and requires commitments to, the provision of a safe and health-enhancing social and physical environment’.

In 1994, the concept of health promoting schools was introduced in South Africa (Medical Research Council ; Flisher and Reddy, 1995; Swart and Reddy, 1999) and guidelines for developing health promoting schools were drafted (Department of Health ). By 2006, schools in all nine provinces of the country were identifying themselves as health promoting schools (Lazarus, 2006). In the process of becoming a health promoting school, it is important for schools to monitor and evaluate changes and developments taking place (Mũkoma and Flisher, 2004; Lazarus, 2006; Departments of Health and Basic Education, 2012; Lee ). Surveys of schools in South Africa have been conducted to gather data on the school infrastructure, for example, the number and type of toilets in a school or the presence of piped water (Department of Basic Education, 2009, 2015); risk behaviours of young people at school such as sexual risk behaviour or nutritional patterns (Reddy ); and targeted health promotion activities, for example, oral health activities in health promoting schools in KwaZulu-Natal (Reddy and Singh, 2015).

Part of monitoring and evaluating changes and developments within health promoting schools is obtaining the opinions of students about their school. However, we could not locate any previous published research regarding such a measurement tool in the South African context. The questionnaire described in this paper, the Health Promoting Schools (HPS) Monitoring Questionnaire, was developed to address this need. This paper reports on the development and psychometric properties, namely reliability and face validity of the HPS Monitoring Questionnaire.

The health promoting school monitoring questionnaire

In 2008, a group of academics from the education and health sciences faculties at the University of the Western Cape, South Africa, initiated a project to develop three high schools as health promoting schools (Preiser ). We recognised the need to determine the opinions of students about their schools which would assist school management to identify strengths and prioritise changes, and developed the HPS Monitoring Questionnaire. The HPS Monitoring Questionnaire was based on the Rapid Assessment Tool for Schools (RATS) which is an unpublished questionnaire produced by the South African practitioner organisation, the Cape Metro Reference Group for Health Promoting Schools. As with the Schools Health Europe Rapid Assessment Tool (Safarjan ), the RATS was designed for use by a working-group in a school, not individuals, to assess or monitor policies and practices related to health promotion in the school. A questionnaire that is answered by a working group has certain advantages; however, generally teachers and local health promotion personnel lack the time and experience to conduct this type of assessment. Furthermore, a questionnaire that is completed by a working group may lack the rigour that a questionnaire or survey that is answered by individuals would provide (Denman ).

The HPS Monitoring Questionnaire was designed for individual use by survey method. This method enables respondents to express their opinions anonymously, without recrimination. It comprises 138 questions, including a section on socio-demographic information (n = 14 items) and a General section, including items about health promotion programmes (n = 17 items). There are a further five sections corresponding to the action areas in the Ottawa Charter (World Health Organisation, 1986), including sections on health related Skills and knowledge (n = 13 items), Policies (n = 18 items), physical and psychosocial Environment (n = 43 items), Community-school links (n = 7 items), and support Services (n = 26 items). The questionnaire uses mainly two types of responses: yes, no, don’t know; or always, sometimes, not at all, don’t know.

The HPS Monitoring Questionnaire was developed in English and then translated into Afrikaans (the language most commonly spoken in the schools where the study was conducted) by a professional translator, and subsequently back translated by an independent translator to ensure accuracy in translation. Discrepancies were resolved through discussion between the translators and the team developing the questionnaire until they reached agreement on the formulation and content of the questions to ensure that the meaning remained consistent.

METHODS

Once the HPS Monitoring Questionnaire was developed, we needed to establish the validity and reliability of the questionnaire for use with students in South African schools.

The validity study

In order to ensure that the HPS Monitoring Questionnaire was perceived to assess opinions about health promoting schools and to identify any ambiguous questions, we conducted a small study to determine the face validity of the questionnaire. Eleven experts working in the field of health promoting schools in South Africa were identified based on their knowledge and experience. They included professionals employed by the Departments of Education and Health. Four of them had been involved in the development of the South African Rapid Assessment Tool for Schools.

In order to determine the experts’ opinions about the HPS Monitoring Questionnaire, we developed a questionnaire that comprised 20 questions: eight socio-demographic questions and twelve questions relating to face validity allowing a yes/no response and space to elaborate. In addition, the experts were requested to rate each item of the HPS Monitoring Questionnaire for clarity on a 3-point scale (1 = very unclear/ambiguous, 2 = unclear, 3 = clear). The experts were contacted telephonically to explain the purpose of the study. On agreeing to participate, the experts (respondents) were given the validity questionnaire together with the HPS Monitoring Questionnaire with three weeks to complete and return the questionnaire in a pre-stamped envelope. After one month, the respondents were reminded by phone and email to complete and return the questionnaire. Two of them requested the questionnaire be sent to them a second time.

Data were analysed descriptively to obtain frequencies, percentages and means. The open-ended comments were analysed through team discussion and changes were agreed upon.

The test-retest study

A test-retest study was used to determine the reliability of the HPS Monitoring Questionnaire. Three high schools that were part of the Health Promoting Schools Project served as the setting. Grade 10 students at the three high schools were purposively selected to participate in the study as it was assumed that grade 10 students, having completed two years of high school, would have sufficient experience to have developed their own informed opinions about their school. Within each school, one grade 10 class was randomly selected to participate from all the grade 10 classes. It was assumed that the classes were of similar size and student composition. Consequently, this method provided a self-weighted sample in which each student had an equal chance of being selected. All students in the class were included in the sample.

The questionnaire was administered in class groups, during school hours, under the supervision of at least two researchers. Respondents completed the questionnaire using a Personalised Digital Assistant (PDA). Students could choose to answer in English or Afrikaans. The average time to complete the questionnaire was 40 minutes. The test and retest surveys were carried out under similar conditions using the same questionnaire. The time intervals between test and retest administrations at the three schools were 28, 29 and 62 days, respectively.

For test-retest reliability, data were analysed for the three schools together. Using a process of matching, only those respondents who took part in both the test and retest surveys were included in the analysis. For the reliability analysis the socio-demographic items were excluded. The Statistical Package for the Social Sciences version 20 (SPSS version 20.0) was used to analyse the data. Means and standard deviations (SD) were reported for continuous measurements. Test-retest agreement was assessed using the Cohen’s kappa coefficient (κ) (Cohen, 1960). The κ values can be characterised as 0–0.20 poor, no agreement beyond chance; 0.21–0.40 fair; 0.41–0.60 moderate; 0.61–0.80 substantial; and 0.81–1.00 almost perfect agreement (Landis and Koch, 1977).

Ethical considerations

The study was approved by the Research Ethics Committee of the University of the Western Cape and the Western Cape Education Department (20080411-0025). Permission was obtained from the principals of the schools. An information letter describing the objectives and procedures and the consent form were sent to parents/guardians to obtain active parental consent. An explanation of the study was given to the students who signed assent forms, with the understanding that they could withdraw at any point. Confidentiality and anonymity were ensured.

RESULTS

Face validity

Seven of the eleven experts responded to our request to examine the validity of the HPS Monitoring Questionnaire. All respondents were female. Three of the seven respondents had been involved in the development of the original Rapid Assessment Tool for Schools; six respondents were employed by the Western Cape Department of Health and one respondent was employed by the Western Cape Department of Education.

All seven respondents agreed that the HPS Monitoring Questionnaire was ‘a good measure of the health promoting schools framework’, and that the questions provided a good depiction of the current status of the school as a health promoting school. Furthermore, all seven respondents agreed that the questions ‘made sense and went together’, that they were ‘clear and appropriate for the topic’, and were ‘listed in an appropriate and useful order’. Six respondents agreed that the questions reflected ‘the theory behind health promoting schools’ and were ‘of equal importance’. Five respondents agreed that the questions were ‘asked in the right way to get the true answers’ and that the questions made it possible to distinguish a health promoting school from one that was not.

However, one respondent questioned the relevance of some items to health promoting schools. For example, ‘Our school provides learners with career guidance and entrepreneurship skills’ and ‘Our learners take part in activities that help them to recognise, understand and value differences between themselves (e.g. cultural, religious and social)’. Another respondent suggested ‘enough taps’ could be replaced with ‘We have adequate water and sanitation: there are taps or drinking fountains on school grounds’. One respondent commented that some items could be asked in a more direct way, for example, ‘Our school has implemented the following programmes: …’ could be expressed as ‘Our school has on-going projects to establish and improve …’. These items were either removed or changed in the final version of the HPS Monitoring Questionnaire. Other suggestions for changes were deemed unnecessary.

Reliability

Of the 94 grade 10 student respondents who participated in the initial test survey, 83 (88%) took part in the retest. Their ages ranged from 15 to 18 years (mean age: 15.73 years; SD: 0.86). Female students comprised 41% (n = 34) of the sample (Table 1).

Table 1:

Demographics of sample

		Test survey (n =  94)			Retest survey (n =  83)
		School A	School B	School C	School A	School B	School C
		n =  37	n =  39	n =  18	n =  34	n =  38	n =  11
Age (years)	Mean	16.1	15.6	15.9	16.0	15.5	15.6
	SD	0.97	0.72	0.99	0.97	0.73	0.69
Gender (%)	Female	59.5	28.2	38.9	55.9	28.9	36.4
	Male	40.5	71.8	61.1	44.1	71.1	63.4

The kappa-coefficients demonstrate mostly a fair (κ-scores between 0.21 and 0.4) to moderate (κ-scores between 0.41 and 0.6) agreement between test-retest General and Environment items; a poor (κ-scores up to 0.2) agreement between the Skills and Community test-retest items, a fair agreement between the Policies items, and for most of the questions focussing on Services a fair agreement was found (Tables 2 and 3). Where κ-scores were poor the questions/items should be checked for interpretation and rephrased where needed.

Table 2:

Overview of the Kappa scores (κ-scores) of the HPS Monitoring Questionnaire

	Item	κ-scores	Agreement
General
Q16.	Our school community has been introduced to the Health Promoting School concept.	0.059	Poor
Q23.	Our school has implemented a physical activity programme.	0.143
Q17.	We have a school based team with a representative who acts as a link with other organisations involved in health.	0.144
Q20.	Our school has implemented a food garden programme.	0.167
Q31.	Our school has implemented a sexuality education programme.	0.173
Q18.	Our school has implemented a hand washing and diarrhoea reduction programme.	0.189
Q29.	Our school has implemented a sexually transmitted infection/HIV/AIDS programme.	0.237	Fair
Q21.	Our school has implemented a prevention of drug abuse, dagga (cannabis) and alcohol programme.	0.241
Q32.	Our school has implemented a leadership programme.	0.268
Q22.	Our school has implemented an oral health and tooth brushing programme.	0.324
Q24.	Our school has implemented a prevention of cigarette use programme.	0.332
Q28.	Our school has implemented an anti-bullying and anti-violence programme.	0.334
Q25.	Our school has implemented a traffic safety programme.	0.342
Q30.	Our school has implemented a TB (tuberculosis) programme.	0.345
Q27.	Our school has implemented a prevention of child abuse programme.	0.362
Q26.	Our school has implemented a recycling programme.	0.445	Moderate
Q19.	Our school has implemented a nutrition and feeding scheme reduction programme.	0.484
	Skills
Q44.	We educate our parents and community in health promotion and the prevention of health problem e.g. diabetes.	0.04	Poor
Q36.	Our Life-skills curriculum provides opportunities for learners to practise coping with stress.	0.085
Q37.	Our school provides learners with career guidance and entrepreneurship skills.	0.087
Q38.	We ensure first aid training of learners.	0.1
Q39.	We ensure first aid training of staff.	0.11
Q34.	Our Life-skills curriculum provides opportunities for learners to practise how to refuse to do things they don't want to do.	0.12
Q40.	We ensure first aid training of parents.	0.128
Q43.	We train our educators in health promotion and the prevention of health problems e.g. diabetes.	0.165
Q45.	Our educators are aware & informed about common health conditions that could affect learners (…) & able to manage.	0.175
Q35.	Our Life-skills curriculum provides opportunities for learners to practise solving problems and making decisions.	0.183
Q41.	We ensure first aid training of community members.	0.3	Fair
Q42.	We educate our learners in health promotion and the prevention of health problems e.g. diabetes.	0.302
Q33.	Our Life-skills curriculum provides opportunities for learners to practise communication.	0.332
	Policies
Q63.	Our policies are monitored and reviewed regularly.	0.05	Poor
Q51.	Our school has policies which prevent or reduce physical, social and emotional problems such as a smoking policy.	0.118
Q48.	Our school knows what to do, and who to refer to if there are problems such as drug abuse and child abuse.	0.124
Q46.	Our school has a basic approach that helps create a healthy and happy environment for the whole school community.	0.164
Q54.	Our school has policies which prevent or reduce physical, social and emotional problems such as a healthy tuck shop.	0.17
Q49.	Our school has policies which prevent or reduce physical, social and emotional problems such as a code of conduct.	0.209	Fair
Q50.	Our school has policies which prevent or reduce physical, social and emotional problems such as an AIDS policy.	0.213
Q47.	Our educators set an example of how to be healthy (e.g. participate in sports …).	0.217
Q60.	Our school has policies which prevent or reduce physical, social and emotional problems such as a child-abuse policy.	0.242
Q52.	Our school has policies which prevent or reduce physical, social and emotional problems such as a TB policy.	0.249
Q62.	Our school ensures that all staff, parents and learners are fully informed of what is in our policies.	0.252
Q53.	Our school has policies which prevent or reduce physical, social and emotional problems such as a drug-free policy.	0.261
Q57.	Our school has policies which prevent or reduce physical, social and emotional problems such as a no bullying policy.	0.262
Q55.	Our school has policies which prevent or reduce physical, social and emotional problems such as a healthy lunchbox policy.	0.28
Q61.	Our school has policies which prevent or reduce physical, social and emotional problems such as a learner pregnancy policy.	0.314
Q56.	Our school has policies which prevent or reduce physical, social and emotional problems such as a no alcohol policy.	0.315
Q59.	Our school has policies which prevent or reduce physical, social and emotional problems such as a sun protection policy.	0.32
Q58.	Our school has policies which prevent or reduce physical, social and emotional problems such as a weapon-free policy.	0.341
	Environment
Q91.	Our school is safe INSIDE: getting into our school is controlled.	0.042	Poor
Q85.	Our school is beautiful: grounds and sports fields are kept in good condition.	0.057
Q99.	Our school is safe OUTSIDE: school transport is safe to be driven on public roads, with legal licensed drivers.	0.077
Q78.	Our school actively involves learners in decisions about how the school is organised and run.	0.136
Q79.	Our learners take part in activities that help them to recognise, understand & value differences between themselves.	0.144
Q94.	Our school is safe INSIDE: first aid kits are fully-stocked and checked every week.	0.152
Q83.	Our school is beautiful: school walls are clean.	0.153
Q74.	Our school prohibits physical punishment as an acceptable disciplinary procedure.	0.158
Q105.	We promote conservation of scarce resources (water, electricity, fuel).	0.165
Q98.	Our school is safe OUTSIDE: playgrounds are monitored during intervals.	0.167
Q72.	Our school provides a friendly, rewarding and supportive atmosphere for parents.	0.173
Q104.	We have adequate water and sanitation: sanitary bins are provided.	0.181
Q107.	We have space and facilities for indoor sports.	0.19
Q77.	Our school encourages the connection between school and home life through involving parents.	0.196
Q71.	Our school provides a friendly, rewarding and supportive atmosphere for staff.	0.197
Q73.	Our school encourages active participation and group work in class.	0.2
Q75.	Our school does not tolerate bullying, discrimination and harassment (including sexual).	0.2
Q90.	The classrooms' noise levels are acceptable.	0.21	Fair
Q76.	Our school provides opportunities for learners to experience creative learning experiences e.g. music, art, drama.	0.217
Q80.	Our school promotes equal opportunities for all irrespective of ethnicity, gender, religion and sexual orientation.	0.218
Q96.	Our school is safe INSIDE: we have a private space to administer medication to learners.	0.219
Q97.	Our school is safe OUTSIDE: fences, building, grounds and equipment are in a good condition.	0.223
Q103.	We have adequate water and sanitation: learners assist in keeping toilets clean, especially by flushing after use.	0.224
Q87.	The classrooms have enough and proper seating, furniture and equipment.	0.226
Q92.	Our school is safe INSIDE: toilets are supervised by an adult or senior learner.	0.243
Q101.	We have adequate water and sanitation: toilets are clean and working.	0.249
Q100.	We have adequate water and sanitation: there are taps or drinking fountains on school grounds.	0.259
Q102.	We have adequate water and sanitation: toilet paper and soap are available for hand washing.	0.261
Q89.	The classrooms have no broken windows.	0.274
Q84.	Our school is beautiful: it is litter free.	0.29
Q65.	Our school welcomes and encourages friendly greetings and good manners at all levels.	0.299
Q86.	The classrooms are clean and in good condition.	0.308
Q88.	The classrooms have proper lighting and ventilation.	0.308
Q82.	Our school is beautiful: our school has ongoing gardening projects.	0.309
Q69.	There are good relationships between staff and parents.	0.319
Q106.	We have a sports field that is used regularly.	0.321
Q93.	Our school is safe INSIDE: toilet doors can be locked.	0.322
Q68.	There are good relationships between staff and principal.	0.327
Q70.	Our school provides a friendly, rewarding and supportive atmosphere for learners.	0.328
Q67.	There are good relationships between staff.	0.351
Q95.	Our school is safe INSIDE: we have a sick bay.	0.392
Q66.	There are good relationships between learners and staff.	0.403	Moderate
Q81.	Our school is beautiful: trees have been planted.	*
	Community
Q110.	Our school invites the participation of parents and local community and all HPS projects and programmes.	0.116	Poor
Q113.	Our curriculum includes health-related activities that involve learners working with their families.	0.147
Q112.	Our school offers its facilities for programmes for the local community.	0.148
Q109.	Our school focuses on health problems that are relevant to the community.	0.179
Q111.	There is good communication with local community about HPS activities and events at the school through the media.	0.179
Q108.	Our school involves the whole school community in efforts to promote health.	0.187
Q114.	Our school links with others schools around health issues.	0.274	Fair
	Services
Q138.	Our school knows whom to contact in a medical emergency.	0.017	Poor
Q139.	Our school displays contact numbers for medical emergencies in a place where all can see it.	0.061
Q137.	Local health services (e.g. clinic …) support the school in implementing local health programmes.	0.069
Q116.	Health tests or examinations are provided at our school for hearing.	0.079
Q140.	We ensure that all learners have been immunised prior to enrolment.	0.135
Q132.	Our school is in contact with services that support a safe and healthy environment: traffic safety.	0.161
Q133.	Our school is in contact with services that support a safe and healthy environment: public works.	0.182
Q121.	Our school is committed to accessing services for learners with special needs.	0.197
Q118.	Health tests or examinations are provided at our school for TB.	0.216	Fair
Q117.	Health tests or examinations are provided at our school for dental care.	0.218
Q122.	We have an updated list of qualified service providers (e.g. psychologist, school nurse) in our area.	0.218
Q125.	We have regular contact with service providers: school doctor.	0.222
Q115.	Health tests or examinations are provided at our school for: vision.	0.226
Q131.	Our school is in contact with services that support a safe and healthy environment: environmental health.	0.231
Q126.	We have regular contact with service providers: social worker.	0.233
Q129.	We have regular contact with service providers: speech and hearing therapist.	0.241
Q124.	We have regular contact with service providers: school psychologist.	0.242
Q119.	Health tests or examinations are provided at our school for: HIV.	0.252
Q130.	We have regular contact with service providers: oral hygienist.	0.279
Q128.	We have regular contact with service providers: physiotherapist.	0.284
Q134.	Our school is in contact with services that support a safe and healthy environment: safer schools project.	0.288
Q135.	Is your school supported by an organisation(s)?	0.305
Q120.	Our school is committed to identifying learners with learning and developmental needs.	0.311
Q123.	We have regular contact with service providers: school nurse.	0.329
Q127.	We have regular contact with service providers: occupational therapist.	0.345
Q136.	By which organisation(s) is your school supported? (Check all that apply)	*

No don't know answers in the pre-test survey data.

Table 3:

Overall summary of the test-retest agreement of the HPS Monitoring Questionnaire

	Questions with poor agreement (0 < κ < 0.2)		Questions with fair agreement (0.21 < κ < 0.4)		Questions with moderate agreement (0.41 < κ < 0.6)
General	6/17	(35%)	9/17	(53%)	2/17	(12%)
(17 questions)
Skills	10/13	(77%)	3/13	(23%)
(13 questions)
Policies	5/18	(28%)	13/18	(72%)
(18 questions)
Environment	17/42	(41%)	24/42	(57%)	1/42	(2%)
(42 questions)*
Community	6/7	(86%)	1/7	(14%)
(7 questions)
Services	8/25	(32%)	17/25	(68%)
(25 questions)*

One question could not be analysed as no don’t know answer was observed in the pre-test survey data.

DISCUSSION

In response to the need for a valid and reliable tool to monitor and evaluate changes in health promoting schools in the South African context, the HPS Monitoring Questionnaire was developed. The questionnaire enables schools to better understand the strengths and challenges regarding schools as health promoting schools, from the perspective of students. The current study examined the face validity and reliability of the HPS Monitoring Questionnaire as there were no previous studies in this specific field in South Africa. As such, the study has provided a starting point to improve the validity and reliability of the tool for use in schools.

Due to the broad, holistic framework used in developing health promoting schools, including the constructs of healthy skills, healthy school policy, healthy physical and psychosocial environments, community links, and appropriate support services, a measurement tool might easily become cumbersome. However, there was consensus among the experts that the HPS Monitoring Questionnaire has satisfactory face validity. Some suggestions were made which were taken into consideration and changes were made to the questionnaire prior to conducting the test-retest survey. Although the HPS Monitoring Questionnaire has face validity, more extensive validation of the HPS Monitoring Questionnaire is required, using other methods of validity testing.

The HPS Monitoring Questionnaire showed overall fair to moderate test-retest Kappa agreement scores. The items in the ‘policy’ section demonstrated the highest agreement with a fair agreement for 72% of the items. The items in the section on ‘services’ provided in the schools, the ‘general health programmes’ section, and the section on physical and psychosocial ‘environment’ in the schools, demonstrated fair or moderate agreement for 68, 65 and 59% of the items, respectively. The items in the section on ‘community’ and the section on ‘skills’ demonstrated poor agreement for 86% and 77% of the items respectively. The slightly higher high kappa in the policy section could be attributed to the likelihood of policies remaining the same over the one to two month period in which the test-retest reliability study was conducted. Similarly, services to the school and health programmes offered in the schools (general section) and the environment are also unlikely to change over this short period of time. Where there was low test-retest reliability, this may be explained by the possibility that respondents may, between test and retest surveys, change their opinions about certain topics, or changes might have been made in the school, for example, new skills may have been learnt. Moreover, students may have discussed their answers with others and subsequently may have changed their opinions. It is also possible that with the administration of the first survey students became sensitised to health promotion issues in their school and thus potentially more critical in their retest survey. Students may have become more aware of health promoting activities in their school and changed their opinions. The questions with poor agreement should be improved for future use.

There is much discussion about the development of suitable approaches for evaluating health promotion in schools (Pommier, Guével and Jourdan, 2010). Judd, Frankish and Moulton (2001, p.368) support ‘the use of a comprehensive, diverse set of standards that reflects different concerns and forms of evidence’. Given these discussion, the HPS Monitoring Questionnaire might be regarded as one component of a multifaceted, systemic evaluation of a health promoting school that incorporates the viewpoints of school staff, students and parents, amongst others. For example, schools may have policies and implemented programmes of which their students are not aware. Using the HPS Monitoring Questionnaire with students alone would not give a realistic evaluation of the school, and one might use the questionnaire with school staff and parents as well. On the other hand, the questionnaire might provide school management with valuable information about their students’ knowledge of policies. In South Africa, although education policies are developed at national level individual schools must develop context-specific policies in consultation with parents, and students are expected to be informed about these policies. For example, corporal punishment has been banned by the Department of Basic Education, but each school must have a code of conduct which indicates ways of maintaining discipline (Department of Basic Education, 2006). Additionally, schools may evaluate themselves against national norms and standards for school infrastructure, which were introduced following an extensive advocacy campaign (Department of Basic Education, 2013).

There were several limitations in this study. Firstly, only seven of the eleven experts contacted for the validity study responded. It would have been preferential to have had a greater range of expertise in the responses. Secondly, due to organisational challenges the period in between test and retest surveys was much longer in one school (62 days compared with 28 and 29 days). Furthermore the testing was before and after examinations in two schools, and before and after a lengthy school holiday in the third school. It is plausible that opinions and/or factors in the schools might have changed during this time. In turn, this may have affected the results of the test-retest reliability and resulted in the low agreement of some questionnaire items. Because of the cyclical nature of the school calendar, if the questionnaire is to be used for monitoring purposes, it would be preferable to use it at the same point in time in the school calendar on each occasion. Thirdly, the study was conducted in only one of the nine provinces of South Africa. Finally, as the questionnaire did not have an item to identify the language used and both were simultaneously available on the PDAs, it was not possible to disaggregate the findings, according to the different language groups (English and Afrikaans); therefore, we could not analyse any potential differences in the two groups.

CONCLUSION

In South Africa, there is a need to understand the opinions of students about their school as a health promoting school in order to monitor and evaluate health promoting changes and developments, identify strengths and weaknesses in the school community, and guide school planning. In response to this need, the current study is a first effort at providing a tool that may be used to monitor and evaluate students’ opinions about changes in health promoting schools. It offers a starting point from which to further develop and improve the HPS Monitoring Questionnaire. This study has shown that the HPS Monitoring Questionnaire has satisfactory face validity, although the results of reliability testing were inconclusive. Therefore, further research is warranted.