Psychosocial screening and monitoring for children in foster care: Psychometric properties of the Brief Assessment Checklist in a Dutch population study.

Children in foster care experience higher levels and rates of psychosocial difficulties than children from the general population. Governments and child welfare services have a responsibility to identify those children in care who have need for therapeutic services. This can be achieved through systematic screening and monitoring of psychosocial difficulties among all children in foster care. However, general screening and assessment measures such as the Strengths and Difficulties Questionnaire (SDQ) and Child Behavior Checklist (CBCL) might not adequately screen for the range of difficulties experienced by foster children. The Brief Assessment Checklists for Children (BAC-C) and Brief Assessment Checklists for Adolescents (BAC-A) are measures designed to screen for and monitor attachment- and trauma-related difficulties among child welfare populations. This article reports psychometric properties of the BAC-C and BAC-A, estimated in a population study of 219 Dutch foster children. The results suggest the BAC-C and BAC-A perform both screening and monitoring functions well. Their screening accuracy, internal reliability and concurrent validity are comparable to those estimated for the SDQ within the same child and adolescent sample. Future research is needed to assess the value of the Brief Assessment Checklists (BAC) compared to other measures and to validate cut-points for the BAC. This study further establishes the BAC-A and BAC-C as valid and useful mental health screening and monitoring measures for use with children and adolescents in foster care.

Children in family foster care manifest high rates of clinically significant psychosocial difficulties. In contrast to children from the general population, foster children are characterized by high levels of internalizing and externalizing behaviour problems (Carbone, Sawyer, Searle, & Robinson, 2007; Goemans, Van Geel, Van Beem, & Vedder, 2016; Pritchett et al., 2016). Exact numbers differ across studies, but proportions of foster children with clinically significant mental health problems have been reported to be over one-third (Maaskant, Van Rooij, & Hermanns, 2014), almost half (Burns et al., 2004) or even over half (Tarren-Sweeney & Hazell, 2006). A recent meta-analysis reported that approximately 40% of the foster children show mental health problems (Vasileva & Petermann, 2016). These high numbers suggest a strong need for mental health services for foster children. An important challenge in this respect is the discrepancy between the need for and receipt of mental health services for children in care (Burns et al., 2004; Janssens & Deboutte, 2009; Stanley, Riordan, & Alaszewski, 2005). Estimates of the proportion of those children who have need for mental health services (denominator) and who do not actually receive a service (numerator) vary considerably, ranging from 12% (Tarren-Sweeney, 2010) to almost 40% (Burns et al., 2004; Janssens & Deboutte, 2009). This discrepancy is a consequence of a variety of circumstances, one of which is system-wide failure to detect and monitor psychosocial difficulties (Tarren-Sweeney et al., 2010). One of the shortfalls of current screening and monitoring practices in foster care is the limited availability and use of measures that are valid for this particular population (Chambers, Saunders, New, Williams, & Stachurska, 2010; Denton, Frogley, Jackson, John, & Querstret, 2017; Tarren-Sweeney, 2007). This is also true for the Netherlands, where recent guidelines for foster care policy and practice emphasize the importance of screening and monitoring, but where there is a lack of specialized measures for youth in foster care (De Baat, Van den Bergh, & De Lange, 2015). This study aims to take a step towards improved screening and monitoring of foster children by reporting the psychometric properties of an existing measure (the Brief Assessment Checklist; Tarren-Sweeney, 2013a) for signalling psychosocial difficulties of children and adolescents in foster care in the Netherlands.

Approaches to systematic screening and monitoring of psychosocial difficulties

The systematic identification of foster children’s psychosocial difficulties requires either clinical/developmental assessment or screening that has acceptable accuracy. However, because the former would require considerable expansion of specialized assessment services for children in care, as well as associated workforce training, this goal is perhaps more aspirational than achievable in the short term. Presently, therefore, systematic identification is best achieved through mental health and developmental screening, wherein screening serves as the first step in a multistage assessment approach, and children who screen positive are then referred for more detailed assessment. Beyond the initial identification of such difficulties, child welfare services also have an ongoing duty of care to monitor children’s development and mental health throughout their time in care. Research has shown that foster children’s development does not necessarily improve while in foster care (Goemans, Van Geel, & Vedder, 2015). So even if the initial screening gave no reason for further assessment, foster children’s development needs to be closely monitored to ensure their well-being. For both screening and monitoring, it is important to make use of measures which have good psychometric properties. For the purpose of screening, accuracy is paramount, with high sensitivity and good specificity essential to identify psychosocial difficulties of foster children.

Psychosocial screening and brief monitoring measures for foster children

The Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997) is frequently used as a child mental health screening measure in the Netherlands and elsewhere in and outside Europe, both for children at large and for vulnerable populations such as children in foster care. There is reasonable evidence that the carer-report SDQ total difficulties score provides an accurate screen for elevated and/or clinical-level mental health difficulties among children in care (Goodman & Goodman, 2012; Lehmann, Heiervang, Havik, & Havik, 2014; Marquis & Flynn, 2009). However, this has not dispelled concern that the SDQ and other standard child mental health checklists, such as the carer-report Child Behavior Checklist (CBCL; Achenbach, 1991), may fail to identify some children in care who need clinical services. Three recent reviews highlight an increased focus on mental health screening for vulnerable children exposed to severe social adversity, including maltreatment (Denton et al., 2017; Lewis, 2014; Milne & Collin-Vézina, 2015). These reviews concluded that in addition to general, standard mental health measures such as the SDQ and the CBCL, assessment of children in care should include measures of various attachment- and trauma-related difficulties that are characteristic of this population. While the SDQ and CBCL have good psychometric properties, they might not adequately capture these latter problems (Lewis, 2014; Milne & Collin-Vézina, 2015).

There are only a few specialized measures that focus on psychosocial difficulties specifically manifested by children in foster care (Levitt, 2009). Among these are the Brief Assessment Checklists for Children (BAC-C) and Brief Assessment Checklists for Adolescents (BAC-A; Tarren-Sweeney, 2013a), which are twenty item checklists derived, respectively, from the Assessment Checklist for Children (ACC; Tarren-Sweeney, 2007) and the Assessment Checklist for Adolescents (ACA; Tarren-Sweeney, 2013b). The ACC and ACA are empirically derived caregiver-report psychiatric rating scales designed to measure problematic behaviours, emotional states and relational difficulties experienced by children in care (and similar vulnerable populations), in comprehensive clinical/psychosocial-developmental assessments. The BAC measures were developed for use as screening and brief monitoring measures by children’s agencies, also covering attachment- and trauma-related difficulties. In the Australian development samples, the Brief Assessment Checklists (BAC) provided accurate screening for elevated and clinical-level mental health difficulties among children in care, comparable to that provided by both the CBCL and the Brief Problem Monitor (BPM, a short-form of the CBCL) (Tarren-Sweeney, 2013a). While the BAC measures may possibly provide enhanced mental health screening and monitoring for foster children, their psychometric properties need to be further established (Denton et al., 2017; Tarren-Sweeney, 2013a). Given they, as yet, have only been established for the Australian development samples, there is need for further population-level research to identify their psychometric properties when used elsewhere in the world. As there is a need for specialized screening and monitoring of foster children in the Netherlands (De Baat et al., 2015), the most feasible option would be to translate the measures and adapt them for use in the Dutch foster care context.

This study

This study examined the psychometric properties (screening accuracy, reliability and concurrent validity) of the BAC-C and the BAC-A, based on data obtained in the third wave of a longitudinal population study of Dutch children and adolescents in foster care. This study analysed pertinent psychometric properties of the BAC to establish its validity for two separate purposes: screening and monitoring.

The validity of the BAC’s test scores as a screening measure was estimated from its screening accuracy in relation to various clinical and social welfare dichotomous outcome criteria. Because the most important purpose of screening is to identify children who have need for therapeutic and clinical support services, we examined how accurate BAC measures classify whether or not foster parents and/or foster children received additional support services or interventions. A secondary purpose of screening is to identify foster placements that require additional support services to reduce the risk of placement disruption. Children’s behaviour problems are related to foster parent stress (Hurlburt, Chamberlain, DeGarmo, Zhang, & Price, 2010; Vanderfaeillie, Van Holen, Trogh, & Andries, 2012), and they account for an increased risk of foster placement disruption (Brown & Bednar, 2006; Farmer, Lipscombe, & Moyers, 2005). Therefore, we also examined the accuracy of the BAC measures to identify whether or not foster parents had increased levels of foster parent stress and whether or not foster parents considered quitting foster care. The screening accuracy of the BAC measures was also compared with the SDQ because the SDQ is an often-used screening and monitoring tool in child welfare and mental health services.

To examine the psychometric properties of the BAC’s test scores as monitoring measures, we examined the measure’s reliability by looking at Cronbach’s alpha. Furthermore, we examined the concurrent validity by analysing the associations between BAC scores and SDQ scores and foster parents’ stress. We expected positive correlations between BAC scores, on one hand, and SDQ problem scales and foster parents’ stress, on the other hand. We expected negative correlations between BAC scores and SDQ prosocial scores. In addition, concurrent validity was examined by looking at the association of BAC scores with additional support services for the foster child and the foster family. It was hypothesized that foster families and foster children receiving additional support services have higher BAC scores (Goemans, Van Geel, & Vedder, 2016; Jones et al., 2011; Newton, Litrownik, & Landsverk, 2000).

Method

Procedure

This study reports the results of the third wave of a larger longitudinal study on the development of children in foster care. The aim of the longitudinal study was to establish why some foster care trajectories are more successful in terms of children’s development and preventing breakdown than others. The SDQ was included in each study wave, while the BAC was only included in the third wave. The study waves were spaced six months apart, with the last wave being conducted in October 2015. The study design was approved by the Leiden University Ethics Review Board.

Foster parents were recruited as study participants to report on a foster child (the ‘index’ child) in their care, as well as their own caregiving, their parental stress and intentions to continue fostering. Families with multiple foster children were asked to fill out the questionnaire for one foster child only to ensure independent observations. We invited all foster care agencies in the Netherlands, and seven agencies (25%) agreed to participate. The main reason cited for non-participation was that they had already participated in other foster care–related research and wanted to prevent overloading their foster families. The participating foster care agencies informed their foster parents about the goal of the study and obtained an informed consent from those parents who wished to participate. The researchers only received contact information for those foster parents who consented to participate. Foster parents of children in both short-term and long-term placements were eligible to participate, but those caring for children in ‘crisis’ placements were excluded. For the third wave of the study (October 2015), we approached 561 foster parents to complete either an online questionnaire (94.3%) or, where email or Internet access was not available, a paper questionnaire (5.7%). All invited foster parents were informed that participation was voluntary and that they could end their participation at any moment. It was clarified that all information given by the foster parents to the research team would be handled confidentially and that the research was performed independently of the foster care agencies. After the invitation, two reminders to complete the questionnaire were sent, on a two-week interval. Three weeks after the last reminder, the online questionnaire was closed. A family excursion to an amusement park and several gift vouchers were raffled off among participating foster families.

Participants and index children

Index children were aged between 4 and 17 years, residing in regular full-time family foster care. For the purpose of the present analyses, the larger study sample (N = 219, response rate = 39.0%) is divided into separate child and adolescent samples, reflecting the BAC-C (4–11 years) and the BAC-A (12–17 years) age ranges. Characteristics of the child and adolescent samples, including information about their foster parents, are presented in Table 1.

Table 1.

Characteristics of the samples.

	Categories	4–11 years (BAC-C sample; n = 118)		12–17 years (BAC-A sample; n = 101)
		N (%)	n missing (%)	N (%)	n missing (%)
Age (years) – M (SD)[a]	–	7.80 (2.15)[a]		14.25 (1.68)[a]
Gender	Girls	44 (37.3%)	17 (14.4%)	41 (40.6%)	15 (14.9%)
	Boys	57 (48.3%)		45 (44.6%)
Placement history – M (SD)[a]	–	1.17 (1.66)[a]		1.04 (0.99)[a]
Duration placement – M (SD)[a]	–	47.77 (31.67)[a]		66.82 (51.31)[a]
Type of foster family	Kinship	29 (24.6%)	18 (15.3%)	30 (29.7%)	17 (16.8%)
	Non-kinship	71 (60.2%)		54 (53.5%)
Family composition	Two-parent family	92 (78.0%)	17 (14.4%)	79 (78.2%)	15 (14.9%)
	Single-parent family	9 (7.6%)		7 (6.9%)
Biological children foster parents (at T1)	No	34 (28.8%)	17 (14.4%)	39 (38.6%)	16 (15.8%)
	Yes	67 (56.8%)		46 (45.5%)
Other foster children (at T1)	No	49 (41.5%)	23 (19.5%)	49 (48.5%)	17 (16.8%)
	Yes	46 (39.0%)		35 (34.7%)
Foster parent thinks about quitting foster care	Often	1 (0.8%)	0 (0.0%)	1 (1.0%)	0 (0.0%)
	Sometimes	16 (13.6%)		17 (16.8%)
	Barely	22 (18.6%)		20 (19.8%)
	Never	79 (66.9%)		61 (60.4%)
	I don’t know	0 (0.0%)		2 (2.0%)
Foster parent is planning on quitting	Yes, concrete plans	0 (0.0%)	0 (0.0%)	4 (4.0%)	0 (0.0%)
	Yes, no concrete plans	3 (2.5%)		6 (5.9%)
	No	114 (96.6%)		89 (88.1%)
	I don’t know	1 (0.8%)		2 (2.0%)
Legal framework	Voluntary	16 (13.6%)	19 (16.1%)	19 (18.8%)	15 (14.9%)
	Mandated	83 (70.3%)		67 (66.3%)
Planning to stay in the foster family	Yes	99 (83.9%)	5 (4.2%)	83 (82.2%)	10 (9.9%)
	No	4 (3.4%)		0 (0%)
	Not clear yet	10 (8.5%)		7 (6.9%)
	I don’t know	0 (0%)		1 (1.0%)
Planning for reunification	Yes	4 (3.4%)	5 (4.2%)	1 (1.0%)	2 (2.0%)
	No	107 (90.7%)		97 (96.0%)
	I don’t know	2 (1.7%)		1 (1.0%)
Intervention foster parents	Yes	33 (28.0%)	0 (0.0%)	19 (18.8%)	1 (1.0%)
	No	85 (72.0%)		81 (80.2%)
Intervention foster child	Yes	51 (43.2%)	0 (0.0%)	41 (40.6%)	0 (0.0%)
	No	67 (56.8%)		60 (59.4%)
Parental contact (at T1)	Yes	93 (78.8%)	17 (14.4%)	17 (16.8%)	1 (1.0%)
	No	8 (6.8%)		83 (82.2%)

BAC-C: Brief Assessment Checklists for Children; BAC-A: Brief Assessment Checklists for Adolescents.

For numerical variables, mean (M) and standard deviation (SD) are presented instead of N (%).

Brief assessment checklists

The BAC-C and the BAC-A are brief (20-item) mental health screening and monitoring scales that yield a single score ranging from 0 to 40 (Tarren-Sweeney, 2013a). The items of the BAC measures are derived from the ACC and ACA measures and selected for their statistical screening accuracy, rather than for their substantive link to specific types of attachment and trauma (see Tarren-Sweeney, 2013a). Most items are derived from a few domains of specific attachment- and trauma-related difficulties. For the BAC-C, the majority of the items is derived from four domains: insecure, indiscriminate, non-reciprocal and pseudomature. For the BAC-C, the items mainly stem from three domains, namely, social instability/behavioural dysregulation, non-reciprocal and emotional dysregulation/distorted social cognition. Sample items for the BAC-C are ‘Fears you will reject him/her’ and ‘Distressed or troubled by traumatic memories’. Sample items for the BAC-A are ‘Relates to strangers as if they were family’ and ‘Resists being comforted when hurt’. The measures contain no subscales and are presented in two parts. Part 1 contains less critical, higher incidence items rated on a 3-point Likert scale: 0 (not true), 1 (partly true) and 2 (mostly true) in the past 4–6 months. Part 2 contains more critical, lower incidence items rated on a different 3-point Likert scale: 0 (did not occur), 1 (occurred once) and 2 (occurred more than once) in the past 4–6 months. In the Australian development samples, the BAC-C and BAC-A were highly accurate in screening for CBCL, ACC and ACA clinical range scores (area under the curve (AUC) >0.88) and moderately accurate in screening for caregiver-reported referrals to clinical services (AUC = 0.74–0.79) (Tarren-Sweeney, 2013b).

Translation procedure

Using a strict translation-back-translation protocol, the BAC measures were independently translated into Dutch by three staff members from the Institute of Education and Child Studies. After discussion and agreement, the Dutch versions were independently back-translated by three other staff members. All translators were fluent in Dutch and English. After discussion and agreement about the back-translations, they were compared with the original versions of the checklists by the scale developer, who approved the translation after adjustment of a few minor points (M. Tarren-Sweeney, personal communication, 29 September 2015).

Other carer-report measures and survey questions

SDQ

The SDQ (Goodman, 1997) was employed as one of two child psychosocial outcome measures, along with the BAC. The SDQ was previously translated and validated for use in the Netherlands (Van Widenfelt, Goedhart, Treffers, & Goodman, 2003). The SDQ comprises 25 items which are rated on a 3-point Likert scale: 0 (not true), 1 (somewhat true) and 2 (certainly true). In line with previous research (Goodman, Lamping, & Ploubidis, 2010) and based on the syntax provided by the SDQ website (http://www.sdqinfo.com/py/sdqinfo/c0.py), we combined the 25 items into three subscales – prosocial behaviours, internalizing problems and externalizing problems. The SDQ total difficulties score is obtained by adding all internalizing and externalizing item scores, yielding a possible score ranging from 0 to 40. Previous studies have shown that the SDQ is a valid screening measure (Achenbach et al., 2008; Van Widenfelt et al., 2003), with good convergent and discriminant validity for the subscales ( Goodman et al., 2010). The Dutch version of the SDQ has been found to have acceptable to good psychometric properties (Muris, Meesters, & van den Berg, 2003; Van Widenfelt et al., 2003).

Parenting stress

Parenting stress that is specific to caring for the index child was measured with the abbreviated version of the Nijmeegse Ouderlijke Stress Index (NOSI-K; De Brock, Vermulst, Gerris, & Abidin, 1992), which is based on the Parenting Stress Index (PSI; Abidin, 1990). The NOSI-K consists of 25 parenting stress–related items which are rated on a 6-point Likert scale ranging from 1 (totally disagree) to 6 (totally agree). Parents answer the items in reference to a specific child. Sample items are ‘Child does things that bother me a great deal’ or ‘Child is more of a problem than expected’. The items of the NOSI-K are derived from scales which have been shown to have good concurrent and discriminant validity. Furthermore, the NOSI-K has been found to have high internal consistency (De Brock et al., 1992; Haskett, Ahern, Ward, & Allaire, 2006).

Other survey questions

Foster parents provided information about the foster child (e.g. age, gender, placement history and duration of the current placement), foster family (e.g. kinship or non-kinship and thinking about quitting foster care) and foster placement (e.g. legal framework and interventions aimed at foster parents and/or foster children). Regarding the questions about interventions, foster parents were asked to indicate whether or not there had been any form of additional support during the past 6 months of the current foster placement. It was clarified that this concerned therapy, training and intervention over and above the regular support from the foster care institution.

Statistical analyses

The BAC’s validation as a screening measure is established through estimates of screening accuracy, as indicated by the area under the receiver operating characteristic (ROC) curve or AUC. The AUC statistic indicates the extent of trade-off between sensitivity and specificity when screening for various dichotomous reference criteria. An AUC of 0.5 means that there is no discrimination (e.g. true- and false-positive proportions are equal), and an AUC of 1.0 means that there is perfect discrimination (Swets, 1988). The BAC’s validation as a monitoring measure rests on classical validity and reliability data. The present analyses were carried out with a view to estimating internal reliability (Cronbach’s alpha), item validity and concurrent validity (BAC-SDQ and BAC-NOSI inter-scale correlations).

Results

The distributions (M, SD and range) and internal consistency of the measured scale scores are listed for the two age groups in Table 2. BAC-C and BAC-A item score characteristics (M, SD and item-total correlation) are listed in Tables 3 and 4, respectively. To compare the item score characteristics of our Dutch samples, we also present the item score characteristics of the New South Wales (NSW) Children in Care Study (CICS) sample. Item means and prevalence were reported previously (Tarren-Sweeney, 2013a), while corrected item-total correlations are published for the first time. Based on the suggested cut-point of 5 (Tarren-Sweeney, 2013a), over three-quarters of children and adolescents were screened positive for clinically meaningful mental health difficulties (BAC-C = 85.6%, BAC-A = 78.2%). Mean SDQ scores for both child and adolescent samples fall within the borderline range (Goedhart, Treffers, & Van Widenfelt, 2003). The proportions of child sample SDQ scores in the normal, borderline and clinical ranges were 42.4%, 17.8% and 39.8%, respectively. The proportions of adolescent sample SDQ scores in the normal, borderline and clinical ranges were 47.5%, 15.9% and 36.6%, respectively. Mean foster parents’ stress levels as estimated by NOSI-K scores for foster parents of the child (M = 57.69, SD = 22.43) and adolescent (M = 57.54, SD = 29.11) samples were both within the ‘average’ range. The proportions of foster parents of children who reported ‘below average’ (0–42), ‘average’ (43–61) and ‘above average’ (62–150) stress were 29.3%, 29.3% and 41.4%, respectively. For the adolescent sample, the proportions were 46%, 15% and 39%, respectively.

Table 2.

Distributions and internal consistency of study measure scale scores.

	4–11 years (BAC-C sample; n = 118)				11–17 years (BAC-A sample; n = 101)
	Min–max	M	SD	α	Min–max	M	SD	α
BAC	1–33	12.09	8.16	.89	0–33	11.45	7.76	.87
NOSI-K parenting stress	25–120	57.69	22.43	.94	15–117	57.54	29.11	.97
SDQ total behaviour problems	1–30	12.93	7.00	.85	1–31	11.74	6.87	.84
SDQ internalizing	0–15	4.67	3.55	.74	0–15	5.48	4.05	.77
SDQ externalizing	1–18	8.26	4.65	.85	0–18	6.28	4.11	.81
SDQ prosocial behaviour	0–10	7.36	2.12	.74	0–10	11.73	2.55	.78

BAC-C: Brief Assessment Checklists for Children; BAC-A: Brief Assessment Checklists for Adolescents; SD: standard deviation; BAC: Brief Assessment Checklist; NOSI-K: Nijmeegse Ouderlijke Stress Index; SDQ: Strengths and Difficulties Questionnaire.

Table 3.

BAC-C item characteristics.

Item		Dutch BAC-C sample (n = 118)			NSW CICS BAC-C sample (n = 347)[a]
		Item mean score (SD)	Prevalence[b]	Corrected item-total correlation[c]	Item mean score (girls/boys)	Prevalence[b]	Corrected item-total correlation[c]
1	Can’t concentrate, short attention span	1.15 (0.78)	76.3	.39	0.84/1.16	66	.44
2	Craves affection	1.41 (0.72)	84.7	.27	1.06/0.90	64	.58
3	Eats too much	0.34 (0.62)	26.2	.23	0.37/0.35	25	.32
4	Fears you will reject her/him	0.97 (0.73)	72.0	.64	0.42/0.38	31	.54
5	Hides feelings	0.78 (0.78)	56.8	.56	0.22/0.19	37	.60
6	Is convinced that friends will reject her or him	0.45 (0.59)	39.8	.58	0.29/0.22	20	.57
7	Lacks guilt or empathy	0.67 (0.74)	50.8	.53	0.52/0.63	39	.57
8	Prefers to be with adults, rather than children	0.32 (0.64)	23.1	.54	0.54/0.34	32	.49
9	Relates to strangers ‘as if they were family’	0.61 (0.79)	42.3	.66	0.73/0.69	47	.55
10	Seems insecure	0.75 (0.72)	59.3	.40	0.53/0.56	44	.60
11	Startles easily (‘jumpy’)	0.46 (0.71)	33.0	.56	0.49/0.38	33	.52
12	Suspicious	0.52 (0.77)	34.7	.69	0.24/0.34	22	.56
13	Too dramatic (false emotions)	0.55 (0.77)	38.1	.70	0.61/0.41	35	.59
14	Too friendly with strangers	0.58 (0.79)	39.8	.62	1.02/1.00	68	.46
15	Too jealous	0.64 (0.79)	44.1	.70	0.57/0.47	40	.58
16	Treats you as though you were the child and she or he was the parent	0.33 (0.64)	23.7	.39	0.40/0.29	26	.44
17	Uncaring (shows little concern for others)	0.31 (0.61)	23.7	.47	0.29/0.53	31	.50
18	Distressed or troubled by traumatic memories	0.78 (0.88)	47.4	.53	0.40/0.40	28	.51
19	Does not show pain if physically hurt	0.29 (0.63)	19.5	.31	0.16/0.23	15	.33
20	Sexual behaviour not appropriate for her or his age	0.18 (0.48)	13.5	.38	0.27/0.17	14	.50

BAC-C: Brief Assessment Checklists for Children; NSW: New South Wales; CICS: Children in Care Study; SD: standard deviation.

Data derived from the Children in Care Study carried out in NSW, Australia. Item means and prevalence were reported previously (Tarren-Sweeney, 2013a), while corrected item-total correlations are published for the first time.

Item prevalence is the percentage of children with item score of 1 (partly true) or 2 (mostly true).

Correlation between the item score and the ‘total score minus the item score’ (i.e. item-rest correlation).

Table 4.

BAC-A item characteristics.

Item		Dutch BAC-A sample (n = 101)			NSW CICS BAC-A sample (n = 230)[a]
		Item mean score (SD)	Prevalence[b]	Corrected item-total correlation[c]	Item mean score (girls/boys)	Prevalence[b]	Corrected item-total correlation[c]
1	Constantly seeking excitement or ‘thrills’	0.41 (0.64)	32.7	.25	0.32/0.34	30	.44
2	Craves affection	1.12 (0.73)	79.0	.08	0.80/0.57	46	.37
3	Does not share with friends	0.44 (0.70)	31.7	.49	0.30/0.43	29	.41
4	Does not show affection	0.62 (0.77)	44.5	.49	0.42/0.52	37	.34
5	Feels victimized or misunderstood	0.68 (0.80)	47.5	.65	0.28/0.46	37	.54
6	Gorges food	0.38 (0.63)	29.7	.29	0.31/0.50	28	.42
7	Hides feelings	1.02 (0.75)	73.3	.53	0.79/0.80	56	.44
8	Impulsive (acts rashly, without thinking)	0.67 (0.72)	52.5	.57	0.68/0.95	68	.69
9	Lacks guilt or empathy	0.75 (0.82)	51.5	.68	0.62/0.76	50	.60
10	Relates to strangers ‘as if they were family’	0.53 (0.75)	38.0	.43	0.36/0.42	30	.50
11	Resists being comforted when hurt	0.36 (0.59)	29.7	.41	0.31/0.39	30	.32
12	Shows intense and inappropriate anger	0.41 (0.62)	33.6	.64	0.42/0.63	47	.69
13	Too friendly with strangers	0.45 (0.72)	32.0	.53	0.61/0.52	50	.43
14	Too jealous	0.40 (0.67)	29.7	.59	0.40/0.42	26	.45
15	Tries too hard to please other young people	0.58 (0.78)	40.0	.60	0.53/0.47	40	.37
16	Withdrawn	0.52 (0.70)	40.6	.34	0.39/0.44	28	.29
17	Appears dazed, ‘spaced out’ (like in a trance)	0.50 (0.77)	32.6	.30	0.34/0.24	24	.40
18	Intense reaction to criticism	0.90 (0.87)	57.5	.65	0.58/0.65	41	.61
19	Sexual behaviour not appropriate for her or his age	0.16 (0.49)	11.0	.38	0.19/0.14	8	.33
20	Sudden or extreme mood changes	0.58 (0.80)	38.6	.48	0.42/0.35	34	.57

BAC-A: Brief Assessment Checklists for Adolescents; NSW CICS: New South Wales Children in Care Study; SD: standard deviation.

Data derived from the Children in Care Study carried out in NSW, Australia. Item means and prevalence were reported previously (Tarren-Sweeney, 2013a), while corrected item-total correlations are published for the first time.

Item prevalence is the percentage of children with item score of 1 (partly true) or 2 (mostly true).

Correlation between the item score and the ‘total score minus the item score’ (i.e. item-rest correlation).

Psychometric properties of the BAC as screening measures

ROC analyses were carried out separately for the child and adolescent samples, examining BAC and SDQ screening properties against three reference criteria: (1) receiving intervention services, (2) high foster parents’ stress in caring for the index child and (3) risk the foster parent will quit fostering. Table 5 lists the AUCs and confidence limits for these ROC analyses, along with comparison AUCs obtained for the BAC and BPM in the Australian development samples (Tarren-Sweeney, 2013a). The results show that screening accuracies of the BAC are comparable between the Dutch and the New South Wales sample and to the Dutch SDQ.

Table 5.

Screening accuracy (area under the ROC curve) [95% confidence interval].

	Interventions	Parental stress	Risk of quitting care
Children
Dutch BAC-C	.72 [.63, .81]	.79 [.71, .88]	.60 [.50, .71]
Dutch SDQ	.72 [.63, .81]	.79 [.70, .88]	.61 [.50, .72]
NSW BAC-C[a]	.74 [.69, .80]	–	–
NSW BPM[a]	.75 [.70, .81]	–	–
Adolescents
Dutch BAC-C	.76 [.66, .86]	.87 [.80, .95]	.73 [.63, .83]
Dutch SDQ	.77 [.67, .86]	.83 [.75, .92]	.67 [.55, .77]
NSW BAC-C[a]	.79 [.73, .85]	–	–
NSW BPM[a]	.79 [.73, .85]	–	–

ROC: receiver operating characteristic; BAC-C: Brief Assessment Checklists for Children; SDQ: Strengths and Difficulties Questionnaire; NSW: New South Wales; BPM: Brief Problem Monitor.

Data derived from the Children in Care Study carried out in NSW, Australia, and reported by Tarren-Sweeney (2013a).

Psychometric properties of the BAC as monitoring measures

The internal consistency (Cronbach’s alpha) of the BAC-C and BAC-A scores was .89 and .87, respectively, while the internal consistency of the SDQ total difficulties scores in the same child and adolescent samples was .85 and .84, respectively. The correlation matrix for the study measure scale scores (BAC, SDQ and NOSI-K) is presented in Table 6, with correlations for the child sample set out below the diagonal and those for the adolescent sample above the diagonal. Moderate to strong correlations were found between BAC-C and SDQ scales (total difficulties score, internalizing problems, externalizing problems and prosocial behaviours), with higher scores on the BAC scales related to higher scores on the SDQ problem scales and lower scores on the SDQ prosocial behaviour scale (see Table 5). The correlations of the BAC measures with the SDQ total difficulties score were the strongest, with .83 for the BAC-C and .80 for the BAC-A. Moderate to strong correlations were also found between the BAC measures and parental stress (NOSI-K). Higher scores on the BAC measures were related to higher levels of parental stress.

Table 6.

Correlations between BAC measures and NOSI-K and SDQ.

	1.	2.	3.	4.	5.	6.
1. NOSI-K parenting stress		.597**	.462**	.540**	−.412**	.712**
2. SDQ total difficulties score	.635**		.838**	.843**	−.411**	.795**
3. SDQ internalizing problems	.468**	.805**		.414**	−.374**	.690**
4. SDQ externalizing problems	.598**	.892**	.448**		−.318**	.648**
5. SDQ prosocial behaviour	−.546**	−.426**	−.307**	−.406**		−.518**
6. BAC	.643**	.831**	.757**	.674**	−.440**

BAC: Brief Assessment Checklist; NOSI-K: Nijmeegse Ouderlijke Stress Index; SDQ: Strengths and Difficulties Questionnaire; BAC-C: Brief Assessment Checklists for Children; BAC-A: Brief Assessment Checklists for Adolescents.

Under the diagonal: BAC-C correlations. Above the diagonal: BAC-A correlations.

p < .01.

Furthermore, we tested whether BAC scores were associated with additional intervention or support services received during the placement. Independent sample t-tests revealed a significant small to medium effect of additional support services for children (t(116) = 2.319, p < .05, d = .43) and a large effect for adolescents (t(99) = 4.233, p < .001, d = .88), with children or adolescents receiving additional support scoring higher on BAC measures (MBAC-C = 14.04, SDBAC-C = 7.74; MBAC-A = 15.24, SDBAC-A = 8.12) than children and adolescents not receiving additional support (MBAC-C = 10.61, SDBAC-C = 8.22; MBAC-A = 8.85, SDBAC-A = 6.36). This was also true for the additional support services for foster parents (BAC-C: t(116) = 3.489, p < .01, d = .68; BAC-A: t(98) = 3.245, p < .01, d = .86), with a medium to large effect for children and a large effect for adolescents. Children or adolescents whose foster parents received additional support services scored higher on BAC measures (MBAC-C = 15.79, SDBAC-C = 6.69; MBAC-A = 16.79, SDBAC-A = 8.08) than children and adolescents whose foster parents did not receive additional support services (MBAC-C = 10.66, SDBAC-C = 8.26; MBAC-A = 10.23, SDBAC-A = 7.21).

Discussion

This study examined the psychometric properties of the Dutch version of the BAC measures. The analyses suggest that the BAC-C and BAC-A perform both screening and monitoring functions well among population samples of Dutch foster children and adolescents. The measures’ reliability, concurrent validity and screening accuracy are comparable to those estimated for the SDQ in the same study samples, as well as to those previously reported for the Australian development samples (Tarren-Sweeney, 2013a).

With respect to screening properties of the BAC, both the BAC-C and BAC-A screened for Dutch foster children and adolescents receiving clinical interventions and support services with comparable accuracy to that attained by the SDQ on the same samples, as well as comparable accuracy to that attained by the BAC-C and BAC-A in the Australian development samples (Tarren-Sweeney, 2013a). While the Dutch and Australian studies measured children’s receipt of clinical services in different ways (such that the reference criterion may not be directly comparable), the present results, nevertheless, suggest the Dutch versions of the BAC-C and BAC-A can be employed as mental health screening measures for children in the care of Dutch-speaking foster parents. Only the AUC value of the BAC-C with respect to the risk of quitting care was poor. However, it was equally poor for the SDQ. The poor screening accuracy for the risk of quitting care might be related to the quality of the indicator, with thinking or considering quitting foster care not being a good predictor of actual breakdown. Another explanation might be the fact that most foster children in our sample resided in long-term foster placements. Although there was large variation, foster children resided on average >4.5 years with their current foster family which might indicate that they are stably settled in their foster families. In line with previous studies (Oosterman, Schuengel, Slot, Bullens, & Doreleijers, 2007), a recent retrospective study examined both foster children’s behaviour problems on admission and after 6 months in relation to breakdown and found that only behaviour problems on admission were significantly associated with breakdown (Vanderfaeillie, Goemans, Damen, Pijnenburg, & Van Holen, n.d.). It would be of interest to examine whether the risk of breakdown, which is most prevalent during the first months of the placement, can be predicted in a longitudinal study using a screening measure such as the BAC.

With respect to the psychometric properties of the BAC as a monitoring measure, the internal consistency of the BAC measures were good, particularly for 20-item scales. They are identical to those reported for the BAC-C and BAC-A in the Australian development samples (Tarren-Sweeney, 2013a), and they are comparable to the internal consistency of child and adolescent SDQ total difficulties scores in this study. The SDQ total difficulties score alphas were at the upper end of previous estimates of the internal consistency of the SDQ obtained in studies of children at large (Stone, Otten, Engels, Vermulst, & Janssens, 2010). With respect to concurrent validity, foster children and/or foster families who received additional support services or interventions also scored higher on the BAC. Furthermore, higher scores on the BAC measures were related to higher levels of foster parents’ stress, higher levels of SDQ behaviour problems and lower levels of SDQ prosocial behaviours, with medium to large effect sizes. The correlations between BAC measures and SDQ total difficulties score were particularly high and highly comparable with the correlations of the BAC measures and the BPM measures (.74 and .83 for the child and adolescent samples, respectively) (Tarren-Sweeney, 2013a). Moreover, the correlations between the BAC measures and the SDQ total difficulties score in this study are very similar to the correlations which are generally found between the total problem scores of the SDQ and CBCL (Stone et al., 2010). These findings suggest the BAC-C and BAC-A have good concurrent validity.

Limitations and implications for future research

We could not make statements about the optimal cut-points for the BAC measures in this study. An initial look at the results suggests that when applying the suggested cut-point of 5 (Tarren-Sweeney, 2013a), the BAC has a higher sensitivity than the recommended SDQ cut-points for borderline/clinical range. However, future research should examine screening accuracy of these measures for Dutch children against further reference criteria, such as the CBCL; trauma- and attachment-related measures, such as the ACC; and more specific recording of received interventions or support services.

Although the BAC and SDQ measures performed similarly in this study, and showed comparable screening accuracy, further analysis is required to establish the extent to which the positive screens for each measure identify the same versus different children and adolescents. This is not just determined by the cut-points employed for each measure. Even if the cut-points were calibrated to yield the same numbers and proportions of positive screens, there is likely to be some discrepancy as to which children are identified as positive screens. This is because the BAC and SDQ are designed to screen for different forms of psychopathology. Nevertheless, in the Australian development study, the BAC measures accurately screened for both ACC/ACA elevated and clinical range scores and for the equivalent CBCL borderline and clinical range scores. So we should also anticipate a fair degree of overlap between those children who screen positive on the BAC and those who do so on the SDQ. An important question for future research, therefore, is whether screening accuracy for detecting mental health difficulties among children in care is meaningfully improved beyond increased reliability using both the BAC and SDQ, in place of a single screening measure. Is the BAC indeed more sensitive with respect to attachment- and trauma-related problems than the SDQ or CBCL?

Conclusion

Considering the results on the psychometric properties of the BAC measures, both BAC-C and BAC-A seem promising and useful tools for screening and monitoring in the foster care context of the Netherlands. However, more studies are necessary to more thoroughly assess the value of the BAC next to existing screening measures and to validate or justify cut-points for the BAC. Furthermore, aside from the discussion about which measures to use for screening and what cut-points to set, we should take the high scores of foster children on the BAC as a serious signal with respect to their psychosocial development. The findings of this study indicate that a large group of foster children might experience serious psychosocial difficulties. This finding is comparable with previously reported (Janssens & Deboutte, 2009; Maaskant et al., 2014; Minnis, Pelosi, Knapp, & Dunn, 2001) and fairly similar to the percentages of children with a CBCL total problem score in the clinical range in the NSW sample (Tarren-Sweeney, 2013a). Screening and the specific measures used for screening are important, but it is equally important to know about and invest in follow-up. Screening is supposed to be the first step in a multistage assessment approach.

The Dutch translations of the Brief Assessment Checklist can be downloaded at www.childpsych.org.uk.