The 3600CHILD-profile, a reliable and valid tool to visualize integral child-information.

A 3600Child-profile, with theoretically ordered, integral child-information visualized in one image, is designed by the Dutch preventive Child and Youth Health Care (CYHC). The introduction of this new data/information carrier gives an important incentive to enhance a transformation towards personalized health care for children and adolescents by supporting the complex medical thought process of CYHCmedical doctors (MD's). This information tool aims to effectively estimate child's functioning, detect emerging health problems and inform parents and caregivers. This pilot study evaluated aspects of inter- and intra-rater reliability and concurrent validity of the 3600Child-profile when used by MD's to estimate functioning and needed intervention of 4-year-old children. After the development process, in January 2015, 3600Child-profiles (n = 26) were assessed by MD's, in the Netherlands. Each MD assessed two Childprofiles twice and was matched to another MD receiving exactly the same two profiles. The paired scores and rater's scores of both time-points were compared. Rater's scores also were compared with the 26 reference tests scores. Reliability results showed Intraclass correlation coefficients between 0.71 and 0.82 (overall functioning), Cohen's kappa's between 0.61 and 0.80 (psychosocial functioning) and 0.46-0.47 (needed intervention). Validity results showed a Spearman's correlation coefficient of 0.78 (overall functioning), Cohen's kappa's of 0.43 and 0.77 (psychosocial functioning) and 0.52 (needed intervention). In conclusion, in some domains, acceptable results regarding reliability and validity are found for the visualization of integral childinformation used by CYHC-MD's to assess child-functioning after only a short training. The 3600Child-profile's value on tracking change in functioning and decision-making on intervention needs further exploration.

Introduction

A transformation of the current conventional health care, with treatment after a diagnosis (“find it and fix it”) to a predictive and personalized health care (“predict it and personalize it”) is needed (Doove et al., 2013, Snyderman and Langheier, 2006). Substantial burden and irreversible damage, present at time of diagnosis, can be prevented by tracking health and early intervention in the evolution of disease (Snyderman and Yoediono, 2006, Auffray et al., 2010). Preventive Child and Youth Health Care (CYHC), with surveillance of individual children's health, is focused on early detection and management of symptoms. However, still a shift is needed towards a more predictive and personalized health care.

To enhance this paradigm shift in practice it is important to understand the continuous multidimensional interactions between biological-genetic vulnerability and many environmental factors, which determine health and functioning (Sameroff, 2010). The CYHC collects information on child's health, biological-genetic vulnerability and environment. However, theoretical ordering is needed to make the individually unique and complex pathogenic processes accessible and tangible (Doove et al., 2013).

A 3600Child-profile (Fig. 1), based on the “International Classification of Functioning, Disability and Health for Children and Youth” (ICF-CY), is designed within the Dutch CYHC.

Fig. 1

a: The 360⁰ CHILD-profile. b: example of a 360⁰ CHILD-profile with information about a child.

On this newly developed child-profile, the child-information, already registered in the Electronic Medical Dossier (EMD) from birth by the CYHC, is ordered and visualized on one digital image.

The ICF-CY, based on the integrated Bio-Psycho-Social model of health, is chosen as a framework for ordering information as it is conceptually focused on health and functioning rather than on diseases and diagnosis like other classification systems, e.g. ICD-11 and DSM-5. Moreover, it offers an internationally standardized language for accessible communication with parents and other caregivers (World Health Organization, 2007).

The 3600Child-profile, designed to promote the CYHC transformation towards future proof care, is not a diagnostic instrument. However, it is a data/information carrier, which supports the professional (medical) thought process of MD's during the surveillance of children's health, development and functioning, early detection of emerging health problems and respectively pathogenic processes and symptom management on an individual level.

Previous pilot studies with a mixed method design (an integration of qualitative and quantitative descriptive data), showed that the comprehensibility of the 3600Child-profile, as well as relevance, acceptance and feasibility is good according to parents, children, CYHC-workers and other caregivers (Weijers and van der Goot, 2013).

For a measurement tool used in clinical practice it is essential to develop knowledge regarding relevant aspects of reliability and validity (de Vet et al., 2013). The underlying theoretical construct of the 3600Child-profile is based on the framework of the ICF-CY, which is focused on child's health and functioning. In daily practice, CYHC-workers decide, based on integral child-information, what level of intervention is needed to promote health and lower risks. The aim of this study of the 3600Child-profile was to investigate the inter- and intra-rater reliability and concurrent validity in a population of CYHC-medical doctors (MD's) to estimate overall- and psychosocial functioning and needed intervention. For concurrent validity the results of the 3600Child-profile were compared with best practice results of MD's who have carried out the actual care of the children (overall functioning and needed intervention), and validated questionnaires (psychosocial functioning).

Methods

Study design

In this pilot study a cross-sectional reliability and validity study was conducted.

Concerning reliability it was hypothesized that the Intraclass Correlation Coefficients (ICC) and quadratic weighted Cohen's kappa's between the scores of both rater's and the rater's scores of both time-points are at least 0.6. It was stated that 0.6 is acceptable in this stage of development as the tool was newly introduced as well as the outcome measure (after only a short training for the participating MD's).

For concurrent validity the formulation of hypothesis, as no references for these specific correlations were available, was based on theoretical grounds related to the constructs of the used measures and practical experience of the authors as epidemiologists and medical doctors in the CYHC. The following hypotheses were formulated:

Spearman correlation coefficient and quadratic weighted Cohen's kappa's between the scores based on the 3600Child-profile and the best practice- and questionnaire scores are between 0.5 and 0.8.

Spearman correlation coefficient between the overall functioning scores based on the 3600Child-profile and the best practice scores is between 0.6 and 0.75.

Quadratic weighted Cohen's kappa between the psychosocial functioning scores based on the 3600Child-profile and the validated questionnaire scores is between 0.5 and 0.65.

Quadratic weighted Cohen's kappa between the needed intervention scores based on the 3600Child-profile and the best practice scores is between 0.65 and 0.8.

Study population and material

In January 2015, all 30 MD's working in two different CYHC-organizations (one for 0–4 year and one for 4–18 year old children) in the region of Southern of Limburg, the Netherlands, were asked to participate. Using a design in which various MD's participated, enables generalization of results to all CYHC-medical doctors. Before performing assessments, the MD's attended a short training with an introduction of the 3600Child-profile and the validated CGAS-scale to indicate overall functioning, exercise assessments and discussion.

The sample of children was selected from a representative cohort of a prospective cohort study within the real life CYHC practice in Maastricht; the “MOM-study” (Doove, 2010), a cohort obtained in 2010 of parents of children living in Maastricht and surroundings, between 1-1-2007 and 1-1-2010. During regular visits, performed by doctor-assistants and nurses of the CYHC, children with specific medical or complex issues are selected to be scheduled for consultation by a CYHC-MD. Therefore, the percentage of children experiencing problems in functioning seen by MD's is higher in comparison to children in the general population. An effort was made to obtain a sample of children reflecting a general population and to reach coverage of a substantial range of the measurement scale. The full cohort was divided into 3 groups with different levels of functioning (“high”, “moderate” and “low”) by using functioning scores obtained during the “MOM-study” (continuous scale 6 (high) to 30 (low)). Cut-off points of 11 and 17 were chosen to obtain a sufficient number of children in each group. From each group 10 children were randomly selected for this study.

For the experimental assessments, the researcher (MW; founder of the 3600Child-profile) filled out 3600Child-profiles of 30 selected 4 year old children out of the cohort. The information, obtained from the EMD at the time the data of the reference tests were collected, were transferred into the child-profile's categories: body structures and functions, activities and participation, personal- and environmental factors (Fig. 1).

Measurements

All MD's that attended the 3600Child-profile's introduction and short training assessed two randomly allocated 3600Child-profiles during a timeframe of 5 min (every child-profile was assessed by two MD's). Based on the information on the 3600Child-profile, they estimated the level of overall and psychosocial functioning and needed intervention of the children. During assessments discussion was prohibited. After a time-interval of 4 months, the exact same profiles were reassessed in a random order by the same MD's during a regular MD-meeting.

The MD's indicated the estimated level of overall functioning of the child on a validated continuous scale: the Children's Global Assessment of Functioning (CGAS)-scale (Shaffer et al., 1983). The MD's indicated the estimated level of psychosocial functioning and needed intervention on a three-point ordinal scale.

The reference tests to assess concurrent validity were performed in the original cohort study. In that study, the MD's who carried out the actual care of the included children, scored the level of overall functioning on a continuous scale and needed intervention on a 3-point scale. In the original cohort study, psychosocial functioning was filled out on a validated questionnaire for parents (CBCL1,5-5: Child Behavior Check List for children between 1,5 and 5 years old) and for teachers (TRF1,5-5: Teacher Report Form) (Schmeck et al., 2001, NederlandsJeugdInstituut. Child Behavior Checklist, 2014). These questionnaires also were used as a criterion in other validation studies (Reijneveld et al., 2006). A flow chart of the study material and population is provided in Fig. 2 and all comparators are described in Table 1.

Fig. 2

Flow chart of study material and population.

Table 1

Categories of comparisons and comparators derived from the hypotheses.

Comparisons:	Inter-rater reliability: Experimental scores at T1: MD1 versus MD2
	Intra-rater reliability: Experimental scores MD's T1 versus T2
	Concurrent validity: Experimental scores MD's at T1 versusReference test scores
Comparators
Experimental scores:Scores of MD'safter perceiving 3600Child-profilefor 5 min	Overall functioning	Estimation indicated on CGAS-scale 0–100:(severe dis-functioning to optimal functioning)
	Needed intervention	Estimation indicated on three-point ordinal scale:(“no intervention”, “advice and/or extra CYHC-contact” or “refer to further help”)
	Psychosocial functioning	Estimation indicated on three-point ordinal scale:(“normal”, “borderline”, “clinical”)
Reference test scores:Best-practice scores:by MD's that carried out actual care of included children&Questionnaires for parents and teachers	Overall functioning	Estimation MD indicated on scale 6–30:(good- severe dis-functioning)
	Needed intervention	Decision MD indicated on three-point ordinal scale:(“no intervention”, “advice and/or extra CYHC-contact” or “refer to further help”)
	Psychosocial functioning	CBCL (parents) and TRF (teacher) questionnaires outcome: (“normal”, “borderline”, “clinical”)

The estimation of the level of functioning of children is a daily task of CYHC professionals, as well as the decision if a follow-up or (preventive) intervention has to be advised to parents.

Statistical analysis

Minimal sample size, calculated according to the formulas for ICC given in de Vet et al. (de Vet et al., 2013) for Spearman correlation (Berger et al., 2002), was 20 objects of study with two raters per object.

Descriptives of participating MD's (year of experience, educational level, and gender), included children (age, gender, level of functioning) and distributions of the scores and missing values were presented. Missing data were handled by pairwise deletion (only the specific missing values and the paired values were not included in the analysis).

To measure inter-rater reliability, scores of the same child at T1 by 2 MD's were compared paired-wise and for intra-rater reliability, the two scores of the same child by the same MD at T1 and T2 were compared. For continuous values of overall functioning, the Intraclass Correlation Coefficient-agreement (ICC-agreement) was calculated because we were not only interested in consistency (i.e. ranking) but wanted to know if the raters draw the same conclusions about the functioning of a child (Terwee et al., 2007). The SCALE-analysis was performed and in addition the VARCOMP-analysis and analysis of variance (ANOVA) to obtain all variance components of the measurements. Standard Error of Measurement (SEM-agreement) was calculated conform the formula given by de Vet (de Vet et al., 2013). Bland and Altman-plots were constructed to define Limits of Agreement (LoA) and systematic error (Bland and Altman, 1986). For ordinal values of psychosocial functioning and needed intervention the quadratic weighted Cohen's kappa's were calculated as it adjusts for differences in misclassification between adjacent categories and distant categories (de Vet et al., 2013).

To test the hypothesis for concurrent validity scores of the 3600Child-profile at T1 are compared with best practice scores (overall functioning, needed intervention) and scores on validated questionnaires (psychosocial functioning). For the continuous value of overall functioning, the Spearman's correlation coefficient was calculated as experimental- and reference scores were not measured in the same units and it minimizes the effect of violation of normality and homoscedasticy. For ordinal values of psychosocial functioning and needed intervention the quadratic weighted Cohen's kappa was calculated.

The software package SPSS 21, SPSS Inc. Chicago, USA, was used for statistical analysis. In addition, the VassarStats-website was used to calculate quadratic weighted kappa's (VassarStats, n.d.).

Results

Descriptives

In total 26 MD's participated at T1 and 21 were also present at the second meeting (T2). Twenty six children were assessed by two different MD's (at T1) and 21 children were assessed by two MD's twice (at T1 and T2) (Fig. 2: Flow chart of study population). Characteristics of the medical doctors and the children are provided in Table 2.

Table 2

Characteristics of participating CYHC-medical doctors and the children of whom information is used to construct the 3600Child-profile profiles.

Medical doctors	Categories	Number per category	N
Education level	CYHC-introduction course	2	26
	Specialist CYHC	22
	Missing	2
Years of experience	0–5 years	1	26
	5–10 years	3
	10–15 years	4
	> 15 years	17
	Missing	1
Gender	Male	3	26
	Female	23
Children
Gender	Boys	10	26
	Girls	16
Level of functioning	“High”:	9	26
	“Moderate”:	8
	“Low”:	9
		Mean (SD)
Age (in months)		47	26

The range of continuous scores was between 40 and 100 (CGAS) and 6–22 (reference scores) with 31–35% of the scores in the highest quintile (high functioning). The majority of scores of psychosocial functioning by the MD's (56%), the teachers (69%) and parents (80.8%) were in the “normal” category. The majority of needed intervention scores by the MD's were in the middle category “advice/extra CYHC-contact” (54%) while for the reference test this percentage was lower (23%).

The assumption linearity was met for all relationships concerning the continuous scores.

At T1 there were two missing values on the score of overall functioning (the MD's forgot to fill it out). For intra-rater reliability additional missing values occurred as at T2 five MD's could not attend the second meeting.

Inter-rater reliability

ICC-agreement concerning overall functioning was 0.71 (95%CI 0.42–0.87). The different variance components were: children 233.44; raters 0.00; and error 94.15 and the SEM-agreement was 9.7. The Bland and Altman-plot (Fig. 3a) showed a mean difference of 1.8 (P = 0.546), LoA of ± 27.3 (39% of mean).

Fig. 3

a Bland Altman plot of difference CGAS-score between two raters (MD's) against the average of both raters.

b Bland Altman plot of difference between CGAS-scores MD's at T1 and at T2 against the average of the two scores.

Concerning the psychosocial functioning and the needed intervention the quadratic weighted Cohen's kappa's were 0.80 (95%CI 0.72–0.88), respectively 0.47 (95%CI 0–0.96) and the % of agreement were 73%, respectively 69% (Table 3, main results).

Table 3

Main results. Correlations between 3600Child-profile scores by two different MD's (inter-rater reliability), by MD's at different time-points (intra-rater reliability) and between 3600Child-profile- and reference-scores (concurrent validity) and agreement and measurement error.

Assessment	Correlation: Value (95%CI)		Agreement/measurement error
Inter-rater reliability
Overall functioning(n = 22)	ICC-agreement*	0.71(0.42–0.87)	LoA** ± 27.3 (39% of mean)SEM-agreement*** 9.7
Psychosocial functioning (n = 26)	Quadratic weighted kappa	0.80(0.72–0.88)
Needed intervention(n = 26)	Quadratic weighted kappa0.47(0–0.96)
Intra-rater reliability
Overall functioning(n = 40)	ICC-agreement	0.82(0.68–0.90)	LoA** ± 19.2 (27% of mean)SEM-agreement***7.14
Psychosocial functioning (n = 42)	Quadratic weighted kappa	0.61(0.39–0.83)
Needed intervention(n = 42)	Quadratic weighted kappa	0.46(0.11–0.81)
Concurrent validity
Overall functioning(n = 48)	Spearman's rho	0.78(0.63–0.86)
Psychosocial functioning(TRF- teacher) (n = 52)	Quadratic weighted kappa	0.77(0.68–0.86)
Psychosocial functioning(CBCL-parents) (n = 52)	Quadratic weighted kappa	0.43(0.11–0.75)
Needed intervention(n = 52)	Quadratic weighted kappa	0.52(0.27–0.76)

*Intraclass correlation coefficient-agreement.

**Limits of agreement.

***Standard error of the mean-agreement.

Intra-rater reliability

Intra-rater reliability for overall functioning the ICC-agreement was 0.82 (95%CI 0.68–0.90). The variance components were: children 242.12; time 2.89; error 48.10 and the SEM agreement was 7.14. The Bland and Altman-plot (Fig. 3b) showed a mean difference of − 3.03 (P = 0.058), LoA of ± 19.2 (27% of mean) and no obvious relationship between the measurement error and the true value. The differences were normally distributed.

Concerning the psychosocial functioning and the needed intervention the quadratic weighted Cohen's kappa's were 0.61 (95%CI 0.39–0.83), respectively 0.46 (95%CI 0.11–0.81) with % of agreement of 62% respectively 71% (Table 3).

Concurrent validity

Concerning concurrent validity for overall functioning the Spearman correlation coefficient was − 0.78 (95% CI − 0.86 − (− 0.63)). For psychosocial functioning the quadratic weighted kappa, with TRF-teachers form as reference, was 0.77 (95% CI 0.68–0.86) and with CBCL-parents form as reference it was 0.43 (95% CI 0.11–0.75). For needed intervention, the quadratic weighted kappa was 0.52 (95% CI 0.27–0.76) (Table 3).

Discussion

This study measured inter-rater and intra-rater reliability and concurrent validity of the newly developed 3600Child-profile when used by CYHC-MD's to estimate functioning and needed intervention of Dutch 4-year-old children in a heterogeneous population. This heterogeneous population is a realistic representation of the population MD's usually see in their practice.

Main findings

Reliability outcomes for overall- and psychosocial functioning are promising with correlations widely above the minimal value in the hypothesis. For needed intervention, reliability outcomes are not satisfactory with correlations lower than hypothesized.

For validity correlations between the experimental scores and reference test scores for overall- and psychosocial functioning (if compared with teacher form) and needed intervention were, as hypothesized, between 0.5 and 0.8. The correlations for overall functioning and psychosocial functioning (if compared with teacher form) were higher and the correlations for psychosocial functioning (if compared with parent form) and needed intervention were lower than expected. Although further research is needed, positive indicators were revealed concerning validity.

Strengths and limitations

This study has several strengths. First of all, this first reliability and validity study of the 3600Child-profile shows that, with only a short training in the child-profile and the CGAS-scale, good results can be reached for estimating overall- and psychosocial functioning. Secondly, the MD's were very willing to participate and enthusiastic about using the 3600Child-profile to assess children, which is a premise for successfully implementing a new tool in practice. Thirdly, outcomes of this study enable direct translation into CYHC-practice. Limitations of this study result from the fact that the tool is newly introduced and no earlier reliability and validity studies were performed yet. The lack of evidence and experience could have lead to suboptimal hypothesis and systematic differences in the scores of the compared measurements. The sample size is rather small, however, the minimal sample size was reached and conclusions can be drawn regarding the continuation of development of the 3600Child-profile. Moreover, the majority of MD's of the participating organizations participated and all Dutch CYHC organizations follow the same national guidelines.

Reflection to other literature

The enthusiastic, positive reaction of the CYHC-MD's on working with the 3600Child-profile is in concordance with earlier positive results of pilot studies on comprehensibility, relevance, acceptance and feasibility in practice (Weijers and van der Goot, 2013). As this was the first study to examine aspects of reliability and validity of the 3600Child-profile, no data are yet available in literature. As the minimum standard for reliability an ICC of 0.7 is considered and regarding the CGAS an inter-rater reliability study showed an ICC of 0.73. Our results are higher then expected: in line with these values and well above the chosen cut-off points in hypotheses (Schmeck et al., 2001, Terwee et al., 2007). No validity studies of the CBCL and TRF were found, but a study of test-retest reliability of these questionnaires showed correlations between 0.94 and 0.97 (Achenbach et al., 2008). The CBCL is used as a criterion for studies to validate short checklists, like the Strengths and Difficulties Questionnaire (SDQ), which is commonly used in CYHC to identify psychosocial problems. For the SDQ test-retest reliability correlations between 0.74 and 0.81 are known, while a correlation between the SDQ and CBCL of 0.68 is reported (Achenbach et al., 2008). Reliability outcomes for the 3600Child-profile of this study are comparable with the results of the SDQ and, as expected, lower than results of the extensive CBCL-questionnaire. A rather strong correlation between the 3600Child-profile and CBCL was found in comparison with the reported correlation between the SDQ and CBCL.

In reflection to findings in literature, results of this study are promising when taking into account that the 3600Child-profile and CGAS-scale were newly introduced and the training given was short.

Clinical implications

This pilot study was conducted to justify the need for further research on this tool. This study showed that this new Child-profile can be useful in daily CYHC-practice to quickly distinguish children in levels of overall- and psychosocial functioning in a population that visit the CYHC-MD.

Although the ICF-CY and 3600Child-profile are not yet commonly used in the CYHC, it seems to fit the field of the CYHC. The positive results on the rapid interpretation of functioning after only a short training enhance good and readily implementation in practice with rather low costs.

An explanation for the suboptimal results for needed intervention could be the fact that MD's might need an additional face to face consult to explore the needs of child and parents in order to decide on the exact intervention needed. Whether the child-profile is useful to select children for whom such a face to face contact is needed and/or supportive during these consultations, should be investigated in the future.

This study was part of a development project, with a continuous process of evaluation and adaptation that started in 2010. This study justifies continuation of development and evaluation of other aspects of the 3600Child-profile (e.g. responsiveness, predictive value) during step by step implementation. Algorithms are now being developed and tested to automatically transfer the most up-to-date information from the EMD into the child-profile's categories. Each version of the child-profile (generated at different time-points) can be stored in the EMD. After implementation the empirical process will carry on and the CYHC-working methods with the support of the 3600Child-profile will be evaluated on efficiency of assessing and tracking child-functioning and early detection of emerging health problems. Eventually it is intended to evaluate the other important purposes of the 3600Child-profile like provision of visual support to transmit integrated child-information to parents, youth and other caregivers, facilitate tailored shared decision-making and early and personalized interventions and provision of standardized language to report data on a population (World Health Organization, 2007).

The goal of implementing this multifunctional tool is to contribute to the transformation towards a personalized care and an evidence-based working method in the Dutch CYHC.

Conclusion

Our pilot study showed promising results regarding reliability and validity on relevant domains for the visualization of integral child-information used by CYHC-MD's to assess child-functioning. The 3600Child-profile's value on tracking change in functioning and decision-making on intervention needs further exploration. This newly developed tool for efficient estimation of functioning, early detection of emerging health problems and respectively pathogenic processes, shared decision making and personalized interventions, is enthusiastically received by CYHC-MD's. The positive reactions of the CYHC-MD's indicate that this original 3600Child-profile is rather easy to implement in practice.