Psychometric properties and contextual appropriateness of the German version of the Early Development Instrument

We tested the psychometric properties and contextual appropriateness of the GEDI in southwest Germany. As preschools in Germany are administered through a variety of mechanisms including private, community-, local government- and faith-based organizations, we recruited schools by contacting the individual responsible at each organization. Recruitment took place from December 2015 to June 2016, with invitations extended by email or telephone to 27 preschool organizations throughout the northern portion of the federal state of Baden-Württemberg (population ca. 11 million). Ultimately, six organizations took part in our study. Of these, three were located in a small city (population ca. 30,000) participating in a larger project to promote health and well-being [29] (http://​www.​ein-gutes-jahr-mehr.​de), and three were located elsewhere in two additional cities. A total of nine preschools belonging to these organizations with 444 children and 60 teachers were available for participation in the study. Inclusion criteria for participating teachers in each preschools were having established a relationship with an eligible child for at least 1 month, having sufficient German language knowledge, and having taken part in a training session prior to the assessment. Eligibility criteria for the children to whom the GEDI was administered included age 3 to 6 years, absence of special needs and parental consent. Sixty teachers from nine preschools completed the GEDI for 225 children (51% of eligible children). Our reporting is based on an extension of the STROBE Statement [30].

The items of the original EDI [31] are rated using 2-point yes/no questions or 3-point scales (often/very true, sometimes/somewhat true, and never/not true; very good/good, average, poor/very poor). Following procedures outlined in the original report [31], we recoded all items for the GEDI-validation data set on a scale of 0 to 10. Mean GEDI scores were calculated for each of the five domains, with higher scores indicating better development. Domain scores were excluded from analyses if a child had three or more missing values for items within a given domain [31]. We did not exclude complete cases because of missing data and followed the same approach to desctibing our sample as in the original EDI calidation paper. In the absence of a normative German sample to establish valid cut-offs, and in line with the original EDI procedures, children who scored lower than the 10th precentile in at least one of the five domains were preliminarily categorized as “vulnerable” in terms of school readiness [32]. However, we were aware that German children scoring below the 10th percentile cut-off might not be vulnerable per se, given the differences between the Canadian and the German educational systems and examined this possibility in an analysis described below.

With the permission of the EDI authors, the GEDI was created through a translation process by the research team and English native speakers consulted for the study, with back translation conducted by a second, independent native English-speaking expert linguist [33]. Differences between the original and back translation versions were discussed until consensus was reached between the translators, back translators and members of the original Canadian EDI research team.

To provide an accurate and meaningful translation, it was necessary to replace three of the 26 supplemental items due to a lack of applicability in the German context. These included assessment of class type, aboriginal/indigenous status and ethnicity. These items were replaced with items to assess group structure, immigration status and country of origin. As the organizational structure and pedagogical objectives of German preschools vary significantly, we included four additional items potentially associated with early childhood development: overall educational goals of the preschool, German as a second language, the availability of additional educational resources (i.e., language skills, art and music instruction, physical activity), and categories for the length of the daily stay at the preschool (up to 5 hours, fige to 7 hours, greater than 7 hours). These characteristics were not reported in the results of the current study, but are rather mentioned for the sake of completeness. Four indicators of socioeconomic status (SES) included in the original EDI (i.e., family income/wealth indicator, parental education, emploment and siblings) were moved to a separate parental survey used to more extensively assess sociodemographic factors as well as the health and family background of the children.

The measurement was adapted from the standard socioeconomic Index [34] (see Table 5) consisting of three components: family income, maternal education, and parental employment. Consistent with federal standards for reporting poverty and wealth and the recommendations for reporting on social cohesion in Europe, household income was determined according to need [34‐36].

Reliability was assessed through checks of internal item consistency, test-retest response and interrater reliability. Internal consistency of items within each of the five domain scales of the GEDI was tested using Cronbach’s alpha. Domain intercorrelations were assessed using Pearson correlation coefficients. To establish test-retest reliability, preschool teachers were asked to complete the GEDI for a second time for a subset of randomly selected children (n = 29) after a two-week interval. To establish interrater reliability, preschool teachers were instructed on how to randomly select a subset of children (n = 27) and the GEDI was completed after an interval of 2 weeks by a different teacher also acquainted with the child for at least 1 month. For both assessments, we confirmed that the child’s data from the first and second measurement time and from both assessors agreed by comparing the following unchanging demographic variables: date of birth, gender, and special needs status.

The validity of the GEDI was explored in several ways: 1) Content validity was assessed using a face-validity approach through qualitative interviews with preschool teachers. 2) Concurrent validity was assessed by comparing Pearson correlation coefficients and by plotting differences using Bland-Altman plots [37] between the mean GEDI scores and those of other previously validated instruments including the SDQ [11] and the DESK 3–6 R [38, 39], administered concurrently (see below). 3) Convergent validity was assessed using exploratory factor analyses (EFA). 4) External validity was assessed through comparison of correlations between GEDI scores and sociodemographic parameters of our sample with those of previous studies using the EDI.

Face-validity was determined following consultation with three preschool teachers, one of whom participated in our study and two who worked at non-participating preschools. Each rated the GEDI in five areas on a 5-point ordinal scale ranging from very bad/low to very good/high regarding the comprehensibility of items, adequacy of examples provided in the items, adequacy of item coverage for key developmental domains, balance of effort and information utility, and usefulness in day-to-day work.

To assess concurrent validity, we collected data from the same individuals using two existing instruments currently applied in some German preschools: the SDQ and the DESK 3–6 R.

To ensure data quality, a 10% random sample of data was selected by the research team and re-entered by a commercial data entry service, resulting in an error rate of 0%. Additionally, congruence of responses given by teachers and parents was confirmed for the child’s date of birth, gender, country of origin, and first language.

To meet the requirement for normality in creating Bland-Altman plots and to enable cross-measure comparisons, we transformed scores for the GEDI (sub-)domains and the corresponding SDQ−/DESK-domains (the outcome variables) into z-scores. To determine whether additional transformation of the scores was necessary, we assessed the distribution of differences between the measures using ladder-of-powers histograms [46, 47]. Bland-Altman plots were then generated using the Stata “-concord” command for all comparisons [48]. Each graph plots the mean of the measures against the difference between the measures. In each plot, the middle horizontal line represents the mean difference between the measures while the outer lines indicate 95% confidence intervals for agreement [49]. The association between each of the GEDI comparisons with SDQ and DESK was examined (i) by considering the mean difference and (ii) the scattering of dots around this line in relation to the latent trait continuum on the x-axis. Bland-Altman plots, generated to enable subgroup analyses by age group (3 years, 4 years, 5 to 6 years), are presented in the appendix (Additional file 1).

All statistical analyses were conducted using the statistical software package STATA (Ver. 13.1 for Mac, StataCorp LP, College Station, TX).

The recruitment process is presented in Fig. 1. Reasons for lack of participation in some preschool organizations included a shortage of staff or the subjective perception of the EDI as a deficit-oriented instrument. Some preschool teachers also noted that involvement in the development of a population-based instrument for primary use in research and policymaking was not a strong motivator for participation. Indeed, some individuals felt the need for an instrument to assess individual children was more important. Due to time constraints, participation in retest- and interrater reliability testing was limited to four preschools and completion of the validated DESK survey for assessment of concurrent validity took place in seven of the nine recruited preschools.

All children to whom the GEDI was administered (N = 225) were additionally assessed with the SDQ. For several reasons, 34 children were excluded from analyses (see Fig. 1) leaving 191 children (85%) in the analytic sample. The DESK was administered to a subgroup of 39 children (17%) (age three: n = 14, age four: n = 15, age five: n = 7, age six: n = 3) in seven preschools.

The average age of children in the analytic sample was 4.72 years (SD 1.05; range: 3 years to 6 years, 9 months; age three (n = 58; 30.4%), age four (n = 60; 31.4%), age five (n = 43; 2.5%), age six (n = 30; 15.7%)). Forty-nine percent of the analytic sample was female, and 2.6, 49.2 and 40.3% had a low, middle or high SES, respectively. Eighteen percent spoke German as a second language.

Descriptive statistics (means, standard deviations, and internal consistency coefficients) for the five domains of the GEDI are presented in Table 1, along with the descriptive statistics reported in parentheses for the original sample used in the development of the EDI [50]. Cronbach’s alphas for each domain of the GEDI were generally very good (> 0.8). Only the domain PHY showed lower, but acceptable internal consistency (0.69).

Test-retest reliability analyses from 18 children and interrater reliability analyses from 19 children did not yield statistically significant differences between the first and second measurements nor between teacher one and two for any of the GEDI domains. Pearson correlations suggested moderate to high test-retest reliability ranging from 0.50 to 0.81 (p < 0.05) and interrater reliability ranging from 0.48 to 0.71 (p < 0.05).

The Bland-Altman method requires normally distributed differences between measures. Ladder-of-power histograms for differences between variable pairs showed approximate normality, indicating that further transformation was not needed. We created plots for domain pairs, in which we expected to observe agreement with each other and with regard to the latent construct. Moreover, we assessed mean differences in plots by different age groups (3, 4, and 5 to 6 years of age) and for the overall sample (Table 3). The overall bias (mean difference) was close to zero for most comparisons except for domain pairs FMO/PHY_3 in three- and five- to six-year-olds, in GMO/PHY_3 in four- and five- to six-year-olds, and in SZK/EMO _1 and AKN/EMO_4 in four- and five- to six-year olds. The table also shows that mean differences for GEDI/SDQ domain pairs predominantly range below zero in younger children (three- and four-year-olds) and above zero in five- and six-year-old children in our sample. For the GEDI-DESK-domain pairs, we could not observe a consistent pattern in the mean differences. Age-specific plots are provided in the appendix (Additional file 1).

Plots A to E in Fig. 2 show dispersion in the extent of agreement between GEDI and SDQ domain score pairs. Generally, we observed good and acceptable agreement in all plots, particularly at the highest end of the latent trait continuum, as points were more tightly clustered around the mean difference line. In the midsection of plots A, B and C and in the lower section of plots D and E, we noted greater dispersion indicating poorer agreement for children with average and lower abilities, respectively.

Plots A to E in Fig. 3 demonstrate dispersion in the extent of agreement between GEDI and DESK domain score pairs. Aside from a few outliers, we observed generally good and acceptable agreement in all plots, particularly at the highest end of the latent trait continuum, where points were more tightly clustered around the mean difference line. We noted strong ageement at the lowest end of plot A. In the midsection of plot B, we observed greater dispersion indicating poorer agreement for children with average abilities.

Several differences were noted when comparing results from the EFA of the GEDI with those from the original EDI report [51] (Table 4): The GEDI, for example, had smaller factor loadings in the main domains PHY, SOC, and EMO, and larger loadings in the main domain LAN and COM. Nevertheless, in all but two of the main domains (PHY and LAN), we found similar factor and subdomain headings. In the main domains PHY and SOC, some items with very small loadings were retained as their content was considered to be strongly related to either physical health or social competence (hungry, is independent in washroom habits most of the time, shows an established hand preference, is able to solve day-to-day problems by him/herself, is able to follow one-step instructions). In contrast, because the item sucks thumb/finger in the main domain PHY had a very small factor loading and its content was considered more closely related to the domain EMO [52], it was excluded. Additionally, a factor analysis of the domain PHY excluding this item resulted in a two factor model explaining a higher proportion of the variance compared with the model including the item. Therefore, our analysis suggested the presence of 15 rather than 16 factors (subdomains) across the main domains. We elected to alter the subdomain headings for the domain LAN from the one used in the original report of the EDI, as we felt the items underlying the factors in our EFA did not refere to language and cognitive abilities per se, but rather to the availability of resources in German preschools to facilitate development of those abilities.

The kernel density plots (Fig. 4) for each domain illustrate the underlying distribution of the data. In the domains PHY, SOC, and COM, the majority of children (> 50%) scored in the upper range of the latent trait continuum (> 8.5). With regard to EMO, more than 50% of children scored above 7.6 and children in the top 10th percentile scored above 9.0. Only the domain LAN did not show a similar distribution, with 50% of children scoring in the lower half of the latent trait continuum (≤5).

Regarding the cut-off of the lowest 10th percentile, we found similarities between the German and original sample cut-offs in all domains except LAN (Fig. 4).

In this study, we were able to demonstrate good reliability and confirmed several aspects of validity for the GEDI.

Specifically, we observed excellent internal consistency and moderate to good test-retest and interrater reliability. Moreover, we confirmed face validity by conducting expert interviews with preschool teachers. Additionally, our findings suggest that most items included in the GEDI are valid indicators for key developmental domains. Testing the GEDI against validated instruments currently in use in Germany showed good correlations between corresponding domains. Bland-Altman plots overall revealed good and acceptable agreement between GEDI and SDQ/DESK-measured domains of child development, respectively. However, some variation in agreement existed across the distribution of scores and between age groups, with the GEDI underestimating scores in younger age groups and overestimating in older age groups. In addition, associations between GEDI scores and specific characteristics such as SES and sex were comparable to previous work [24, 26, 28, 32, 53], suggesting good convergent validity. Lastly, our density plots displayed left-skewed distributions of GEDI scores across domains, as might be expected. Scores in the lowest 10th percentile were largely similar between the German and the original EDI in all domains except LAN.

Although internal consistency was generally good, some exceptions indicate a need for careful consideration. Two domains in particular should be given a closer look: PHY and LAN.

In terms of content, it might be that the domain PHYcontains more than a single latent variable; however, the original item structure of the EDI, which we used as a template, treated this as one domain. Within this domain, four items loaded below 0.3.: (i) the loading for Sucks thumb/finger was near zero and the item-total correlation was quite low (α = 0.077). Similarly, the original EDI report [51] as well as a report from Hagquist et al. (2013) showed poor loading (0.401) of this item in factor analyses and identified it as the most poorly fitting item in the domain [54]. Excluding this item from our analyses, however, did not result in significantly higher internal consistency as the item response category used by 92% of our sample was “never or not true”. One potential explanation for the latter finding is that the vast majority of children in our sample came from families with a higher SES, a setting which might provide greater emotional stability [55]. Given that theories of developmental psychology consider this item reflective of emotional conditions in children such as anxiety or depression [52, 56], we recommend shifting the item to the domain EMO. Similarly, three items within the domain PHY showed a lack of variablity in responses: (ii) child arrives hungry, (iii) is independent in washroom habits most of the time, and (iv) shows an established hand preference. We attribute this to selection issues in our study, with the majority of our sample coming from higher SES households. Previous work suggests that children in high-SES families have more of the resources needed to support their positive development than those from lower SES households [57]. Correlations between SES and child development in our study are to be interpreted with caution, however, and future studies implementing the GEDI or an adaptation of it should ensure that children from diverse social backgrounds are sampled.

While many studies show that a secure and organized parent-child attachment is positively associated with the social, emotional and cognitive skills of children [58], only a few items that relate to these skills exist in the GEDI and the EDI as originally reported. Thus, including additional items in the GEDI covering aspects of familial support may, as others have suggested [59], improve the GEDI.

According to our interpretation, the poor performance of three items within the LAN domains across all age groups (is able to read complex words, is able to read simple sentences, and is able to write simple sentences [response of “never or not true” in almost all cases [96, 96, 97% respectively]]) might be related to differences in the structure and learning objectives in German versus Canadian preschools. As the context in which the original instrument was developed, Canadian preschools focus from the outset on promoting advanced language and math skills, whereas German preschools emphasize free play during preschool time and introduce children to basic numeracy and literacy skills only in the last year for the oldest children. Advanced reading and writing abilities do not represent pedagogical aims in German preschools, which may explain why children have neither developed nor are expected to have these skills before the first grade in elementary school. A previous study [54] from Sweden, where preschools take a similar approach to that of Germany [60], reports similar results for two of the items mentioned. Moreover, children enrolled in German preschools typically range from 3 to 6 years of age (mean of our sample 4.7), while the age range for children participating in the sample used to develop the original instrument was higher (4 years, 11 months to 6 years, 4 months) [50]. This would suggest, therefore, that preschool children in Germany would be even less likely to show these competencies. Therefore, these items might have to be excluded in order to produce a reliable, contextually appropriate instrument documenting early development and vulnerability in Germany.

Our content validity results and a factor structure very similar to that reported in the original study [31] underscore one of the key tenets of developmental psychology - that children’s development is universal [44]. Thus, most of the GEDI items seem to be transferable to children across industrialized countries.

Significant negative correlations between two domains of the SDQ (i.e., SOC and EMO) with comparable domains in the GEDI indicate acceptable concurrent validity. Significant positive correlations of GEDI domains with corresponding DESK domains (all p-value < 0.05) indicate good validity in younger children who comprised a majority of the subsample (74%, n = 29), whereas older children had very low and in some cases negative correlations in half of the corresponding DESK domains (5/11). We attribute this either to the small sample size of older children, or to the fact that those DESK domains mainly include group performance tasks while GEDI items are based on teacher report [39].

Results from Bland-Altman plots show good to moderate agreement between the GEDI and SDQ/DESK, though with some variation aross the distribution of scores. This was expected, as the measures capture constructs that are similar but not identical. For SDQ, agreement values are derived from the total sample. Acceptable agreement only in the highest scores of GEDI-SDQ domain pairs might be related to scorings in the higher ranges of the latent constructs as shown in the density plots. Furthermore, based on the mean differences, preschool teachers tended to underestimate younger children’s development (age groups 3 and 4 years) and to overestimate older children’s development (age groups five and 6 years) with the GEDI compared to assessments applying the SDQ. This finding suggests that GEDI assessment in Germany should be administered using age-specific questionnaires. In the small subsample in which DESK was administered, we found similar results, except for domain pair GEMO/PHY_3, where the mean difference for three-year-olds was near zero and for four-year-olds was almost half a standard deviation below zero. This inconsistency might be attributed to methodological discrepancies and should be interpreted with caution. Taken together, these findings suggest that further adaptation will be necessary for future use of the GEDI, especially for enabling valid measurement of development in middle and lower score ranges of the latent construct.

Despite a factor structure very similar to the one reported for the EDI [51] (16 instead of 15 factors/subdomains), we observed loadings that were generally lower. This might be due to a smaller sample size of and to a younger mean age in our sample. Therefore, a future study to confirm convergent validity of an adapted GEDI should try to achieve a larger sample and a more uniform age distribution.

The density plots reveal distributions as expected in the domains PHY, SOC, EMO and COM. For the domain PHY (also containting items that cover the general state of health and motor abilities of children) our results are in line with national statistics on the general health status of children in this age range [61], in which 57.1, 38.6 and 4.3% report having a very good, good or poor health status, respectively. Another nationally representative study reported, that 5 to 11% of preschool children have noticeable problems in their motor development [62]. This statistic is consistent with findings in our sample, in which a majority of children (> 90%) scored in the upper range of the specific latent trait continuum (right-skewed distribution).

Further, around 17% of children in Germany are affected by psychological health issues [61, 63]. This in consistent with our results from domains SOC and EMO, in which approximately 75% of our sample scored higher than 8.7 and 6.8, respectively.

For COM, only 10% scored below 5 in our sample, a finding consistent with representative monitoring data, in which 3 to 16% of children showed difficulties in the development of communication skills such as vocabulary, speech comprehension, articulation, and oral fluency [64].

For the domain LAN, the picture is different: If we had used the score threshold applied in the report of the original EDI, 50% of the children would have scored below 5 and would have been identified as vulnerable. Report from previous study conducted in Germany, however, suggests that only 20.7% showed deficits in languge and cognitive development [12]. We interpret this as a lack of contextual appropriateness of several items in the domain LAN, as discussed previously.

Our results show that the GEDI was acceptable in the German preschool setting, most items were valid and thus, with further adaptations the GEDI promises to offer a useful tool for monitoring child development at the popoulation level in Germany [31]. In terms of detecting developmentally delayed children, one of the biggest advantages of the GEDI is that it is designed for teacher proxy reports, which makes assessment independent from parental availability related to language barriers or other factors. Moreover, the instrument allows the preschool teacher to reflect on the development of the individual child. If concerns regarding development delay arise from the GEDI assessment, preschool teachers are well-positioned to determine the relevance of the issue for a specific child in question given their own professional capacity.

Cut-off scores to determine vulnerability rates for German preschoolers would have to be generated with a contextually appropriate version of the GEDI adapted to the context in the ways we describe.

In fact, previous studies on the EDI in other countries [24‐26, 28] computed vulnerability rates by classifying children in their study population as developmentally vulnerable if they scored in the lowest 10th percentile in at least one domain. In these countries, the age ranges of children in preschools were somewhat narrower (4–6 years), or they assessed only children from one age group, making comparison with the original EDI sample easier.

Taken together, to establish vulnerability cut-off scores that are contextually appropriate for the German system, the adaptation of the GEDI has to account for both (i) the pedagogical objectives of the German preschool context and (ii) age-appropriateness.

We provide the first evidence of the reliability and validity of a German translation based closely on the internationally renowned EDI instrument. Despite this strength, we acknowledge several limitations to our work. While psychometric evaluations of existing instruments do not require representative samples, for example, a selection bias in our sample makes it difficult to derive reference values and describe child development at the population level. In Germany, a legal requirement exists for active instead of passive consent from children or their parents [65]. While similar selection biases exist in other studies [66, 67], the net result is that participation is greatest among higher SES parents. However, to be useful as a population-based measure, future data should be anonymized and routinely collected in preschools.

While most of our measurements reflected moderate to high reliability, long intervals between the first and second measurement points of our test-retest and interrater reliability check are also a limitation of our study. These were related to the competing time demands of preschool teachers. Nevertheless, correlation values between test and retest were moderate to high, indicating good consistency of data over time. With regard to internal consistency, we followed the recommendations of the COSMIN checklist [68] and were able to show good Cronbach’s alpha values similar to those of the developers [32]. Nevertheless, Cronbach’s alpha has some limitations in assessing the internal consistency of latent variables [69].

While we performed a comprehensive EFA [68], the sample size impeded conducting a confirmatory factor analysis. Moreover, given our small subsample of children assessed with the DESK, we were not able to draw any definitive conclusions on concurrent validity using the DESK. However, using the SDQ as a comparison for the GEDI domains “social competence” and “emotional maturity” in the whole sample exhibited good correlations.

In order to develop an adapted, contextually and age-appropriate version of the GEDI, we suggest the application of Item Response Theory (IRT) [70]. This method has been used for adapting the Swedish [54] and Australian [71] versions of the EDI and may also be suitable for a successful adaptation of the GEDI. We expect IRT analysis to result in a shorter instrument, including only those items with a high information value for age-specific latent trait scopes. This could increase the feasibility of the GEDI for population monitoring in Germany.