Qualitative multi-stakeholder evaluation of the adoption, implementation and sustainment of the school-based dietary intervention “Jump-in”

Part of the larger obesity prevention program, the Amsterdam Healthy Weight Approach (AHWA [20]), Jump-in was originally designed and shown to be effective in stimulating PA [21, 22, 24]. Following this original positive evaluation, Jump-in was expanded to also include a healthy dietary component and to be further shaped after the Health-Promoting School model (HPS) [23, 25, 26]. Alongside HPS components such as “structurally involving parents” Jump-in comprises three main behavioral change intervention components: stimulating healthy dietary habits, stimulating PA and stimulating active recess play. The City of Amsterdam primarily implements Jump-in in the low-SEP areas of Amsterdam, where overweight/obesity rates are higher than the Dutch national averages. The Jump-in dietary intervention component consists of a set of healthy school nutrition policies, see Table 1. More details on Jump-in [21, 22, 27] and the AHWA [20, 28] have been published elsewhere.

This study used a combination of interviews and focus groups to distill lessons learned from the intervention’s design, adoption, implementation, and sustainment (or embedding). This study is part of a larger mixed methods evaluation study of the Jump-in program [27].

Semi-structured (group) interviews and focus group discussions were conducted approximately 12 or 24 months after the start of the intervention’s implementation. Their focus was to gain insight into participants’ overall experience with the intervention and the contextual factors that (potentially) hampered or facilitated its adoption, implementation and sustainment processes. By involving such a broad range of relevant actors, we aimed to obtain different perspectives on the adoption, implementation and sustainment processes of the intervention. All interviews and focus group discussions were conducted in person within the confines of participants’ school, by two researchers in different combinations (FT and V.B., V.D., V.T. or S.J.). The interview and focus group guides were based on the implementation model of Fleuren et al.: Determinants of Innovation Model [30] with additions from previous interviews conducted by De Meij et al. [22] and Van Nassau et al. [31]. Per school, individual interviews were held with the school principal, Jump-in coordinator, two teachers, and the involved HPP. In some cases, interviews with different respondents were combined into one interview upon participants' request. We collected their views on contextual factors (e.g., socio-political, organizational, user and innovation context, and innovation strategies) across adoption, implementation, and sustainment phases. An example question was: “What factors are hindering the implementation of the Jump-in nutrition policy at this school?".

In addition, for each school, one focus group discussion was held with approximately six to eight parents of 4-to-12-year-old children and another with the same number of 8-to-12-year-old children. We decided for sessions of 6–8 individuals to both provide a variety of perspectives and allow everyone a chance to speak and share their views. However, in some cases, we were unable to reach the minimum of six participants, and we conducted a group interview instead. Discussions with parents mainly focused on the alignment of the intervention with existing school culture (socio-political context), clarity of intervention guidelines (innovation context), and their perception of the workshops and promotional materials (innovation strategies). We gathered opinions on contextual factors (e.g., socio-political, innovation and innovation strategies) across intervention phases using statements, sticky notes and plenary discussions. An example statement was:”I agree with the school restricting regulations with regards to healthy eating and drinking in the classroom, And why (not)?”. When transitioning from a focus group to group interviews, the topics remained similar, the guide was simplified, and statements and sticky notes were not used. The discussions with children mainly focused on their opinions on the intervention (innovation context) and its impact on their (school) lives (socio-political context), and their opinions on workshops and promotional materials (innovation strategies). For children, we used more creative methods such as drawings and sticky notes, along with plenary discussions. An example exercise was: "Draw your life after the Jump-in nutrition policy" after which the drawings were discussed. Detailed information on the interview guide, codebook and researcher characteristics are described in the Consolidated criteria for reporting qualitative studies questionnaire (COREQ) [32] in additional file 1 and the study protocol [27]. The topic list for the interviews and/or focus groups (in Dutch) are available on request.

Schools’ enrollment in the Jump-in intervention occurred during the school year 2016–2017. Schools were eligible to participate in the evaluation study based on identical criteria to those for participation in the Jump-in intervention: schools where obesity rates exceed national averages and that are not participating in other health promotion interventions. At the time of this study, Jump-in was already (being) implemented at more than 100 other primary schools in Amsterdam. From all 10 participating schools, we recruited a convenience sample of teachers, parents and children. The schools’ usual communication channels (e.g., website, newsletters) were used to recruit children and parents, and further adjusted in consultation with the Jump-in coordinator to promote participation. Furthermore, we aimed for a diverse sample of participants, including teachers (across gender and teaching grade), parents (across gender and their child’s age), and children (across gender and age). On a few occasions, we exceeded the required subscription (i.e. two teachers, eight parents and children), enabling us to select a sample with more diversity in terms of gender, grade levels and children’s age, If the number of subscriptions allowed it, we also attempted to prevent the participation of both parent and child from the same family. The interviews and focus groups took place between November 2017 and July 2019.

The interviews and focus group discussions were recorded, transcribed verbatim and analyzed in MaxQDA 2018. The latter was done independently by two researchers. We used Directed Content Analysis [33], a methodology that enabled us to build upon existing research and theories as well as distill novel codes derived from our findings. Prior to data collection, we developed a backbone coding scheme using Fleuren’s Determinants of Innovation Model [30], yet allowing for a significant degree of open coding. Per phase, this model describes determinants and processes at the level of the sociopolitical context, organization, user, and intervention/innovation, i.e., adoption, implementation, and sustainment. The comprehensive coding of all data according to this model, identifying barriers and facilitators within each phase, generated an extensive results section characterized by substantial overlap due to the abundance of data. Subsequently our approach involved thematically summarizing and analyzing the data to distill overarching themes.

Many of the factors (either barriers or facilitators) that we found in our study did not act as stand-alone, individual influences, but rather as parts of specific, sometimes elaborate, interactions impacting the program’s adoption, implementation or sustainment. For example, some parents wished for an expansion of the healthy policies by including a selection of foods and drinks that were considered more culturally inclusive, which simultaneously complicated the teacher’s role that included checking whether all children adhered to the school policies. Also, teachers emphasized the importance of educating children about nuances, such as instances where exceptions to the guidelines are permissible. However, children perceived differences in how teachers enforced the nutrition guidelines as unfair. Qualitative evaluations of health promotion interventions often present lists of barriers and facilitators [18, 30, 39], yet rarely note their complex interactions, which sometimes leads to oversimplified recommendations. Instead, Darlington and colleagues describe five types of interactions: hindering, moderating, counterbalancing, enabling and neutral [35]. Such distinctions may help report and understand how factors are positioned within more complex mechanisms of change. With our thematic analyses, we aimed to understand the influence of certain barriers and facilitators, in a broader context. Such considerations and views could benefit the development of context- and phase-specific strategies to help implement interventions in a real-world setting [18].

A recurring theme in our study was the challenge to implement the intervention as intended (i.e., maintaining intervention fidelity) while simultaneously allowing for, and even stimulating, local adaptation, ownership, and shared responsibility. Allowing for local adaptations makes an intervention more (culturally) appropriate, stimulating feelings of ownership and intrinsic motivation among actors, which may also benefit effectiveness and long-term sustainment. For HPPs this was challenging, confirming HPS intervention studies [9, 40, 41]. Jump-in’s HPPs aimed to adapt the intervention to the local context together with relevant stakeholders while building on evidence-based behavioral change techniques [42, 43] and their peer-to-peer learning network. In addition, there is a need for tools that HPPs can use to distinguish crucial versus adaptable intervention components, so that the intervention’s effectiveness is secured, e.g., Intervention Mapping Adapt [44, 45]. Taking the lead in the implementation processes, however, also hampered the process of ensuring schools feel the sense of ownership and responsibility required to create long-term embedment of the desired system changes [17, 41]. We recommend more research on practical tools for HPPs that enable school actors to feel ownership while simultaneously facilitating smooth implementation processes.

Several hampering factors might be better understood when viewed in the context of a systems approach. For instance, challenges encountered by some teachers in playing the role of “police officers”. Each school represents a unique context, and it is therefore also important to address contextual differences between schools. For example, some teaching teams perceived it as a barrier when they were not involved in the decision-making process, whereas teachers at other schools preferred a small group to handle this, as long as they were kept updated. Additionally, variations were observed in the extent to which parents accepted information from health professionals as accurate. It is therefore crucial to consider the context within schools. Accomplishing sustained changes in schools – as in most complex adaptive systems – often requires coordinated, complementary actions on different socioecological levels [17, 25, 26, 28]. Viewing schools as such in public health promotion efforts can advance the evolution of the symbiosis between health and education [9, 17, 46]. Looking at the implementation processes via the systems view of the Action Scales Model (ASM) [47] shows how aiming to change certain system structures without also dealing with root system goals is unlikely to lead to durable, effective changes. This model states that a system is shaped by a combination of concrete, visible elements (i.e., actions and structures, the tip of the iceberg) and the underlying beliefs and goals that shape them. Schools, for example, revolve around the belief that providing a child with the best possible education is vital to its success in life. Given that belief, and consequent goals, structures, and actions (or: events) naturally emerge to achieve those goals. Therefore, changing certain system structures (e.g., implementing certain healthy nutrition policies in schools) is more likely to succeed when aligned with the underlying system beliefs and goals (e.g., education). Requiring teachers to place their role of health promotor above that of educator in a system that revolves around providing education is unlikely to be successful and sustainable. However, results indicate this is what happens when teachers are expected to argue with parents over healthy nutrition guidelines, thereby endangering their parent-teacher relationship. Creating more lasting impact is likelier to succeed with a simultaneous integration of intervention change objectives focused on targeting underlying systems goals and relevant actors’ beliefs. It would help, for instance, to get school boards to prioritize children’s health alongside their educational achievements. This way, taking such a systems perspective, can provide new insights into potential new leverage points to intervene on and help to create more durable, impactful health promotion initiatives. Therefore, future research could benefit from integrating such approaches during the initial phases of HPS evaluations.

A strength of the current study is its large size. With 39 individual and group interviews and 13 focus groups with a total of 131 participants, it provides a broad range of perspectives from all main actors involved in the intervention’s adoption, implementation, and sustainment processes in a real-world setting. Shaping the data collection and analysis with a clear theoretical framework also added to the study’s strength. In addition, the qualitative evaluation was generally carried out one to two years after intervention adoption, which provided sufficient time to experience recurring, structural barriers and lessons learned. Yet, this timeframe was most of the times not sufficient to capture sustainment.

The study also had certain limitations. Firstly, as is generally the case in research, parents that were willing to participate were potentially more likely to have relatively extreme opinions on the intervention leading to an overrepresentation of their strong views, a phenomenon known as self-selection bias [48]. In addition, despite efforts to mainly select parents from a low SEP, informal data on occupation suggest parents from high SEP backgrounds also participated in the focus groups discussions. However, due to the sampling size this influence is likely minimal and still allowed for obtaining a representative view on the perspectives from both low and high socioeconomic position (SEP) groups.