Guidance to develop a multidisciplinary, international, pediatric registry: a systematic review

The most common purpose of pediatric registries was to monitor the incidence of congenital malformations in newborns (n = 7) [15‐21]. Other purposes were data-collection for performance reporting (n = 3) [17, 22, 23], research (n = 4) [17, 24‐26], health monitoring (n = 2) [17, 27], and complications monitoring (n = 1) [24]. Two registries described the main purpose of creating an inter-operable multicenter framework for data collection [28, 29]. Prince et al. reported the change of a paper-based registry into electronic-based registry as the main goal of their study [30]. Other motivations for developing pediatric registries were improving international cooperation between specialized centers [28] assisting parent or caretakers of pediatric patients to access resources for support [16] Gauvrit et al. defined the purpose of the registry to report general indicator of cochlear implants in pediatric and adult patients [31].

Funding was essential for the development and durability of a registry [23]. Continues funding was necessary to train, hire and retain registry team members [17]. From the included studies, the majority received funding deriving from a local or national government (N = 6) [17, 20, 22, 25, 32, 33]. Other sources were nonprofit organizations [34], or via a combination of multiple sources such as government run grants, and private (industry) trusts [21, 26‐28, 30‐32, 35]. However, a lack of funding, or biased funding could both be detrimental to the functioning of a registry [23]. Nembhard et al. described that the change from a temporary research grant to long term government funding greatly benefitted the overall success of the registry [17].

In total, nine studies reported on the governance of a registry [17, 25, 28‐33, 35]. Two studies noted a registry committee consisting of both patients and experts to ensure governance aligned with the purpose of the registry [28, 29]. Nembhard et al. noted that clinicians entering data felt more comfortable entering data once ownership of the registry was relinquished to government [17]. One study mentioned joint ownership of the data between two or more centers [23]. Four studies mentioned that data was owned and stored at one medical center [28, 30, 33, 35]. One study mentioned that the entire dataset was owned by the principal researcher [32], whilst another noted that data was owned by a private company funded by a patient foundation [29].

Seven studies noted the importance of a solid legal basis for the collection, storage, and use of data [17, 28‐30, 32, 35, 36]. Three studies mentioned the registry gaining approval for implementation after consulting national authorities pertaining to medical legislation [17, 26, 28, 30, 31]. Furthermore, four studies noted that consent forms were required prior to inclusion of patients into the registry [28‐30, 35]. The study by Hassan et al. mentions that implementation of the registry was performed in accordance with local jurisdiction, whilst not attaining to the medical and health research act of 2008 [32]. Lazem et al. ensured all participating centers signed data sharing and cooperation agreements as well as the national pediatric nephrology society [25].

When storing patient data on an electronic database, multilevel security was advised when identifiable patient data were included [37]. These security measures consisted of passwords (n = 2) [17, 35], limiting access to the database to members of the registry team and only allowing de-identified patient data to be used by researchers (n = 7) [17, 22, 28, 30, 32, 34, 35]. Furthermore, in one instance data logs were used to monitor the access and behavior of users [30]. In case of a paper-based framework, secure storage, limited access, and legal identification prior to data access were most frequently used (n = 3) [21, 32, 38]. Finally, one study advised contracts regarding privacy legislation to be signed by affiliated centers prior to participation into the registry [30]. Four studies mentioned the use of servers for data storage [22, 28‐30]. Most of these studies used MySql for as the management system for the database [22, 29, 30]. Hassan et al. describes the use of the DHIS2 software package [32].

Labor needed for the successful running of a registry range between 0.2 full time equivalents (FTE) to 5.0 FTE [17, 22, 32, 34]. Members of registry teams reported are medical experts; principal investigators and co-researchers; and patient representatives [22, 28‐30]. Responsibilities of a registry team entailed mainly data quality controls [17, 32, 34] Other responsibilities were: to improve the overall quality by updating patient case information on a regular basis, and to ensure continued completeness of data [17]. Auditing of researchers prior to relinquishing data was mentioned to be valuable to ensure data was used ethically and securely [17, 28, 33]. Furthermore, centralized registry teams produced periodic benchmark reports and structured feedback to and from participating centers. [22, 23, 30, 33]. Finally, documenting procedures and policies by the current registry team helps potential successors to work effectively after retirement of previous registry team members [17].

Most of the studies described registries that were based on an electronic database (n = 12) [22, 23, 25‐27, 29‐31, 33‐36] or a combination of paper-based entries and direct entries into an electronic database (n = 8) [17, 19, 20, 24, 28, 32, 38, 39]. However, two studies report the use of exclusively paper-based data entries. When classifying countries using the World Bank classification [40]. After 2005,

electronic registries used to be more frequent in high-income countries than in middle income countries and low-income countries. The popularity of electronic databases in low-income countries has increased in the past decade. However, not without issues, as Mallon et al. describe frequent power outages and poor Wi-Fi connections even after distributing laptops to participating centers [26]. Furthermore, Hassan et al. stated the implementation of an electronic database was difficult in LMIC due to the lack of existing electronic frameworks [32]. Having the option of both paper-based and electronic-based entries helped include more centers in participating with the collection of registry data [15‐17, 19, 20, 24, 28, 32, 33, 38, 39]. Electronic databases were therefore an appropriate choice in countries with a solid electronic healthcare framework and had many benefits, such as direct data entries and high data quality [18, 22, 23, 27, 29, 30, 33‐36]. Paper-based data entries hindered participation and had lower efficiency compared to electronic-based data entries [17]. Furthermore, initial design and conceptualization of the registry needed to be profound since later changes to the system could be simple in concept but proved difficult in execution [39]. However, changes in outcome measures after implementation of the registry should be expected in pediatric registries [33]. Shahian et al. even described regular updates of common data elements to evaluate current variables and adding new variables [33]. Finally, three studies stated that registry design flaws caused innate bias and incomplete definition of complex malformations by using low-quality data sources [16, 18, 19].

Eighteen registries discussed the implementation of data quality checks. Our thematic analysis identified four categories: manual data checks prior to the data inclusion; periodic manual data reviews; periodic auditing visitations; automatic data quality checks. Manual data checks performed [16‐18, 20‐23, 25‐35]by the registry staff upon data-entry into the registry were the most common (n = 10) [16, 20‐22, 25, 28, 31‐34]. Periodic auditing visitations were described in three studies [21, 28, 32]. Alternatives mentioned were automated data quality checks, consisting of plausible ranges, mandatory answer options and duplicate removal (n = 4) [22, 28, 33, 35]. Periodic manual data reviews were also used in two examples [23, 26, 32]. Furthermore, in one registry, after initial data entry, a summary was presented to the participant to confirm the correctness of the data [30]. Regarding internal validity of data in the registry, erroneous or inaccurate data entries should be anticipated and prevented [15‐18, 21, 34]. On top of that, several studies advised training of the medical staff at the affiliated centers [21, 26, 28, 32].

Concerning answer options, multiple designs were suggested. The use of some form of standardized answer options (SAO) was mentioned in eighteen studies[15‐19, 21‐23, 25, 26, 28‐30, 33‐35, 38, 39]. Several variations were described, of which tick-box answers, and international classification of Disease (ICD) codes [41, 42] were commonly used [16, 18, 21, 32, 38].

Three studies reported the use of SAO exclusively, while eight studies implemented both questions with free text answer options as well as SAO. Facilitating both options allowed for more profound and descriptive information to be captured [15, 17‐19, 22, 25, 26, 28‐30, 33, 34, 39]. In registries concerning more complex multi-malformations or diseases with phenotypical variety, this strategy was reported to be especially useful [35]. As a supplement to written descriptions, the inclusion of drawings, photographs or x-photographs were of added value [18]. Changing the framing of questions substantially changed the outcome of the questions and should thus be done cautiously according to Minassian et al. [27].

Various data-entry options were discussed in the included studies. Most registries (n = 9 reported that data was entered exclusively by clinicians or medical professionals [18, 20, 21, 23, 25, 26, 28, 30‐32, 34, 39]. Two studies reported the data entries were performed by a dedicated registry team [17, 28]. Seidlin et al. reported that data was entered solely by patients [29]. In the latter case, continues monitoring of data entries was advised, since disease specific surveys can contain questions prone to misunderstanding. To increase correct data entries, patients were provided with a glossary and photographs of the most major manifestations of the disease [29]. Moreover, Hassan et al. concluded that the amount of data that should be entered has to be considered as well: extensive lists of variables that were not part of routine documentation prior to the registry were viewed as tedious by medical professionals. This caused missing data entries and even low-quality data entries in the registry [32].

In total, eleven studies mentioned some form of linkage. Most of these registries allowed for linkage to more than one database or population-based registry, to increase data completeness (N = 8) [17, 19, 20, 24, 27, 33, 36, 38]. Designing the registry with personal identifiers was advised in three studies [19, 20, 34], since this allows for linkage to other registries or population-based databases. Linkage of databases was described as a powerful tool to improve data completeness [15, 17, 19, 20, 24, 27, 33, 36, 38].

Completeness of data was limited by missing data; incomplete data; failure to complete data after linkage or failure to follow-up during data quality checks (n = 6) [16, 21, 22, 24, 27, 28]. Six registries contacted and implored participating centers to correct dubious data entries and false entries, or complete missing entries [20, 22, 28, 32, 33, 35]. Furthermore, to promote participating centers to enter high quality and complete data, two studies used immediate feedback forms after data entries to allow for instantaneous correction by the participants [30, 33]. Completion was improved via routine linkage to other databases or registries in three studies, [17, 18, 27] or via active data ascertainment by a dedicated research team [17]. In case of repeated incomplete or dubious data entries, one registry used a regional coordinator to provide support to the participating center [35]. In case of patient entered data, frequent reminders to complete or update personal data increases data entries were sent to patients [29]. Finally, Mallon et al. report that regular site visits to the participating centers helped increase data completeness [26].

Findings regarding participation were noted in sixteen studies. The thematic analysis showed that participation was influenced by different factors. First, the affiliated centers must feel that the active participation positively affects health outcomes, without causing a disproportionate increase in workload [32]. To ensure involvement of affiliated centers, two studies organized periodic meetings to inform on the functioning and future perspectives of the registry at the center’s site [20, 39] Nembhard et al. suggested active feedback ascertainment [17]. This appeared to improve participation of medical professionals and patients, as well as data quality. Financial reimbursements for completing data entries were mentioned to improve overall motivation of clinicians to participate [34]. Mallon et al. reported financially reimbursing participating centers after more than fifty new case entries to compensate for time [26]. Lastly, the study by Seidlin found a correlation between patient entered data entries and a yearly reminder e-mail to complete data entries [29].

The use of periodic benchmarking reports was mentioned in five studies [22, 23, 30, 32, 33]. The advantages reported were the allowance for continued comparisons between participating centers, for caveats in performance to be addressed and to improve resource utilization [22, 23, 30, 33, 39]. Furthermore, benchmarking or performance reports allowed participating centers to routinely review data in their respective database [22, 25]. In one study, participating centers could voluntarily choose to share personal benchmarking reports with patients or other participating centers. This transparency allowed patients to make better decisions regarding healthcare providers [33].

The purpose of a registry needs be clear for all participating parties and predetermined prior to the implementation of a registry [43]. This purpose can be diverse, such as health monitoring, quality control or scientific research. Currently, registries are getting used more in research for cost-effective data collection. Furthermore, registries are becoming more popular when flexible research design is needed over randomized controlled trials [43‐47]. To warrant durability and functionality of a registry, unbiased funding is required [43]. Funding can be procured via private-sector funding, charities, and non-profit organizations such as the World health organization (WHO) as well as via the government [47]. However, a combination of different sources is not uncommon and expected to become more frequent in the future [43]. Long term funding is especially important for the long-term success of a registry. The design, size and purpose of the registry all influence the amount of funding needed to succeed [43]. Therefore, it is essential to include funding in the entire process of creation and implementation of a registry. To ensure the success of a registry governance is needed. Governance can provide guidance in the form of committees consisting of medical specialists, researchers, and patient representatives. These committees can provide high-level decision making, relating to purpose, funding, and data relinquishing [43]. Furthermore, the WHO suggests publicly stating ownership and governance structure with regards to a registry. (Source) When a registry is operating on an international scale, local data ownership is preferred, due to the legal implications associated with patient data. Privacy laws are strict and can be nation or center specific, thus international ownership of data can prove very difficult practice [48]. Data custody without data ownership can therefore be an appropriate solution to this problem [49]. Furthermore, it is important to consider which data are to be entered in the registry. Variables exceeding standard practice, at least in Europe are subject to more extensive and strict requirements [50]. To guarantee security of patient data, data access should be limited via multilevel security. Limiting full data access to registry to a data committee ensures patient privacy is preserved. Passwords and user logs provide additional protection. Further data protection can be realized by creating data logs and user behavior screening. Numerous free to use software for database provide such services [51‐53]. Furthermore, for local storage, secure servers hosted by the medical centers themselves are recommended. However, in the future, this could change via the General Data Protection Regulation of the European Union. The GDPR ensures that data protection throughout Europe is compliant to the highest standards [54]. Possibly, this could facilitate centralized data ownership in Europe in the future. Another key factor for long-term success of a registry is a registry team. Such teams can be used for manual quality control of entered data. A registry team can create periodic benchmarking reports and consistently perform visitations to participating centers and audit researchers and perform active data ascertainment. Like other registry committees, registry teams can consist of different members, such as medical experts, researchers, and patient representatives. The required full-time equivalents (FTE) required by the registry team to run a successful registry is highly depended on the purpose and size of the registry [43].

The design of a registry is dependent on the purpose for which the registry is created. When designing a registry, it is important to consider what operating framework is best suited for the purpose, the patient population, and the area in which the registry operates This systematic review shows that an electronic based registry is preferred over a paper-based registry provided local ICT facilities allow to do so. An electronic registry allows for higher data quality, higher participation, and immediate data entries. Furthermore, in an international setting, electronic databases allow for instantaneous global functionality. However, it is important to realize that within Europe, internet access and digital skills are not equally distributed throughout the continent [55]. Additionally, even in countries with an electronic infrastructure, health centers can still rely on paper-based records [56]. When multidisciplinary outcome measures are to be captured, the framework of a registry should be accommodating to change, since different specialties might change or expand on the outcome measures. This especially holds true for pediatric registries, since it requires long term follow-up of patients. For instance, tools used to assess patients to change or improve over the years, thereby changing the collected outcome measures over time. However, changing outcome measures should ideally be kept to a minimum since this can be easy in concept but prove difficult in execution. For instance, new data additions might require changes in the framework, training personnel on sight, adjustments to registry protocols and amendments at the medical ethics board. This also results in more financial recourses being used [43]. To ensure the data included in the registry are of high quality and suited for the predetermined purpose of the registry, quality checks need to be implemented. Automated and manual data checks prior to inclusion of data into the registry, periodic manual data checks, periodic auditing visitations and automated data checks are all advised to guarantee continued high data quality. The training of staff at the participating centers is also a viable way to ensure data quality is high. Finally, it is important to consider what answer options are best suited for the registry. By using free-text answer options, SAO, and additional data entries such as photographs, a more profound description of malformations can be achieved. To ensure participants remain motivated to put in the effort of completing data entries over longer time periods, the effort to complete data entries should be reasonable and in line with de purpose of the registry. Therefore, it is important to distinguish between a core data set, ideally consisting of the minimum requirements for achieving the purpose, and additional more extensive data entries useful for further research initiatives. Thus, the purpose of the registry equally dictates the number of variables included in the registry. The WHO suggests appointing a “responsible registrant” to perform and check data-entries to ensure high-quality data is continuously uploaded in the registry [47]. Furthermore, the complexity of clinical outcome measures being captured influences the quality checks required to examine the data entries [57].

The use of personal identifiers is recommended within a local server system to ensure the possibility of linkage to other local or national databases and registries. This can help increase the overall completeness of the collected data in the registry. High completeness of data is mentioned as an indicator of registry success [58‐60]. Using entries by both clinicians and patients improves the overall understanding of malformations. However, in case of patient entered data, careful monitoring is necessary to prevent erroneous data to be entered into the registry. This requires a dedicated team [43]. Furthermore, extensive lists of variables are viewed as tedious and are at risk of not being entered correctly. This not only influences data quality since this negatively influences participation of medical centers. However, by organizing periodic meetings to inform on projects and research efforts arising from the data entered in the registry, participation can be increased. Furthermore, participation was reported to increase when active feedback ascertainment was organized by the registry team. Participation can also be increases by allowing participating centers to co-author research initiatives [60, 61]. Lastly, compensating participating centers after completing a pre-determined amount of complete data entries can improve participation [26]. Another frequently noted way of improving completeness and participation was via benchmarking reports. Benchmarking reports are a useful tool to allow for inter-center comparison in performance and help improve participation and health outcomes for patients. Lastly, if possible, financial reimbursements for participating centers after complete and correct data entries are proven to be effective in increasing participation.