Development of a competency-based clinical assessment instrument for exit level Oral Hygiene students at the University of Western Cape

Assessment has been defined as “the systematic collection and analysis of information to improve student learning”, that allows for the recognition of effective teaching and learning practices to reflect one’s pedagogy and to record evidence of meaningful learning [15]. Dental education had witnessed a paradigm shift from a predominantly content-based curriculum to one that is competency focused [16]. The primary undertaking of competency assessments is to produce an entry-level professional capable of independent function. Competency-based assessment implements have extensive acceptance as a measure for assessing a student’s task- related ability [17] and offers more accountability, flexibility, and importantly, it is learner-centred [18]. It encompasses gathering of evidence to validate judgements regarding the student’s progress, for the development of proficient standards and the mastery of skills. It also reflects the student’s ability to translate theory into clinical practice and demonstrate this ability in a work-based context such as the clinic [19]. However, the potential to assess all the dimensions that undergraduates are anticipated to demonstrate is a major challenge. To ensure transparency for both students and teachers, it is imperative that the assessment is aligned to learning objectives and module outcomes.

Aligning assessments with learning objectives and outcomes will not only provide insights on the extent to which the outcomes have been achieved but it will also facilitate student learning whilst ensuring validity and accountability [20]. Outcomes and objectives refer to what is expected of the student to know and be able to do whilst the assessment will reveal to what extent the student has achieved those goals [21]. If assessment criteria are mismatched with the outcomes, measuring the achievement of learning outcomes as well as student progress can become problematic. Hence explicit assessment criteria are seminal to the assessment instrument to create transparent and explicit achievement benchmarks for individual learning outcomes. It serves to postulate how tasks can be assessed [22].

The assessment criteria must also reflect what a student should know and be able to demonstrate as reflected in Miller’s Pyramid which differentiates between knowledge at the lower levels and action in the higher levels. A recent review of this pyramid reflects domains of Bloom’s Taxonomy [11]. The core purpose of Bloom’s taxonomy is to present the learning exemplar progression [23] and to illustrate the progression of learning into higher levels of thinking. This taxonomy is pursued during theoretical and practical assessment components in dental education such as in formative assessments.

Another important aspect on facilitating or supporting learning through assessment is the formative element that allows for the provision of the continuous evaluation of a student’s learning and progress [24]. This kind of assessment practice necessitates the evaluation of learning outcomes on numerous occasions and facilitating content and skill-specific evaluation [25]. An important component of formative assessment is feedback, which has been defined by Boud & Molloy  as “A process whereby learners obtain information about their work in order to appreciate the similarities and differences between the appropriate standards for any given work, and the qualities of the work itself, in order to generate improved work”. These authors propose that a new paradigm shift in feedback should incorporate a more constructivist and student-oriented approach [26].

When developing an assessment instrument, ideally it should provide the assessor with sufficient information to gauge a student’s progress, and simultaneously offer the student enough feedback to leverage their learning. Various models of feedback have been advocated, with each possessing their individual merits and challenges. Irrespective of the model implemented, feedback should follow specific guidelines [27]. The repeated emphasis and focus in literature on student learning with regards to addressing learning gaps, the role of this practice in teaching and learning as well as the role of feedback for improvement has been noted. However, research has also alluded to the noticeably reduced focus on the confluence of each of these aspects [28].

In assessment development, validity is generally measured by how well the instrument measures what it intends to measure and whether the decision about a candidate’s competence is justified [29]. The authors argue that it is necessary to determine the conditions that undermines the integrity of the assessment, by exploring explanations that account for the good or bad performance and finally channelling this information into the review process so as to minimize errors when assessing. They further highlight several ways to ensure reliability and validity. Firstly, in developing an instrument, there must be evidence of how validity and reliability has been tested and built into the design and use of the instrument. Secondly instruments must be reflected on and reviewed periodically to increase validity. Thirdly that the instrument must be designed for the group it was intended and not expect similar outcomes for a different group. Fourthly that assessments should be mapped against policy documents, such as, outcomes of a curriculum. Fifthly that the instrument be subjected to the input from experts (face and content validity). Lastly, one should not mistake authenticity for validity as even a well-developed model is no guarantee of the quality of the items. Constructs need to be defined accurately so that it truly measures the skills and knowledge it is intended to measure [29].

The reviewing of the existing assessment instrument was thus intended to consolidate and build on the strengths of the current system but work towards developing an integrated and authentic model of clinical teaching and assessment that would meet the criteria of being outcomes-based and student-centred with dimensions of being patient-centred. This paper documents the pedagogy and the process involved in developing a formative competency-based assessment instrument making it more robust and fit for purpose for exit level (BOH3) students. The authors proposed that a purposely representative model of knowledge and skills employing a judiciously constructed “blueprint” based on a structured process be developed and implemented [14].

The NGT facilitated the gathering of qualitative data [30] and was the technique of choice as it “seeks to provide an orderly procedure for obtaining qualitative information from target groups who are most closely associated with a problem area” [31]. The study was aligned to the NGT [32] which foregrounds inclusivity and equal contribution when gaining consensus and typically involves several steps: the individual generation of ideas, recording of all ideas and suggestions, group discussion of the ideas, preliminary selection of pertinent ideas and reconvening with group discussion until consensus is reached and a final decision made. In this study, the NGT process started with an initial meeting on an online platform with the academics which allowed for group members to initially present their own perspectives on the existing assessment instrument, rather than as a group. Data from this meeting framed the agenda for the next meeting and were considered within the context as starting points for discussion. Multiple online and face to face group discussions that were iterative in nature allowed members to go back and forth with the development of the instrument. The group members were allowed equal opportunity to express their views and debate on the suggestions of all members. Where there were differences of opinion, individual members were allowed further opportunity to engage in discussion so as to reach consensus from all members. Subsequently, group consensus was reached and the instrument was refined as members became better informed through discussions and consultation. All meetings were recorded and minuted. Audio recordings and typed notes served as data and was analysed by formulating themes which were categorized according to ‘layout and flow of the instrument’, ‘criteria and wording for each domain of assessment’, ‘grading and weighting’ and ‘general information’.

The existing assessment instrument, the literature and group consensus was used to guide the identification of the final domains for the development of the new instrument.

The development of the assessment instrument was conducted in five phases and was informed by the assessment principles of the University and Faculty [5], which include, alignment of assessment criteria with learning outcomes, feedback and transparency and validity.

These phases highlighted below will be discussed in detail as part of the reporting and discussion:

Five main domains were identified to assess clinical competence based on the provision of holistic care and framed within the scope of the Oral Hygienist. These domains included ethics and professionalism; patient assessment; diagnosis and treatment planning and treatment implementation. The domains were not different from the previous assessment instrument used. However, its application was to vary in terms of refined criteria and weighting in the revised instrument.

The first domain of ethics and professionalism is consistently identified as an important competence. It is featured here as an individual domain but it is also assessed as an implicit feature that is underpinned in all the domains. With the increasing emphasis on teaching and assessment of professionalism within the Oral Hygiene curriculum, this aspect is assessed as a core competency that is reflected as a mark denoting the overall level of ethics and professionalism displayed by the student. The second domain, on patient assessment relates systematic collection, documentation and analysis of the information attained through a range of interventions in order to make a diagnosis and draft a corresponding patient management and treatment plan. The process includes a patient interview to gather social, medical and dental histories, dietary analysis, general health appraisal, comprehensive clinical extra and intra-oral, the charting of dental and periodontal status, risk assessments for caries and periodontal disease and radiographic evaluation. The third domain is that of diagnosis. It focuses on the ability of the student to perform an evidence-based analysis and interpretation of the assessment performed in order to make a summation of the patient’s oral health status as well as the recognition of the etiological factors contributing to ill health. The fourth domain is that of treatment planning which involves the selection of interventions in response to the diagnosis and specific health / oral health problems the patient presents with. This domain includes the plan of action within a comprehensive care plan that is cognisant of the etiological factors that must be addressed. It also includes referrals for treatment procedures not within the scope of the Oral Hygienist. The fifth domain focuses on the implementation of educational, preventive, and therapeutic oral care services to support and attain optimal oral health for the patient. It includes the process of performing an evaluation of the patient in response to intervention. Throughout the patient management process, the student must demonstrate the ability to translate knowledge into practice, and use critical thinking skills and clinical reasoning.

Phase 3, involved the refining of the instrument and gaining consensus on the criteria that was created to assess in each domain. Thereafter the criteria within each domain were checked for alignment with the learning outcomes. The outcomes include the knowledge, skills, and values to be attained and as stipulated in the module descriptor of the clinic practice module. These learning outcomes were developed on the basis of the Oral Hygiene Scope of practice (HPCSA) [34] and SAQA exit level outcomes [33]. The process of alignment (Table 1) was guided by the principles described by (Gosling and Moon, 2002) [35].

The assessment criteria were further refined following the interrogation of the assessment instrument which contributed towards its validation. The process allowed for opportunity to gain consensus on areas revealed to be vague, ambiguous and biased.

The level descriptors in the assessment criteria included: “not achieved”, “partially achieved”, “achieved” and “exceeds expectations” with corresponding scores ranging from 1–10.

Appendix 1 highlights the criteria and corresponding scores that were developed to facilitate the students learning towards achieving the set outcomes.

Phase 4 involved the gaining of consensus on the grading and weighting of each domain. The process was informed by the need to provide an assessment that was more student-centred and holistic and that embroidered on knowledge, skills and values but also allowing for the potential to judge the student’s competency in a varied range of tasks [36]. To this end, a score was to be assigned based on an inclusive decision of the student’s work. The blueprint of this clinical assessment was constructed as a rubric that was based on the five domains. The domains for the clinical assessment instrument were weighted according to the complexity of learning and the skills needed to demonstrate the level of competency. The group collectively agreed upon the weighting and scoring (Appendix 1). The assessment instrument was divided into two sections. The first section included the domains of ethics and professionalism weighted at (15%). Assessment (30%), Diagnosis (25%) and Treatment planning (25%) which totalled to 100% for part one. In part two (treatment procedures) all clinical procedures implemented was equally weighted and an average calculated. A summative total was obtained by summing up 50% of part one and part two, respectively (Appendix 1).

Phase 5, focused on aligning the assessment instrument in terms to the assessment principles. The review of the clinical assessment process as depicted in Table 2 was aligned to and located within a framework of assessment principles as set out by the Council of Higher Education [6], UWC Assessment guidelines and the Assessment and Moderation Policy of the Faculty of Dentistry [5].

The main strength of the review and development process of the assessment instrument is that a structured and collaborative process was followed within a context of multilevel consultation. Consultation was with literature, expert academics (clinical teachers) who are involved in assessment as well as the students. Consensus at interim points assured some degree of calibration as the staff using the instrument are the same as the group involved in the development of the instrument. To some extent this process assured reliability in that the instrument would be least affected by influences from different clinical teachers.

Transparency was a further assessment principle that was incorporated in the assessment development process and it relates to the extent to which the assessment plans are shared with the recipient, in this case, the students being assessed. In this project, it began with the sharing of the expected learning outcomes, the assessment instrument, the assessment criteria as well as the mark allocation. The act of transparency also contributed to the validation process of the assessment.