First implementation of a German-language progress test in an undergraduate dental curriculum

Nähere Beschreibung des Projekts

Introduction

Based on the Bologna process, which was initiated 20 year ago, higher education institutions throughout Europe have changed their study programs significantly, targeting on harmonization and comparability. The study programme of “Dental medicine” was introduced in Austria in 1998 and has, since then, also been offered in Graz. In the preclinical years scientific basics are taught. In the clinical part an all-encompassing practical education, by means of a vocational training, preparing students to become safe and independent dental practitioners, stands in the foreground. Students engage with the main dental disciplines including restorative dentistry, oral surgery, periodontology, prosthodontics and orthodontics. Despite the subjective impressions of offering a high qualitative dental education, student outcomes do not always meet the expectations of the educators on the daily basis. A prospective study was undertaken with the undergraduate dental students as a form of quality control. The intervention facilitated the evaluation of knowledge levels and differences

Study design

This study reflects a prospective investigation, aimed at the development and implementation of a Dental Progress test (DPT). The tool of DPT implements a special form of assessment method, developed for the clinical oral surgery education at the Dental school of Medical University of Graz, to evaluate the teaching and learning quality.

Duration and setting

This study was performed within the years 2016 and 2017 at the Dental School of the Medical University of Graz. Therefore, all 7th to 12th term, respectively 4th- to 6th-year, students were recruited for participation in this prospective project. The attendance was planned to be compulsory and accordingly the approval was given by the Local Advisory Committee on Dental Study Affairs. Participants consisted of male and female trainees, mainly from Austria. Depending on the admission to studies, students from Germany and Southeastern European countries were also included.

Ethics approval

According to the Ethics Committee of Medical University of Graz, no concerns about this study were identified.

DPT development

The project was developed by a senior staff member at the Division of Oral Surgery and Orthodontics who has 10 years of experience in dental education and is specially trained in the formulation of multiple-choice questions. First a pool of single-best and true/false (K-type) MC items, 375 in number, was de- signed within a period of one year. This was mapped against the local document of learning content and to current teaching literature. Storage was password protected at the local available IMS2 (Item Management System - Umbrella Consortium for Assessment Networks, Heidelberg, Germany). Every single item contained an explanatory introduction (also known as case vignette) and the question text itself. The author decided on five to six possible answers with only one correct key answer. As a special feature of a progress test, a “don’t know” option was included. In context on the field, the author chose either three or four distractors. Negative as well as double negative formulations were avoided. The difficulty level of all items was at final exam level targeting the four fields of “oral surgery”, “oral medicine”, “oral radiology” and “cases.” Depending on the local catalogue of learning objectives and the subjects pursued during the local clinical dental training, fields also included correlating subcategories. Clinical images and radiographs were also included in 95 items for a better connection to the daily dental practice and to reach a higher discrimination.

MCQ review

A multistage review process was performed question by question for all items. First a factual review was done conducted internally, with a group of four senior academics in-house. A second independent review followed. Finally, senior academics at Dental School of Medical University of Vienna did a third review round. An additional formal review by the local examination department, after which final question pool was completed, was also carried out.

Test schedule and content details

Three progress tests were administered within three terms from the summer term 2016, over the winter term 2016/2017 to the summer 2017. 100 items were randomly chosen for each test. The selection was made on the basis of a predesigned blueprint, referring to the four categories “oral surgery”, “oral medicine”, “oral radiology”, and “cases”. Thereby a specific number of items per test was chosen from each category, described as follows. 30 items came from “oral surgery” targeting on issues such as diagnostics, treatment indica- tions, surgery techniques, instruments, complication a d risk management as well as implant surgery. A further 30 were chosen from “oral radiology” including radiography techniques, radiation protection and image interpretation. The next 20 came from “cases” reflecting clini- cal vignettes as encountered in clinical practice, and another 20 items were selected from “oral medicine”, which also included five “local anaesthesia” and five “acute pain management” items. A repetition of any item was prevented. All tests were computer-based with an overall limited time of three hours. A primary a score of +1 was given for correct answers and -1 for incorrect answers. “Don’t know” options were scored with 0. Students participation was mandatory, but based on the formative design, DPT performance did not influence any pass/fail decision. Feedback was given on student’s scores concerning number of correct, incorrect and “don’t know” responses. Their rank in class and their rank in the total cohort was also communicated separate to each student. The best performers were rewarded, to enhance student motivation.

Post-test review

A post review was completed after each test. Exclusion criteria were defined and applied prior the final analyses of the results. For instance, technical issues during test administration such as problems with the image visualization or items with errors in the assignment of the correct answering options, were considered as a justification for an item exclusion.

Statistical analysis

Data analysis was done anonymously and blinded after the third test, using descriptive and explorative analyses as well as IRT analysis. First student’s response behavior was comput- ed as mean, median, standard deviation, minimum and maximum values for each test sepa- rated in terms and years. Furthermore, the range of results was also described. Due to the changing cohorts and items per test, the response behavior was not calculated for all three tests together. Further test data were calculated as median and inter-quartile range (IQR) or absolute and relative numbers. After the extensive descriptive and exploratory analyses, Rasch analysis (IRT) was used to evaluate the test and to identify misfitting items. After that, in preparation for Rasch analysis, the response categories “don’t know” (0) and “false” (-1) were changed and summarized into only “false”. In the case of only correct or only false answers questions were to be excluded from investigation. Item parameters and person parameters were estimated using response patterns and were expressed on a common log- odds scale. Important requirements for Rasch analysis are uni-dimensionality of data, local independence and sample independency. If all items tap into only one dimension, uni-dimensionality is given. Only in this case the interpretation of an overall score is meaningful. Local dependency, which can inflate reliability, refers to the fact, that another dimension causes dependency among responses. Therefore, responses to the items are not only influenced by the analyzed dimension but also by another. The tests include four the- matic fields. Therefore, local dependency may be introduced by these thematic fields. An advantage of item response theory to classical test theory is, that estimated parameters are not dependent on the sample used for estimation. This so-called sample independency has to be analysed. Further, infit and outfit measures (mean square statistics) and the Wald test were also applied. The reason for this was to identify items that did not match within a uni- dimensional model or had varying item parameters in subsamples of respondents (sample independence). To analyse local independency a second analysis was made grouping the items of the four thematic fields into four polytomous item. To evaluate the assumption of parallel item characteristic curves (ICC), Andersen’s likelihood-ratio tests for goodness-of-fit with mean split criterion were calculated. Furthermore, the preparation of person-item maps followed: a person-item maps shows the ability, distribution of the examinees and the distribution of difficulty parameters of the items. Therefore, the fit of the difficul- ties to the tested sample can be evaluated. The person separation reliability was calculated for the presentation of the internal consistency. For the datasets of four polytomous items the reliability was also calculated. The analyses were done test per test with the appliance of the R-package eRm (Version 0.15-7) and mirt (Version 1.27.1).

Results

Overall 173 students, (male: female/ 1:0.7), participated in all three tests. In the post-review process a total of 6 items was excluded. Item responses resulted in similar levels at all three test time points for the categories “correct”, “false” and “don’t know” (Test 1: 61.6%, 26.6%, 11.8%; Test 2: 56.0%, 27.8%, 16.3%; Test 3: 62.1%, 26.5%, 11.4%). These results were comparable to those reported in the literature. A significant increase in “correct” answers from 4th to 5th year with p <.001 and from 5th to 6th year with p =.002 was observed, however only in Test 2. A significant decrease of “don’t know” answers was seen in Test 1 and 2 from year 4 to 5 with p =.003 and p <.001 and from year 5 to 6 with p <.001 at both. Concerning “false” answers, even a significant increase occurred, however only in Test 1 from year 4 to 5 with p = .009 and from year 5 to 6 with p = .022. The reliability ranged from 0.82- 0.88 at all three tests. Within the Rasch analyses the assumption of parallel ICC was met and item difficulties for the thematic fields were similarly distributed across the latent dimensions.

Conclusion

Targeting the aims of the presented study, our DPT resulted in the first standardized presentation of student knowledge in undergraduate oral surgery education at the Medical University of Graz. Furthermore, it is the first performed DPT in the German-language counties of Germany, Austria and Switzerland in this form.

Currently, knowledge levels and variation of performance during the clinical educational programme could be documented. Although sample size and the duration of the study were limited, essential information was acquired concerning both learning and teaching quality in undergraduate oral surgery education.

Further, a homogeneous response behaviour, a uniform distribution of included and ex- cluded items within the separate tests and fields, and an accurate range of difficulty of the questions could be drawn from the collected data. This means sufficient and sat- isfactory quality of the newly created item bank, which will allow usage in the future.

The DPT provided and facilitated an innovative a comprehensive survey of students, which mandates new requirements for the educational programme at our dental school. As a result, curricular and educational changes have already been launched in the third study section.

Fulfilling the thought of Bologna process, an extension at least national and as a next step to the German-speaking countries or further should be focused. As a result of this, as with the established PT for medical students, dental student cohorts could also be compared between universities. This in turn could increase comparability and competition and force institutions to focus more on high quality in education.

Accordingly, this could give rise to a positive effect in terms of harmonisation of dental education throughout Europe.

National and international collaboration to extend the item pool and sharing resources would definitely be potential options for future developments in the continued use of DPT.