First implementation of a German-language progress test in an undergraduate dental curriculum

Umgesetztes Projekt

Ziele

Basierend auf dem vor 20 Jahren initiierten Bologna-Prozess, haben die Europäischen Universitäten wesentliche curriculare Änderungen, vor allem in Richtung Harmonisierung und Vergleichbarkeit, vorgenommen. Dementsprechend wurde auch das Zahnmedizinstudium in Österreich 1998 implementiert und wird seither auch in Graz angeboten. Im präklinischen Abschnitt wird theoretisches Basiswissen vermittelt. Im klinischen Teil steht im Sinne einer Berufsausbildung eine umfassende praktische zahnmedizinische Ausbildung als Vorbereitung auf ein sicheres und unabhängiges Arbeiten als Zahnarzt/Zahnärztin im Vordergrund. Die Studierenden durchlaufen dabei im Wesentlichen die Fachbereiche Restaurative Zahn- medizin, Orale Chirurgie, Parodontologie, Prothetik und Kieferorthopädie. Trotz des subjektiven Eindrucks, eine qualitativ hochwertige Ausbildung anzubieten, entspricht die Qualifikation der Absolventen nicht immer den Erwartungen der Lehrenden. Aus diesem Grund wurde im Sinne einer Qualitätskontrolle von Lehr- und Lernerfolg die folgende prospektive Studie initiiert. Dabei wurden mittels einer speziellen longitudinalen Prüfungsmethode Wissensniveaus und Wissensunterschiede in der zahnmedizinisch-oralchirurgischen Lehre standardisiert evaluiert und den Studierenden ihre individuellen Ergebnisse, sowie ihr Rang innerhalb der Kohorte übermittelt. Die Methode nennt sich "Progress Test Zahnmedizin" und wurde erstmals im deutschsprachigen Raum für die Zahnmedizinausbildung an der Medizinischen Universität Graz entwickelt und implementiert. International wird ein derartiger Test auf dem Gebiet der Zahnmedizin nur an der University of Plymouth (UK) durchgeführt.

Folgende Ziele wurden im Rahmen des präsentierten Projektes verfolgt :

Ad hoc Überprüfung der studentischen Wissensniveaus im 3. Studienabschnitt

Überblick über Wissensunterschiede im Verlauf des 3. Studienabschnittes

Evaluierung, ob der entwickelte Test für die Zielgruppe geeignet ist

Vergleich des Tests mit Daten aus der Literatur

Initiierung von curricularen Veränderungen basierend auf den Testergebnissen, um für die didaktischen Herausforderungen der nächsten Dekade gerüstet zu sein

Initiierung einer Prüfungsmethode zur Förderung der nationalen und internationalen Vergleichbarkeit, der interfakultären Kooperation und Harmonisierung von Curricula im Sinne des Bologna Prozesses und der Verbesserung des pan-europäischen Studierenden- und Lehrendenaustausches.

 

Das Projekt wurde im July 2018 publiziert:

Kirnbauer B, Avian A, Jakse N, Rugani P, Ithaler D, Egger R

First reported implementation of a German-language progress test in an undergradu-ate dental curriculum: A prospective study.

Eur J Dent Educ. 2018 Jul 1. doi: 10.1111/eje.12381

Kurzzusammenfassung (dt.)

In diesem prospektiven Projekt geht es um die Entwicklung und Implementierung des sogenannten "Progress Test Zahnmedizin" (DPT). Dieses ist ein longitudinales Prüfungsverfahren, welches für die Evaluierung der Lehr- und Lernqualität im Zahnmedizinstudium an der Medizinischen Universität Graz entwickelt wurde.

Die Durchführung des Projektes erfolgte in den Jahren 2016 und 2017 an der Univ.-Klinik für Zahnmedizin und Mundgesundheit Graz, wobei sämtliche Studierende der Semester 7-12 (Studienjahr 4-6) zur Teilnahme verpflichtet wurden.

Das Projekt wurde von einer erfahrenen wissenschaftlichen Mitarbeiterin der Abteilung für Orale Chirurgie und Kieferorthopädie entwickelt, welche 10 Jahre Berufserfahrung als Zahnärztin vorweist und spezielle Erfahrung im Bereich der Multiple-Choice-Fragenerstellung hat.

Die statistische Auswertung erfolgte nach Abfolge von 3 Tests unter Anwendung deskriptiver und explorativer Verfahren und der Item Response Theorie (Rasch Analyse).

Insgesamt nahmen 173 Zahnmedizinstudierende (Männer:Frauen/ 1:0.7) an dem Projekt teil. 6 Fragen wurden im Post-Review ausgeschlossen. Das Antwortverhalten resultierte in ähnlichen Ergebnissen zu allen 3 Testzeitpunkten für die Kategorien "Richtig", "Falsch" und "Weiß nicht", welche auch mit Daten aus der Literatur vergleichbar sind.

Der entwickelte DPT lässt sich als erstes standardisiertes, longitudinales Prüfungsverfahren in der zahnmedizinisch-oralchirurgischen Lehre an der Meduni Graz präsentieren.

Kurzzusammenfassung (engl.)

This study reflects a prospective investigation, aimed at the development and implementation of a Dental Progress test (DPT). The tool of DPT implements a special, longitudinal form of assessment method, developed for the clinical oral surgery education at the Dental school of Medical University of Graz, to evaluate the teaching and learning quality.

This study was performed within the years 2016 and 2017 at the Dental School of the Medical University of Graz. Therefore, all 7th to 12th term, respectively 4th- to 6th-year, students were recruited for participation in this prospective project.

The project was developed by a senior staff member at the Division of Oral Surgery and Orthodontics who has 10 years of experience in dental education and is specially trained in the formulation of multiple-choice questions.

Data analysis was done after the third test, using descriptive and explorative analyses as well as IRT analysis (Rasch analyses).

Overall 173 students, (male: female/ 1:0.7), participated in all three tests. In the post-review process a total of 6 items was excluded. Item responses resulted in similar levels at all three test time points for the categories “correct”, “false” and “don’t know”, also comparable to those reported in the literature.

Our DPT resulted in the first standardized presentation of longitudinal student knowledge in undergraduate oral surgery education at the Medical University of Graz.

 

Nähere Beschreibung

Introduction

Based on the Bologna process, which was initiated 20 year ago, higher education institutions throughout Europe have changed their study programs significantly, targeting on harmonization and comparability. The study programme of “Dental medicine” was introduced in Austria in 1998 and has, since then, also been offered in Graz. In the preclinical years scientific basics are taught. In the clinical part an all-encompassing practical education, by means of a vocational training, preparing students to become safe and independent dental practitioners, stands in the foreground. Students engage with the main dental disciplines including restorative dentistry, oral surgery, periodontology, prosthodontics and orthodontics. Despite the subjective impressions of offering a high qualitative dental education, student outcomes do not always meet the expectations of the educators on the daily basis. A prospective study was undertaken with the undergraduate dental students as a form of quality control. The intervention facilitated the evaluation of knowledge levels and differences

Study design

This study reflects a prospective investigation, aimed at the development and implementation of a Dental Progress test (DPT). The tool of DPT implements a special form of assessment method, developed for the clinical oral surgery education at the Dental school of Medical University of Graz, to evaluate the teaching and learning quality.

Duration and setting

This study was performed within the years 2016 and 2017 at the Dental School of the Medical University of Graz. Therefore, all 7th to 12th term, respectively 4th- to 6th-year, students were recruited for participation in this prospective project. The attendance was planned to be compulsory and accordingly the approval was given by the Local Advisory Committee on Dental Study Affairs. Participants consisted of male and female trainees, mainly from Austria. Depending on the admission to studies, students from Germany and Southeastern European countries were also included.

Ethics approval

According to the Ethics Committee of Medical University of Graz, no concerns about this study were identified.

DPT development

The project was developed by a senior staff member at the Division of Oral Surgery and Orthodontics who has 10 years of experience in dental education and is specially trained in the formulation of multiple-choice questions. First a pool of single-best and true/false (K-type) MC items, 375 in number, was de- signed within a period of one year. This was mapped against the local document of learning content and to current teaching literature. Storage was password protected at the local available IMS2 (Item Management System - Umbrella Consortium for Assessment Networks, Heidelberg, Germany). Every single item contained an explanatory introduction (also known as case vignette) and the question text itself. The author decided on five to six possible answers with only one correct key answer. As a special feature of a progress test, a “don’t know” option was included. In context on the field, the author chose either three or four distractors. Negative as well as double negative formulations were avoided. The difficulty level of all items was at final exam level targeting the four fields of “oral surgery”, “oral medicine”, “oral radiology” and “cases.” Depending on the local catalogue of learning objectives and the subjects pursued during the local clinical dental training, fields also included correlating subcategories. Clinical images and radiographs were also included in 95 items for a better connection to the daily dental practice and to reach a higher discrimination.

MCQ review

A multistage review process was performed question by question for all items. First a factual review was done conducted internally, with a group of four senior academics in-house. A second independent review followed. Finally, senior academics at Dental School of Medical University of Vienna did a third review round. An additional formal review by the local examination department, after which final question pool was completed, was also carried out.

Test schedule and content details

Three progress tests were administered within three terms from the summer term 2016, over the winter term 2016/2017 to the summer 2017. 100 items were randomly chosen for each test. The selection was made on the basis of a predesigned blueprint, referring to the four categories “oral surgery”, “oral medicine”, “oral radiology”, and “cases”. Thereby a specific number of items per test was chosen from each category, described as follows. 30 items came from “oral surgery” targeting on issues such as diagnostics, treatment indica- tions, surgery techniques, instruments, complication a d risk management as well as implant surgery. A further 30 were chosen from “oral radiology” including radiography techniques, radiation protection and image interpretation. The next 20 came from “cases” reflecting clini- cal vignettes as encountered in clinical practice, and another 20 items were selected from “oral medicine”, which also included five “local anaesthesia” and five “acute pain management” items. A repetition of any item was prevented. All tests were computer-based with an overall limited time of three hours. A primary a score of +1 was given for correct answers and -1 for incorrect answers. “Don’t know” options were scored with 0. Students participation was mandatory, but based on the formative design, DPT performance did not influence any pass/fail decision. Feedback was given on student’s scores concerning number of correct, incorrect and “don’t know” responses. Their rank in class and their rank in the total cohort was also communicated separate to each student. The best performers were rewarded, to enhance student motivation.

Post-test review

A post review was completed after each test. Exclusion criteria were defined and applied prior the final analyses of the results. For instance, technical issues during test administration such as problems with the image visualization or items with errors in the assignment of the correct answering options, were considered as a justification for an item exclusion.

Statistical analysis

Data analysis was done anonymously and blinded after the third test, using descriptive and explorative analyses as well as IRT analysis. First student’s response behavior was comput- ed as mean, median, standard deviation, minimum and maximum values for each test sepa- rated in terms and years. Furthermore, the range of results was also described. Due to the changing cohorts and items per test, the response behavior was not calculated for all three tests together. Further test data were calculated as median and inter-quartile range (IQR) or absolute and relative numbers. After the extensive descriptive and exploratory analyses, Rasch analysis (IRT) was used to evaluate the test and to identify misfitting items. After that, in preparation for Rasch analysis, the response categories “don’t know” (0) and “false” (-1) were changed and summarized into only “false”. In the case of only correct or only false answers questions were to be excluded from investigation. Item parameters and person parameters were estimated using response patterns and were expressed on a common log- odds scale. Important requirements for Rasch analysis are uni-dimensionality of data, local independence and sample independency. If all items tap into only one dimension, uni-dimensionality is given. Only in this case the interpretation of an overall score is meaningful. Local dependency, which can inflate reliability, refers to the fact, that another dimension causes dependency among responses. Therefore, responses to the items are not only influenced by the analyzed dimension but also by another. The tests include four the- matic fields. Therefore, local dependency may be introduced by these thematic fields. An advantage of item response theory to classical test theory is, that estimated parameters are not dependent on the sample used for estimation. This so-called sample independency has to be analysed. Further, infit and outfit measures (mean square statistics) and the Wald test were also applied. The reason for this was to identify items that did not match within a uni- dimensional model or had varying item parameters in subsamples of respondents (sample independence). To analyse local independency a second analysis was made grouping the items of the four thematic fields into four polytomous item. To evaluate the assumption of parallel item characteristic curves (ICC), Andersen’s likelihood-ratio tests for goodness-of-fit with mean split criterion were calculated. Furthermore, the preparation of person-item maps followed: a person-item maps shows the ability, distribution of the examinees and the distribution of difficulty parameters of the items. Therefore, the fit of the difficul- ties to the tested sample can be evaluated. The person separation reliability was calculated for the presentation of the internal consistency. For the datasets of four polytomous items the reliability was also calculated. The analyses were done test per test with the appliance of the R-package eRm (Version 0.15-7) and mirt (Version 1.27.1).

Results

Overall 173 students, (male: female/ 1:0.7), participated in all three tests. In the post-review process a total of 6 items was excluded. Item responses resulted in similar levels at all three test time points for the categories “correct”, “false” and “don’t know” (Test 1: 61.6%, 26.6%, 11.8%; Test 2: 56.0%, 27.8%, 16.3%; Test 3: 62.1%, 26.5%, 11.4%). These results were comparable to those reported in the literature. A significant increase in “correct” answers from 4th to 5th year with p <.001 and from 5th to 6th year with p =.002 was observed, however only in Test 2. A significant decrease of “don’t know” answers was seen in Test 1 and 2 from year 4 to 5 with p =.003 and p <.001 and from year 5 to 6 with p <.001 at both. Concerning “false” answers, even a significant increase occurred, however only in Test 1 from year 4 to 5 with p = .009 and from year 5 to 6 with p = .022. The reliability ranged from 0.82- 0.88 at all three tests. Within the Rasch analyses the assumption of parallel ICC was met and item difficulties for the thematic fields were similarly distributed across the latent dimensions.

Conclusion

Targeting the aims of the presented study, our DPT resulted in the first standardized presentation of student knowledge in undergraduate oral surgery education at the Medical University of Graz. Furthermore, it is the first performed DPT in the German-language counties of Germany, Austria and Switzerland in this form.

Currently, knowledge levels and variation of performance during the clinical educational programme could be documented. Although sample size and the duration of the study were limited, essential information was acquired concerning both learning and teaching quality in undergraduate oral surgery education.

Further, a homogeneous response behaviour, a uniform distribution of included and ex- cluded items within the separate tests and fields, and an accurate range of difficulty of the questions could be drawn from the collected data. This means sufficient and sat- isfactory quality of the newly created item bank, which will allow usage in the future.

The DPT provided and facilitated an innovative a comprehensive survey of students, which mandates new requirements for the educational programme at our dental school. As a result, curricular and educational changes have already been launched in the third study section.

Fulfilling the thought of Bologna process, an extension at least national and as a next step to the German-speaking countries or further should be focused. As a result of this, as with the established PT for medical students, dental student cohorts could also be compared between universities. This in turn could increase comparability and competition and force institutions to focus more on high quality in education.

Accordingly, this could give rise to a positive effect in terms of harmonisation of dental education throughout Europe.

National and international collaboration to extend the item pool and sharing resources would definitely be potential options for future developments in the continued use of DPT.

Positionierung des Lehrangebots

3. Studienabschnitt Diplomstudium Zahnmedizin an der Medizinischen Universität Graz, Studienjahre 4-6, Semester 7-12.

Das Beispiel wurde für den Ars Docendi Staatspreis für exzellente Lehre 2019 nominiert.