Рhysical therapу graduate students' and examiners' perception of objective structured clinical examination: a feedback for process improvement perception of objective clinical a

Purpose: to analyze the first experience of administering Objective Structured Clinical Examination (OSCE) for the students (studs) enrolled in the master's degree program majoring in physical therapy at National University of Ukraine on Physical Education and Sport. Material and methods. The survey included 21 examiners (examrs) and 46 studs involved in OSCE administering. The questionnaire consisted of the questions assessing the quality of station equipment, clarity of the exam format, other aspects. Results. All the examrs informed of total clarity of the examination format, but only 13 (61.9%) of them noted a similar level of clarity for the studs. Т he examrs identified 3 main reasons for unsuccessful attempts to perform a skill/task: insufficient training, new format of checking knowledge/examination, slightly different focus of training. The distribution of answers concerning the clarity of the format of the examination and its tasks provided by the studs did not differ statistically from those provided by the teachers. The vast maj ority of the examrs indicated that the skills were of sufficient difficulty for the purpose of the examination. 5 examrs had significant difficulties assessing skill performance according to the provided form, while 7 of them had slight difficulties and the rest had no difficulties. The majority of the studs noted that the number of stations and equipment was sufficient (no statistical differences were determined between the answers of the studs and the examrs). Conclusions. The survey revealed the need for introducing certain changes in OSCE assessment form, in the educational program, conducting special preparatory classes for studs. The obtained results can be used to improve the development of OSCE in physical therapy in Ukraine.


Introduction
Ukraine is currently implementing a healthcare reform, which includes developing a strategy to change the medical education system. The latter involves, among other things, adjusting of educational and professional training programs for physical therapists to European practice. Ukraine has been providing educational programs in physical therapy since 2017. However, they do not meet the requirements of the European Region of the World Confederation for Physical Therapy [1]. Improving training quality is a very important issue since physical therapy involves a wide range of patients with various disorders of the nervous [2,3], musculoskeletal [4,5] and cardiovascular systems [6][7][8][9], as well as congenital pathologies [10].
One of the fitting criteria is a mandatory certification of the students enrolled in the master's academic program and majoring in "22 Health care" branch of knowledge, carried out in the form of a unified state qualification exam (USQU OSCE was hypothesized by Harden in the 1960s, first reported in the British Medical Journal [13] and introduced in 1975 [14]. Thereafter, it has been used as a valuable tool to assess students' clinical skills in medical, dentistry, nursing and pharmacology schools around the world [13]. Being the model for assessing results in a number of medical professions OSCE is a type of examination that provides students with a series of scenarios or stages, each including a standardized task which should be performed within a specific timeframe, often involving interaction with a standardized patient or other types of models [15].
Students rotate round the stations where they perform certain practical tasks, being observed and scored by the examiners according to the checklist. The examination is more objective, and grading strategy is determined in advance. The examination results in improved feedback to students and staff [16].
The main concerns that this exam should address are validity for clinical skills, not just for knowledge; high reliability; fairness to students; compliance with learning objectives; possibility to create in medical universities using existing or available technologies [15].
The aim of the examination is to conduct an observational pseudo-quantitative assessment of students' clinical skills and competencies in a standardized, simulated environment before they proceed to a clinical internship. The standardized nature of OSCE is aimed to provide a possibility to determine whether a trainee has reached the threshold level of competence that will ideally ensure safe and effective practice in real-life environment. However, students describe OSCE as one of the aspects of their physical therapy training that is of most anxiety [17].
In September 2019, the National University of Ukraine on Physical Education and Sport (NUUPES) started implementation of a pilot master's degree program in specialization 227.1 "Physical Therapy" for the second (master's) level of higher education within the project "Innovative Rehabilitation Education -Introduction of new master's degree programs in Ukraine" (REHAB) No. 598938-EPP-1-2018-1-LV-EPPKA2-CBHE-JP of the Erasmus+ program, funded by the European Union. During May 20-25, 2021, its graduates (77 students) passed the pre-pilot version of OSCE. Six testing stations were equipped (physical therapy for the diseases of the nervous system, physical therapy for the diseases of the musculoskeletal system, assessment methods in neurology and orthopedics). The duration of one station was 6 minutes; transition time was 30 seconds. Six students were evaluated within one station. Twenty-one examiners and 18 standardized patients were trained and involved in the evaluation process. Task performance was assessed according to the developed checklist and 24 competencies, grouped into the following categories: communication skills -4, ethic skills -4, demonstration of knowledge and skills -10, ability to interpret collected data correctly -6. Each of the competencies was graded according to the developed checklist. Maximum score for demonstrating one skill was 20 points.
Рurpose: to analyze the first experience of administering OSCE for the students enrolled in the master's degree program majoring in physical therapy at NUUPES. 66 Health, sport, rehabilitation Здоров'я, спорт, реабілітація Здоровье, спорт, реабилитация

Participants
The survey included 21 examiners (six examiners were PhD and one was a Doctor of Science; two examiners were Associate Professors and one was a Professor) and 46 students involved in OSCE administering. Participants of the research signed an informed consent form. The research was approved by the University Ethics Committee (№ 2/2020).

Procedure
The survey was conducted with the help of online service "Google Forms". The questionnaire consisted of the questions assessing the quality of station equipment, clarity of the exam format, compliance of the behavior of standardized patients with the tasks, as well as other aspects of the exam administration.

Statistical Analysis
The obtained results were processed by means of mathematical statistics using IBM SPSS Statistics 21 application. Since the results did not correspond to the law of normal distribution, calculations included measuring median value (Me) and upper and lower quartiles (25%; 75%), as well as mean value and standard deviation (M±SD). Taking into account its specifics, the survey included frequency analysis and determining proportions of the patients according to their answers to the questions. Chi-square test (χ²; nominal and ordinal variables) and Mann-Whitney U test (quantitative indicators) were used to compare the groups' answers to similar questions.

Results
Consider the results of surveying the examiners. Their academic experience varied from one to 22 years. M±SD values comprised 7±6.39 years, Me (25%; 75%) indicators were 4 (2; 11) years, which indicates that both young and experienced specialists were involved in the study. It should be noted that apart from academic activities all the young specialists dealt with practical activities of physical therapy with patients of different nosological groups. Examination administration received 8.85±1.33 points on a ten-point scale from the examiners, with Me (25%; 75%) indicators being 9 (8; 10) points.
The vast majority of the examiners noted that the video camera did not distract them (90.5%) and the students (85.7%). Other examiners chose the answer "partially". The largest share of the examiners (71.4%) noted that the room where the examination was conducted met the necessary requirements. Four respondents informed of partial compliance with the requirements, whereas two of them denied any compliance. All the examiners informed of total clarity of the examination format, but only 13 (61.9%) of them noted a similar level of clarity for the students. Eight examiners (38.1%) noted partial clarity of the format for the students.
Nineteen examiners (90.5%) reported that the format of the tasks was totally clear to them, while the others noted that it was partially clear. Eleven respondents (52.4%) thought that the format of the tasks was completely clear to the students, nine examiners (42.9%) noted that it was partially clear and one of them stated that the format was not clear.
The same shares of the examiners rated the attitude of the students to the examination as good/with interest and uncertain (47.6%), and only one respondent chose a negative attitude. The vast majority of the examiners (71.4%) reported that they were not distracted by extraneous noise or interference, while others indicated that this was rare. A similar question, but concerning the students, received similar answers -71.4% and 28.6% of the respondents, respectively. The answer options "often" and "constantly" were not chosen by any of the respondents.
The average score of tasks/skills performance by all the students was evaluated by the examiners at the level of 60.71±21.05 points on 100point scale; Me (25%; 75%) indicators comprised 65 (40; 75) points. At the same time, the examiners identified three main reasons for unsuccessful attempts to perform a skill/task (Fig. 1). 67 Health, sport, rehabilitation Здоров'я, спорт, реабілітація Здоровье, спорт, реабилитация 8(2) Fig. 1. Main reasons for unsuccessful attempts to perform a skill/task by the students in the opinion of the examiners Almost a quarter (23.8%) of the respondents noted that the number of stations (examination tasks) should be increased, while the others considered it sufficient. All the examiners informed of the relevance of the skills that were selected for the examination.
Eleven respondents noted that the equipment of the stations was sufficient to perform the skill, while the other ten considered it necessary to increase its number. A number of questions concerned the patient model. Sixteen (76.2%) examiners considered it totally possible for a student to perform the skill on a standardized patient, while the others chose the answer "partially".
The examiners noted that the standardized patient should be given better and more specific instructions on complaints and medical history (including their absence) -90.5% of responses, the need to indicate in the medical history counterindications to perform the skill -66.7%, testing conditions (inpatient, outpatient admission) -57.1%, following only student's instructions (because the standardized patient could perform the skill correctly, although the student provided vague explanations) -76.2%.
Ten examiners indicated that the assigned time was sufficient to perform all the skills. A similar proportion of the respondents answered "enough time for most of the skills", only one respondent chose "enough time for half of the skills".
The vast majority of the respondents (81%) indicated that the skills were of sufficient difficulty for the purpose of the examination. The answers "not difficult enough" and "too easy" considering the purpose of the examination were chosen with the same frequency (9.5%).
Almost all the examiners noted that they were not biased against the students during the examination (90.5%). Only two teachers chose the answer "biased to a certain extent".
Five examiners had significant difficulties assessing skill performance according to the provided form, while seven of them had slight difficulties and the rest had no difficulties.
In addition, the examiners were inquired about the need for additional instructions on assessing skill performance according to the provided form in a specific situation. Thirteen respondents (61.9%) confirmed the need for additional instructions in a situation where the student performed a skill not specified in the task, but did it properly and in accordance with the assessment purpose (e.g. assessment of strength, stamina, range of motion, pain); 38.1% of the respondentsin a situation where the student performed a skill not specified in the task and not in accordance with the assessment purpose, but did it properly; 47.6%when the student performed a specified skill, but partially (only on one side or limb, during insufficient amount of time); 33.3%when the student explained how to perform the skill, but was distracted by talking to the patient and did not perform it.
Besides, the examiners were inquired about the need to change the form of skill assessment. The majority of the respondents (57.1%) confirmed that the first assessment unit (communication skills) should be left unchanged. Other respondents chose the answers "to reduce the number of grades from 4 to 2" (14.3%), "to reduce the number of grades from 4 to 1" (14.3%), "to reduce the number of grades and change the form" (14.3%). None of the respondents chose the options "to increase the number of grades" and "to delete the unit". Openended questions in the first assessment unit included recommendations and comments concerning the fact that the form should include only those communication skills that are necessary Health, sport, rehabilitation Здоров'я, спорт, реабілітація Здоровье, спорт, реабилитация 8(2) to perform the skill; these skills should be taught and assessment points should be explained to the students; the purpose of the examination is to assess students' practical skills, therefore this unit is not a key one.
The majority of the respondents (57.1%) confirmed that the second assessment unit (ethic skills) should be left unchanged. Other respondents chose the answers "to reduce the number of grades from 4 to 2" (19%), "to reduce the number of grades and change the form" (14.3%), "to delete the unit" (4.8%), "task format does not always enable to assess ethic skills" (4.8%). None of the respondents chose the options "to reduce the number of grades from 4 to 1", "to increase the number of grades" and "to combine the unit with the first one".
Open-ended questions in the second assessment unit included recommendations and comments concerning the fact that the form should include only those ethic skills that are necessary to perform the skill; ethic skills should be taught and assessment points should be explained to the students; this unit should be assessed directly in a medical institution, rather than automatically completed at the examination.
The majority of the respondents (61.9%) confirmed that the third assessment unit (demonstration of knowledge and skills) should be left unchanged. Other respondents chose the answers "to reduce the number of grades" (9.5%), "to reduce the number of grades and change the form" (14.3%), "to increase the number of grades" (9.5%), "to change the content of grades to make them more objective" (4.8%).
Concerning the fourth unit (the ability to correctly interpret the collected data), the same shares of the respondents (33.3%) chose the answers "to leave the unit unchanged" and "to reduce the number of grades and change the form". Other respondents chose the answers "to reduce the number of grades" (19%), "to increase the number of grades" (4.8%), "to delete the unit" (9.5%). Sixteen examiners noted they did not interfere with the skill performance.
Consider the results of surveying the students. The group involved 54.3% of males. The age of the students ranged from 21 to 43 years. M±SD indicators comprised 26.8±6.25 years, and Me (25%; 75%) indicators were 24 (23; 27.5) years. The distribution of answers concerning the clarity of the format (χ²=2.243, p=0.326) of the examination and its tasks (χ²=1.411, p=0.494) provided by the students did not differ statistically from those provided by the teachers (Fig. 2). Health, sport, rehabilitation Здоров'я, спорт, реабілітація Здоровье, спорт, реабилитация 8(2) stations should be increased (15.2%) and decreased (15.2%). No statistical differences were determined between the answers of the students and the examiners (Table 1). The largest share of the students (58.7%) noted that the equipment of the stations was sufficient to perform the skill. The need to increase the amount of the equipment was chosen by 39.1% of the students, while the rest of them considered it necessary to reduce its amount. No statistical differences were determined between the answers of the students and the examiners ( Table 2). Sixteen students (34.8%) considered it possible to perform the skill on a standardized patient. The answer "partially" was chosen by 45.7% of the students, "impossible"by 4.3%, while the other 15.2% confirmed that the patient distracted/prevented them from demonstrating the skill.
Almost half (47.8%) of the students noted that the assigned time was enough to perform all the skills; 21.7% of them chose the answer "enough to perform most of the skills"; 17.4% -"enough to perform half of the skills"; 6.5% -"not enough to perform most of the skills"; 6.5% -"not enough to perform all the skills". No statistical differences were determined between the answers of the students and the examiners (Table 3). Concerning the performed skills, 19.6% of the students informed of the absence of skills that were not taught within their bachelor's and master's degree programs. 37% of the students chose the answer "a small number of the skills"; 19.6% -"a half of the skills"; 19.6% -"majority of the skills"; 4.3% -"all the skills were not taught". 70 Health, sport, rehabilitation Здоров'я, спорт, реабілітація Здоровье, спорт, реабилитация 8(2) The vast majority of the students (71.7%) noted that the skills were difficult enough for the purpose of the exam. The answer "not difficult enough" was chosen by 21.7% of the respondents, and "too easy"only by one student. Two students (4.3%) considered the skills too difficult to perform. No statistical differences were determined between the answers of the students and the examiners (Table  4). The majority of the students noted that they were not biased during the exam (54.3%). 39.1% of the students were biased to some extent, while the others were strongly biased.

Discussion
The obtained results can be used to improve the development of a standardized OSCE in physical therapy in Ukraine and to elevate the integrity of future surveys of students and teachers. The study was limited by the amount of samples that can be increased in further studies to improve generalizability of the results. It was previously reported that feedbacks from students and teachers are invaluable and contribute to a critical review and modification of the content of OSCE stations and carrying out expert examination [14]. According to the surveys, OSCE is positively perceived by most students [18][19][20]. One of the previous studies showed that OSCE is perceived as a fair assessment tool by both students and teachers [13].
The results of the survey confirmed the need to improve the format of assessing students' skill performance. The difference between assessment results of the examiners and the students, as well as the level of understanding of the format of the examination and the tasks emphasize the need for administering special training courses or classes for students, implementing changes in the educational curricula. On the other hand, this difference can be explained by the results of the previous studies where students reported that OSCE is more difficult 1 [14,21] and more honest [14] than the usual teacher assessment (Clerkship ratings).
In terms of simplifying the assessment, we should note its specifics in one of the studies: the results of students' performance were assessed according to the criteria for each station (the validity and content of each checklist were established by a group of senior pediatricians through conducting a review and reaching a consensus); criteria-based scoring was used, with each checklist item being rated as 0 (omitted, incorrect or inadequate) or 1-2 (correct or adequate) [14].
The presented results confirm that the majority of the examiners and the students believed that there was enough time to perform all or most of the skills. Only a few students noted that there was not enough time to perform all or most of the skills. One of the previous studies [14] stated that faculty members considered that students' concerns about the distribution of time at the station and the degree of stress were caused by insufficient preparation for the exam, especially in competencies that were not previously assessed at the "traditional" examination.
At the same time, the obtained results actualize the issues of task design, clarity of instructions, and the problem of individual variability among examiners. In order to make OSCE reliable, it is necessary to conduct a thorough review of the content and design of the tests, training of examiners and implementation conditions. A survey conducted in Nigeria confirmed that 40.4% of the students easily understood written instructions at OSCE stations; 70.2% of the students believed that the time allocated to each station was sufficient. Besides, 56.3% informed that OSCE enhanced their communication skills; 53% informed of nervousness caused by OSCE; and 48.3% expressed concerns about the change of examiners at the stations. At the same time, only 37.1% of the students believed that OSCE should be used more often than other forms of assessment [19].
A survey of the teachers who conducted OSCE for the students of the Department of Internal Medicine revealed that 80% of the students perceived OSCE as a better assessment tool than traditional long/short case exams. A large number of stations and areas of tasks (nosologies) allowed to identify weaknesses and explain them by the peculiarities of training and the prevalence of pathology. It should be noted that communication skills were assessed at one of 21 stations (10 history-taking stations, 6 physical examination stations and 5 data interpretation stations) [13]. Taking into account the obtained results and teachers' comments, the organization of a specific station for the assessment of communication skills can result in improving the organization of the examination on the whole.
One of the studies grouped medical competencies into six categories: medical knowledge, patient care, professionalism, interpersonal and communication skills, systembased practice, practice-based learning and improvement [22]. OSCE is considered an important tool for assessing patient care, interpersonal and communication skills, and professionalism. It is also a reliable method of assessing practice-based learning and improvement, system-based practice, but not medical knowledge [22,23]. These data can also be used to improve examination structure, namely the number of stations and their content.

Conclusions
Students' feedbacks on the use of tools to assess their knowledge and skills are of great importance, being thus an effective method to improve education system. The survey revealed the need for introducing certain changes in OSCE assessment form, conducting special preparatory classes for students, providing changes in the educational program. The obtained results can be used to improve the development of OSCE in physical therapy in Ukraine, to enhance master's degree program in specialization 227.1 "Physical Therapy" at NUUPES and advance its acceptance by the European Region of the World Confederation for Physical Therapy.
The study is funded with the support of the European Commission. The article reflects the views of its authors only, and the Commission cannot be held responsible for any use of the information contained therein.