Background: There is a pressing need to evaluate the performance of surgical resident in the operation theater objectively by using a structured assessment tool. This will eliminate the possibility of bias which might be a concern when the assessment is subjective. An Objective Structured Assessment of Technical Skills (OSTAT) based tool was tested using Global Rating of Thyroid Surgical Skills (GRTSS) in terms of feasibility, reliability and construct validly in evaluating otolaryngology head and neck surgery residents at various postgraduate years.
Objectives: A prospective observational study using a global rating evaluation form developed in conjunction with the OSATS to assess the thyroid surgical skills of residents. The study was conducted in King Abdul Aziz University hospital, Riyadh, KSA. A total of 23 thyroid surgeries performed by 7 residents at three different levels were assessed. The participants’ scores were calculated as percentages and were entered using SPSS (Statistical Package for the Social Sciences, version 17.0) for reliability and construct validity measures.
Methods: Construct validity was established by comparing senior and junior years (P<0.001, mean difference: 11.375, 95% confidence interval 6.25-16.49) and by comparing same residents in two different levels (P<0.005, mean difference: 18.8, 95% confidence interval 7.45-30.14). Proved reliability with a value of Cronbach alpha of 0.707.
Conclusions: This pilot study has shown this tool to be a valid, feasible, and reliable instrument to assess the thyroidectomy surgical skills. Potential applications will be useful in tracking resident development throughout postgraduate training and certifying operative skills.<
Keywords: Pilot study; Assessment tool; Thyroid surgical skills (GRTSS); Thyroid surgery; Resident
Objective assessment of surgical skills of trainees during a residency program is crucial in the education and development of the surgical skills of trainees. Methods of assessing the performance of the residents in the operating theatre is vital in order to achieve competency in technical skills and acquisition of surgical expertise ultimately ensuring best optimal patient safety . Efforts have been made to design assessment tools aiming to evaluate surgical skills in the operating theater that meet the standards of feasibility, reliability and validity. The Objective Structured Assessment of Technical Skills (OSATS) was developed by Winckel et al.  to conduct a formal assessment of surgical resident’s technical skill, which was later modified by Martin et al. .
Numerous researches in the surgical literature has been able to develop OSATS-based assessment tools with proven feasibility, reliability and validity in different surgical disciplines [3-5]. In otolaryngology head and neck surgery, such objective tools have been successfully examined as being reliable and valid in different procedures such as mastoidectomy, sinus endoscopic surgery and laryngoscopy [6-8].
Thyroid surgery can be considered as an ideal representation of head and neck procedures in which surgical residents need to have integrated knowledge of the challenging anatomy and requires development of meticulous surgical technical skills, hence the demand for a structured observational assessment is undeniable . Intending to address this issue, an OSATS-based instrument using Global Rating of Thyroid Surgical Skills (GRTSS) has been designed in this pilot study to assess the resident’s expertise in technique for thyroid surgery. The purpose of this pilot study was to test this tool’s feasibility, reliability and construct validly in evaluating otolaryngology head and neck surgery residents at various postgraduate levels.
After obtaining an approval from the ethical committee to carry out this prospective study, consent was obtained from residents at various levels of training and enrolled. The staff assessors were informed about the study and agreement was reached about their participation.
The design was a prospective, observational study within the operating theatre of a teaching hospital. Trainees were directly observed performing thyroidectomy.
A 2-page evaluation form was developed in conjunction with the OSATS evaluation form that was originally developed by Winckle et al and modified by Martin et al. [2,3] (Table 1) to assess the surgical skills of residents. A one to five (five = excellent) scale was used for evaluation.
|Criterion||Grade 1||Grade 2||Grade 3||Grade 4||Grade 5|
|Respect for tissue||Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments||Careful handling of tissue but occasionally caused inadvertent damage||Consistently handled tissues appropriately with minimal damage|
|Time & Motion||Many unnecessary moves||Efficient time/motion but some unnecessary moves||Economy of movement and maximum efficiency|
|Instrument handling||Repeatedly makes tentative or awkward moves with instruments||Competent use of instruments although occasionally appeared stiff or awkward||Fluid moves with instruments and no awkward movements|
|Knowledge of instruments||Frequently asked for the wrong instrument or used an inappropriate instrument||Knew the names of most instruments and used appropriate instrument for the task||Obviously familiar with the instruments required and their names|
|Flow of operation and forward planning||Frequently stopped operating or needed to discuss the next move||Demonstrated ability for forward planning with steady progression of operative procedure||Obviously planned course of operation with effortless flow from one move to the next|
|Knowledge of specific procedure||Deficient knowledge.Needed specific instructions at most operative steps||Knew all important aspects of the operation||Demonstrated familiarity with all aspects of the operation|
|Use of assistants||Consistently placed assistants poorly or failed to use assistants||Good use of assistants most of the time||Strategically used assistants to the best advantage at all times|
Table 1: The global rating score (Martin et al.).
The (GRTSS) evaluation instrument (Table 2) was designed with input from academic otolaryngologist, fellowship trained head and neck surgeons and experts in medical education. Their comments and suggestions were incorporated, establishing face and content validity. Finally 16 items of task performance were selected with a five-point Likert scale linked to specific descriptors. The number five in the scale was equivalent to the performance of a resident who can perform that step independently and safely.
|NO||Surgical Step||Not done
|1||General knowledge of the patient:
Identified Patient, checked indication for surgery, checked consent.
|3||Proper communication with anesthesia ,ETT, muscle relaxants, local xylocaine injection|
|4||Identified landmarks pre-op|
|5||Checked cautery parameters and adjusts accordingly|
|6||Maintained a dry field safely|
|7||Creation of flaps inSubplatysmal plane properly|
|8||Dividing Strap Muscles in midline|
|9||Delivery of Thyroid lobe|
|10||Use of Laryngeal rotation maneuver|
|11||Control ofThyroid Vessels|
|12||Identification ofRLN landmarks and dissection|
|13||Identification & preservation of Parathyroid gland(s)|
|14||Subcutaneous & Skin closure|
|15||Proper marking of surgical specimen|
|16||Outlined Post-op orders clearly and appropriately|
Table 2: Global Rating for Thyroid Surgical Skills (GRTSS). Please mark the candidate’s performance in the above scale. Scale 5 is equivalent to a resident that can perform the step independently and safely.
The participants’ scores were calculated as percentages and were entered using SPSS (Statistical Package for the Social Sciences, version 17.0) for reliability and construct validity measures.
A total of 23 thyroid surgeries were performed by 7 residents at three different levels of postgraduate years; R3, R4 and R5. They were assessed by a single assessor using two different assessment tools the Global Rating Score (GRS) and Global Rating of Thyroid Surgical Skills (GRTSS) displayed in Tables 1 and 2 respectively.
The assessors reported that completing the assessment forms were convenient and not time consuming which is a major issue in an operation theater setting. The tools were also described as being simple, brief and to the point by the assessors. Based on these observations the instrument can be considered as feasible.
The interterm reliability was determined by the assessment tool’s internal consistency. It was evaluated with Cornbach’s α, which proved the tool’s high reliability, 0.707.
It has proven to be construct-valid as senior trainees achieved higher scores than junior trainees. This was measured comparing means using independent t sample test (P<0.001, mean difference: 11.375, 95% confidence interval 6.25-16.49). The GRS also showed statistical significance (P<0.005 mean difference: 6.958, 95% confidence interval 2.787-11.129).
Furthermore, to provide evidence of construct validity the a set of residents performing the surgeries were assessed by the same assessor in two separate training years, first in year 4 and next in year 5. The scores was compared between the trainees attending senior and junior years. Notable improvement was observed in the scores of the same trainees in senior year (Figure 1), proving the construct validity of the instrument. This was measured comparing means using independent t sample test (P<0.005, mean difference: 18.8, 95% confidence interval 7.45-30.14). However, the in this set, GRS did not show statistical significance (P<1.80 mean difference: 12.2 95% confidence interval 6.96-31.36).
The emphasis on work-place assessment methods has been growing as measuring a trainee’s knowledge, skills, judgment or professional behavior is of determinative values in the educational or training programs . Standardized methods of assessing the surgical skills of trainees in the operating theatre require valid and reliable instruments to ensure that graduates’ performance is in compliance with the accepted professional demands . To address the growing need for a structured assessment tool to properly evaluate the operative skills of otolaryngology trainees/residents performing thyroid surgery this assessment tool was developed, which proved to be valid and reliable as well as acceptable to assessors. The feasibility and cost-effectiveness of the assessment tool developed in this study were present, as these are essential components for an assessment method to be convenient and for proper implementation of it in a training program.
Considering the reliability, the internal consistency was found to be significant when compared to the known global rating scale that was used as a second tool in this study.
Since measuring the progress of trainees over time is crucial in certifying competence, this instrument was able to demonstrate achievement of higher score with advancement in training years. It was also able to capture progress within the same participant after gaining more experience and knowledge with subsequent evaluations.
Limitations to this study include the small sample size, however, increase in the number of participants would be considered to confirm findings in future studies as this is a preliminary pilot one. Another limitation which too will be addressed in the subsequent larger scale study is that inter-rater agreement was not possible to be calculated since every participant was assessed by a single assessor.
The otolaryngology residents were directly observed performing thyroid surgeries on patients in presence of the assessors who were consultants, in contrast to other studies that were done at other different settings using bench models or live anaesthetized animals .
This tool’s components are broadly similar to those developed by Diaz Voss Varela et al. , however, this instrument has longer and separate items that would play an important role in accurate evaluation of operative skills of the residents. This study’s results were consistent with those developed by Diaz Voss Varela et al. in terms of feasibility, reliability and construct validity .
The experience of otolaryngology head and neck surgeons in developing assessment tools for evaluating technical skill and knowledge of trainees performing various procedures have been successful as these tools were found to have proven statistical significance [7-9]. This ensures that such validated instruments can be adopted with good educational impact in otolaryngology practice.
This instrument shows promising results to be a reliable, valid and feasible assessment method for testing the surgical skills of the residents within the operation theater, performing thyroidectomy. It provides detailed evaluation with global considerations to the decisive and crucial steps in the surgical procedures that requires competence of the residents. Such instrument can contribute to the educational development in the surgical field as it gives insight and feedback about the competency level of trainees.