Using Multiple Assessors To Evaluate Core Competencies Of Nursing Students: A 360° Evaluation Approach

Introduction

Evaluation of nursing students in the clinical field requires the clinical teacher to make judgments regarding student progress in a number of areas. The key to feeling confident about judgments is to use multiple data sources for evaluation. Faculty tends to see clinical performance through their own clinical 'spectacles'. Therefore summative evaluation from a single faculty member should not be relied upon. When interpersonal, communication, professionalism and team work behaviors is to be assessed, 360 degree feedback is the best approach to be adopted to evaluate and guide the performance.

Performance assessment provides a measure of an individual's competence. Clinical competence is a complex construct and it requires the use of knowledge, technical skills, communication, clinical reasoning, abilities in daily practice. It can be argued that there is no single method of assessment that can successfully evaluate the clinical knowledge, skills and abilities of nursing students.¹ Traditionally, the evaluation of nursing students is completed by faculty evaluators.These evaluations are limited in their application as they do not consider the patient, nursing staff and colleagues/peers. 360 degree feedback or multi source feedback increases individual's awareness of own performance, how their performance is viewed by their peers, nurses and patients.

Obstetrics and gynecology residents were assessed on their interpersonal and communication skills by nurses, faculty members, allied health professional staff, medical students, patients, and co-residents. In addition, each resident completed a self-assessment. The researchers found good correlation between evaluations within each group of evaluators as well reasonably strong agreement among evaluators regarding each resident's rank among the peer group. Interestingly, there was a negative correlation between the rankings by faculty, staff, and medical students with the rankings given by peers. On selfassessment, junior residents typically rated themselves highly while senior residents rated themselves average or low. ²

A study aimed to determine if non faculty ratings of resident professionalism and interpersonal skills differ from faculty ratings using a 360 degree rating scale among pediatric residents. This study found high ratings for resident professionalism and interpersonal skills. However, different members of the health care team rated residents differently, and ratings are not correlated. These results provide evidence for the potential value of 360-degree evaluations.³

A project was undertaken to determine if the addition of peer, self, and nurse evaluators would enhance faculty assessment of resident performance. There was a low degree of correlation between attending and selfevaluations and attending and nurse evaluations. This small study supports the use of peer evaluations in addition to attending evaluations for Obstetrics and Gynecology residents in training. It also demonstrated that residents may benefit from doing self-evaluations to improve their ability to honestly appraise their clinical and interpersonal skills.⁴

The 360-feedback technique used in residency training has been widely described in medical education journals. Very few studies have been done to assess the use of 360 degree assessment in nursing education. 360-degree feedback can be an extremely effective tool for in nursing education, as learners will often communicate differently in the presence of a nursing faculty and when independently communicating with peers, staff and patients.

The purpose of this study is to compare the evaluation of the nursing students' evaluations using 360 degree evaluation by patients, nursing staff, peer and self evaluation.

Materials and Methods

A total of 374 final year nursing students were enrolled for the study. All students had completed their classroom training and were in clinical placement in Obstetrical Nursing units, Medical Surgical units and community. Verbal consent was obtained from the study subjects and students not attending any practical experience were excluded from the study. Administrative Permission was also obtained from the colleges to conduct the study. Each subject was being evaluated by the patient, nursing staff, peers and self evaluation. For the subjects having practical experience in the community, clients assigned, nursing faculty, peer and self were considered for evaluation. The completed questionnaires were compiled into a final report for the subjects. Anonymity for those completing the report helped ensure that ratings and comments are fair as well as skill and behavior-based.

Raters: The number of raters is important on two counts. First, the assessment has to be based on a large enough sample to ensure that it is valid; if it is too small, there is a danger that one rater's view will have a major impact on the overall results. Second, the sample of raters needs to be large enough that individual sources cannot be identified; a minimum of three to five people, depending on the circumstances. The feedback from Nursing staff, Clients, Peers and Self assessment were included. As some of the subjects were also having practical experience in the community and they undergo 8 weeks of training in the community, it was considered necessary to include it in the competency assessment. As some community areas did not have a nursing staff, the clinical supervisor of community health nursing completed the questionnaire.

Rating scales : The 360 degree feedback comprised of 4 rating scales to be completed by the Nursing staff, client, self and peers. It is a 3 point rating scale and a separate column for items that could not be evaluated. The rating scale for nursing staff and client encompassed 4 domains -Valuing Human Beings, Professional Nursing practice, therapeutic communication and interpersonal relationship, professional and ethical framework. It had a total of 21 items and 20 items respectively. The rating scale for evaluation by peer and self had 9 competency domains (Valuing Human Beings, Professional Nursing practice, therapeutic communication and interpersonal relationship, professional and ethical framework, Knowledge and application of knowledge, collaborative therapeutic practice, and management of nursing services, research utilization, and professional advancement) and had a total of 40 items.

Structure of Feedback : The ratings of the different groups are presented separately, and the range of the ratings (i.e. highest and lowest) as well as the averages included so that these differences in perspective are identified. As there are enough raters involved, this will not compromise anonymity.

Scoring criteria: The final scoring was done using Bondy's criteria modified by Holaday and Buckley ⁵ (Self Directed -4 Supervised-3, Assisted-2, Novice-1, Dependent-0) as its established validity and reliability has contributed greatly to nursing education and were major factors in the decision to use the scale.

Results:

Discussion:

In this study, all the evaluators scored the students competency level above the novice dependent level. It is similar to the findings of Chandler et al. Chandler et al in a 360 degree rating of pediatric residents by peer, patients, self and MD's found the residents were scored highly by all the raters.

The students received the lowest ratings from the RN's. This is consistent with the study conducted by Brinkman et al wherein nurses rated the residents lower than the other raters.

Patients ratings of students were lower than the self and peer ratings of the students. This is consistent with the findings of Chandler et al³ wherein the patient/families rated the residents lower than the MD's. Woods et al studied the 360 degree assessment of radiology residents and found that there was negative relationship between resident and patient rating.⁴

Lelliot et al in a study of multisource feedback of consultant psychiatrists found a correlation between colleague rating and patient rating(r=0.33, p<0.001). In this study, there was no correlation between patient and peer rating.⁷

The purpose of 360 degree rating is to enhance the feedback process and obtain different perspectives on student evaluations that cannot be obtained only through individual faculty evaluations. However, in this study different assessors evaluated differently and there was no correlation between some assessors. This is consistent with the different literature reviews done by the author that found very few studies had significant correlations between all the assessors.⁶

There are certain limitations of this study. First, the subjects were rated after one week of the posting. Domains like interpersonal relationship and professional, legal and ethical practice should use information collected over a long period of time. All clinical supervisors were given a 2 week period to get the rating scales completed by the multi sources. It could be possible that different sources evaluated the subjects at different times. Another limitation was that the subjects were providing the care for the clients at the time of assessment. This may have affected the anonymity and rating. The number of evaluators per student also needs to be considered. Weinrich et al suggested that approximately 10 to 15 ratings from the nurses need to be considered.⁸ In this study, only one nurse evaluator who is supervising the student was considered.

Conclusion:

Moving away from the traditional single faculty evaluation system is essential in order to bring the perspectives of clients, colleagues, nurses into the evaluation system. 360 degree assessment focuses on different aspects of care such as professionalism, interpersonal relationship and collaborative practice that cannot be evaluated by a single faculty. There is no gold standard to evaluate clinical ability. The results of the above study suggest that 360 degree assessments can provide additional useful information on student performance and evaluation of different perspectives of care.