How do I decide which Personal Result method to apply in my peer assessment activity?

The choice of calculation method for determining a team member’s personal result is determined by the teacher's preference for compensating more strongly team members who have contributed significantly to their teams, and under-rewarding team members who are peer assessed as weak contributors. The figure illustrates the statistical features, such as team average, range, and standard deviation, associated with each method.

Alternative calculation methods for Personal Result (PR) illustrating effect on team average and spread for a given Team Result

 

The teacher can select either the Peer Assessed Score (PA Score) or Peer Assessed Index (PA Index) if they wish to exclude a team result in calculating the Personal Result (PR).

More usually, the Peer Assessed Score and Team Result (TR) are combined mathematically to produce a Personal Result. There are three alternative methods. As the figure illustrates, the Indexed Personal Result (IPR) is the least discriminating method, whilst the Rank-Based Personal Result (RPR) is the most discriminating in terms of favouring significant team contributors and penalising weak contributors. Most teachers select the Normalised Personal Result, often with a spread factor of 1.5 to 2.0.

In contrast to the graphical illustration earlier, the following table summarises the example calculations presented through a series of FAQ that present the mathematical definition and example calculations for each method.

 

Comparison of Personal Results calculated by several methods in a team of four members

 

ASSESSEE

   

ASSESSOR

Bridget

Julian

Lydia

Nigella

Mean

Range

Rank Reversed

1

2

4

3

   

Peer Assessed Score, PA Score

54

74

82

78

75

28

Peer Assessed Index, PA Index

66

90

100

95

88

34

Team Result, TR

50

50

50

50

50

0

Indexed Personal Result, IPR

33

45

50

48

44

17

Normalised Personal Result, NPR

(SpreadFactor = 1)

39

51

56

54

50

17

Normalised Personal Result, NPR

(Spreadfactor = 2)

28

52

62

58

50

34

Rank-Based Personal Result, RBR

20

40

80

60

50

60

 

Source: FAQ: How are peer assessment and personal results calculated and defined mathematically?

 

Definitions and features of calculation methods used in Peer Assess Pro

Attribute (X1)

Abbreviation (X1)

Definition (X1)

Peer Assessed Score

PA Score

A relative measure of the degree to which a team member has contributed to their team's overall achievement, team processes, and leadership. The Peer Assessed Score (PA Score) is calculated for each team member directly from their Average Team Contribution (ATC) and Average Leadership Contribution (ALC). That is, from the ten components of Team and Leadership contribution survey in the peer assessment.

 

A Peer Assessed score is generally used to compare the relative contribution of students WITHIN the same team, rather than BETWEEN teams. The Team Result has NO impact on the value of the Peer Assessed Score. Values for the PA Score range from zero through 100.

Peer Assessed Index

PA Index

The Peer Assessed Score (PA Score) is indexed upwards so that the person in the team with the highest Peer Assessed Score is awarded a Peer Assessed Index of 100. All other team members receive a proportionally lower PA Index in the ratio PA Score / max(PA Score). The Team Result has NO impact on the value of the Peer Assessed Index.

Team Result

TR

The result awarded to the team for the outputs of their work. The teacher typically derives the Team Result (TR) from grades for team reports, presentations, and results of Team Readiness Assurance Tests.

 

The teacher may select to combine a student's Peer Assessed Index (PA Index) with their team's Team Result (TR) to calculate a Personal Result (PR) for each student, reflecting their relative contribution to the Team Result as assessed by their peer team members. Peer Assess Pro enables the teacher to select from several methods to combine the Team Result and Peer Assessed Index (PA Index) to produce a Personal Result: the Indexed Personal Result (IPR), the Normalised Personal Result (NPR), and the Rank Based Personal Result (RPR).

Measures of a student's personal result

   

Personal Result

PR

A student's personal result gained from combining their Peer Assessed Index (PA Index) and, optionally, their Team Result (TR).

 

The teacher selects from one of several Calculation Methods to calculate the Personal Result that incorporates the Team Result. These methods are Indexed Personal Result (IPR), Normalised Personal Result (NPR), and Rank-Based Personal Result (RPR).

 

The choice of method is determined by the teacher's preference for compensating more strongly students who have contributed significantly to their teams, and under-reward students who are peer assessed as weak contributors. Figure 1 illustrates the statistical features, such as team average, range, and standard deviation, associated with each method. The IPR is the least discriminating method, whilst the RPR is the most discriminating in terms of favouring significant team contributors and penalising weak contributors, as the figure illustrates.

Indexed Personal Result

IPR

The Indexed Personal Result is calculated from the Team Result (TR) combined with the student's specific Peer Assessed Index (PA Index). The Indexed Personal Result method awards the Team Result to the TOP RATED student in the team, since, by definition, their Peer Assessed Index is 100. All remaining students in the same team earn the Team Result downwards, directly proportional to their PA Index.

 

The Indexed Personal Result calculation means that NO team member can earn an Indexed Personal Result greater than the Team Result. That is, values for the Indexed Personal Result range from zero up to the Team Result.

Normalised Personal Result

NPR

The Normalised Personal Result is calculated from the Team Result combined with the student's specific Indexed Personal Result (IPR). However, in contrast to the IPR method, the Normalised Personal Result method awards the AVERAGE student in the team the Team Result (TR). All remaining students are awarded a Personal Result ABOVE or BELOW the Team Result depending on whether their IPR is above or below that team's average.

 

Features of the Normalised Personal Result are that (a) In contrast to the IPR method, the Normalised Personal Result method calculates a Personal Result ABOVE the Team Result for the above-average peer rated students in the team (b) The average of the team's Normalised Personal Results matches the Team Result (c) The spread of the team's Normalised Personal Results matches the spread of the Indexed Personal Results (IPR) that is calculated for that team. Spread is measured by the standard deviation statistic. .

 

Optional feature: To enhance the effect of rewarding high contributors and penalising weak contributors the tutor can increase the Spread Factor (SF) from the default value of 1.0. Increasing the Spread Factor increases the spread of the results centred around the Team Result. However, an increase in the Spread Factor will maintain a team average NPR that matches that team's Team Result. A Spread Factor of 1.5 to 2.0 is recommended, especially in classes where team members are reluctant to penalise weak contributors and/or reward the highest contributors through their peer assessment rating responses.

 

Values for the NPR range from zero to 100. Calculations that exceed these ranges are clipped to fit within zero to 100

Rank Based Personal Result

RPR

The Rank Based Personal Result is calculated from the Team Result combined with the student's specific Rank Within Team based on that student's Peer Assessed Score. Like the Normalised Personal Personal Result the RPR method awards the AVERAGE student in the team the Team Result. All remaining students are awarded a personal result above or below the Team Result depending on whether their Rank Within Team is above or below that team's middle-ranked student.

 

Features of the Rank Based Personal Result (PR) calculation method are that (a) A team's RPR values are spread over a MUCH WIDER range than the NPR and IPR methods. Small differences in PA scores within a team are amplified significantly by this method (b) In contrast to the IPR method, the RPR method calculates a Personal Result significantly ABOVE the Team Result for the top ranked student in the team (c) Like the NPR method, the average of the team's RPR values matches the Team Result. Values for the Rank Based Personal Result range from zero to 100. Calculations that exceed these ranges are clipped to fit within the range zero to 100.

 

Note that in the Xorro PAP app, we have renamed the following Personal Result Methods

Renaming of terms for Xorro Peer Assess Pro

Xorro Peer Assess Pro

Abbreviation

Google Peer Assess Pro

Abbreviation

Peer Assessed Score

PA Score

Team Based Learning Score

TBL Score Score

Peer Assessed Index

PA Index

Team Based Learning Index

TBL Index

 

Also see:  Participant    Teams and Teamsets    Group    Peer Assessment   

What is the Purpose of Peer Assessment?   What questions are asked in the peer assessment survey?    How is the Peer Assessed (PA) Score calculated?    How are peer assessment and personal results calculated and defined mathematically?

Running Peer Assessments from Xorro-Q    Importing Participants, Teams and Groups    How do I view or correct the Team Composition in a running peer assessment activity?    How do I decide on which Personal Result method to apply in my peer assessment activity?

Peer Assess Pro website    Peer Assess Pro Reference Guide (PDF)

Categories: Xorro-X, Facilitators.
Tags: peer assessment.