What does a negative discrimination index mean

Point-Biserial – The point-biserial correlation is the Pearson correlation between responses to a particular item and scores on the total test (with or without that item). The Biserial Correlation models the responses to the item to represent stratification of a normal distribution and computes the correlation accordingly. Like the Discrimination Index the range is -1.0 to 1.0. Generally 0.2 and above is considered to have high correlation and positive association with overall performance on the assessment; lower levels are acceptable for mastery; and 0.3 or higher are best for discriminating questions.

In our discussion about correlational item discrimination, I mentioned that there are several other ways to quantify discrimination. One of the simplest ways to calculate discrimination is the High-Low Discrimination index, which is included on the item detail views in Questionmark’s Item Analysis Report.

To calculate the High-Low Discrimination value, we simply subtract the percentage of low-scoring participants who got the item correct from the percentage of high-scoring participants who got the item correct. If 30% of our low-scoring participants answered correctly, and 80% of our high-scoring participants answered correctly, then the High-Low Discrimination is 0.80 – 0.30 = 0.50.

But what is the cut point between high and low scorers? In his article, “Selection of Upper and Lower Groups for the Validation of Test Items,” Kelley demonstrated that the High-Low Discrimination index may be more stable when we define the upper and lower groups as participants with the top 27% and bottom 27% of total scores, respectively. This is the same method that is used to define the upper and lower groups in Questionmark’s Item Analysis Report.
The interpretation of High-Low Discrimination is similar to the interpretation of correlational indices: positive values indicate good discrimination, values near zero indicate that there is little discrimination, and negative discrimination indicates that the item is easier for low-scoring participants.

In Measuring Educational Achievement, Ebel recommended the following cut points for interpreting High-Low Discrimination (D):

In Introduction to Classical and Modern Test Theory, Crocker and Algina note that there are some drawbacks to the High-Low Discrimination index. First, it is more common to see items with the same p value having large discrepancies in their High-Low Discrimination values. Second, unlike correlation discrimination indices, High-Low Discrimination can only be calculated for dichotomous items. Finally, the High-Low Discrimination does not have a defined sampling distribution, which means that confidence intervals cannot be calculated, and practitioners cannot determine whether there are statistical differences in High-Low Discrimination values.

Nevertheless, High-Low Discrimination is easy to calculate and interpret, so it is still a very useful tool for item analysis, especially in small-scale assessment. The figure below shows an example of the High-Low Discrimination value on the item detail view of the Item Analysis Report.

To determine the difficulty level of test items, a measure called the Difficulty Index is used. This measure asks teachers to calculate the proportion of students who answered the test item accurately. By looking at each alternative (for multiple choice), we can also find out if there are answer choices that should be replaced. For example, let's say you gave a multiple choice quiz and there were four answer choices (A, B, C, and D). The following table illustrates how many students selected each answer choice for Question #1 and #2.

* Denotes correct answer.

For Question #1, we can see that A was not a very good distractor -- no one selected that answer. We can also compute the difficulty of the item by dividing the number of students who choose the correct answer (24) by the number of total students (30). Using this formula, the difficulty of Question #1 (referred to as p) is equal to 24/30 or .80. A rough "rule-of-thumb" is that if the item difficulty is more than .75, it is an easy item; if the difficulty is below .25, it is a difficult item. Given these parameters, this item could be regarded moderately easy -- lots (80%) of students got it correct. In contrast, Question #2 is much more difficult (12/30 = .40). In fact, on Question #2, more students selected an incorrect answer (B) than selected the correct answer (A). This item should be carefully analyzed to ensure that B is an appropriate distractor.

Another measure, the Discrimination Index, refers to how well an assessment differentiates between high and low scorers. In other words, you should be able to expect that the high-performing students would select the correct answer for each question more often than the low-performing students.  If this is true, then the assessment is said to have a positive discrimination index (between 0 and 1) -- indicating that students who received a high total score chose the correct answer for a specific item more often than the students who had a lower overall score. If, however, you find that more of the low-performing students got a specific item correct, then the item has a negative discrimination index (between -1 and 0). Let's look at an example.

Table 2 displays the results of ten questions on a quiz. Note that the students are arranged with the top overall scorers at the top of the table.

"1" indicates the answer was correct; "0" indicates it was incorrect.

Follow these steps to determine the Difficulty Index and the Discrimination Index.

1. After the students are arranged with the highest overall scores at the top, count the number of students in the upper and lower group who got each item correct. For Question #1, there were 4 students in the top half who got it correct, and 4 students in the bottom half.

2. Determine the Difficulty Index by dividing the number who got it correct by the total number of students. For Question #1, this would be 8/10 or p=.80.
3. Determine the Discrimination Index by subtracting the number of students in the lower group who got the item correct from the number of students in the upper group who got the item correct.  Then, divide by the number of students in each group (in this case, there are five in each group). For Question #1, that means you would subtract 4 from 4, and divide by 5, which results in a Discrimination Index of  0.
4. The answers for Questions 1-3 are provided in Table 2.

Now that we have the table filled in, what does it mean? We can see that Question #2 had a difficulty index of .30 (meaning it was quite difficult), and it also had a negative discrimination index of -0.6 (meaning that the low-performing students were more likely to get this item correct).  This question should be carefully analyzed, and probably deleted or changed. Our "best" overall question is Question 3, which had a moderate difficulty level (.60), and discriminated extremely well (0.8).

What does the discrimination index tell you?

What does a difficulty index of 0.10 mean?