Measures That Are Not Reliable Are Not Valid
At 2:14 pm, the thermometer on the left "measured" the temperature at
51 degrees, whereas the one on the right "measured" the temperature at
54 degrees. Obviously, both cannot be correct.
From this picture alone, you might think that one temperature gauge
is simply biased. For example, you might think that one bank has set their
gauge to read warmer temperatures so that people will associate good news
or warmth with their bank. Or, perhaps one bank's gauge is in the sun,
but the other is always in the shade.
However, it's quite possible that the two temperature gauges differ
due to random error.
After comparing the temperature's for a period of time, we found that their
readings varied due to random error. That is, one gauge did not read consistently higher than the other.
Regardless of whether the difference is due to bias or to random error,
we know that at least one of the gauges is not perfectly accurate.
Without looking at both gauges, we wouldn't have known about the error.
We might have assumed that the one we saw was giving us accurate and objective information.
Even if both gauges gave us the same reading, that doesn't necessarily
mean that they are both right. They might both be wrong! Thus, reliability
does not guarantee validity (accuracy).
There are at least two ways that the measures could be biased and still produce a high inter-observer reliability.
Thus, high inter-observer reliability does not necessarily mean
that your measure is free from systematic error ( bias).
If one gauge was consistently more biased than the other (one always read
3 degrees lower than the other), then the correlation between their ratings
would have been perfect. That is, the inter-observer(thermometer) reliability
would be 1.
- If both measures were equally biased (e.g., running three degrees too warm), both inter-observer reliability and inter-observer agreement would be perfect.
If both thermometers are unreliable, then averaging them would give us
the best estimate of the true temperature. Thus, in the example above,
we would say that the true temperature down town is 52.5 degrees (the average of 51 and 54).
To Quiz over this material
Back to Chapter 5 Menu
Back to Research Design Explained Home Page