Measures That Are Not Reliable Are Not Valid 

At 2:14 pm, the thermometer on the left "measured" the temperature at 51 degrees, whereas the one on the right "measured" the temperature at 54 degrees. Obviously, both cannot be correct.

From this picture alone, you might think that one temperature gauge is simply biased. For example, you might think that one bank has set their gauge to read warmer temperatures so that people will associate good news or warmth with their bank. Or, perhaps one bank's gauge is in the sun, but the other is always in the shade.
 
However, it's quite possible that the two temperature gauges differ due to random error.

Note that:
1. Regardless of whether the difference is due to bias or to random error, we know that at least one of the gauges is not perfectly accurate.
2. Without looking at both gauges, we wouldn't have known about the error. We might have assumed that the one we saw was giving us accurate and objective information.
3. Even if both gauges gave us the same reading, that doesn't necessarily mean that they are both right. They might both be wrong! Thus, reliability does not guarantee validity (accuracy).

• There are at least two ways that the measures could be biased and still produce a high inter-observer reliability.
  1. If one gauge was consistently more biased than the other (one always read 3 degrees lower than the other), then the correlation between their ratings would have been perfect. That is, the inter-observer(thermometer) reliability would be 1.
  2. If both measures were equally biased (e.g., running three degrees too warm), both inter-observer reliability and inter-observer agreement would be perfect.
  Thus, high inter-observer reliability does not necessarily mean that your measure is free from systematic error ( bias).
• If both thermometers are unreliable, then averaging them would give us the best estimate of the true temperature. Thus, in the example above, we would say that the true temperature down town is 52.5 degrees (the average of 51 and 54).

To Quiz over this material

Back to Chapter 5 Menu

Back to Research Design Explained Home Page