7 Questions to ask when evaluating a research claim

1. Is prediction testable like experimental hypothesis or is their prediction untestable because:

1. It is too vague

2. It is subjective rather than objective

3. They are trying to prove null hypothesis (not finding something doesn't mean it doesn't exist--it may exist but you failed to find it).

2. Is there objective evidence or results supporting the claim?

3. Is the objective evidence presented typical of what happens or an exception?

1. Beware of argument by example. Remember that the lesson of the case study is that anything can happen once.

2. Look for statistical significance (a tool for seeing whether the pattern in the results is reliable ),

but realize that statistical significance does not necessarily mean large or important

4. Ask about control groups (lesson from the experiment: if you want to say that one group did better, you need to be able to ask  "better than what")

1. Is there one --or are they assuming they know what the control group would do?

 

2. Is it fair?

a. Was it established by random assignment?

b. Is it treated like the treatment group in every respect except that it doesn't get the treatment?

c. Were double-blind techniques used? If not, the results could be due to a placebo effect.

5. Are they trying to insinuate that correlation means causation? Remember, A-->B is only one of several possibilities for a statistical relationship between A and B.

6. Are they reporting what they actually saw

or

are they reporting their interpretation of what they saw?

(The lesson of observation: objective observation is difficult)

7. Are their results based on a sample that would apply to you?

(The lesson of the survey: the sample matters)

---