Bonus Appendix 1 | Research Design Explained

Bonus Appendix C: Conducting a Study

PlanningYour Procedures

You have reviewed the literature, developed a hypothesis,operationalized your variables, and given sound reasons for testing yourhypothesis. However, your preliminary work is still not done. You must nowdecide exactly what specific actions you will take. In other words, althoughyou probably have decided on the general design (such as a simple experiment ora 2 X 2), your plan is not complete until each detail has been thought throughand written down.

You shouldwrite down exactlywhat procedures you will follow. For example, what instructions, word for word,will you give participants? Where will you test them? Will participants be runin groups or individually? In answering these questions, you must take intoaccount issues of validity. However, your paramount concern must always beethics. You do not have the right to harm another.

EthicalConsiderations: Human Research

Ethics should be the foundation of your research plan. Therefore, youshould read Appendix A (Ethics) before conducting a study.

In additionto reading Appendix A, you must be extremely careful not to harm yourparticipants. Ideally, your participants should feel just as well when theyleave the study as they did when they began the study. Unfortunately, even inthe most innocuous studies, protecting your participants from discomfort ismuch easier said than done.

Weighing the Risks

Realize that any experience may be traumatic to someparticipants. Trauma can occur from things you would never think of as beingtraumatic. Because any study has risks and because you will not know all of therisks, do not run a single participant without your professor’s permission.

To begin tosensitize yourself to the risks involved in your proposed study, list the 10worst things that could possibly happen to participants. If you are using humanparticipants, be aware that not all participants will react in the same way.Some may experience trauma because the study triggers some painful memory. Someparticipants may feel bad because they think they did poorly. Otherparticipants may feel bad because they think their behavior ruined your study.Realize that some of your participants may be mentally unbalanced and anyattack on their self-esteem might lead to disastrous consequences. Becauseparticipants are often fragile, you should list some serious consequences inyour worst-case scenario.

Reducing Risks

Because any study has the potential for harm, the possibility of severeconsequences does not mean that your professor will not allow you to do thestudy. However, you and your professor should think about ways to minimize therisks.

Screening Participants

One method of minimizing risks isto screen out “vulnerable participants.” For instance, if there isany reason to believe that your study may increase heart rate or bloodpressure, you may want to make sure that only people in good health participatein your study. If your study might harm people with low self-esteem, you maywant to use only well-adjusted participants who have high levels ofself-esteem. Therefore, you might give a measure of self-esteem to potentialparticipants to eliminate those with low self-esteem.

Informed Consent

Not only should you screenparticipants, but you should also let participants screen themselves. That is,participants should be volunteers who give their informed consent:They should know what the study is about before volunteering for it.

Howinformed is informed consent? Very informed, when it comes to tellingparticipants about any unpleasant aspect of the study. If participants aregoing to get shocked or exposed to loud noises or extreme cold, they should beinformed of this before they volunteer. Consequently, if your study doesinvolve unpleasantness, you may have difficulty getting participants tovolunteer.

Informedconsent is considerably less informed when it comes to more innocuous aspectsof the study. After all, the study would be ruined if participants kneweverything that would happen (and why it happened) before it happened. So,although participants are usually told the truth, they are not always told thewhole truth. For example, a memory experiment’s description would mentionthat participants have to memorize words, but might omit the fact that theresearcher is looking at the order in which facts are recalled or that there isa surprise recall of all the lists at the end of the study.

Becauseparticipants are not fully informed about your study, there may be some thingsabout it that they dislike. For example, suppose a participant finds the tasktoo difﬁcult or finds it upsetting to try the surprise recall task. Whatcan you do?

Oneprotection against these unexpected problems is to make sure participantsunderstand that they can quit the study at any time. So, before theparticipants begin your study, tell them that if they find any aspect of thestudy uncomfortable, they can and should escape this discomfort by quitting the study.Assure them that it is their duty to quit if they experience discomfort andthat they will still get full credit.

Modifying the Study

You have seen that you can minimize ethical problems by lettingparticipants know what they are in for and by letting participants gracefullywithdraw from the study. You should also minimize harm by making your study ashumane as possible. You can make your study more ethical by reducing thestrength of your treatment manipulation, carefully selecting stimulusmaterials, and by being a conscientious researcher.

Reducing the Treatment Strength

Although using extreme levels ofyour predictor variable may help you get a significant change in the criterionvariable, extreme levels may harm your participants. For example, 24 hours offood deprivation is more likely to cause hunger than 12 hours. However, 24hours of deprivation is more stressful to the participant. If you plan anunpleasant manipulation, remember your participants’ welfare and minimizethe unpleasant consequences as much as possible. Consider using levels of thepredictor variable that are less severe than you originally intended.

Modifying Stimulus Materials

By modifying your stimulusmaterials, you may be able to prevent them from triggering unpleasant memories.For instance, if you were interested in the effects of caffeine on memory forprose, you would not want the prose passage to cover some topic like death,divorce, alcoholic parents, or rape. Instead, you would want to use a passagecovering a less traumatic topic such as sports. If the sports article referredto someone’s death or hospitalization, you might want to delete thatsection of the article.

The Conscientious Researcher

Often, it is not the study that causes ethical problems, it is the researcher’s arrogance. For example, an arrogantresearcher may rush through research sessions providing only superficialexplanations and almost no time for questions and feedback. Although we know ofa few participants who were hurt as a direct result of a research manipulation,we know of many more who were hurt because the researcher treated them likedirt. To ensure that you are sensitive, courteous, and respectful to all ofyour human participants, you should do two things.

First, whenscheduling your research sessions, make sure you leave a 10-minute gap betweenthe end of one session and the beginning of the next session. Someinvestigators feel that, like a physician, they should efficiently schedulepeople one after another. Their attempt at efficiency results in participantshaving to wait for the investigator, the investigator having to rush throughthe formalities of greeting participants, or—even worse—theinvestigator rushing through debriefing. Thus, the overly efficient investigator,like the overly efficient physician, is seen as unconcerned. Although thisconduct does not become physicians, it is intolerable for psychologicalresearchers. After a research participant has given an hour of his or her time,you should be more than willing to answer any questions the participant has.Furthermore, if you rush through greeting or debriefing each participant, theparticipants will see you as uncaring. Consequently, participants will be less likelyto tell you about any psychological discomfort they felt and less likely toaccept any aid you might offer. Thus, the first step is to walk, rather than torun, participants through your study.

Second,give the participants power. That is, allow participants to rate your study ona scale such as the one in Table 1–1 (below). Give each participant’srating sheet to your instructor. Following this simple procedure helps you tobe a conscientious and courteous researcher.

Table1–1

SampleDebriefing Rating Scale

Being aparticipant in psychology studies should provide you with a firsthand look atresearch. On the scale below, please indicate how valuable or worthless youfound being in today’s study by circling a number from +3 to –3.

WORTHLESS:–3 –2 –1 +1 +2 +3 :VALUABLE

If you wish to explain your ratingor to make comments on this study, either positive or negative, please do sobelow.

NOTE: Thisscale is a slightly modified version of a scale that has been used at The OhioState University.

Debriefing

Although you should try to anticipate and prevent every possible badreaction a participant may have to being in your study, you will fail. .Inevitably, your procedures will still cause some unpleasantness. After thestudy is over, you should try to remove this unpleasantness by informingparticipants about the study, reassuring them that their reactions were normal,and expressing your appreciation for their participation.

You shouldalso listen to participants and be sensitive to any unexpected, unpleasantreactions to your study. By being a good listener, you should be able to undoany damage you have unwittingly done. This process of informing yourparticipants about the study and removing any harm done is called debriefing.Occasionally,ordinary debriefing will not undo the harm caused to the research participant.In those cases, there are several steps you may take to alleviate distress. Forparticipants who are upset with their responses, you should ask them whetherthey want you to destroy their data. For participants that you cannot calmdown, you should take them to talk to a professor, counselor, orfriend—even if this means canceling a research session you had scheduled.

In summary,you should be very concerned about ethics. Since ethics involves weighing thecosts of the study against the potential benefits, you should do everything youcan to minimize the risk of participants becoming uncomfortable. If, despiteyour efforts, a participant experiences discomfort, you should try to reducethat discomfort during debriefing.

EthicalConsiderations: Animal Research

With animal participants, you incur the same responsibilities that youdid with human participants—you must protect animal participants fromundue stress and discomfort. In many ways, you have even more responsibility toanimal participants because they depend on you for their mere existence. Youmust keep them fed, clean, warm, and comfortable—24 hours a day. Tofulfill your responsibility to animal participants, you must follow APA’sguidelines for proper housing, food and water, and handling (see Appendix A).

Furthermore,because your animal subjects n have neither the power to give their informedconsent, nor the power to quit the study, you must carefully question the valueof your study. Ask yourself and your professor this question: “Is thepotential knowledge gained from the study worth the cost to the animals?”Finally, if you must euthanize (kill) your animal participants at the end ofyour study, follow APA’s guidelines to ensure that you euthanize youranimals in the most humane way.

Maximizethe Benefits: The Other Side of the Ethics Coin

We havediscussed ways of minimizing harm to participants. However, minimizing harm isnot enough to ensure that your study is ethical. For your study to be ethical,the potential benefits must be greater than the potential harm. Thus, anextremely harmless study can be unethical if the study has no potentialbenefits. In other words, just as you owe it to your participants to reducepotential harm, you owe it to your participants to maximize the potentialbenefits of your study. You maximize that potential by making sure your studyprovides accurate information. To provide accurate information, your studyneeds to have power and validity.

Power IsKnowledge

One of themost serious obstacles to obtaining accurate information is lack of power.Remember, null results do not prove the null hypothesis. They only make peoplewonder about the study’s power. There is no point in doing a study thatis so powerless that it will lead to inconclusive, null results.

To havepower, you should use a strong manipulation, a sensitive dependent measure,well-standardized procedures, a sensitive design, and enough participants.

SampleSize: There Is Power in Numbers

Perhapsyour most important obstacle to finding a significant effect is a lack ofparticipants. As a general rule, you should have at least 16 participants ineach group.[1] However, thenumber of participants you need in each group will be affected by thesensitivity of your design, the heterogeneity of your participants, the numberof observations you get from each participant, the size of the difference youexpect to find between conditions, and the sensitivity of your dependentmeasure.

If you havea within-subjects design, a reliable and sensitive dependent variable, andexpect a rather large difference between your conditions, you may be able touse fewer than 16 participants per group. If, on the other hand, you are usinga simple, between-subjects design, heterogeneous participants, a manipulationthat may have little effect, and a relatively insensitive dependent measure,you may want at least 100 participants per condition.

Huntingfor Participants

Gettingenough participants to have enough power can be a challenge, especially if youare at a small school. However, even at a large school, getting enough studentsto volunteer to be in your study can be difficult.

Using a Department Participant Pool

Your institution may, however, havemade it is easier for you to get enough participants. Many colleges stronglyencourage students in introductory psychology courses to volunteer to beresearch participants. In fact, most psychological research used thisparticipant recruitment strategy. If your school has such a pool ofparticipants, count yourself among the blessed. All you have to do is ask yourprofessor how to use the participant pool.

Enlisting Volunteers

If your school does not have aparticipant pool, an effective way of getting participants is to ask professorsto request volunteers from their classes. Many professors will gladly recruit participants for you. Indeed, your main problem may be that professors may be too eager. For example, if students feel they must participate, then they are not volunteers and you have violated the principle of informed consent. Even if professors merely encourage students to participate by giving extra credit, you are on ethically shaky ground unless (a) you have, in your IRB application, explained that students may receive extra credit and (b) there is an alternative extra credit assignment for those students who choose not to participate in your study.

NonCollege Samples

But what if you do not want to usecollege students in your study? For example, suppose you want to study childrenor retirees? Alternatively, suppose you agree with the skeptics who claim thatif you study college students, you cannot apply your results to normal people.Then, you might look beyond college classrooms for participants. However, weshould warn you that getting real-world participants make take as much work andcreativity as planning your study.

Children

If you want to study children, you maybe able to take advantage of the “captive” audience approach. Afterall, most children have to go to school. However, obtaining access to thosechildren may turn into a nightmare of red tape. You will have to obtainpermission from all or many of the following: the school board, thesuperintendent, principal, teacher, parent, child, your professor, anduniversity. If you are going to get these permissions in time for your study,you will need to plan ahead—and be very lucky.

Adults

Findingadult participants can be even more challenging than finding children. Forexample, one of your textbook’s authors wanted an adult population forher doctoral dissertation. Her first thought was to contact a major company andgain access to its employees. This tactic failed. Next, she tried to run anewspaper ad asking for volunteers. One newspaper refused to print it. Anotherwould only run it in the “Personal” section. Thus, her appeal forparticipants appeared with ads for astrological advice, massage services, andpeople wanting dates. Although a few “volunteers” called, mostwanted either a date or an obscene conversation. We do not recommend newspaperads—especially if your goal is to get a representative sample of theadult population.

OlderAdults The authors have had greater success recruiting elderly participants.Nutrition centers, retirement communities, friendship networks, and nursinghomes have been fruitful sources of participants. In addition, we recommend the“grandmother connection”: having an older relative or friendintroduce you to other prospective participants.

Obviously,finding human participants will take planning, perseverance, and luck. Once youcontact prospective participants, you should explain your study to them. Beforethey participate, they must sign a permission form. Signing the permissionform—called an informed consent form—indicates that the participantunderstands what the study is about and has voluntarily agreed to participate.If you properly design the form (by following the instructions on page 475 of Research designexplained andfollowing the model presented in Box A-2, page 476 of Research design explained), the informed consent form willprotect both you and your participants—if your participants are collegestudents who are 18 years old or older. If minors are participating in yourstudy, you need to have separate forms for both the participants and theirparents.

AnimalSubjects

Our experiences with recruiting humanparticipants might have increased your enthusiasm for animal research. In someways, research with animal subjects is less stressful than doing research withhuman participants. You do not have to worry about permission slips, extracredit, or obscene phone calls. Consult with your instructor about obtaininganimals for your research. Often, schools have rat colonies or purchase animalsfor student research.

Reducing Threats to Construct Validity

After ensuring that your study has adequate power,we would like to be able to tell you that you can take it easy and relax.Unfortunately, however, you cannot relax. Power is not your only concern whenconducting psychological research. You must also ensure that the constructvalidity of your results is not destroyed by

1. Researchersfailing to conduct your study in an objective, standardized way, or

2. Participantsreacting to how they think you want them to react to the treatment, rather thanreacting to the treatment itself.

Researcher Effects

If you use more than one investigator, you may be able todetect researcher effects by including the researcher as a factor in yourdesign. In other words, randomly assign participants to both a condition and toa researcher. For example, if you have two treatment conditions (Aand B)and two researchers (1 and 2), you would have four conditions: (1) A1,(2) B1(3) A2and (4) B2. After having Research 1 run conditions 1 and 2 andResearcher 2 run conditions 3 and 4, you could do an analysis of variance (ANOVA) using researcher as afactor to see whether different researchers got different results.[2]

Using ANOVAto detect researcher effects can be useful. However, there are at least tworeasons why using ANOVA may not eliminate researcher effects. First, thisstatistical approach will only tell you if one researcher is getting differentresults than other researchers. If all your researchers are biased, you may notget a significant researcher effect. (Besides, if you are the only researcher,you cannot use researcher as a factor in an ANOVA.) Second, and moreimportantly, detecting researcher effects is not the same as preventingresearcher effects.

To preventresearcher effects, you must address the three major causes of researchersfailing to conduct studies in an objective and standardized manner. What arethese causes? First, researchers may not know how to behave because you havenot spelled out the exact procedures that the researchers should follow. Second,researchers may not follow those procedures. Third, the researchers maystrongly expect participants to behave in certain ways.

Loose-Protocol Effect: TheImportance of Developing a Protocol Often, the researchers are not behaving inan objective and standardized way because of the loose-protocol effect:The instructions are not detailed enough. Fortunately, the loose-protocoleffect can avoided.

Before youstart your study, carefully plan everything out. As a first step, you shouldwrite out a set of instructions that chronicles the exact procedure for eachparticipant. These procedures should be so specific that by reading andfollowing your instructions, another person could run your participants thesame way you do.

To makeyour instructions specific, you might want to write a computer program based onthese instructions. Since computers do not assume anything, writing such aprogram forces you to spell out everything down to the last detail. If youcannot program, just write the script as if a robot were to administer thestudy. Write out each step, including the actual words that researchers willsay to the participants. The use of such a script will help standardize yourprocedures, thus reducing threats to validity.

Once youhave a detailed draft of your protocol, give it a test run. For example, toensure that you are as specific as you think you are, pretend to be aparticipant and have several different people run you through the study using only your instructions. See how thedifferent individuals behave. This may give you clues as to how to tighten upyour procedures. In addition, you should run several practice participants.Notice whether you change procedures in some subtle way across participants. Ifso, adjust your instructions to get rid of this variability.

At the endof your test runs, you should have a detailed set of instructions that you andany co-investigator can follow to the letter. To double-check your protocol,see Table 1–2 (below).

Table1–2 ( To Table 1–3)

Protocol Checklist

How willyou manipulate your treatment variables?

How willyou measure your dependent (criterion) variables?

How manyparticipants will you need?

Do you haveyour professor’s permission to conduct the study?

Do you havea suitable place to run your participants?

How willyou get your participants?

If you areusing animals, how will they be cared for?

What willyou do with your animals after the study?

If you areusing human participants, how will you make your sign-up sheets available topotential participants?

Have youincluded a description of the study (including how long it takes) on thesign-up sheet?

Willparticipants be rewarded for volunteering to be in your study (such as money orextra credit)?

If you areconducting an experiment, how will you assign participants to condition?

Have youwritten out a detailed research protocol?

If you areusing human participants, have you developed a consent form?

If you areusing human participants, have you written out the oral instructions you willgive your participants?

If you areusing human participants, have you written out what you will say duringdebriefing?

If volunteers are college students seeking extracredit, how will you notify professors about which students participated?

Will you informparticipants about the outcome of your study? How?

Inspiring the Troops to AvoidResearcher Effects.

Unfortunately, even if you writeout your protocol (procedures) in detail, you or yourco-investigators may still fail to follow that protocol. To avoid the researcherfailure-to-follow-protocol effect, you need to make surethat (a) All investigators know the procedures, and (b) that everyone ismotivated to follow the procedures.

To makesure investigators learn the procedures, you should hold training sessions.Supervise investigators while they practice the procedures on each other and onpractice participants.

Onceresearchers know the right way to run the study, the key is to make sure thatthey are motivated to run the study the same way every time. To increaseresearchers’ motivation to be consistent, you might have them work inpairs. While one researcher runs the participants, the other will listen inthrough an intercom or watch through a one-way mirror. You may even wish torecord research sessions.

If yourresearchers still have trouble following procedures, you may need to automateyour study. For instance, you might use a computer to present instructions,administer the treatment, or collect the dependent measure. Computers have thereputation for following instructions to the letter, so using a computer mayhelp standardize your procedures. Of course, computers are not the onlymachines that can help you. Some of the machines that could help you giveinstructions and present stimuli include automated slide projectors, taperecorders, and videotape players. Countless other devices could help you recorddata accurately, from electronic timers and counters to noise-level meters.

Researcher-Expectancy Effect Thefinal source of researcher bias is the researcher-expectancy effect:Researchers’ expectations are affecting the results. You can take threesteps to prevent the researcher-expectancy effect:

1. Be very specific about howinvestigators are to conduct themselves. Remember, researcher expectanciesprobably affect the results by changing the investigator’s behaviorrather than by causing the investigator to send a telepathic message to theparticipants.

2. Do not let the investigators knowthe hypothesis.

3. Do not let investigators know what condition the participant isin—making the investigator “blind.” Although makinginvestigators blind is easiest in drug experiments where participants takeeither a placebo or the real drug, you can make investigators blind in nondrugexperiments.

Forexample, if you present stimuli in booklets, you can design your booklets sothat booklets for different conditions look very similar. In that way, aninvestigator running a group of participants might not know what condition eachparticipant is in. For some studies, you may be able to use a secondinvestigator who does nothing except collect the dependent measure. You couldeasily keep this second investigator in the dark about what treatment theparticipant received.

Review ofResearcher Effects.

Whether you are the only investigator orone of a team of investigators, researcher effects may bias your results. Therefore,you should always try to prevent the loose-protocol effect, thefailure-to-follow-protocol effect, and the researcher-expectancy effect.

ParticipantEffects

Unfortunately,in psychological research, you must be aware not only of researcher effects,but also of participant effects: Participants may see through the study and try to playalong with what the investigator wants. Fortunately, there are various ways ofpreventing participants’ expectancies from biasing your results.

PreventingParticipants’ Expectancies For starters, you might make your researcherblind to reduce the chance that the participant will get any ideas from theresearcher. Thus, the techniques for reducing research effects that we justdiscussed may also reduce the effects of participants’ expectancies.

Inaddition, you may also be able to prevent participants’ expectancies byskillfully choosing your research design. In experimental investigations, forexample, you might use a between-subject design rather than a within-subjectdesign because participants who are exposed to only one treatment condition areless likely to guess the hypothesis than participants who are exposed to alltreatment conditions.

PlaceboTreatments

Anotherdesign trick you can use to reduce the impact of participants’expectancies is to use placebo treatments. Placebo treatments preventparticipants from knowing that they are in the “no-treatment”condition. Therefore, if you have comparison condition(s), use placebotreatment(s) rather than no-treatment condition(s). That way, all groups thinkthey are receiving the treatment. Thus, any treatment effect you find will notbe due to participants changing their behavior because they expect thetreatment to have an effect.

UnobtrusiveRecording

Participantsare less likely to know the hypothesis if they do not know what you aremeasuring. Obviously, if, as in some field experiments, participants do noteven know you are observing them, participants will not know what you aremeasuring. Thus, if your hypothesis is an obvious one, you might consider afield study. Although field studies lend themselves to unobtrusive recording,unobtrusive recording can even occur in a laboratory study. That is,participants will assume that if you are not in the room with them, you are notobserving them. However, thanks to one-way mirrors and intercoms, you canmonitor participants’ behavior from the next room.

UnobtrusiveMeasures

Even if the participant knows you arewatching, the participant does not have to know what you are watching. That is,you can use unobtrusive measures. For example, you might put the participant infront of a computer and ask the participant to type an essay. Although theparticipant thinks you are measuring the essay’s quality, you could havethe computer programmed to monitor speed of typing, time between paragraphs,number of errors made, and times a section was rewritten. In addition, youmight also have tape-recorded and videotaped the participant, monitoring his orher facial expressions, number of vocalizations, and loudness of vocalizations.

ExperimentalRealism

Rather thantrying to obscure or confuse participants as to the purpose of the study, youmight try to prevent participants from thinking about the purpose of the study.One way to stop participants from thinking about the study’s purpose isto design a study that has a high degree of experimental realism: a study that involvesparticipants in the task. Experimental realism means that participants are notconstantly saying to themselves: “What does the researcher really want meto do?” or “If I were a typical person, how would I behave in thissituation?” Note that experimental realism does not mean the study islike real life; it means that participants are engrossed in the task. As youknow from video games, even an artificial task can be very high in experimentalrealism.

EthicsSummary

Before now,you might have been surprised to see experimental realism and other strategiesfor reducing participant effects in a section on ethics. However, you now knowthat planning an ethical study involves taking into account many factors. Notonly must you ensure the safety of your participants, but you must alsodemonstrate the validity of your methods. To avoid overlooking an importantethical consideration, consult Table 1-2, Table 1-3 (below), Appendix A, andyour professor.

Table1–3

Research With Human Participants: An EthicsChecklist

Is aphysically unpleasant stimulus going to be used in your study? If so,

1. Isthis fact clearly stated

a. on the sign-upsheet?

b. on the consentform?

2. Haveyou considered alternatives that would be less unpleasant?

3. Haveyou limited the intensity of this stimulus?

4. Haveyou taken steps to reduce potential harm to your participants caused by aphysically unpleasant stimulus?

Are you goingto use stress of some sort (such as sense of insecurity or failure, assaultupon values, fatigue, or sleep deprivation) in your study? If so,

1. Isthis fact clearly stated

a. on the sign-upsheet?

b. on the consentform?

2. Haveyou considered alternatives that would be less stressful?

3. Haveyou limited the intensity of this stimulus?

4. Haveyou taken steps to reduce potential harm to your participants caused by apsychologically unpleasant stimulus?

What willyou do if participants exhibit signs of harm (for instance, crying, disorientedbehavior)?

Are youprepared to describe the purpose and nature of your study to your participantsduring debriefing?

Will youuse deception in your study? If so, what will you tell participants duringdebriefing?

Are youaware that participants can quit your study at any time? If a participant doesdrop out, will you give your participants credit for participating? Is thisfact stated on the informed consent form? Is this fact part of yourinstructions to the participants?

Whateducational gain do you think participants will obtain from participating inyour study?

How will you ensure the confidentiality of eachparticipant’s data?

Beyondthe Proposal: The Pilot Study

Even afteryou have carefully designed your study, modified it based on comments from yourinstructor, and been given your professor’s go-ahead to run it, you maystill want to run several participants (friends, family members, other membersof the class) just for practice. By running practice participants, you will getsome of the “bugs” out of your study. Specifically, by running anddebriefing practice participants, you will discover

1. whether participants perceived yourmanipulation the way you intended;

2. whether you canperform the study the same way every time or whether you need to spell out yourprocedures in more detail;

3. whether you areproviding the right amount of time for each of the research tasks and whetheryou are allowing enough time in between tasks;

4. whether yourinstructions were clear;

5. whether your coverstory was believable;

6. whether you need torevise your stimulus materials;

7. how participantslike the study; and

8. how long it takes you to run anddebrief a participant.

In short,running practice participants helps you to fine-tune your study. Becauserunning practice participants is so useful, many professional investigators runenough practice participants to constitute a small study—what researcherscall a pilot study.

Conductingthe Actual Study

The dressrehearsal is over. You have made the final changes in your procedures and yourproposal. Now you are ready for the real thing—you are ready to conductyour study. This section will show you how.

Establishing Rapport

As you may imagine, some of your prospective participants may beapprehensive about the study. Participants often are not sure whether they arein the right place, or even whether the researcher is a Dr. Frankenstein.

To put yourparticipants at ease, let them know they are in the right place, and becourteous. You should be both friendly and businesslike. The expertinvestigator greets the participant warmly, pays close attention to theparticipant, and seems concerned that the participant knows what will happen inthe study. The expert investigator is obviously concerned that each participantis treated humanely and that the study is done professionally.

Beingprofessional does not hurt how participants view you. Why? First, mostparticipants like knowing that they are involved in something important.Second, some will view your professionalism as a way of showing that you valuetheir time—which you should.

So, how canyou exude a professional manner? Some novice investigators think that theyappear professional when they act aloof and unconcerned. Nothing could be lessprofessional. Participants are very turned off by a disinterested attitude. Theyfeel that you do not care about the study and that you do not care about them.

To appearprofessional, you should be neatly dressed, enthusiastic, well-organized, andprompt. “Prompt” may be an understatement. You should be ready andwaiting for your participants at least 10 minutes before the study is scheduledto begin. Once your participants arrive, concentrate exclusively on the job athand. Never ask a participant to wait a few minutes while you socialize withfriends.

What do youlose by being a “professional” investigator? Problem participants.If you seem enthusiastic and professional, your participants will also becomeinvolved in doing your study—even if the tasks are relatively boring.Thus, if you are professional in your manner and attitude, you will probablynot even have to ask the participants to refrain from chatting throughout thestudy. Similarly, if you are professional, participants will stop askingquestions about the study if you say, “I will explain the purpose at theend of the study.”

After youhave established rapport, you need to give your participants instructions. Toget participants to follow instructions to the letter, you might

1. be repetitive,

2. have participants read theinstructions,

3. orally paraphrase those instructions,

4. run participants individually,

5. invite participants to askquestions, and

6. have participants demonstrate thatthey understand the instructions by quizzing them or by giving them a practicetrial before beginning the study.

Once thestudy has begun, try to follow the procedure to the letter. Consistentlyfollowing the same procedures improves power and reduces the possibility ofbias. Therefore, do not let participants change your behavior by reinforcing orpunishing you. For instance, imagine you are investigating long-term memory. You want to expose participants toinformation and then see what they can write down. However, if you do this,participants may be writing down information that is in short-term memory. Thus, you would not be assessinglong-term memory. Therefore, you add a counting backwards task that shouldvirtually eliminate all of the information from short-term memory.Specifically, in your memory study, participants are exposed to information,are supposed to count backwards from a number like 781 by 3s for 20 seconds,and then are asked to recall the information. Ideally, their recall willrepresent only what they have in long-term memory. Unfortunately, manyparticipants will aﬁnd the counting task unpleasant, embarrassing, orsimply an unwanted nuisance. Consequently, some participants will thank you fortelling them they can stop; others will plead nonverbally for you to stop. Clearly,you cannot let any of these strategies stop you from making them countbackwards for the full 20 seconds. If you vary your procedures from participantto participant based on each participant’s individual whims, your studywill have questionable validity.

Debriefing

Once the study is over, you should debrief your participants. Indebrieﬁng, you should ﬁrst try to ﬁnd out whether theparticipants suspected the hypothesis. Simply ask participants what theythought the study was about. Then, explain the purpose of your study.

If youdeceived your participants, you need to make sure they are not upset about thedeception. You also need to make sure that they understand why deception wasnecessary. Participants should leave the study appreciating the fact that therewas one and only one reason you employed deception: It was the only way to getgood information about an important issue.

Making sureparticipants accept your rationale for deception is crucial for three reasons.First, you do not want your participants to feel humiliated or angry. Second,if they get mad, they may not only be mad at you, but also at psychologists ingeneral. Perhaps that anger or humiliation will stop them from visiting apsychologist when they need help. Third, the unhappy participant may spread theword about your deception, ruining your chances of deceiving otherparticipants.

Afterexplaining the purpose of the study, you should answer any questions theparticipants have. Although answering questions may sometimes seem like a wasteof time, you owe it to your participants. They gave you their time, now it isyour turn.

Afterparticipants’ questions and doubts have been dealt with, give them anopportunity to rate how valuable they felt the study was. Letting participantsrate your study encourages you to be courteous to your participants, ( lets youknow whether your study is more traumatic than you originally thought, and makes participants feel that yourespect them because you value their opinions.

Afterparticipants rate your study, you should assure participants that their responsesduring the study will be kept conﬁdential. Tell them that no one but youwill know their responses. Then, ask the participants not to talk about thestudy because it is still in progress. For example, you might ask them not totalk about the study until next week. Finally, you should thank yourparticipants, escort them back to the waiting area, and say goodbye.

ProtectingData: Confidentiality

You might think that once a participant leaves the study, yourresponsibilities to that participant end. Wrong! You are still responsible forguaranteeing the participant’s privacy. Knowledge about a givenparticipant is between you (the investigator) and the participant—noone else. Neverviolate this conﬁdentiality. To ensure conﬁdentiality, you shouldtake the following precautions:

1. Assign each participant a number.When you refer to a given participant, always use the assignednumber—never that participant’s name.

2. Never store a participant’sname and data in a computer—this could be a computer hacker’sdelight.

3. If you have participants writetheir names on booklets, tear off and destroy the cover of the booklet afteryou have analyzed the data.

4. Store a list of participants andtheir numbers in one place and the data with the participants’ numbers onit in another place.

5. Watchyour mouth. There is rarely a reason to talk casually about aparticipant’s behavior. Even if you do not mention any names, otherpeople may guess or think they have guessed the identity of your participant.We realize that it is hard to keep a secret. But to talk freely about someonewho participated in your study is to betray a trust. Furthermore, keepingsecrets will, for many of you, be an important part of your professional role:Therapists, researchers, consultants, lawyers, and physicians all must keeptheir clients’ behaviors confidential.

[1] Having more participants will giveyou more power. Indeed, some (Cohen, 1990) would consider 64 participants pergroup to be a reasonable minimum. We have talked about minimums. Are theremaximums? Could you have a design that was too powerful? Some would argue that,in some cases, researchers use so many participants that even the smallest ofeffects, no matter how practically and theoretically insignificant, would bestatistically significant. However, having an overpowered design is rarely aproblem for novice researchers.

[2] You may want to consult with yourprofessor as to the type of ANOVA you should use. Experts argue about whetherone should use a conventional ANOVA model or a “random effects”ANOVA model.