BACK TO OVERVIEW

RESEARCH PROBLEM STATEMENT DUE SEPTEMBER 23 WILL YOU GATHER DATA THIS SEMESTER? IF YES, YOU MUST APPLY TO FSU'S HUMAN SUBJECTS COMMITTEE. CLICK HERE FOR MORE INFO. FSU'S LAB SCHOOL HAS ITS OWN HUMAN SUBJECTS COMMITTEE. SEE ME FOR MORE DETAIL.

BACK TO SYLLABUS

Most social psychologists are empiricists, that is, we test out our conceptually-derived hypotheses with systematically gathered data. If the data contradict or disconfirm our hypotheses, then typically we revise and retest the theory. On the other hand, social psychologists are eclectic. Most of us use a variety of methods, such as experiments or surveys, depending on the topic at hand. While it is true that psychologists are more likely to use experimental methods and sociologists are more likely to use survey research, ethnographies, or observational methods, we tend to switch methods depending on the kind of research we do.

Let me give some personal examples.

• When I wanted to study group processes in religious congregations, I used participant observation. I coordinated over 30 observers in 38 churches and synagogues, who attended weekly services, socials, and other congregational events. We gathered data on interaction patterns, the formation of smaller groups within each congregation, how often individuals attended, and we monitored invitations to lunch, conregational parties, and other occasions. I used participant observation because I wanted to see how each group interacted in a natural setting.

• When I study science attitudes, generally I work with public opinion survey research data. While there are a variety of ways to study how someone feels (including unobtrusive observational methods), probably the most direct method is to ask them! Survey questionnaires allow the researcher to gather considerable amounts of information in a relatively short time. Further, I am dealing with individuals, not groups, which makes it appropriate to use an individual questionnaire.

• On the other hand, people often are unable to tell you what reasoning processesthey use because these processes may be preconscious (there is interesting work about training people to reason aloud, but that is for another day) or poorly understood by the individual. In such instances, I have used experiments, varied the conditions, and assessed how reasoning was affected.

Causality is critical to the research enterprise. Much of the research process centers around designating the "true causal" or “independent variables.” What we initially may consider to be “true causal” variables may, instead, turn out to be artifacts of the research process (e.g., evaluation apprehension in response to being an experimental subject), the particular group that we studied (college undergraduates), or thinking that we had measured variable one while we had, in fact, studied variable two. A great deal of behavioral science consists of ruling out alternative causes or explanations.

Establishing causality is by no means agreed-upon even among  scientists, let alone most members of a particular culture. If you subscribe, at least in part to the "rational laws" approach, you also probably accept controlled experiments, logic, "reasonable arguments,” and statistical control as suggestive of "proof." Science is one form of knowing and one generic way of gathering evidence that either disconfirms or suggests causality.

According to science rules, definitive proof via empirical testing does not exist. Science uses the term "proof" (or, rather, "disproof") differently from the way attorneys or journalists do. Our measurements could be later shown to be contaminated. A correlation could have many causes, only some of which have been identified. Later work can show earlier causes to be spurious, that is, both cause and effect depend on some prior causal (often extraneous) variable. Statistics are NEVER EVER considered to "prove" anything although statistical results CAN disconfirm.
 

In order to make any causal assessments in your research situation, you must have reliable measures, i.e., stable measures. If the random error variation in your measurements is so large that there is almost no stability in your measures, you can't explain anything! Picture an intelligence test where an individual's scores ranged from moronic to genius level. No one would place any faith in the results of such a "test" because the person's scores were so unstable or unreliable.

Reliability is a requirement to make statements about validity. However, reliable measures could be biased and hence "untrue" measures of a phenomenon) or confounded with other factors such as acquiescence response set. Picture a scale that always weighs five pounds too light. The results are reliable, but inaccurate or biased. Or, picture an intelligence test on which women or people of color always score lower (even if this doesn't occur on other tests). Again, the measure may be reliable but biased.

Internal Validity

Internal validity addresses the "true" causes of the outcomes that you observed in your study.Strong internal validity means that you not only have reliable measures of your independent and dependent variables BUTa strong justification that causally links your independent variables to your dependent variables. At the same time, you can rule out extraneous variables, or alternative, often unanticipated, causes for your dependent variables.Thus strong internal validity refers to the unambiguous assignment of causes to effects. Internal validity is about causal control.

Laboratory "true experiments" have the potential to make very strong causal control statements. Random assignment of subjects to treatment groups (see below) rules out many threats to internal validity. Further, the lab is a controlled setting, very often the experimenter's "stage." If the researcher is careful, nothing will be in the laboratory setting that the researcher did not place there. When we leave the lab to do studies in natural settings, we can still do random assignment of subjects to treatments, but we lose control over potential causal variables in the study setting (dogs bark, telephones ring, the experimental confederate just got run over walking against the "don't walk" sign on Stadium Drive.)

External Validity

External validity addresses the ability to generalize your study to other people and other situations. To have strong external validity (ideally), you need a probability sample of subjects or respondents drawn using "chance methods" from a clearly defined population (all registered students at Florida State University, for example). Ideally, you will have a good sample of groups (e.g., classes at all ability levels). You will have a sample of measurements  and situations (you study who follows the confederate who violates the "don't walk" sign on Woodward Avenue at different times of day, different days, and different locations on campus.)  When you have strong external validity, you can generalize to other people and situations with confidence.
 

Each major method that social psychologists use has its advantages and disadvantages. A method may work better to study some areas than others. For example, it is critical to assess whether you are studying individuals or groups. A method that can work well to study individuals, such as a survey questionnaire, can work poorly to study groups.

EXPERIMENTS

• stress CAUSAL CONTROL (internal validity) both in the study setting (think stage play) and in manipulating independent variables (causes).
• A "true experiment" randomly assigns subjects to treatment conditions, that is, the treatment an individual receives is determined by a coin flip, a random number table, or some other similar chance device.  Which treatment someone receives is basically determined by a process akin to a lottery.
• Random assignment or "randomization" controls differences in life experiences, etc. In randomized groups, ON THE AVERAGE, there will be about the same percentage of women, the same percentage of highly test anxious people, the same percentage of good athletes, the same percentage of "high risk" people, etc.

However, laboratory experiments have low mundane [everyday] reality. Most laboratory situations do not resemble the circumstances that subjects encounter in their everyday lives.

Lab subjects typically have too similar backgrounds (e.g., college students or military recruits are about the same age and the same educational level) to allow high external validity.

Lab experiments are reactive (obtrusive). Subjects are acutely aware that their behavior is under scrutiny. They may react with anxiety, with the desire to "fake" good (social desirability) and appear smarter, more attractive, or more tolerant than they typically do.

Better use a manipulation check to be sure that on some level the subjects noticed the manipulation. If you manipulate a variable and the different treatments never register, the treatment probably won't make a difference. For example, if you use a "buddy system" in a stop-smoking group, we would expect the subjects to rate the group as friendlier than subjects who don't receive the "buddy system" treatment.

Field experiments

• occur in natural settings but still preserve random assignment of subjects to treatments correct some of these problems.
• have less control over the study setting than laboratory experiments.
• are typically short in duration, making it difficult to research complex, in-depth events with this method. It will be impossible to ascertain changes over time.
• since you usually do not even have the names of experimental subjects, any kind of followup becomes impossible. This is critical if the phenomenon you are studying has "sleeper effects," i.e., the effects of a treatment take time to develop and may not be apparent until days, or even weeks, after the initial treatment period.

• In these methods, data are collected in natural settings.
• They have high mundane reality and greater INSIGHT, especially participant observation, which usually studies groups in-depth.
• In field observations, participant observation, or natural experiments, we  observe and infer rather than manipulate causal factors so these methods have lower causal control.
• Participant observers join the studied group and share many group activities. Maintaining empathy and objectivity simultaneously is difficult in this case. Participant observation is typically one facet of conducting an ethnography, or a description of group culture.
• In natural experiments, you have virtually no causal controlover the independent variables. A natural experimental study takes advantage of a naturally occurring event and may, at best, have limited access to data before the event occurred. For example, in the late-1990s, Texas graduate and professional schools were forbidden by state law to consider ethnicity in admissions. Within one year after passing this law, Hispanic and Black applications to the University of Texas Law School dropped by 25 percent. The guess is that changes in the law caused changes in the ethnic composition of applicants--but without a control group, this conclusion is a tenuous one. Other factors could have been responsible for this drop.
• In field observations, observations using systematic categories for group or individual behavior are made in a natural setting. There is not the same kind of intense involvement as there is in participant observation and, in fact, observers may not even be noticed by research subjects or respondents. For example, someone who codes seatbelt wearing behavior may not be noticed by motorists.

Public opinion SURVEYS typically take REPRESENTATIVE SAMPLES thus have relatively high external validity. In surveys, questions are asked and the answers are recorded, sometimes by interviewers and sometimes using paper-and-pencil questionnaires.

• In most professional surveys, the results may be generalized to a known population with known levels of error.
• Establishing internal validity is fair and often depends on question wording. Response rate is important here too, especially if there is any indication that respondents differ from non-respondents.
• Mundane reality is medium. Survey questions often discuss a respondent's daily life BUT the interview itself can be  reactive and can evokedemand effects(e.g.,social desirability or evaluation apprehension).
• Standardized tests are a form of survey research and are subject to most of the same problems such as question format bias.

Researchers may use content analysis to study media or other documents. For example, you might study gender stereotypes in sports magazines or the frequency of unmarried couples cohabiting on television situation comedies. Causal control is low but mundane reality can be high. External validity can be high with careful sampling. Content analysis has been cautiously combined with surveys to show how watching television can influence adult attitudes and behaviors.

Simulations occur in constructed settings. Participants may role play or act "as if" they held certain roles such as a parent, a prison guard or head of state. A classic example is the Haney, Banks and Zimbardo prison study; you can access a slide show of this study at Philip Zimbardo's web site HERE.

• Since subjects may have limited exposure to such roles, their behavior will have low mundane reality and may be artificial.
• When participants are randomly assigned to the roles they play, the simulation is experimental and thus has high causal control.
• Simulations can be dramatic and INSIGHTFUL but the "in-role" behavior may reflect the fantasies that participants have about the role or TV portrayals if the roles are unfamiliar ones.This may have happened in the Haney, et al., prison study; the Stanford University students may have applied stereotypes about prisoners and prison guides that they learned from movies or TV.
• Conversely, simulations can be very revealing if the situations are ones the subjects have encountered first hand. Family role-playing is one example, wherein the subjects project their family experiences and expectations into the simulation situation.

The chart below is a synopsis of the relative advantages and disadvantages of each method. Notice that a strong point for one method, such as strong internal validity for laboratory experiments, is often balanced by a weak point, such as the limited external validity in most laboratory experiments because of the reliance on "captive" subjects such as undergraduate college students. This means that the wise researcher will use a variety of methods. It also means that you should carefully consider the evidence in the studies that you read about:

• Would you feel comfortable generalizing the data to different groups or individuals? Would the same results be expected for women and men? Youth and adults? Different ethnic groups? Did the researchers even consider different populations?
• How tight was causal control? Were control groups truly comparable to the experimental groups? How conscious were subjects that their behavior was being monitored?
• Was there any followup on subjects or respondents? Were the changes in behavior that were observed stable?
• Was the unit of study a group or an individual? Were the measures appropriate? Generally, surveys do not do a good job of describing group behavior. It is difficult to infer individual feelings from simply watching interaction patterns.

THE UNIT:Since we study individuals, but we also study groups and organizations, it is imperative to make your measures consistent with the unit you have chosen.

THE TIME FACTOR:Because some results fade quickly (e.g., your specific attitudes about almost anything political) and others are "sleepers" or increase over time, you should assess whether followup over time was needed--and whether it was done.

REACTIVITY:A classic social psychology study found that simply viewing oneself in a mirror changed behavior. Picture, then, the effects of being in a research study! Was there any control for reactivity? Sometimes that does mean we use deception in experiments. After all, if you told people "this experiment is to assess your attractiveness," we can be pretty sure that people would immediately "up their image."

NOTE: The chart below refers to the "usual" or "typical" professional  study. Any specific study, of course, may be "better" or "worse".

 

ALPHABETIC ABBREVIATION  IN CHART BELOW	TYPE OF METHOD
A	Laboratory Experiments
B	Field Experiments
C	Participant Observation (often ethnographic)
D	Field Observations
E	Surveys
F	Secondary Analysis
G	Simulations
H	"Natural" Experiments

METHOD	A	B	C	D	E	F	G	H
Causal Control  (Internal Validity)	Excellent	Very Good	Fair to Good	Fair	Fair to Good	Poor	Can be  Good	Poor
Control over Other Factors	Good	Fair	Fair	Poor	Fair	Fair	Can be  Good	Poor
Represents Population  (External Validity)	Often Poor	Fair to Good	Fair to Good	Fair	Very Good	Can be Good	Poor	Varies
Time Follow-up	Often Poor	Very Poor	Good	Fair	Can be Good	Can be Good	Poor	Can be  Good
Mundane Reality	Often Low	High	High	High	Fair to High	Varies	Low	High
Reactivity	High	Low	Middle	Middle	MAY be high	Low	High	Low
Unit of Study	Person or Group	Person or Group	Group	Person or Group	Person	**	Person or  Group	Often Group

**Varies with the original method of collecting data which is specific to each research study. For example, the unit may be a person, a group, a cultural reproduction such as a television show or magazine, or a created cohort.
 

TO OVERVIEW

TO SYLLABUS

This page was built with Netscape Composer.
It is best viewed with Netscape Navigator
600 X 800 display resolution.
Susan Carol Losh September 2 2002