 |
| | | | | | |
Mass Media and Politics:
A Social Science Perspective,
First Edition
Jan E. Leighley, Texas A&M University
| | | | | | | | | | | |
Appendix B
By Christina Suthammanont
Experimental Research. Experimental research differs from quantitative analysis in two fundamental ways. First, experimenters have more control over their research because they directly manipulate the independent variables (i.e. factors that they think influence an outcome). In quantitative studies, such as those relying on survey research, researchers do not have direct control over the independent variable(s). Moreover, experimental research provides researchers with a method of testing theories under controlled conditions and in a unique manner.
At the most basic level, experiments involve taking action and observing the consequences of that action. For example, you may be curious to find out how your best friend will react if you say you don't like her brand new car. The observation you may make is that you no longer get a ride to class. However, the most conventional type of experiment has three pairs of components: (1) independent and dependent variables, (2) pre-testing and post-testing, and (3) experimental groups and control groups.
An independent variable is the item that the researcher controls or manipulates. In the example above, it is whether you do or do not tell your friend how you feel about her new car. Whether you tell or do not tell will (presumably) influence whether you continue to get rides to class; a ride is the dependent variable. In political science, you may want to know how people will respond (dependent variable) to different colored voting ballots (independent variable). In short, the independent variable is the cause and the dependent variable is the effect.
Pre-testing and post-testing is when participants are measured, perhaps through the use of questionnaires or observation, in terms of a dependent variable, (pre-tested), exposed to a stimulus (the independent variable), and then measured again in terms of the dependent variable (post-tested). Differences between the first and last measurements on the dependent variable are then attributed to the independent variable (Babbie 1998, 234). If you kept track of the number of rides your best friend gave you in the week before you told her that you didn't like her new car and then counted the number rides she gave in the following week, the difference between the two weeks could be attributed to the fact that said you didn't like her new car.
The third pair of components in an experiment is the presence of experimental groups and control groups. Participants within the experimental group receive the treatment, or variation of the independent variable, while participants in the control group do not. Let's say you want to know whether prejudice toward African-Americans is the result of ignorance of African-American history. In order to test this theory you first need to establish levels of prejudice. To begin, each group -- the experimental and the control groups -- are given a questionnaire designed to measure prejudice against African-Americans. While the experimental group is shown a documentary on African-American history, the control group is not shown the documentary on African-American history. But both groups are subsequently given a post-test of prejudice. This approach allows the researcher to compare the experimental group's levels of prejudice after exposure to the documentary to the control group's levels of prejudice (Babbie 1998, 235).
Using the control group allows the researcher to detect any effects of the experiment itself upon the participants, which is known as the Hawthorne effect. The Hawthorne effect occurs when participants in experiments are affected by the experimental procedure itself. The realization that the experiment might be producing an effect is the result of a study on employee satisfaction conducted by Roethlisberger and Dickson in 1939. These researchers studied working conditions in the telephone "bank wiring room" of the Western Electric Works in the Chicago suburb of Hawthorne, Illinois. The researchers discovered that if they brightened the room, worker productivity increased. Upon further brightening, productivity increased more. When the lights were dimmed, productivity increased again! The workers were responding more to the attention of the researchers than to the level of brightness in the room. As a result of this experiment, researchers realized that they needed to expose one group to a treatment (such as levels of brightness) while maintaining one group that did not receive the treatment.
In addition to the Hawthorne effect, experimenters must guard against pre-determining the results of the experiment. Just as participants may respond to the experiment itself, researchers may unwittingly influence the outcome of an experiment with researcher bias. One method of guarding against this is to use the double-blind experiment in which neither the experimenter nor the participants know which is the experimental group and which is the control group. Let's say you want to know whether reading liberal or conservative newspapers will make people more or less liberal or conservative. In this case, the experimenter may be friendlier toward the group receiving the newspaper that is closer to her own ideological preferences. As a result, the group treated in a more friendly manner might "become" even more liberal or conservative than otherwise would have been the case while the group treated less friendly would provide even less favorable responses due to the difference in treatment. The double-blind experiment prevents this phenomenon by the virtue that the researchers who are responsible for administering the newspapers and noting changes in ideological leanings would not be told which group is receiving the liberal newspaper and which is receiving the conservative newspaper.
Using the double-blind experimental process decreases sources of internal invalidity. Internal invalidity refers to the possibility that the conclusions drawn from experimental results inaccurately reflect what has gone on in the experiment itself. The potential for internal invalidity is present whenever anything other than the experimental stimulus (i.e. the independent variable) can affect the dependent variable. Some factors that may skew the results of an experiment are history, maturation, testing, instrumentation, selection biases, experimental mortality, and diffusion or imitation of treatments.
Historical events may occur during the course of the experiment that will confound the experimental results. In the example of the liberal-conservative newspaper experiment, if both types of newspapers contained stories of the Rodney King beating then both groups may become more liberal despite the conservative orientation of one of the newspapers. Maturation occurs when the experiment takes place over relatively long periods of time and, during the experimental process, the participants grow and change. These changes may affect the results of the experiment. Results of an experiment that is initiated at the beginning of the semester and concludes at end of the semester, for example, may be affected by the knowledge and experience that students gain during the semester. The process of testing and retesting will also influence people's behavior. In the prejudice pre-test, for example, the control group may demonstrate lower or higher levels of prejudice simply as a function of completing the prejudice questionnaire. This is one reason why experimental questionnaires and surveys must be carefully constructed to obscure the purpose of the experiment. Yet, the way in which the experiment is carried out may also affect the results. This is known as instrumentation. For example, if the post-test differs from the pre-test then any differentiation cannot only be attributed to the stimulus. Diffusion or imitation of treatments is when the control group is "contaminated" by the experimental group. If the control group in the prejudice experiment, for example, overhears participants in the experimental group discuss the documentary on African-American history, the results of the control group may mirror, or imitate, the experimental group.
Still another source of internal invalidity is that of selection biases. Comparisons across groups are meaningless unless they are comparable. According to Babbie "the cardinal rule of subject selection and experimentation concerns the comparability of experimental and control groups" (1998, 237). In other words, experimental and control groups should be as similar as possible. There are several ways to accomplish well-balanced groups. Randomization is one process by which experimenters may achieve comparable groups. From the total number of participants, the experimenter randomly assigns them either to the control or to the experimental group. Randomization may be accomplished by numbering all of the participants serially and selecting numbers through a random number table, or the experimenter might assign the odd-numbered subjects to the experimental group and the even-number participants to the control group. However, successful randomization is largely contingent on the total number of participants. It hardly makes sense to try to randomize placement of a total of six participants.
An alternative to the randomization process is that of matching. This process closely resembles the process of quota sampling. Let's say you have 12 white males show up for your experiment. Quota sampling means that you would place six in the experimental group and six in the control group. Matching, on the other hand, is achieved through the creation of a quota matrix that is constructed based on all of the most relevant characteristics of your sample. Ideally, the matrix would be constructed in such a way that an even number of participants end up in each cell; this even number is distributed equally between the control and experimental groups (see Babbie 1998, 239 for illustration). Whether through randomization, matching, or quota sampling, it is crucial to the internal validity of an experiment that the groups are comparable.
These potential sources of internal invalidity are extremely important considerations in experimental research. Historical events, maturation, testing and retesting, instrumentation, selection biases, experimental mortality, and diffusion or imitation of treatments must each be considered when designing and executing an experiment. As Campbell and Stanley note, "internal validity is the basic minimum without which any experiment is uninterpretable" (1963, 5).
In addition to internal validity, there is also the issue of external validity. External validity refers to the generalizability of your sample population, which is the total number of participants in your experiment. It asks the questions of to what populations, settings, treatment variables (independent variables), and measurement variables can your effect be applied, or generalized? If all of your participants are white, male, undergraduates, can the results of your experiment be generalized to the real population comprised of diverse races, ethnicities, ages, educational levels, and females? Are your results limited to a particular college campus at a certain time of year or can they be applied to any university campus year-round?
At least one of the factors that challenge external validity is the same as that which poses a problem for internal validity. Testing and retesting may affect external validity because it is unlikely that the general population, to which you wish to generalize your results, would be tested on the same issue twice and under such conditions. That which Campbell and Stanley call the reactive effects of experimental arrangements precludes generalization about the effect of the independent variable upon people in a non-experimental setting. For example, in the colored ballot experiment, participants may vote for a different party when the ballot is red as opposed to blue but in the real world it is unlikely that the color of the ballot will dissuade voters from voting for their preferred candidate. Another potential source of external invalidity is a follow-up to the reactive effects. Within the experiment a participant may be exposed to multiple treatments and this may lead to multiple-treatment interference. This is likely to occur because the effects of prior treatments are not usually erasable. Thus, once the experimental group in the prejudice experiment views the documentary they cannot be expected to "erase" the documentary from their mental bank.
Both issues of internal and external validity are important to generating reliable, informative, scientific experiments. However, of the two, internal validity is essential; without it, the results of an experiment are unreliable and, therefore, useless for the advancement of research. While the issue of external validity will never be entirely resolved, the experimenter must take steps to ensure that the highest level of external validity is achieved.
Some people may have had the impression that experimental research consisted only of electrocuting subjects or psychological evaluations. While early experimental research certainly incorporated some of these elements into their designs, experimentation has evolved -- some might say matured -- into one of the most fascinating methods available for research of political theories and questions that cannot always be answered with data sets.
| | | | | | | |
|
|
|
