47.269 Research I:  Basics

Factorial Designs

Lecture 1.  Thinking Inside the Box

This material is complicated.  It requires you to put together all of the other material we have learned so far.  Be prepared to read this through several times.

Let’s say you wanted to compare the ability to study under three different conditions:  quiet, instrumental music, instrumental plus vocal music.  What do you have?  A control condition (quiet) and two different treatments.  So far, so good.

Naturally you want the instrumental and instrumental+vocal conditions to be as much alike as possible on all other aspects.  So you decide to use the same music with and without the accompanying vocals for both of these conditions.

Now you decide how to measure how studying is evident in your experiment.  For this experiment the operational definition will be the number of correct answers to questions following each of 10 brief reading passages containing new information to the participants (i.e., reading comprehension).

You have 60 participants and randomly assign each to one of these three conditions.  You can picture placing their scores in one of the boxes below.  When the data were collected, each box would contain 20 numbers.  We would characterize the performance of that group by the mean and standard deviation (central tendency and variability) of those scores.

You can also think about these three conditions as three levels of one factor.  If you consider the factor to be background music, the levels would be quiet (i.e., no background music), instrumental only, and instrumental plus voice.   

When we think of them as one factor with three levels it is probably more appropriate to picture them like so:

Again, there are 20 scores in each of the boxes. 

Now that the boxes are lined up, they are often referred to as “cells.”

So far we have proposed research that:

· Is an experiment

· Is between subjects

· Has one factor (Independent variable) with three levels

But, wait.  You have heard about this Mozart effect for which there is evidence in children at least for some types of mathematics.  So, you decide to add another type of variable:  type of material to be learned.  Now you want to have half of your participants solving math problems and the other half solving reading comprehension (i.e., verbal) problems.  Every participant still winds up with a score, i.e., the total number correct out of 100.

What do you have now?  Two types of independent variables—or factors—instead of one.  How many subjects do you need now? How would you think about the boxes for another factor?

Now you have 6 cells arranged in a 2x3 configuration of (A) Material by (B) Background Music. Notice that we put Material first; we simply put the factor or variable with the lower number of levels first by convention.

If you still have 20 participants per cell, you will now have 120 (2x3x20) participants.   Think about what that allows you to do.  You can look at

· Average math compared to verbal scores regardless of what is going on in the background, or one comparison of 60 vs. 60 scores (All Math vs. All Verbal).  We think of this as the main effect of type of material.

· Average quiet, instrumental and instrumental plus verbal scores regardless of what type of material they come from, or three comparisons of 40 vs. 40 (Q vs. I, Q vs. I+V, I vs. I+V).  We think of this as the main effect of type of background music.

· Whether the average score depends on both factors somehow, not just the type of material but the type of music.  If, for example, the music made a difference for math but not for verbal, then the effect of music would also depend on the level of material used.  If scores were higher for Instrumental plus Vocal compared to Instrumental alone when it was verbal, but lower when it was math, then that score also depends on the value of both factors.  We think of these kinds of effects as the interaction of (A) background music and (B) type of material.  These are sometimes referred to as AxB interactions.

Wait, wait, wait.  Everyone knows that college students differ in their math and verbal abilities. Maybe it would be interesting to see if there are effects of the major of the student participant. Let’s look at three different majors.  Again, we can look for main effects as well as interaction effects.

How to picture this 2x3x3 study?  Add a third dimension of course.  Notice that we now have our cells stacked three deep to represent each of the types of majors we have chosen (note that major is not an independent variable to which you can randomly assign participants, but an antecedent variable).  How many participants do we have now if we want to keep 20 in each condition?  2x3x3x20 = 360.  So 360 scores would be distributed among the 18 cells below.

So, are you done yet? 

Nope.  You find empirical evidence out in the literature that suggests that gender—again, an antecedent rather than an independent variable—might play a role in verbal compared to math skills.  So, you decide to examine gender too.  That will add one factor with two levels.

How do we draw in the fourth dimension?  Well, we can repeat the boxes as below. 

Males (n = 360)

Females (n = 360)

Our experiment has become a 4 factor design and no longer a pure experiment.

It is a 2x2x3x3 factorial design for (A) Gender x (B) Material x (C) Background Music x (D) Major.  Notice that we can look at main effects for A, B, C, or D by averaging across the other factors. We can also examine AxB interactions, AxBxC interactions, or AxBxCxD interactions as well as any subset thereof (e.g., AxC or BxD or BxCxD).

Notice also that this is a design with both experimental and correlational features.  We randomly assign and manipulate background music and type of material, but gender and major are already characteristics of our participants. We sometimes call this a quasi-experiment.

Now how many participants do we have? 2x2x3x3x20 = 720.  That is a lot of person hours!

We could reduce variance and decrease the number of participants by introducing a within subjects component.  We have two variables that are candidates.  Gender and Major??  Nope.  We cannot randomly assign gender or major either practically or ethically.  But we could have the same subjects do math and verbal tests within their background music condition.  That would give us a 2x2x3x3 design with three between- and one within-subjects factors.

So, how many subjects do we need now?  Remove the 2 conditions of material from the equation because the same people are in both conditions.  We wind up with B x C x D x n or 2x2x3x3x20 or 360 participants.

In fact, if we tested the same people in both Math and Verbal under conditions of Quiet, Instrumental Music, and Instrumental plus Vocal Music, then we would have two within and two between factors and we would go back down to 120 participants (2 x 3 x 20 = 120).

So why on earth might you choose not to?  If you were very worried about practice effects, you might choose a between subjects design.  Even if you are not so worried, you must clearly counterbalance the order and sequence of conditions that each participant experiences across the participants (within major and gender) to be able to gauge the contribution of those effects.

When you think about a research design, comparing groups or conditions, try to think inside these boxes!