LING440 – LAB3c
In Lab 3c, you will analyze the self-paced reading data that you collected in Lab 3b. In order to make sure that you have a reasonably large dataset to analyze, we recruited 12 additional subjects from Mechanical Turk to supplement the 15 subjects that the class collected, for a total of 27. Two of them had accuracy levels below 80%, so we have excluded them from the current dataset, for a total of 25 subjects.
This lab will assume that you are using Microsoft Excel for analysis, although you are welcome to use any similar kind of program (contact me if you need the data in a different format). Note that different versions of Excel have slightly different menu options, so don’t freak out if the screenshot in the lab is not identical to yours. Try using common sense or googling ‘pivot tables’ for your Excel version first, but if you can’t figure out how to do something in your version, let us know and we can help!
For convenience, below we re-copy the logic of the experimental design that you are analyzing.
1a SgAttGr: The slogan about the husband was designed to get attention.
1b SgAttUngr: The slogan about the husband were designed to get attention.
1c PlAttGr: The slogan about the husbands was designed to get attention.
1d PlAttUngr: The slogan about the husbands were designed to get attention.
1e CommaAttGr: The slogan about the kind, caring husband was designed to get attention.
1f CommaAttUngr: The slogan about the kind, caring husband were designed to get attention.
The first four conditions will serve as controls and will let us determine whether our population is showing the standard agreement pattern from previous literature: the slowdown between SgAttGr and SgAttUngr should be larger than the comparison between PlAttGr and PlAttUngr, because people should erroneously retrieve the non-subject plural from memory.
Conditions e and f allow us to test a new prediction based on recent new work by Zoe Schlueter. She found that you also observe a moderate number of errors even when the attractor is not plural, if it has an and in it: The slogan about the kind and caring husband were designed to get attention. Hypothesis 1 to explain this data is that people use and as a shallow (good-enough) cue to plurality. Hypothesis 2 to explain this data is that people just stop noticing agreement errors at all if the subject and the verb are too far away from each other.
Q1: These hypotheses make different predictions for what will happen in the class experiment in conditions e and f. Explain what outcome each hypothesis would predict.
Preliminaries: Data structure and Pivot Tables
This section will give you an understanding of how the data is structured and how to use the Pivot Table tool, for doing the rest of the lab. If you happen to already know how to use Pivot Tables, you may be able to skip some of this part.
1. Open ‘Lab3_S16_data.xlsx’ in Microsoft Excel.
2. To understand the structure of the datafile, we should think about the experimental task, and what data was recorded during the experiment. In the experiment, participants read one sentence at a time. Individual words in the sentences were masked, and participants pressed the spacebar to reveal one word at a time. Every time they pressed the spacebar, the computer recorded their reaction time (how long it took them to press it). So, for every word in every sentence that a participant saw, we have a RT number. We also get an RT for their response to the question at the end of the sentence. Each individual RT recording gets its own line in our datafile.
So, for every subject in the experiment, there are n lines of data, which is calculated by the equation below. S is the number of sentences a subject saw, and R is the number of regions per sentence.
n = S x R
Note that in the class experiment, sentences varied in length; some sentences had only 8 regions (10 for the adjective condition) and some were longer and had more regions. In the Lab3c datafile, we have removed regions that aren’t shared by all sentences, so every sentence will have data from the 8-10 common regions only.
The datafile you have in front of you is just a series of individual recordings (RTs), tagged with information about where it comes from. Information is stored in the columns:
- ‘sentence’ == This column gives the whole sentence that the data point came from
- ‘subj’ == This column specifies the subject that the data was collected from.
- ‘item’ == a numeric code for each sentence item
- ‘cond’ == the code for the experimental condition the sentence was from
-‘cond_name’ == the name for the condition
-‘lurenum’ == whether the ‘lure’ or ‘attractor’ phrase was singular or plural
-‘verbnum’ == whether the verb was singular or plural
- ‘reg’ == the numeric region within the sentence that the RT corresponds to
- ‘reglabel’ == the label for the region within the sentence that the RT corresponds to
- ’word’ == the word presented or the button response (Y or N)
- ’RT’ == the amount of time spent on the word before the button-press, in milliseconds
In this ‘long’ format, it is very hard to see any structure in the data, right? But Excel has a magical tool called ‘PivotTables’ which we are going to be using to do most of this lab assignment.
3. Creating a PivotTable:
- Make sure your cursor is on some cell that has data in it.
- Select “Data > PivotTable ...” from the menu bar.
- Specify the range of data you want to use. Excel will often specify all of the relevant data by default.
- Here or in the next step, you’ll need to decide where to place the PivotTable. It’s a good rule to always place it in a new worksheet to avoid confusion. It’s often a good idea to rename that worksheet later so you can keep everything straight.
- Now you need to select the layout of your PivotTable. In older versions of Excel, you need to press the ‘Layout’ button that will appear after you press ‘Next’ on the earlier window. In newer versions of Excel, the empty data frame and PivotTable options window will pop up automatically.
Choosing the layout is a critical step which you will be messing around with a lot in this lab, so you should get familiar with this step. In the “Field name” box of the PivotTable window, there should be a list corresponding to the column names from the datafile. You can now drag-and-drop these menu items into the appropriate position to create your desired tables. The three choices are, what will the columns be, what will the rows be, and what will the actual values be that you put in the table.
In this lab, you’re basically always going to be interested in reaction time measures, so you will always be dragging ‘RT’ into the box labeled ‘Values’ or ‘Data’, depending on your version of Excel. Most of the time you will be interested in looking at different kinds of RT averages, so you need to make sure that you change from the usual default of ‘Sum’ or ‘Count’ to ‘Average’. You can do this by double-clicking on ‘RT’ in older versions of Excel or just by clicking on the Values subfield in new versions of Word.
Most of the interesting choices will be in what you decide for your rows and columns. For example, let’s say I wanted to create a table that showed me the average RT of words in a sentence by condition. I would just drag “condition” into my “ROW” subfield in the PivotTable Menu, and “RT” into my “VALUES” subfield. If I wanted the average RT by condition and item, I might drag ‘condition’ into my “COLUMN” subfield and ‘item’ into my “ROW” subfield.
Now you know how to create pivot tables!
Part 1: Filler-gap Experiment
If you closed it, re-open ‘Lab3_S16_data.xlsx’ .
Q2. As a first order of business, before looking at our contrasts of interest, let’s take a moment to consider the between-subject variability in our data. Create a pivottable that displays average RTs for each subject (note: sometimes it’s hard to look at the numbers with lots of decimal points, so you may want to highlight the values and change the cell formatting to get rid of the decimals). Include a screenshot of this table. Are there any values that seem drastically different from the others? Why might this be? Is this a concern?
Q3. Similarly, create a pivottable for average RT for each item and include a screenshot. What does this show us? Are there any values that seem drastically different from the others? Why might this be?
Q4. So far we have been averaging RTs across all the words in all the sentences to get a birds-eye view of the data. Now, before we go on, take a minute to think about the logic of this experiment. What region(s) of the sentence are going to be critical for answering these questions, if we break the sentences up as indicated below? What conditions are we comparing between, and what pattern of RTs would you expect to see across conditions, based on previous studies?
1
Det_1 / 2
N_1 / 3
prep / 4
Det_2 / 5
N_2 / 6
V / 7
V+1 / 8
V+2
Now create a cond_name x reg pivottable, so that we can look at the by-region RTs between each conditions (hint: make condition the ‘column’ and region the ‘row’ for a cleaner table, as in illustration below). Note that conditions e and f have two extra regions for the adjectives, and that means that for those two conditions the verb is region 8 rather than region 6.
To make it easier to get a quick view of the data you will now plot the RTs across regions. Eventually you will want to make three separate plots. This is because the three different attractor conditions each have different words before the critical region, so this could cause differences in RT that have nothing to do with the question of interest. So you will want one plot for SgAtt conditions, one plot for PlAtt conditions, and one plot for Comma conditions.
However, we want to start out by just looking at the singular attractor conditions a and b (the ‘control’ conditions) to see if there are any signs of trouble in our data. Copy the data for those two conditions to a separate area of the spreadsheet below, as illustrated in the example. Don’t include the ‘Grand Totals’.
Q5. Now, highlight the new sub-table and select the line graph option from the Chart menu to plot the data. Include a screenshot. What might immediately strike you as odd about this pattern of data?
Sometimes just a couple extreme data points—outliers—can have an outsize effect on your data. Go back to the original sheet with all the data (Sheet1), click anywhere in the data, and go to Data -> Sort, to sort by RT.
Q6. Scroll all the way to the bottom to look at the longest RTs—on the order of 2-77 seconds. Can you imagine any possible explanation for these extreme RTs? What impact might this have on our results?
As a rule of thumb for this lab, we’ll say that anything above 2 seconds on a word is likely to reflect responses that are not indicative of normal reading. You should see about 20 data points like this, which is just a tiny fraction out of the total number of ~7600 data points. Go ahead and delete these rows.
Q7. Now create a new region x condition pivot table, copy the data for the singular attractor conditions a and b into a new sub-table, and create a new plot. Include a screenshot. How did the plot change based on excluding these outlier data points?
Now, on the same sheet, copy plural attractor conditions c and d into a new sub-table, and create a new plot. In order to make an accurate visual comparison between the singular attractor (a/b) plot and the plural attractor (c/d) plot, it is critical that you adjust the y-axis so that both plots have exactly the same scale. A good choice for the current dataset would be to set the minimum and maximum values for the axis to 250ms and 450ms. Also, add a title to each plot, and feel free to annotate them in any other way that you find visually helpful.
Q8. Include a screenshot of the singular attractor and plural attractor plots. Do these data look like approximately what would be expected on the basis of previous results? Why or why not?
Finally, copy the data from the new ‘comma’ conditions e and f into a new sub-table and create a new plot for these critical conditions. Make sure to set the y-axis to have exactly the same scale as the other two tables, and make all other parameters of the plot the same.
Q9. Include a screenshot of the ‘comma’ condition plots. What pattern do you observe? Which hypothesis does this support and why?
Q10. Considering the dataset as a whole, are there any puzzling aspects of the results that might raise concerns about the data? Can you think of any ways that this experiment might be improved, either in design, implementation, or analysis? What might be an interesting follow-up study?