Supplemental Information Document for “Discussion Networks, Issues, and Perceptions of Polarization in the American Electorate”
To supplement the discussions and analyses presented in the main text, in this document we provide extended comments on several points, a series of robustness checks to the models, and a number of additional exploratory specifications (considering interactive effects). We present these efforts as additional information for the interested reader, and hope that they help paint a more complete picture of the effects of network disagreement as they relate to perceived distributions of polarization.
Contents: / Page(s):A (slightly) longer comment on polarization in government / 1
A comment on the reliability and validity of the histogram exercise / 2
Estimates using survey weights / 3-5
Specifications including interactive effects / 6-7
Discussion of DVs and potential issues / 8-17
Other Ways of Assessing Whether Disagreement Can Influence Perceptions / 18-19
Using a Partisan Measure of Disagreement / 20-21
Alternative Ways of Measuring Perceived Distributions / 22-24
Are Perceived Distributions Consequential? / 25-27
Discussing Spuriousness / 28-30
Effect Size Figures / 31-32
A (Slightly) Longer Comment on Polarization in Government
While the root causes and the extent of polarization remain contested, there is little doubt that on some dimensions – particularly in Congress – Democrats and Republicans have drifted apart. Using roll call votes to measure voting behavior, there is abundant evidence that elites are ideologically distinct, and that there is far less overlap among the parties than there used to be (Rohde 1991; McCarty, Poole, and Rosenthal 2006; Poole and Rosenthal 1997; Jacobson 2000; Fleisher and Bond 2000; though see Brady and Han 2006). Whether this trend is a function of the southern realignment (Rohde 1991; Jacobson 2000), emerging racial and socioeconomic differences at the district level ((McCarty, Poole, and Rosenthal 1997; Gimpel and Schuknecht 2001), or the behavior of party leaders (Aldrich and Rohde 1997, 2000), the idea of polarization in government is well-accepted/generally uncontested.
A Comment on the Reliability and Validity of the Histogram Exercise
Naturally, concerns about the reliability and validity of the distribution exercises arise – in using these items, we questioned how respondents negotiated such measures. Fortunately, a similar battery was first tested in affiliation with the 2006 ANES Pilot Study, and Judd et al. (2012) provide an extensive discussion of measurement and response issues using those data.
Notably, they found that in open-ended questions asked following the distribution battery, that most respondents had a positive response; there was not confusion about what was being asked of individuals. The non-response rates on the questions were not especially high either (82% provided meaningful answers), indicating that there did not appear to be large numbers of respondents skipping the battery because of potential confusion. Finally, high correlations across a series of histograms (that varied both in the number of bars to be raised and lowered) and across issues suggested a high level of reliability within respondents.
In sum, based on pilot study data, the histogram activity appears to be an internally reliable and valid measure of perceptions (and one that does not overwhelmingly confuse or turn off respondents). Please see Judd et al. (2012) for a complete discussion of these measurement issues/the full results from the battery conducted in affiliation with the 2006 ANES Pilot Study.
Estimates Using Survey Weights
Our first robustness check (below) examines whether the results hold when we use the recommended survey weights for the 2008-2009 ANES. These weights adjust the sample to better generalize to the voting eligible population in the United States. We use the ‘wgtc13’ weights in the first tables as we are looking at outcomes from wave 13. Below we replicate Tables 2 and 3:
SI Table 1: Exploring the Effects of Network Disagreement on Perceived Distance Between Parties
GovernmentTroopIllegal
Healthcare Withdrawal Wiretapping Immigration
Net. Avg. Disagreement-.124-.268-.420*-.240+
(.198)(.194)(.167)(.143)
Net Avg. Education-.013.003.008.001
(.017)(.016)(.015)(.016)
Gender (Male=1).084.060.074.075
(.067)(.071)(.062)(.059)
Race (Non-White=1)-.128-.190*-.130+-.071
(.085)(.095)(.076)(.085)
Education.126*.114*.143*.090*
(.032)(.032)(.030)(.026)
Age -.008*-.007*-.007*-.006*
(.002)(.003)(.002)(.002)
Interest.141*.149*.114*.121*
(.036)(.040)(.034)(.039)
Network Size.109+.069.063*.036
(.056)(.055)(.049)(.056)
Strength of Partisanship.061+.015-.008.007
(.034)(.037)(.032)(.035)
Constant.099.189.073.043
(.152)(.144)(.135)(.117)
N1,0031,0031,0031,003
R-Squared.13.11.12.08
Notes: p*<.05, p+<.1, two-tailed tests. Models estimated using ‘wgtc13’ weights.
Comment: The general constellation of results presented in Table 2 of the main text largely holds with the application of the survey weights. That said, several differences do emerge for the disagreement coefficient across the four issues: while coefficient estimates remain uniformly negative, they are larger with the application of weights, and the coefficient in the Troop Withdrawal model moves out of significance, while it goes from being significant at the p<.1 level to being significant at the p<.05 level for the Wiretapping model. Next we replicate Table 3 with the survey weights:
1
SI Table 2: Effects of Network Disagreement and Education on Perceptions of Issue Standard Deviations
In-Party Perceptions Out-Party Perceptions
Government Troop Illegal Government Troop Illegal
HealthcareWithdrawalWiretappingImmigrationHealthcareWithdrawalWiretapping Immigration
Net. Avg. Disagreement.051.071.059.037-.009.091.075.010
(.053)(.063)(.059)(.052)(.067)(.074)(.058)(.064)
Net. Avg. Education-.003-.007-.004-.005.009+-.002.000.001
(.005)(.006)(.005)(.007)(.005)(.007)(.005)(.007)
Gender (Male=1)-.033-.001-.024-.035+-.042-.012-.051*-.052*
(.021)(.023)(.020)(.020)(.026)(.026)(.022)(.022)
Race (Non-White=1)-.000.051*.046+.019.051.034.026-.002
(.028)(.026)(.027)(.021)(.032)(.035)(.024)(.035)
Education-.023*-.019*-.028*-.015*-.019+-.012-.022*-.011
(.009)(.008)(.008)(.007)(.010)(.011)(.009)(.008)
Age.002*.000-.001-.000.002*.001.001.001
(.001)(.001)(.001)(.001)(.001)(.001)(.001)(.001)
Interest-.015.004.012.019+-.018-.022-.009.001
(.011)(.013)(.012)(.011)(.016)(.016)(.011)(.014)
Network Size-.002.000-.005.011-.031-.003-.017-.006
(.017)(.021)(.017)(.023)(.016)(.021)(.015)(.023)
Strength of Partisanship-.015.006-.007-.019+-.010-.024.000-.008
(.013)(.015)(.012)(.011)(.016)(.016)(.012)(.015)
Constant1.40*1.34*1.43*1.37*1.30*1.40*1.41*1.31*
(.057)(.075)(.057)(.055)(.067)(.070)(.050)(.065)
N869869869869869869869869
R-Squared.04.03.05.04.05.03.04.02
Notes: OLS regression coefficients, standard errors in parentheses. p*<.05, p+<.1, two-tailed tests. Models estimated using ‘wgtc13’ weights.
Source: 2008-09 ANES Panel Study.
1
Comment: In these models, the signs on the disagreement coefficient estimates remain consistent and unchanged (with the one exception of the Out-Party Government Healthcare measure). The size of the coefficient estimates for most issues are either the same (1 issue) or larger (4 issues) than those in the unweighted analysis, though it is important to note that none obtain statistical significance at conventional levels. Thus, the loss of statistical significance appears to be largely a function of the weighting reducing the number of independent observations in the analysis by almost 60% (from 2,032 to 869), increasing the estimated standard errors considerably.
Summary Comment: Across the different sets of estimates, the overall pattern does not change markedly when we apply weights – our essential narrative obtains. The biggest exception to this statement comes in the replication of the standard deviation results (shown here in SI Table 2 – though again, the coefficients remain consistent in direction, and generally undiminished in size). Given these differences, we exercise some caution when interpreting the findings on perceptions of heterogeneity (please see footnote 12 and the discussion/conclusion sections). Again, however, we find that the general constellation of results remains well-supported.
Note: In a previous version of the manuscript, we had included a series of models in the final section of the (main) paper that modeled political interest as a function of polarization perceptions (the unweighted versions of these appear in the penultimate section of this SI, along with additional discussion on their specification, etc.). Per the helpful suggestions of the reviewers and editor, we removed those models from the main paper to make room for additional – and frankly, more important – discussion of issues related to our (central) findings concerning discussion networks and polarization perceptions. The models predicting political interest from polarization perceptions suggest that perceptions may be consequential/important in structuring things like political engagement. In our original formulation, we included them to help demonstrate additional ways in which polarization perceptions may be substantively important in and of themselves. The results were and remain more suggestive than anything else, and for these reasons we have removed them from the paper. We have included them here for the interested reader, and would suggest that others explore the relationship between engagement and (polarization) perceptions.
Specifications Including Interactive Effects
In this section, we explore whether there are possible interactive effects between network disagreement and network sophistication (i.e., education). There are substantive reasons to suspect that network sophistication and network disagreement could pair in meaningful ways, with network sophistication helping individuals “contextualize”/process the information about groups that they get from disagreeable discussion (e.g., McClurg 2006). In SI Table 3, we present the constituent and interaction terms for the mean distance and standard deviation models. All of the same control variables that have been used previously are included in the estimation of these models – we do not report those coefficients in the interest of saving table space.
Despite our expectations, we find fairly limited evidence that network sophistication moderates the effects of disagreement – this is particularly the case with the “spread” (standard deviation) analyses. That said, we do find hints of conditional effects for the distance models. In those cases, only one of the interactions (Perceived Distance on Government Healthcare) is statistically significant. The negative effect of network disagreement (which works to reduce the perceived difference between the parties) is attenuated as networks become more sophisticated.
While the coefficients on the interaction terms in the distance models generally fail to achieve statistical significance, it is worth noting that they are not zero, and that visualizations of the marginal effects (not presented) provide some evidence of similar attenuating effects for other issues. The suggestion from all of this is that much of the effect of disagreement on changing perceptions may be concentrated in lower sophistication networks (and, network disagreement may exert diminished influence in more sophisticated networks). The broader importance, of course, is the more general recognition that some of the disagreement findings we have observed may be further conditioned by the presence of other network characteristics, namely sophistication/expertise.
1
SI Table 3: Effects of Network Disagreement and Education Interactions on Perceived Distance Between Means and Standard Deviations
Perceived Distance Between Means
Government Troop Illegal
HealthcareWithdrawalWiretappingImmigration
Net. Avg. Disagreement-1.32*-.848+-.657-.481
(.459)(.446)(.423)(.380)
Net. Avg. Education-.012.004-.004-.002
(.010)(.010)(.009)(.008)
Net. Avg. Disagreement X Educ..112*.061.047.031
(.042)(.041)(.039)(.035)
N2,2952,2952,2952,295
R-Squared.13.12.10.06
In-Party Perceptions of Standard Deviations Out-Party Perceptions of Standard Deviations
Government Troop Illegal Government Troop Illegal
HealthcareWithdrawalWiretappingImmigrationHealthcareWithdrawalWiretapping Immigration
Perceived Standard Deviations
Net. Avg. Disagreement.070.053.191.134.237.255+.129.152
(.149)(.146)(.147)(.129)(.161)(.151)(.147)(.142)
Net. Avg. Education-.003-.004-.003-.003.003-.002-.002-.001
(.003)(.003)(.003)(.003)(.004)(.003)(.003)(.003)
Net. Avg. Disagreement X Educ.-.003.000-.011-.004-.021-.019-.006-.010
(.014)(.013)(.013)(.012)(.015)(.014)(.013)(.013)
N2,0322,0322,0322,0322,0322,0322,0322,032
R-Squared.05.04.03.02.05.04.03.02
Notes: OLS regression coefficients, standard errors in parentheses. The same set of control measures are used in these models but are not reported here in order to save space and focus on the interaction terms and component terms. p*<.05, p+<.1.
Source: 2008-09 ANES Panel Study.
1
Discussion of DVs and Potential Issues
How do the respondents use the distribution building exercises? First, we want to know if they are constructing distributions that align with how they “should” look. That is, we want to know if they are constructing distributions where they perceive more Republicans with conservative attitudes, and more Democrats with liberal attitudes. We can look at this across all four issues, and see whether the respondents are identifying this kind of partisan and ideological pattern. SI Figure 1 shows the average distributions that are constructed for all four issues.
SI Figure 1: Average Distributions Across Issues
All issues are coded such that the conservative response is one the right side of the figure. This is those who “strongly oppose” the US government paying for all necessary healthcare for Americans, setting a deadline for withdrawing troops from Iraq, requiring the US government to get a court order before listening to phone calls of American citizens who are suspected of terrorism, and allowing illegal immigrants to work in the US for up to three years, after which they would have to go back to their home country. The far left column is the more liberal response, or those who “strongly favor” these same statements. What we see in these plots is that respondents do perceive of Republicans as being more “concentrated” with conservative issue stances, and Democrats being more “concentrated” with liberal issue stances. There is some variation in how strong these patterns are across issues. Perhaps not surprisingly, the issues of government healthcare and Iraq withdrawal show more distinct concentrations of partisans at the extremes, while the differences are more muted for wiretapping and illegal immigration. In short, this gives us some confidence that respondents are understanding and using the exercise in the way that we would hope.
An additional way that we can look at the levels of constraint that respondents have across issues and parties is to see what percentage identify the Republicans as being to the right of the Democrats on these issues. On the issue of government healthcare, 91.8% of respondents who moved the bars identified the Republican mean to the right of the Democratic mean, and 92.2% identify the Republican mean to the right of the Democratic mean on the Iraq Troop Withdrawal issue. Both of these issues show very high levels of partisan ideological constraint. For wiretapping and illegal immigration there are still sizeable majorities that identify Republicans as being to the right of Democrats, with 71.1% making this distinction for wiretapping and 70% for illegal immigration (though these are less distinct differences than for the other two issues). In short, respondents appear to be building the kinds of distributions that we would expect, on average.
A (potential) issue with the dependent variables that we have constructed pertains to their distributions. The four measures of perceived differences in partisan means have a relatively strong positive skew, especially at the “zero” value. That is there is a somewhat sizeable number of respondents who perceive no difference between the party means. The standard deviation measures have a negative skew. This raises two potential issues. The first of these issues is a potential threat to our assumptionsin a linear regression framework. A non-normally distributed dependent variable could produce problems for the errors. Below we show residual plots from all regressions in Tables 2 and 3 in the main document to explore this possibility. SI Figure 2 shows plots of residuals versus predicted values from the four models where distance between means is the dependent variable, for our primary independent variable of concern (i.e., network average disagreement). The top-left figure is for the government healthcare dependent variable, top-right is for Iraq troop withdrawal, bottom-left is wiretapping, and bottom-right is illegal immigration.
SI Figure 2: Residual versus Predicted Values Plots: Distance Between Party Means
We can see that there is some skew to the residuals, but any problems do not appear to be especially egregious. In addition to visually assessing the nature of the residuals, we can calculate some test statistics to see whether heteroskedasticity appears to be an issue. We turn to the Breusch-Pagan test of constant error variance, which reveals significant F-statistics for all four models, suggesting that we have some reason to be concerned that our errors are not constant.
Next we turn to the same sets of plots for the models where we are using the standard deviation dependent variables. These are shown in SI Figure 3. The left column is in-party standard deviation and the right column is out-party standard deviation. The first row is the government healthcare dependent variable, second row is the Iraq withdrawal dependent variable, third row is the wiretapping dependent variable, and fourth row is the illegal immigration dependent variable.
SI Figure 3: Residual versus Predicted Values Plots: Standard Deviations
Again, the visual inspections shows some level of skew to the residuals, but they are not striking. However, the Breusch-Pagan test reveals significant F-statistics, again suggesting that we have some reason to be concerned that the error variance is not constant.
We are left with several options to remedy this potential threat to inference. First, we could transform the dependent variables by logging them or taking their square root. We have explored this approach, but neither effectively transforms the difference in party means dependent variables – i.e. we are still left with a variable that has a strong positive skew. As a result, we are left with re-estimating the main models using robust standard errors. In SI Tables 4 and 5 we show the results from Tables 2 and 3 of the main document, using robust standard errors. Our significance tests for network average disagreement are unchanged. We have run a host of additional models to address this issue, including using FGLS, which also returns results that are largely similar to those we have presented. While these are imperfect solutions to the problem, we believe that these robustness checks lend additional confidence that the main inferences we present in the paper are sound.
SI Table 4: Exploring the Effects of Network Disagreement on Perceived Distance Between Parties
GovernmentTroopIllegal
Healthcare Withdrawal Wiretapping Immigration
Net. Avg. Disagreement-.119-.193*-.159+-.144+
(.095)(.093)(.089)(.076)
Net Avg. Education.002.011.001.002
(.009)(.008)(.008)(.008)
Gender (Male=1).095*.082*.068*.073*
(.034)(.033)(.031)(.028)
Race (Non-White=1)-.197*-.164*-.166*-.090*
(.048)(.047)(.041)(.038)
Education.146*.135*.123*.074*
(.016)(.016)(.014)(.013)
Age -.009*-.009*-.006*-.003*
(.001)(.001)(.001)(.001)
Interest.129*.123*.115*.080*
(.019)(.018)(.017)(.016)
Network Size.079*.054+.064*.043
(.031)(.031)(.028)(.028)
Strength of Partisanship.067*.029+.028+.023
(.017)(.017)(.016)(.014)
Constant.079.154+-.057.042
(.085)(.082)(.076)(.070)
N2,2952,2952,2952,295
R-Squared.12.11.10.06
Notes: OLS regressions coefficients, robust standard errors in parentheses. p*<.05, p+<.1, two-tailed tests.
Source: 2008-09 ANES Panel Study.
1
SI Table 5: Effects of Network Disagreement and Education on Perceptions of Issue Standard Deviations
In-Party Perceptions Out-Party Perceptions
Government Troop Illegal Government Troop Illegal
HealthcareWithdrawalWiretappingImmigrationHealthcareWithdrawalWiretapping Immigration
Net. Avg. Disagreement.042.057+.074*.092*.009.047.061*.045
(.030)(.030)(.031)(.024)(.032)(.031)(.030)(.028)