Christopher Jarrold and Debbora Hall

1

Commentary: theoretical and methodological challenges to the study of working memory in developmental disorders: A comment on Rhodes et al. (2012)

Christopher Jarrold and Debbora Hall

School of Experimental Psychology, University of Bristol, UK

Rhodes and colleagues’ decision to investigate working memory profiles in children with ADHD, ODD, and a combined diagnosis (see preceding article in this issue: Rhodes, Park, Seth and Coghill, 2011), is certainly timely, as there is an increasing awareness of the role of working memory capacity in educational achievement and classroom behaviour. Indeed, it is often suggested that poor working memory capacity might underpin apparent problems of inattention in the classroom, and it may well be the case that the working memory profiles of these two conditions are different from one another. It is therefore undoubtedly important to understand the pattern of deficits in children with ADHD and ODD, and where they arise, in order to direct educational support to these individuals in an appropriate way.

In addition, the approach taken by Rhodes et al. is much more theoretically motivated than one often sees in studies of this kind. It makes excellent sense to decompose working memory performance into its executive control and non-executive storage aspects, the latter of which appears to have distinct verbal and spatial components. There is a growing consensus in the literature that long-term memory plays a role in constraining working memory performance, so the inclusion of long-term memory measures in this battery is very welcome. Determining the factor structure that emerges from these tasks, and using this to drive most of the analysis of group differences, is also a very sensible analytic strategy.

We therefore fully support the principle of using factor analysis in work of this kind. However, we know from our own experience that it is all too easy to generate an inappropriate factor structure from a dataset, and, from that, questionable estimates of latent variables, and we would argue that aspects of Rhodes et al.’s confirmatory factor analysis may be open to question. In any working memory battery one would expect to see higher correlations between tasks tapping the same storage domain, for example between two spatial storage tasks or between two verbal storage tasks, than between tasks tapping different domains. In this data set, there are a number of relatively small correlations between tasks tapping verbal storage only in contrast to somewhat higher correlations between verbal and spatial storage only measures (e.g., r = . 30 for the correlation between e-prime verbal storage and verbal recognition free recall, r = .40 for the correlation between e-prime spatial storage and verbal recognition free recall, Table 3). This suggests that the tasks may not be as pure indicators of verbal and spatial skills as one might like. Although the confirmatory factor analysis supports a model that includes separable domains for visual and verbal material (p. 5), the model itself arguably does not give an entirely parsimonious account of the data because it also includes a general short-term memory latent variable (Figure 1). One might argue that it makes sense to dissociate domain-specific short-term memory content from the domain-general processes that support the ordering of that content (Majerus et al., 2010), and if so then there may be a theoretical rationale for this type of model. However, it is also likely that the fit indices in a model such as this will show a good fit to the data, as it encompasses all possible relations between variables, in a sense holding two potentially separate theoretical positions simultaneously. Some form of nested model in which verbal and spatial functioning latent variables were partialled out of the working memory storage latent variable might be a more theoretically coherent way of modeling these data.

A further issue with the factor analysis employed in this paper is the use of a single solution for four apparently distinct groups. If the authors are correct in assuming that children in the different subgroups show different patterns of impairment, one would presumably not expect the intercorrelations between factors to be consistent across subgroups; a selective impairment in verbal ability will clearly alter the relative relationships between factors in comparison to a global delay. Latent variable modelling is a powerful technique for showing independence of domains and correlations between latent variables, but it is not clear to us whether it is entirely appropriate to aggregate all the individuals in this study for the purposes of this analysis. Certainly some evidence of a similar pattern of correlations in the four subgroups when considered separately would also be helpful and reassuring.

The advantage of the latent variable approach in this study is it then allows the authors to examine group differences in these latent constructs; even if there is some concern as to the theoretical plausibility of this set of constructs, one can still interpret group differences on these measures in the light of the tasks that load on each latent variable. One of the main claims in the paper that follows from this is that the working memory deficits associated with ADHD and ODD are distinguishable from one another, with selective deficits in the spatial domain in ADHD and more global problems in ODD, and that individuals with ADHD+ODD show a combined, additive profile of difficulties in both the verbal and spatial domains. However, while both interesting and potentially important, we would question whether these conclusions are entirely supported by the data. First, in every analysis of variance reported in the paper there is no significant difference in performance between these three atypical groups. While it is the case, for example, that numerically fewer of the boys with ADHD show very poor ‘verbal memory’ than in the ODD and ADHD+ODD groups, the difference in verbal memory across these three groups is not significant. Indeed, Figure 2a suggests that these groups display very similar verbal short-term memory performance.

Our reading of the results of these group analyses is more in line with the authors’ suggestion that there is “significantly impaired performance across a broad range of aspects of memory functioning in boys with ADHD and/or ODD” (p. 7). However, impaired performance does not necessarily imply atypical functioning of the underlying memory systems. It may well be that the suggestive evidence of differential impairments seen in this study would emerge as significant differences between larger ADHD and ODD samples. Alternatively, any impairment in performance, which statistically is seen to the same extent in all of the clinical participants on every task, may reflect the fact that these groups were not matched to the typically developing boys for level of intellectual functioning. Table 1 shows that while the groups were well matched for age, the clinical participants had significantly lower BPVS II percentile scores than the typical boys. This is a concern given that vocabulary is closely linked to verbal short-term memory performance and may also be a reasonable surrogate for level of general intelligence in ADHD and ODD groups. In addition, the study reports no details on individuals’ socio-economic status. In the absence of any evidence that the groups were matched for SES, matching for vocabulary would allay concerns that the groups differed in intellectual ability as a result of demographic factors (Noble, McCandliss, & Farah, 2007).

We are aware that matching for age alone is a reasonably common practice in this area, and that the authors address the worry of differences in BPVS performance by covarying out BPVS scores from their analyses; doing this makes no substantial difference to the pattern of results. However, analysis of covariance is not an ideal substitute for close matching for some index of ability. Analysis of covariance is subject to a series of rigorous assumptions (such as homogeneity of regression slopes) that are not always easy to meet in a multiple groups design. In addition, ANCOVA designs are an appropriate way of reducing the variance that follows from non-significant group differences in a matching measure, but are not always interpretable when groups differ significantly on the ‘to-be-covaried’ variable (Miller & Chapman, 2001).

In summary, the authors should be commended for attempting to profile the memory impairments in ADHD and ODD, and work of this kind is undoubtedly needed. The suggestion of global memory deficits in ODD and more specific spatial problems in ADHD is certainly an interesting one, but one that we would argue is not unequivocally supported by the current data. To properly confirm these suggestions, future work should ensure that populations are well matched on a suitable measure of general intelligence, not least because evidence of impairment in a given domain, such as memory, is always more persuasive in the context of preserved ability in some other area. Factor analytic techniques provide a potentially powerful way of reducing a large data set into its components, but are essentially atheoretical approaches. Consequently, any factor structure needs to be tested against existing theoretical models. In the case of working memory previous evidence would suggest that domain-specific storage and domain-general executive factors should emerge from such an analysis, and future work should also look to separate out potentially domain-specific item memory from potentially domain-general order memory. Finally, when studying atypical groups on a set of experimental measures one normally needs to compare performance with that seen in typically developing individuals. The same holds when comparing factor scores, and one way of doing this that we would suggest adopting in future work would be to first extract the factor structure inherent in just the typically developing data, and then use that structure to drive the measurement of performance on the emerging domains of interest in any atypically developing participants.

Correspondence:

Christopher Jarrold or Debbora Hall, School of Experimental Psychology, University of Bristol, 12a Priory Road, Bristol BS8 1TU, UK. Email: ,

References

Majerus, S., D'Argembeau, A., Martinez Perez, T., Belayachi, S., Van der Linden, M., Collette, F., . . . Maquet, P. (2010). The commonality of neural networks for verbal and visual short-term memory. Journal of Cognitive Neuroscience, 22, 2570-2593.

Miller, G. A., & Chapman, J. R. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110, 40–48.

Noble, K. G., McCandliss, B. D., & Farah, M. J. (2007). Socioeconomic gradients predict individual differences in neurocognitive abilities. Developmental Science, 10, 464-480.

Rhodes, S.M, Park, J., Seth, S & Coghill, D.R. (2012). A comprehensive investigation of memory impairment in attention deficit hyperactivity disorder (ADHD) and oppositional defiant disorder (ODD). Journal of Child Psychology and Psychiatry, 52. Advance online publication. doi: 10.1111/j.1469-7610.2011.02436.x.