Faye Huie
EDRS 797: Mixed Methods
Module 1 Assignment: Readings value and critique memo
A memo and critique of Castle, Fox, and Souder (2006): A comparative study of PDS
I think that the best way I learn about methods is reading a journal article that utilizes one of those methods and then critique it based on the textbook information about that methodological approach. Although I do enjoy the more theoretical articles assigned in this class, I have to admit that I enjoy and gain the most from the empirical research articles. The article that I felt was most beneficial to my learning about mixed methods research thus far was Castle, Fox and Souder (2006). In this memo of the article written by Castle et al., (2006) I will reflect on how their approach to writing and methodology has impacted my approach to research and writing.
Castle et al. (2006) discuss a very important and current issue in education: professional development schools. In the face of NCLB, it is extremely important that teachers and preservice teachers are prepared and able to teach with pedagogically effective strategies and are well versed in their topic area. Castle et al (2006) make it clear that both factors (e.g., knowledge in both pedagogy and content knowledge) are critical to the effectiveness of teaching. However, in order to get a clear idea of if and how? professional development schools do, in fact, make an impact on teacher effectiveness a mixed methods approach to understanding the issue is required. Although this point is not made explicitly, it is made implicitly throughout their introduction and rationale (e.g., variance: do teachers in PDS differ from teachers in non-PDS schools, and process: how and why do they differ). Understanding these issues will ultimately help teacher educators to better prepare teachers to teach. However, if we just look at the research question “…To what extent do PDS and non-PDS teacher candidates differ in planning, instruction, management, assessment, professionalism, and reflection as defined by the INTASC standards?” (p 66) it leads the reader to believe that this is just a variance approach to examining this issue.
I think that this type of writing and conceptualization of research is rampant throughout many of the research articles written and published in education, especially in the quantitative pieces. That is, when researchers talk about self-efficacy, or stereotypes, or self-regulation, these concepts are intrinsically qualitative (Maxwell, XXXX) and researchers do not address that. Because they turn them into variables. Castle et al. (2006) would have been much clearer in leading the reader if their research question was split into two parts: 1) Do PDS teachers differ from non PDS teachers in terms of their standards stated by the INTASC; and 2) How do PDS teachers differ from non-PDS teachers in terms of their reflections and ways of thinking about teaching? I do not feel that their current research question is appropriate and addresses the “meat” of the paper, which I feel is about looking deeper into the issues of how PDS teachers differ from non-PDS teachers. Good point.
Personally, I can write much better when I have a clear idea of what I am writing about and what my argument is. The introduction in this paper is too short to make a “so-what” argument. Although I feel that it provides a very good and broad argument of why PDS is important and why we need to examine it, the specific ideas covered in this research are not addressed (e.g., why is it important for teachers to connect reflections with teaching?). I feel that there are just so many concepts in this study that are not thoroughly addressed and doing so shows that the authors make an assumption that the readers already understand these issues and why they are important. However, this assumption may be due to two main factors: page limits required by the journal, or writing directed toward a specific audience. Although these are legitimate reasons, I do believe that there needs to be a balance between establishing a more specific while simultaneously addressing a broader “so what” factor to provide a stronger case for the research and its findings. Nice analysis.
In terms of the methodology, I find several problems with the both the quantitative and qualitative analyses as well as the connection between them. First, in terms of the quantitative analyses, I feel that there are two major flaws: 1) sample size and focus on statistical significance and 2) measurement. In terms of the sample size, I think that comparing PDS teachers (n = 47) and non PDS teachers (n = 20) is inappropriate. There are just not enough people in each of the comparison cells to yield accurate statistical comparison. I don’t understand this point. Do you mean that it lacks statistical power? This problem is further amplified by the authors focus on only the statistically significant results. Specifically, statistical significance depends on the sample size. With such a small sample size, very few of the results will be statistically significant. Therefore, I think that the authors should have reported effect sizes for all of their comparisons to estimate the relationship between the two groups ignoring sample size. I think that doing so would not only provide a more accurate account of group differences, but also would have made the article more interesting in terms of discussing those differences. J
Another problem I saw was their measures and how each analysis was done on one item. I think that this issue goes back to how all quantitative measures are intrinsically qualitative (e.g., preparation/planning, personal/professional development is intrinsically qualitative!) which I believe, cannot be accurately captured with one specific item. Instead of just reporting on specific items, I think the authors should have conducted a factor analysis to see how those items clustered together followed by assessing any differences between the different subscales that emerged. But I think that a factor analysis would have required an even larger sample. I think that by doing so would have provided a more accurate account statistically as well as provide a stronger theoretical framework for their study.
I also feel that there are several problems with their qualitative approach. First, they do not provide a thorough description of their procedures in terms of how they analyzed the portfolios (only one paragraph!). But a long and fairly detailed paragraph. This is more than many journal articles contain. Specifically, as a reader, I am left questioning what a “passage” was. Although the authors describe the definition of a passage (e.g., an idea segment or reflective insight), the definition was still vague and more information was needed in order for me to be convinced that the emergent themes were valid. One important piece of information that I felt was left out was: who made the judgment to classify a segment of the qualitative data as a passage? Were there any validity checks where other trained researchers can go over the data to see the degree of agreement between what was considered as and as not a passage (e.g., interrater reliability)? See attached paper. I think that the procedure is especially important in qualitative research because of the judgment calls and what Dr. Maxwell always says, that the researcher becomes the instrument. Another issue with the qualitative data analysis that I noticed was how a theory of teacher effectiveness was imposed onto the data instead of letting themes emerge from the data and interpret those themes freely. Specifically, the authors state that the themes that did emerge from the analysis of the passages were organized according to INTASC standards. Good point, but the themes were identified independently of the INTASC standards, and only later organized according to these standards. Although I understand that the purpose of the paper was to understand how PDS impacted teachers practice of INTASC standards, it would have been important to see what other factors emerged that were different from INTASC standards. I think that doing so would have made the paper more interesting and may even provide a stronger contribution to the field of preservice teacher preparation and education. Also, this makes me wonder how the INTASC standards impacted the researchers’ analysis and interpretation of the data, and how it may have been different if the authors did not keep in mind the INTASC standards.
The final issue that I noticed with the data analysis is the degree to which the quantitative and qualitative analyses and interpretations complement each other. This study taught me that the results from either qualitative or quantitative do not need to inform the other mode of analyses. Specifically, when I first read this study, I expected that since the authors discussed the quantitative analysis first, I thought that they would continue to the qualitative analysis by only focusing on the important findings from quantitative analysis. For example, since the authors found a significant difference in the materials that PDS and non-PDS teachers used to prepare or plan their classes, I thought that the authors would go into the qualitative data to examine how the teachers differed in their material use. Instead, however, it was almost as if the authors entered the qualitative analyses with no consideration with the quantitative analyses to identify the emergent themes, and then afterwards assessed how the results from the two methods of analysis complement each other. I like this way of approaching the research in that there is less restriction on the qualitative data analysis and you are left to explore freely the different themes that emerge. However, I also think that the authors could have made a stronger effort to connect the quantitative and qualitative analyses. Good point.
I think that the biggest lesson that I learned from reading this article is the idea of balance in both the writing and analyzing the variance and process elements of mixed methods research. If the authors had taken all the issues that I noted into consideration, their article would probably exceed the page limits of many journals and people would probably get very tired of reading… J Therefore, I think that the balancing act between different concepts in mixed methods research is an extremely important process. Although critiquing and reflecting on a deeper level about this article has made me feel a bit overwhelmed at the ideas that one must take into consideration when doing mixed methods research, I am excited to put what I learned into practice.
Faye:
Nice work on this. You do a good job employing the “doubting game”; a little more of the “believing game” might balance it better.
Grade: A