SUPPLEMENTARY APPENDIX
Summary of Methods Used in Empirical Studies
ParticipantsAuthor(s), Year / Design / Methods / Number / Level / Subject Matter
Blanco & Niaz, 1997 / Qualitative / Questionnaire / 7 T
89 S / College / Chemistry
Brickhouse, 1990 / Qualitative / Case studies / 3 T / Middle &
High / General science
Physics
Cady, Meier, & Lubinski,2006 / Mixed methods
Longitudinal / Surveys, interviews / 12 T / Elem. & Middle / Mathematics
Cunningham, Perry, Stanovich, & Stanovich, 2004 / Quantitative / Knowledge tests, surveys / 722 T / Elem. / Reading
Duschl & Wright, 1989 / Qualitative / Ethnographic / 13 T / High / Science
Elby, 2001 / Quantitative / Surveys / 106 S / High / Physics
Fishhoff, Slovic, & Lichtenstein, 1977 / Quantitative / Questionnaire / 361 T / College / Domain general
Gallagher, 1991 / Qualitative / Ethnographic / 27 T / High / Science
Glenberg & Epstein, 1985 / Quantitative / Reading task, questionnaire, survey / 85 S / College / Domain general
Hammer, 1995 / Qualitative / (Self) Case study / 1 T
22 S / High / Physics
Harzler-Miller, 2001 / Qualitative / Case Study / 1 T / High / History
Hashweh, 1996 / Mixed methods / Survey, Questionnaire / 35 T / Mix / Science
Johnston, Woodside-Jiron, & Day, 2001 / Qualitative / Class observations, interviews / 2 T
12 S / Elem. / Literacy
Kang & Wallace, 2004 / Qualitative / Multiple case study / 3 T / High / Chemistry
Physics
Physical Science
Koriat, Lichtenstein, & Fischhoff, 1980 / Quantitative / Survey, questionnaire / 73 S / College / Domain general
Laplante, 1996 / Qualitative / Case studies / 1 T / Elem. / Science
Lampert, 1990 / Qualitative / Action research; interpretive / 1 T
1 C / Elem. / Mathematics
Lichtenstein & Fishhoff, 1977 / Quantitative / Survey, questionnaire / 345 S / College. Grad. / Domain general
Lichtenstein & Fishhoff, 1980 / Quantitative / Intervention, survey / 12 S / High, College / Domain general
Lidar, Lundqvist, & Ostman, 2005 / Qualitative / Analysis of class interactions / 1 T
3 S / Middle / chemistry
Louca, Elby, Hammer, & Kagey, 2004 / Qualitative / Analysis of class interactions / 1 T
1 C / Elem. / Science
Maor & Taylor, 1995 / Qualitative / Interpretive / 2 T
2 C / High / Science
Pieschl, Stahl, & Bromme, 2008 / Quantitative / Questionnaire, survey, knowledge test / 51 S / College / Biology
Radigans, 2002 / Qualitative / Analysis of class interactions;
Interviews / 3 T
4 C / High / English
Rosenberg, Hammer, & Phelan, 2006 / Qualitative / Analysis of class interactions / 1 T
11 S / Middle / Science
Schraw & Olafson, 2003 / Mixed Methods / Interviews, questionnaire / 24 T / Mostly
Elem. / Domain General
Stahl, Pieschl, & Bromme, 2006 / Quantitative / Questionnaires / 72 S / College / Biology
Tsai, C., 2006 / Qualitative / Multiple case studies / 4 T
4 C / Middle / Science
VanSledright, 2002 / Qualitative / (Self)Case Study / 1 T
1 C / Elem. / History
Wadsworth, 2007 / Quantitative / Questionnaires / 144 T / Middle / Mathematics
Wilson & Wineburg, 1993 / Qualitative / Task performances / 2 T / High / History
Yaeger & Davis, 1996 / Qualitative / Interview / 3 T / Middle & High / History
Yerrick & Pedersen, 1998 / Qualitative / Surveys, interviews, classroom observations / 1 T
3 S / High / Physics
*Note.T=Teacher; S=Student; C=Class.
SUPPLEMENTARY THEORETICAL MATERIAL
Situating Epistemic Cognition, Epistemic Beliefs, and Calibration
The conceptual roots of constructs like metacognition, self-regulation, and self-regulated learning and the difficulties of reaching clear and consistent definitions in current literature have been explored by Dinsmore, Alexander, & Loughlin (this issue). For the purpose of this paper, it is interesting to consider how epistemic cognition, epistemic beliefs, and calibration situate themselves in relation to these constructs and note how these relations have evolved over time. The relation between metacognition and epistemic cognition emerged first and it was intertwined with the very definition of metacognition (see Kitchener, 1983 and Kuhn, 1983 for a full discussion of this issue). Kitchener (1983), for example, noted that metacognition was usually defined in very broad terms (e.g., as knowledge about knowledge), encompassing knowledge about cognitive phenomena and any kind of monitoring of the cognitive processes. Yet, the study of metacognition tended to focus more narrowly on the self-monitoring of one’s own cognitive processes during specific task (e.g., monitoring of comprehension during reading tasks).
The case for broadening the definition of metacognition beyond the monitoring of one’s own cognitive tasks (e.g., computing, memorizing, and perceiving) was made by several researchers when this line of studies began to develop (Flavell, 1979; Kitchener, 1983; Kuhn 1983). This call responded in part to the realization that competent individuals are not only able to execute a sequence of strategies that enable them to complete tasks successfully; perhaps more importantly, these people are also able to evaluate the appropriateness of specific strategies in the particular situations (Kuhn, 1983), an observation further supported by studies on the development of expertise in academic domains (Alexander, 2003). However, beyond this common understanding, researchers differed in how they conceptualized the relation between metacognition, epistemic cognition, and epistemic beliefs and much empirical work is still needed to test the models proposed.
Studying how people dealt with ill-structured problems, Kitchener (1983) proposed a three-level model of cognitive processing, defining epistemic cognition as the process responsible for monitoring “the epistemic nature of problems and the truth value of alternative solutions” (p. 225). She proposed that it is at this level of cognition that individuals consider whether the problem is solvable under certain sets of conditions and strategy use and evaluate the limits of these solutions.
While Kitchener conceptualized epistemic cognition as a sort of meta-metacognition, Flavell (1979) seemed to place it within metacognitive knowledge related to the task at hand. In particular, he observed that metacognition comprises knowledge about the characteristics of the information available, such as its abundance, organization, and trustworthiness and characterized metacognitive knowledge in relation to the task as an understanding of “what such variations imply for how the cognitive enterprise should best be managed” (p. 907).
The role played in the cognitive process by the object of knowledge was further sharpened by Kuhn (1983). Revisiting Piaget, she suggested that the core question to be asked about the nature of the cognitive process regards how the mind and reality come to be coordinated with one another in such a way that the mind can know reality as it is (p. 102). She proposed that children gradually build an understanding of both their own actions and the external world through a series of interchanges between the subject (knower) and the object (known). Through this process, they aim at obtaining a better fit between their beliefs and the external world. The processes that individuals employ to monitor the appropriateness of their cognitive strategies in relation to the goal of gaining knowledge about a particular object are intrinsically epistemic since they lead to an evaluation of the nature of the knowledge obtainable in that particular context.
The process of judging the fit between beliefs and the external world is a kind of calibration. Early research on calibration considered it an individual skill influenced by cognitive, motivational, and contextual factors. Learners make evaluations based on their goals for a particular task, their perceptions of task difficulty, and their comfort with the task conditions. Calibration has often been measured through the administration of confidence scales and ratings of understanding in conjunction with particular knowledge or reading tasks (Fischhoff, Slovic, & Lichtenstein, 1977; Glenberg & Epstein, 1985; Lichtenstein & Fischhoff, 1977). In particular, Lichtenstein and Fischhoff (1977) considered it to be one of three measures of probability assessments, or statements of confidence in one’s knowledge.
This early body of work on calibration suggested that it is a rational judgment, meant to reflect the state of one’s knowledge at the time of answering a question, or completing a task. None of the studies explicitly mentioned calibration as part of metacognition, or one’s knowledge about cognition (Flavell, 1979). However, a study by Koriat, Lichtenstein, and Fischhoff (1980), exploring reasons why adults tend to be poorly calibrated in the direction of being overconfident, was included in the metacognitive monitoring section of a book entitled Metacognition: Core Readings (1992). Conceptualized as a kind of metacognitive evaluation, as its main consideration is “evaluating the validity of a degree of confidence” (Fischhoff et al., 1977, p. 552), calibration has been considered an element of the regulation component of metacognition (Baker & Brown, 1984).
Other perspectives on where calibration fits within metacognition followed. Nelson and Dunlosky (1991) described the accuracy of judgments of learning (JOLs), or calibration, as critical because JOLs are a mechanism of metacognitive monitoring, which directs metacognitive control processes. Monitoring and control would both seem to fall under regulation of metacognition as defined by Baker and Brown (1984), but in this case calibration is considered a type of monitoring rather than a type of evaluation. Nietfeld and Schraw (2002) similarly referred to calibration as monitoring accuracy, but oriented it within a self-regulated learning framework rather than a metacognition framework. Self-regulated learning was defined by Zimmerman and Schunk (1989) as individuals' thoughts, feelings, and actions directed towards attainment of particular goals.
Beginning in the ‘90s, epistemological research began to focus on beliefs that people entertain about the nature of knowledge and knowing (e.g., Schommer, 1990; Schraw, Dunkle, and Bendixen, 1995),studying their relation not only with metacognition but, more recently, also with self-regulation and self-regulated learning. For example, Hofer (2004) noted that beliefs about the nature of knowledge (usually identified in the literature as beliefs about the certainty and simplicity of knowledge) and beliefs about the self as knower could be located within metacognitive knowledge (i.e., knowledge about cognition, strategies, task variables, and the self as learner), while beliefs about the nature of knowing (commonly identified in the literature as evaluations about the source of knowledge and justification for knowing) might be seen as part of metacognitive judgments and monitoring (i.e., processes regarding judgment of task difficulties, monitoring of comprehension, and confidence assessment). Finally, epistemic considerations may be seen as concurring in the regulation of cognition and self-regulation, referred to as the “planning, strategy selection, allocation of resources, and volitional control” (p. 48). Calibration would fit in the category of regulating cognition during knowledge construction, contributing to answering the question “Do I know what I need to know or do I need to know more?” To answer such a question, individuals must accurately judge what they think knowledge and knowing are for a specific domain. Individuals are accurate (and thus highly calibrated) if they demonstrate an understanding that reflects what they claimed they understood. They are inaccurate (and thus poorly calibrated) if they demonstrate either more or less understanding than they claimed.
More recently, researchers have included calibration, as a metacognitive process by which individuals monitor their planning and goal setting, in models of self-regulated learning (Muis, 2007; Pieschl, Stahl, & Bromme, 2008; Winne & Hadwin, 1998). This implies an entirely different conceptualization of calibration. From reflecting an internal judgment of correctness of knowledge than can be measured against external, agreed upon criteria, calibration is now seen in relation to plans and goals and measured against individually determined standard. That standard can be assumed to vary due to epistemic beliefs, motivation, and prior knowledge for tasks within particular domains. The latter conceptualization of calibration, situated within a framework of self-regulated learning, is a more effective means to consider the relation between epistemic constructs and metacognitive monitoring and control.
A self-regulated learning model containing both epistemic beliefs and calibration has been proposed, and preliminary data has thus far been reported (Pieschl et al, 2008; Stahl, Pieschl, Bromme, 2006). The model places metacognitive calibration processes, described as “the alignment between external conditions and individual learning processes (Pieschl et al. 2008, p. 20)” at the center of several recursive components of learning processes. More specifically, calibration of metacognitive monitoring and control processes is both influenced by, and influences task standards, evaluations of performance, strategies employed, and awareness of external and internal task conditions. Pieschl et al. included epistemic beliefs in internal task conditions. They hypothesized that epistemic beliefs have an effect on calibration processes; however they did not find support for this hypothesis in their preliminary study of undergraduates studying biology hypertext.
More recently, Muis (2007) proposed a model of self-regulated learning that explicitly includes epistemic beliefs. Self-regulated learning has been defined as “learning that results from students’ self-generated thoughts and behaviors that are systematically oriented toward the attainment of their learning goals” (Schunk, 2001, p. 125). Muis’ model situated epistemic beliefs among the cognitive and affective conditions that influence how individuals define a task and thus affect the standards they set for themselves in the specific learning situation. These standards guide the strategies individuals choose in order to pursue their goals and influence how they monitor and evaluate their understanding and the veracity of the information they encounter. The model further hypothesizes that epistemic standards influence the level of engagement in metacognitive processing. For example, more tentative views of knowledge may lead to examining and comparing multiple sources, thus prompting increased levels of metacognitive processing during all the phases of self-regulated learning (i.e., task definition, planning and goal setting, enactment, and evaluation).
Muis (2007) called for closer examination of specific mechanisms by which epistemic beliefs influence self-regulated learning. Examining calibration, as a mechanism of the evaluation phase of the model, is a unique opportunity to do this, as evaluation occurs during all other phases. As suggested by Stahl, Pieschl, and Bromme (2006), individuals should approach learning tasks differently depending on how well-matched their epistemic beliefs are with task demands. Thus, self-regulated learning should vary by the epistemic beliefs of the individual, and the subsequently perceived types of learning processes necessary to achieve particular standards dictated by those epistemic beliefs. Individuals are likely to become better calibrated when provided immediate feedback and training (Lichtenstein & Fischhoff, 1980), increasing flexibility of learning processes to meet different task demands, and therefore improving performance on learning tasks.
References
Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10-14.
Baker, L., & Brown, A. L. (1984). Metacognitive skills and reading. In P. D. Pearson, M. Kamil, R. Barr, & P. Mosenthal (Eds.), Handbook of reading research (Vol. 1, pp. 353-394). White Plains, NY: Longman.
Dinsmore, D. L., Alexander ,P. A., & Loughlin, S. M. (this issue). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational Psychology Review.
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3(4), 552-564.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906-911.
Glenberg, A. M., & Epstein, W. (1985). Calibration of comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 702-718.
Hart, J. T., Nelson, T. O., & Gerler, D. (1992). Metacognitive monitoring. In T. O.
Nelson (Ed.), Metacognition: Core readings (pp. 131-231). NeedhamHeights,
MA: Allyn & Bacon.
Hofer, B. K. (2004). Epistemological understanding as a metacognitive process: Thinking aloud during online searching. Educational Psychologist, 39(1), 43-55.
Kitchener, K. S. (1983). Cognition, metacognition and epistemic cognition: A three-level model of cognitive processing, Human Development, 26, 222-232.
Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6(2), 107-118.
Kuhn, D. (1983). On the dual executive and its significance in the development of developmental psychology. In D. Kuhn, & J. A. Meacham (Eds.), On the development of developmental psychology (pp. 81-110).Basel, NY: Karger.
Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about how much they know? Organizational Behavior and Human Performance, 20, 159-183.
Muis, K.. R. (2007). The role of epistemic beliefs in self-regulated learning. Educational Psychologist, 42(3), 173-190.
Nelson, T. O., & Dunlosky, J. (1991). When peoples’ judgments of learning (JOLs) are extremely accurate at predicting subsequent recall: The “delayed JOL-effect.” Psychological Science, 2(4), 267-270.
Nietfeld, J. L., & Schraw, G. (2002). The effect of knowledge and strategy training on monitoring accuracy. The Journal of Education Research, 95(3), 131-142.
Pieschl, S., Stahl, E., & Bromme, R. (2008). Epistemological beliefs and self-regulated learning with hypertext. Metacognition and Learning, 3, 17-37.
Schommer, M. (1990). Effects of beliefs about the nature of knowledge on comprehension. Journal of Educational Psychology, 82, 498-504.
Schraw, G., Dunkle, M. E., & Bendixen, L. D. (1995). Cognitive processes in well-defined and ill-defined problem solving. Applied Cognitive Psychology, 9, 523-538.
Schunk, (2001). Social cognitive theory and self-regulated learning. In B. J. Zimmerman & D. H. Schunk (Eds.), Self regulated-learning and academic achievement: Theoretical perspectives (2nd ed., pp. 125-151). Mahwah, NJ: Lawrence Erlbaum.
Stahl, E., Pieschl, S., & Bromme, R. (2006). Task complexity, epistemological beliefs and metacognitive calibration: An exploratory study. Journal of Educational Computing Research, 35(4), 319-338.
Winne, P. H. & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277-304). Mahwah, NJ: Lawrence Erlbaum Associates.
Zimmerman, B. J., & Schunk, D. H. (1989). Self-regulated learning and academic achievement: Theory, research, and practice. New York: Springer.