Rawson & Dunlosky, online supplemental materials 6
MATERIALS FOR ONLINE SUPPLEMENT:
Table S1. Summary of Testing Effects Research Published from 2000-2010.
# of Number and Kind Practice Studied Retention
Exp. # Sessions of Practice Trials Test Materials Interval
Agarwal et al. (2008) 1 1 0 T, 1 T, 1 TR SA text 7 days
2 1 0 T, 1 T, 1 R, 2 R, 1 TR SA text 7 days
Bahrick & Hall (2005) 1 4 TR to 1 correct per session CR word pairs 2 weeks
2 4 1 TR per session CR word pairs 7 days
Balota et al. (2006) 1 1 5 T CR word pairs 5 min
2 1 5 TR CR word pairs 5 min
3 1 3 TR CR word pairs 5 min
Barber et al. (2008) 1 1 0 T, 1T SA text 2 days
Barcroft (2007) 1 1 2 R, 2 TR CR picture-word min, 2-7 days
Bishara & Jacoby (2008) 1, 2 1 3 R, 3 TR CR word pairs min
Butler et al. (2007) 1 1 0 T, 1 T, 1 TR MC text 1 day
2 1 0 T, 1 T, 1 TR MC text 7 days
Butler et al. (2008) 1 1 0 T, 1 T, 1 TR MC facts 5 min
2 1 0 T, 1 T, 1 TR MC facts 2 days
Butler et al. (2006) 1 1 3 RT MC word lists 8 min
2, 3 1 1 T MC text 6 min
Butler & Roediger (2007) 1 1 0 T, 1 T, 1 R MC, SA video lecture 1 month
Butler & Roediger (2008) 1 1 0 T, 1 T, 1 TR MC text 7 days
Carpenter (2009) 1, 2 1 1 T, 1 R CR word pairs 5 min
Carpenter & DeLosh (2005) 1a, 1b 1 1 T, 1 R CR face-name 5 min
2 1 3 T, 3 R CR face-name 5 min
Carpenter & DeLosh (2006) 1 1 1 T, 1 R FR, CR, RG word lists 5 min
2, 3 1 1 T, 1 R CR word lists 5 min
Carpenter & Pashler (2007) 1 1 100 seconds of TR or R CR map features 30 min
Carpenter et al. (2009) 1 1 0 T, 1 TR, 1 R SA facts 9 months
Carpenter et al. (2006) 1, 2 1 1 TR, 1 R CR word pairs 1-2 days
Carpenter et al. (2008) 1, 2 1 1 TR, 1 R CR facts 5 min, 1-42 days
3 1 1 TR, 1 R CR word pairs 5 min, 1-42 days
Cepeda et al. (2009) 1 2 TR to 2 correct in S1, then 2 TR in S2 CR word pairs 10 days
2 2 3 TR in S1, 2 TR in S2 CR facts, picture-name 6 months
Cepeda et al. (2008) 1 2 TR to 1 correct in S1, then 2 TR in S2 CR facts 1-50 weeks
Chan (2009) 1 1 0 T, 2 T SA text 20 min, 1 day
2 1 0 T, 2 T CR text 20 min, 1 day
Chan (2010) 1 1 0 T, 2 T FIB text 20 min, 1-7 days
Chan & McDermott (2007) 1a, 1b 1 0 T, 1 T FR word lists min
2 1 0 T, 1 T FR word lists 15 min
3 1 0 T, 1 T CR word lists min
Chan et al. (2006) 1, 3 1 0 T, 2 T, 2 R SA text 1 day
2 1 0 T, 2 T SA text 1 day
Cranney et al. (2009) 1, 2 1 0 T, 1 R, 1 TR MC, FIB video 7 days
Cull (2000) 1 1 0 T, 3 T, 3 R, 3 TR CR word pairs 1 min
2 1 0 T, 3 T, 3 R, 3 TR CR word pairs 10-30 min
3, 4 4 TR to 1 correct in S1, then CR word pairs 3-14 days
0 T, 1 T, 1 R, or 1 TR in S2-S4
Fazio et al. (2010) 1 1 0 T, 1 T MC text min, 7 days
Finn & Metcalfe (2010) 1, 2, 3 1 1 T, 1 TR CR facts min, 1-2 days
Fritz et al. (2007a) 1 1 4 TR, 10 R CR word pairs 3 min, 3 days
2 1 3 TR, 30 seconds of R CR word pairs 1 day
3 1 4 TR, variable R CR word pairs min, 7 days
Fritz et al. (2000) 1 3 1 T, 1 R, 1 RT, 1 TR per session FR text 7 days
Fritz et al. (2007b) 1 1 0 T, 4 TR CR toy-name min
2 1 4 TR, 4 R CR toy-name 1 min, 1-2 days
Helder & Shaughnessy (2008) 1, 2 1 0 T, 3 T CR face-name min, 1 day
Johnson & Mayer (2009) 1 1 1 R, 1 T FR, SA text 5 min, 7 days
Kang et al. (2007) 1 1 0 T, 1 T, 1 R MC, SA text 3 days
2 1 0 T, 1 TR, 1 R MC, SA text 3 days
Karpicke (2009) 1, 2 1 TR to 1 correct, then 2 T or 2 R CR word pairs 7 days
3 1 0 T, 1 T, 2 T CR word pairs min
Karpicke & Roediger (2007a) 1 1 1 T, 3 T CR word pairs 10 min, 2 days
2 1 1 TR, 3 TR CR word pairs 10 min, 2 days
3 1 1 T, 4 T CR word pairs 10 min, 2 days
Karpicke & Roediger (2007b) 1 1 5 RRRT, 5 RTRT, 5 RTTT FR word lists 7 days
2 1 4 RTRT, 4 RRTT, or RT to 1 correct FR word lists 7 days
and then variable R or T
Karpicke & Roediger (2008) 1 1 4 RT, or RT to 1 correct and then CR word pairs 7 days
0 T, variable R, or variable T
Karpicke & Roediger (2010) 1 1 3 T CR text 7 days
2 1 0 T, 1 T, 4 T CR text 7 days
Karpicke & Zaromb (2010) 1, 2, 3, 4 1 1 T, 1 R CR word pairs 5 min
Kornell (2009) 1 1 4 TR CR word pairs 1 day
2 4 2 TR per session CR word pairs 1 day
3 5 2 TR per session CR word pairs 1 day
Kornell & Bjork (2008) 1 1 10 minutes of TR CR word pairs imm, 7 days
2, 3, 4 1 10 minutes of TR CR word pairs 5 min
Kornell et al. (2009) 1, 2 1 1 TR, 1 R SA facts 5 min
3, 4, 6 1 1 TR, 1R SA word pairs 5 min
5 1 1 TR, 1R SA word pairs 1 day
Kornell & Son (2009) 1 1 1 T,1 TR, 1 R, RR CR word pairs 5 min
Kromann et al. (2009) 1 1 30 minutes of T or R perform procedure weeks
Leeming (2002) 1 1 1 T SA text, lecture weeks
Logan & Balota (2008) 1 1 1 T v 3 T CR word pairs 8 min, 1 day
Marsh et al. (2009) 1, 2, 3 1 0 T, 1 T MC facts 5 min
McDaniel et al. (2007) 1 1 0 T, 1 TR, 1 R MC, FIB text up to weeks
McDaniel et al. (2009) 1 1 1 R, 1 TR FR text 3 min, 7 days
2 1 1 R, 1 TR FR, MC, SA text 3 min, 7 days
Metcalfe & Kornell (2007) 1, 2 1 1 R, 1 T, 1 TR CR definition-word min
Metcalfe et al. (2009) 1 1 1 T, 1 TR CR definition-word 1-6 days
2 1 1 T, 1 TR CR definition-word 2-10 days
Metcalfe et al. (2007) 1 variable 35 minutes of R or TR CR definition-word 7 days
2, 3 variable 30 minutes of R or TR CR word pairs 7 days
Morris & Fritz (2000) 1, 2 1 variable TR or R CR face-name 30 min, 2 weeks
11 months
Morris & Fritz (2002) 1 1 or 2 variable TR or R CR names 30 min, 2-8 weeks
Morris et al. (2004) 1 1 variable TR CR names 30 min
2, 3 1 variable TR or R CR names 30 min
Morris et al. (2005) 1 1 3 T, 3 R CR names 5 min
2 1 3 TR, 3 R CR picture-name 5 min
Neuschatz et al. (2005) 1, 3 1 4 T, 4 R CR face-name 15 min
2 1 4 T, 4 R CR face-name 2 days
Odegard & Koen (2007) 1, 2 1 1 T MC text 5 min
Pashler et al. (2005) 1 1 0 T, 2 T, 2 TR CR word pairs 7 days
Pashler et al. (2003) 1 1 2 TR CR word pairs 1 day
2 1 2 TR CR word pairs 7 days
Pavlik (2005) 1 1 1 TR, 1 T, 1 R CR word pairs min
Pavlik & Anderson (2005) 1 1 1 TR, 2 TR, 4 TR, 8 TR CR word pairs 1-7 days
Pavlik & Anderson (2008) 1 3 60 minutes of TR CR word pairs 7 days
Pilotti et al. (2009) 1 1 0 T, 1 T, 1 R MC facts min
Pyc & Rawson (2007) 1 1 3 TR, TR to 1 or 2 correct CR word pairs 40 min
2 1 3 TR, TR to 1 correct CR word pairs 40 min
Pyc & Rawson (2009) 1, 2 1 TR to 1, 3, 5, 6, 7, 8, 10 correct CR word pairs 25 min, 7 days
Pyc & Rawson (2010a) 1 1 3 TR, TR to 1 correct CR word pairs 7 days
2 2 3 TR or TR to 1 correct in each session CR word pairs 7 days
Pyc & Rawson (2010b) 1 1 3 TR, 3 R CR word pairs 7 days
Richland et al. (2009) 1, 2, 3, 5 1 1 TR, 1 R SA facts min
4 1 1 TR, 1 R SA facts 7 days
Roediger & Karpicke (2006) 1 1 1 T, 1 R FR text 5 min, 2-7 days
2 1 3 T, 3 R FR text 5 min, 7 days
Roediger & Marsh (2005) 1 1 1 T MC text 5 min
Rohrer et al. (2005) 1 1 5 TR, 20 TR CR city-country 1, 3, or 9 weeks
2 1 5 TR, 10 TR CR word pairs 1-4 weeks
Rohrer et al. (2010) 1 1 5 TR, 5 R match map region-label 1 day
2 1 5 TR, 5 R match roads-cities 1 day
Smith & Kimball (2010) 1 1 1 T, 1 TR, 1 TT CR facts 7 days
Storm et al. (2010) 1a, 1b, 2 1 4 T FR text 7 days
3 1 3 T FIB text 7 days
Sumowski et al. (2010) 1 1 2 TR, 2 R CR word pairs 45 min
Szpunar et al. (2007) 1 1 0 T, 1 T FR word lists 30 min
Szpunar et al. (2008) 1, 2 1 0 T, 1 T FR word lists 30 min
3 1 0 T, 1 T, 1 R FR word lists 30 min
Taylor & Rohrer (2010) 1 1 11 TR CR math formulas 1 day
Toppino & Cohen (2009) 1 1 1 T, 1 R CR word pairs 2 min, 2 days
2 1 1 T, 1 R CR word pairs 5 min, 2 days
Wheeler et al. (2003) 1 1 3 T, 3 R FR word lists 5 min, 2 days
2 1 4 T, 4 R FR word lists 5 min, 7 days
Note. Number of Sessions refers to the number of sessions in which a given item was practiced; in this column, variable indicates that the number of sessions in which a given item was practiced was not controlled or manipulated. Number and Kind of Practice Trials summarizes the schedule of practice that took place after initial study; in this column, T = test trial, R = restudy trial, S = session, and variable indicates that the number of trials per item was not controlled or manipulated. Practice Test indicates the format of the tests administered during practice; in this column, CR = cued recall, FIB = fill in the blank, FR = free recall, match = matching cues to targets, MC = multiple choice, perform = performing a procedure, RG = recognition, and SA = short answer. Studied Materials indicates what was initially studied. Retention Interval indicates the delay between the last practice session and the final test; in this column, min = minutes.
Rawson & Dunlosky, online supplemental materials 6
Table S2.
List of testing effect studies published prior to 2000 examined for summary of literature.
Rawson & Dunlosky, online supplemental materials 6
Abbott (1909)
Allen, Mahler, & Estes (1969)
Atkinson & Paulson (1972)
Bahrick (1979)
Bahrick, Bahrick, Bahrick, & Bahrick (1993)
Bahrick & Phelps (1987)
Balch (1998)
Bartlett (1977)
Berger, Hall, & Bahrick (1999)
Birnbaum & Eichner (1971)
Carrier & Pashler (1992)
Cull, Shaughnessy, & Zechmeister (1996)
Darley & Murdock (1971)
Deputy (1929)
Duchastel (1981)
Duchastel & Nungester (1981)
Duchastel & Nungester (1982a)
Duchastel & Nungester (1982b)
Estes (1960)
Fishman, Keller, & Atkinson (1968)
Fitch, Drucker, & Norton (1951)
Foos & Fisher (1988)
Gates (1917)
Gates (1918)
Glenberg (1976)
Glover (1989)
Glover, Krug, Hannon, & Shine (1990)
Götz & Jacoby (1974)
Hanawalt & Tarr (1961)
Hogan & Kintsch (1971)
Izawa (1966)
Izawa (1967)
Izawa (1970)
Izawa (1971)
Keys (1934)
Kika, McLaughlin, & Dixon (1992)
King, Zechmeister, & Shaughnessy (1980)
Kratochwill, Demuth, & Conzemius (1977)
Kuo & Hirshman (1996)
Kuo & Hirshman (1997)
Lachman & Laughery (1968)
Landauer & Ainslie (1975)
Landauer & Bjork (1978)
LaPorte & Voss (1975)
Lockhart (1975)
MacLeod (1985)
Maloney & Ruch (1929)
Mandler & Rabinowitz (1981)
Masson & McDaniel (1981)
McDaniel & Fisher (1991)
McDaniel, Kowitz, & Dunay (1989)
McDaniel & Masson (1985)
Modigliani (1976)
Myers (1914)
Nelson, Leonesio et al. (1982)
Nungester & Duchastel (1982)
Raffel (1934)
Rea & Modigliani (1985)
Rees (1986)
Rose (1992)
Rosner (1970)
Rothkopf (1966)
Royer (1973)
Runquist (1983)
Runquist (1986)
Schacter, Rich, & Stampp (1985)
Shaughnessy & Zechmeister (1992)
Slamecka & Katsaiti (1988)
Sones & Stroud (1940)
Spitzer (1939)
Thompson, Wenger, & Bartling (1978)
Thorndike (1914)
Tulving (1967)
Wenger, Thompson, & Bartling (1980)
Whitten & Bjork (1977)
Whitten & Leonard (1980)
References
Abbott, E. E. (1909). On the analysis of the factors of recall in the learning process. Psychological Monographs, 11, 159–177.
Agarwal, P. K., Karpicke, J. D., Kang, S. H. K., Roediger, H. L. III, & McDermott, K. B. (2008). Examining the testing effect with open- and closed-book tests. Applied Cognitive Psychology, 22, 861-876.
Allen, G. A., Mahler, W. A., & Estes, W. K. (1969). Effects of recall tests on long-term retention of paired associates. Journal of Verbal Learning and Verbal Behavior, 8, 463-470.
Atkinson, R.C., & Paulson, J. A. (1972). An approach to the psychology of instruction. Psychological Bulletin, 78, 49-61.
Bahrick, H. P. (1979). Maintenance of knowledge: Questions about memory we forgot to ask. Journal of Experimental Psychology: General, 108, 296-308.
Bahrick, H. P., Bahrick, L. E., Bahrick, A. S., & Bahrick, P. E. (1993). Maintenance of foreign language vocabulary and the spacing effect. Psychological Science, 4, 316-321.
Bahrick, H. P., & Hall, L. K. (2005). The importance of retrieval failures to long-term retention: A metacognitive explanation of the spacing effect. Journal of Memory and Language, 52, 566-577.
Bahrick, H.P., & Phelps, E. (1987). Retention of Spanish vocabulary over 8 years. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13, 316-321.
Balota, D. A., Duchek, J. M., Sergent-Marshall, S. D., & Roediger, H. L. III (2006). Does expanding retrieval produce benefits over equal-interval spacing? Explorations of spacing effects in healthy aging and early stage Alzheimer’s disease. Psychology and Aging, 21, 19-31.
Barber, S. J., Rajaram, S., & Marsh, E. J. (2008). Fact learning: How information accuracy, delay, and repeated testing change retention and retrieval experience. Memory, 16, 934-946.
Barcroft, J. (2007). Effect of opportunities for word retrieval during second language vocabulary learning. Language Learning, 57, 35-56.
Berger, S. A., Hall, L. K., Bahrick, H. P. (1999). Stabilizing access to marginal and submarginal knowledge. Journal of Experimental Psychology: Applied, 5, 438-447.