Moving SoTL Projects Forward: Applying what you already know and do to SoTL - Methodologies and Literature Searches

Alix Darden,

This workshop will focus on the basics needed to begin ones’ own research and scholarship in teaching and learning. Participants will be engaged in a hands-on minds-on workshop and are encouraged to bring questions, materials and/or SOTL projects they are engaged in or are thinking about. It is assumed that participants have a question in mind or in progress. The main focus of the workshop will be on finding appropriate methodologies and relevant literature. Other topics that will be covered include human subject considerations, finding mentors and collaborators and qualitative and quantitative assessment of data.

Assessment definitions

Function of assessment

Diagnostic Assessment – The gathering of information at the outset of a course or program of study to provide information for both the teacher and student about students’ prior understanding. Diagnostic assessment can help to maximize learning by: 1) revealing what students already know and don’t know about a subject so that the teacher can focus lessons appropriately; 2) revealing what students already know and don’t know about a subject so that the student can focus their study appropriately; 3) exposing misunderstandings and misconceptions in prior knowledge; and 4) making it clearer to students the type of understanding that the teacher values for this subject. This type of assessment, coupled with summative assessment, is used to determine the value-added of an educational experience.

Formative Assessment - The gathering of information about student learning-during the progression of a course or program and usually repeatedly-to improve the learning of those students. Formative assessment supports student learning through constructive feedback by: 1) diagnosing student difficulties; 2) measuring improvement over time; and/or 3) providing information to inform students about how to improve their learning. Example: reading the first lab reports of a class to assess whether some or all students in the group need a lesson on how to make them succinct and informative.

Summative Assessment - The gathering of information at the conclusion of a course, program, or undergraduate career to improve learning or to meet accountability demands. Ideally, summative assessment reflects the culmination of the scaffolding process of learning provided by formative assessment throughout the course. When used for improvement, impacts the next cohort of students taking the course or program. Examples: examining student final exams in a course to see if certain specific areas of the curriculum were understood less well than others; analyzing senior projects for the ability to integrate across disciplines.

Assessment targets

Assessment of Individuals, Assessment of Assignments, Assessment of Courses, Assessment of Programs, Assessment of Institutions.

Assessment at a point in time v. longitudinal assessment

Assessment tools and techniques

Qualitative Assessment - Collects data that does not lend itself to quantitative methods but rather to interpretive criteria

Quantitative Assessment - Collects data that can be analyzed using quantitative methods, i.e. numbers, statistical analysis

Validity – A measure of how well an assessment relates to what students are expected to have learned. A valid assessment measures what it is supposed to measure and not some peripheral features.

Reliability – A measure of the constancy of scoring outcomes over time or over many evaluators. A test is considered reliable if the same answers produce the same score no matter when and how the scoring is done.

Direct Assessment of Learning - Gathers evidence, based on student performance, which demonstrates the learning itself. Can be value added, related to standards, qualitative or quantitative, embedded or not, using local or external criteria. Examples: most classroom testing for grades is direct assessment (in this instance within the confines of a course), as is the evaluation of a research paper in terms of the discriminating use of sources. The latter example could assess learning accomplished within a single course or, if part of a senior requirement, could also assess cumulative learning.

Indirect Assessment of Learning - Gathers reflection about the learning or secondary evidence of its existence. Example: a student survey about whether a course or program helped develop a greater sensitivity to issues of diversity.

Embedded Assessment - A means of gathering information about student learning that is built into and a natural part of the teaching-learning process. Often uses for assessment purposes classroom assignments that are evaluated to assign students a grade. Can assess individual student performance or aggregate the information to provide information about the course or program; can be formative or summative, quantitative or qualitative.

Assessment Methodologies – You need to be very clear as to your research objective – What specifically do you want to measure/evaluate? Is your assessment method a valid and reliable measurement of what you want to measure?

Assessing learning

  • Objective tests
  • Pre/Post tests
  • Rubrics
  • Classroom assignments
  • Skills competency
  • Student Portfolios
  • Audio taping
  • Video taping

Surveys – student self-report

Types of questions

  • Likert scale questions – rate 1-7, from best to worst. Can be statistically analyzed
  • Open-ended questions – “code” the written answers

Types of surveys

  • satisfaction
  • values, attitudes, expectations
  • confidence
  • motivation
  • learning

Assessing learning development

  • Bloom’s taxonomy
  • SOLO taxonomy

Focus Groups/interviews

Audio taping

Observations

  • Video taping
  • Think alouds

Triangulation/ Mixed-methodology

Bibliography

On-Line Searching

  • To find on-line SoTL resources, Google “SoTL” or “Your discipline”+SoTL
  • ERIC – Educational Resources Information Center - Search a bibliographic database of more than 1.1 million citations on education topics going back to 1966.

Assessment resources

  • Georgetown University’s Visible Knowledge Project toolkit for analyzing qualitative data.
  • NSF 2002 user-friendly Handbook for Project Evaluation -
  • Rubistar – a free tool to help a teacher make quality rubrics -
  • Angelo, T.A. and K. P. Cross. 1993. Classroom Assessment Techniques. 2nd ed. Jossey-Bass, 427 pp. Paperback.
  • Beckman, TJ and Cook DA. 2007. Developing Scholarly projects in education: a primer for medical teachers. Medical Teacher, 29:210-218.
  • Biggs, J. B. and Collis, K.F. 1982. Evaluating the quality of learning: the SOLO taxonomy. Academic Press, New York.
  • Bloom, B. S. 1956. Taxonomy of educational objectives, handbook I: The cognitive domain. David McKay Co. Inc., New York.
  • Brookhart, SM. 1999.The art and Science of Classroom Assessment: The missing part of Pedagogy. ASHE-ERIC Higher Education Report (Vol 27, No. 1)Washinton DC:The George Washinton University, Graduate School of Education and Human Development.
  • Dunn, L., Morgan, C., O’Reilly, M. and Parry, S. 2004. The student assessment handbook: new directions in traditional and online assessment. Routledge Falmer, New York.
  • Epstein, RM. 2008. Assessment in Medical Education. N Eng J Med. 356(4):387-396.
  • Geisler, C.A. 2003. Analyzing Streams of Language: Twelve Steps to the Systematic Coding of Text, Talk, and Other Verbal Data (Paperback) - expensive brand new, but can get used copies on line
  • Huba, M. E., and Freed, J. E., 2000, Learner-centered assessment on college campuses shifting the focus from teaching to learning, Allyn and Bacon, Boston, MA.
  • Krueger, R. A., and Casey, M. A., 2000, Focus groups: a practical guide for applied research, Sage Publications, Thousand Oaks, Calif.
  • Stevens, D.D. and Levi, A.J. 2005, Introduction to Rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning. Stylus Publishers, Inc., Sterling, VA.
  • Walvoord, B. E. F., and Anderson, V. J., 1998, Effective grading a tool for learning and assessment, Jossey-Bass Publishers, San Francisco, CA.

Development of the Learner

  • Baxter Margolda, MB. 1999. Creating Contexts for Learning and Self-Authorship: Constructive-Developmental Pedagogy.VanderbiltUniversity Press.
  • Bransford, JD, Brown, AL and Cocking, RR, eds. 2000. How People Learn: Brain, Mind, Experience and School. NationalAcademy Press, WashingtonDC.
  • Nelson, C 1999. On The Persistence Of Unicorns: The Tradeoff Between Content And Critical Thinking Revisited. In B. A. Pescosolido and R.Aminzade, eds. The Social Worlds of Higher Education. Pine Forge Press. Application of Perry in the classroom.
  • Perry, WG[1970] 1998. Forms of Intellectual and Ethical Development in the College Years, A Scheme. New introduction by Lee Knefelkamp. Jossey-Bass.

Some classic SoTL studies

  • R. R. Hake. 1998. Interactive-engagement vs. traditional methods: A six- thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics 66, 64. Classic study showing the impact of active learning using the mechanics concept inventory as pre/post tests.
  • P. U. Treisman. 1992. Studying Students Studying Calculus: A Look at the Lives of Minority Mathematics Students in College. College Mathematics Journal 23: 362-72. [UC Berkely, Workshop Calculus, African-American D/F/W rate from 60 percent to 4 percent.] See also R. E. Fullilove and P. U. Treisman. 1990. Mathematics Achievement Among African American Undergraduates at the University of California, Berkeley: An Evaluation of the Mathematics Workshop Program. Journal of Negro Education 59: 463-78.
  • M. D. Sundberg and M. L. Dini. 1993. Science majors vs. nonmajors: Is there a difference? Journal of College Science Teaching Mar./Apr. 1993:299-304. [Multiple sections and instructors. Both courses taught with traditional pedagogy, but with different intensities of ‘coverage.’ “The most surprising, in fact shocking, result of our study was that the majors completing their course did not perform significantly better than the corresponding cohort of nonmajors.” [Note: Less wasn't more, without pedagogical change, but more wasn't more either—and student attitudes suffered.]
  • J. Russell, W. D. Hendricson, and R. J. Herbert. 1984. Effects of lecture information density on medical student achievement. Journal of Medical Education 59:881-89. Three different lectures on the same subject. 90 percent of the sentences in the high-density lecture disseminated new information as did 70 percent in the medium and 50 percent in the low. Remaining time used for restating, highlighting significance, more examples, and relating the material to the student's prior experience. Students randomly distributed into the 3 groups (no significant differences in prior GPA or on knowledge base pretest). Students in low treatment learned and retained lecture information better. Here less is more.

Active Learning

Allen, D and Tanner, K. Infusing active learning into the large-enrollment biology class: Seven strategies, from the simple to complex. Cell Biology Education. 4:262-268. 2006.

Angelo, TA, KP Cross: Classroom assessment techniques: a handbook for college teachers, 2nd edn. San Francisco: Jossey-Bass Publishers; 1993.

Bean, JC: Engaging Ideas.The Professors Guide to Integrating Writing, Critical Thinking, and Active Learning in the Classroom. San Francisco: Jossey-Bass Publishers; 2001.

Bess, JL: Teaching alone, teaching together transforming the structure of teams for teaching, 1st edn. San Francisco, CA: Jossey-Bass; 2000.

Bonwell, CC and JA Eison: Active Learning: Crating Excitement in the Classroom. Washington DC, The George Washington University; 1991.

Bowman, S. Preventing Death by Lecture. 2005. Bowperson Publishing Co. Glenbrook, NV.

Bransford, JD, AL Brown, RR Cocking, eds: How People Learn. Brain, Mind, Experience and School. Washington DC, National Academy Press; 2000. Available free online at:

Davis, BG: Tools for teaching, 1st edn. San Francisco, CA: Jossey-Bass Publishers; 1993.

Ebert-May, D, C Brewer, S Allred: Innovation in Large Lectures--Teaching for Active Learning.Bioscience 1997, 47:601-607.

Fink, L. Dee: Creating Significant Learning Experiences. San Francisco, CA: Jossey-Bass; 2003.

Ghosh, R: The Challenges of Teaching Large Numbers of Students in General Education Laboratory Classes Involving Many Graduate Student Assistants.Bioscene 1999, 25:7-11.

Knight, JK and Wood, WB: Teaching more by lecturing less.Cell Biology Education. 2005. 4:298-310.

MacGregor, J: Strategies for energizing large classes from small groups to learning communities. San Francisco, CA: Jossey-Bass; 2000.

McKeachie, WJ, G Gibbs: McKeachie's teaching tips strategies, research, and theory for college and university teachers, 10th edn. Boston, MA: Houghton Mifflin Co; 1999.

Redish, EF, JM Saul, RN Steinberg: Student Expectations in Introductory Physics. 2000.

Stanley, CS, ME Porter: Engaging large classes: strategies and techniques for college faculty. Bolton, Mass.: Anker Pub. Co.; 2002.

Zull, JE: The Art of Changing the Brain. Sterling, VA: Stylus Publishing, 2002.

1