TITLE: Exploring Learning of Assessment during a “Process Based” Faculty Development Workshop

AUTHOR LIST: Robert M. Carson, Scott K. Metlen, Tris Utschig, Donald F. Elger

ABSTRACT:

I.INTRODUCTION

  1. Present Situation
  2. Students and educators value meaningful learning and growth in performances that are relevant to professional practice. Both learning and professional growth are accelerated by effective assessment (add citations).
  3. Because of this, assessment is part of the ABET and is considered a best practice in education.
  4. By assessment we mean the process of analyzing and providing feedback about a performance or work product. The purpose of assessment is to help the assessee improve their performance or work product of interest.
  5. Performance in assessment can span a continuum from ineffective to effective. Some of the criteria for effective assessment are:
  6. The process of assessment is welcomed by the assessee.
  7. The process of assessment meets the motivational, cognitive, and social needs of the assessee
  8. Assessment is ongoing. When plants are watered and fertilized regularly, they thrive. Similarly, ongoing assessment leads to high-performing, vibrant and growing people.
  9. The process of assessment produces for the assessee a useful combination of learning, growth, and improvements in performances, and improvements in work products.
  10. The process of assessment impacts learning and growth beyond the assessee. That is, assessment impacts the assessor and other members of the professional community.
  11. Faculty are not usually taught assessment
  12. Assessment is high level performance and thus is not easily learned by faculty
  13. Needs
  14. Need for learners (professors, students, managers, …) to apply effective assessment practices. To develop into effective assessors, people need knowledge of assessment plus the ability to apply this knowledge in context.
  15. Need for a culture in high ed that values assessment with this valuing evidenced by the regular practice of assessment throughout the organization of a university.
  16. Need for the practice assessment in a way that allows people to take ownership of their growth and improvement of their work products. .
  17. Research Question
  18. For adult learners who are improving their assessment practice, what factors mediate in learning and performance improvements during a workshop?
  19. What challenges do they face?
  20. What are the cultural norms for assessment practice?
  21. What are the challenges to transfer beyond the workshop boundaries?
  22. What are the motivations of the faculty?
  23. What factors of the leaning environment are important?
  24. How important is practice w/ feedback?
  25. What is their conceptualization of assessment when thy enter the workshop
  26. Context
  27. Faculty development workshop, UI, 35?, facilitated by Pacific Crest.The present study uses case study (add), one of the five traditions of qualitative social research (Creswell, 1998; Creswell, 2003.). The study was conducted at the University of
  28. Research Team comprised on a engr. manufacturing manager , an operations management specialist with significant experience, and a learning specialists. POVs that are brought by the team
  29. Performance is what motivates people. True of students and true of faculty.
  30. People learn performances best by doing and then receiving feedback
  31. Focus on people and their learning is key. This leads to results, whereas a direct focus on results usually fails because the people expected to produce the results lack the performances.

II.LITERATURE REVIEW

To understand learning by professors, researchers have created various theoretical frameworks. Here, we review three frameworks: learning as a process of social participation in a community of practice, learning by an organization, and adult learning.

A.Adult Learning

Cross (1981) presents the Characteristics of Adults as Learners (CAL) model in the context of her analysis of lifelong learning programs. The model attempts to integrate other theoretical frameworks for adult learning such as andragogy ( Knowles ), experiential learning ( Rogers ), and lifespan psychology.

The CAL model consists of two classes of variables: personal characteristics and situational characteristics. Personal characteristics include: aging, life phases, and developmental stages. These three dimensions have different characteristics as far as lifelong learning is concerned. Aging results in the deterioration of certain sensory-motor abilities (e.g., eyesight, hearing, reaction time) while intelligence abilities (e.g., decision-making skills, reasoning, vocabulary) tend to improve. Life phases and developmental stages (e.g., marriage, job changes, retirement) involve a series of plateaus and transitions which may or may not be directly related to age.

Situational characteristics consist of part-time versus full-time learning, and voluntary versus compulsory learning. The administration of learning (i.e., schedules, locations, procedures) is strongly affected by the first variable; the second pertains to the self-directed, problem-centered nature of most adult learning.

Knowles' theory of andragogy is an attempt to develop a theory specifically for adult learning. Knowles emphasizes that adults are self-directed and expect to take responsibility for decisions. Adult learning programs must accommodate this fundamental aspect.

Andragogy makes the following assumptions about the design of learning: (1) Adults need to know why they need to learn something (2) Adults need to learn experientially, (3) Adults approach learning as problem-solving, and (4) Adults learn best when the topic is of immediate value.

In practical terms, andragogy means that instruction for adults needs to focus more on the process and less on the content being taught. Strategies such as case studies, role playing, simulations, and self-evaluation are most useful. Instructors adopt a role of facilitator or resource rather than lecturer or grader.

Rogers distinguished two types of learning: cognitive (meaningless) and experiential (significant). The former corresponds to academic knowledge such as learning vocabulary or multiplication tables and the latter refers to applied knowledge such as learning about engines in order to repair a car. The key to the distinction is that experiential learning addresses the needs and wants of the learner. Rogers lists these qualities of experiential learning: personal involvement, self-initiated, evaluated by learner, and pervasive effects on learner.

To Rogers, experiential learning is equivalent to personal change and growth. Rogers feels that all human beings have a natural propensity to learn; the role of the teacher is to facilitate such learning. This includes: (1) setting a positive climate for learning, (2) clarifying the purposes of the learner(s), (3) organizing and making available learning resources, (4) balancing intellectual and emotional components of learning, and (5) sharing feelings and thoughts with learners but not dominating.

According to Rogers, learning is facilitated when: (1) the student participates completely in the learning process and has control over its nature and direction, (2) it is primarily based upon direct confrontation with practical, social, personal or research problems, and (3) self-evaluation is the principal method of assessing progress or success. Rogers also emphasizes the importance of learning to learn and an openness to change.

B.Learning by Organizations

Organizational learning (OL) is an area of knowledge within organizational theory that studies models and theories about the way an organization learns and adapts [copied from web:

Generally, one can distinguish between two different processes of organizational change that are associated with OL:

  • adaptive learning, i.e. changes that have been made in reaction to changed environmental conditions and
  • proactive learning, i.e organizational changes that have been made on a more willful basis. This is learning which goes beyond the simple reacting to environmental changes.

In general, it is assumed that adaptive learning comes along with a lower degree of organizational change. This means that adaptive learning is seen as a process of incremental changes. What is more, adaptive learning is also seen as more automatic and less cognitively induced than proactive learning. [Copied from web:

How organizations learn

Communication: horizontal vertical within and without, team and personal

Changes tacit to explicit

Grows tacit

Changes personal to corporate

Recognize that one person learns in dialogue with another, together a system is created, together we learn in that system. What we learn and do changes the system.

Anderson, Marilynne (1995) Ahead of the Wave: Valuing Gender Perspective in Learning Cultures.Edited by Chawla, Sarita & Renesch, John Learning Organizations. Productivity Press, Portland, OR

Doing: to increase success, have to increase failures

There is no set theory of organizations except that organizations will always have to change to adapt to an ever dynamic environment. To do so, the organization must hire those who embrace the unknown, and provide an environment where consent, not control is the rule. No matter how large and dispersed the organization gets everyone understands that they are together, all part of the same system. Many mistakes because they are exploring (doing), but they learn from them. They learn because they recognize that learning is the only way to live.

Handy, Charles (1995). Managing the Dream. Edited by Chawla, Sarita & Renesch, John Learning Organizations. Productivity Press, Portland, OR

Over arching set of rules and procedures that allow/foster culture of doing and communication

Leaders need to establish an environment where employees experience increased self esteem, connection, dignity and security, can create, be listened to, cared for, live their values, self discover

Rolls, Jayme 1995 The Transformational Leader: The Wellspring of the Learning Organization. Edited by Chawla, Sarita & Renesch, John Learning Organizations. Productivity Press, Portland, OR

The culture is one where it is safe to say what you think (respectfully) about products and processes and generative conversation and coordinated action (language is device for connection, invention, and coordination) is fostered. An understanding of how your actions affect total system, not just your part of the system is needed to understand enough about the process and system to make suggestions.

Kofman, Fred, Senge, Peter M., (1995). Communities of Commitment: The Heart of Learning Organizations. Edited by Chawla, Sarita & Renesch, John Learning Organizations. Productivity Press, Portland, OR.

Over arching set of values, purpose, and vision that are understood and accepted by all

Values: love (acceptance of others as legitimate beings), wonder (problems that fate throws at you are opportunities), humility (always room to improve), compassion (recognize others view points as valid)

Kofman, Fred, Senge, Peter M., (1995). Communities of Commitment: The Heart of Learning Organizations. Edited by Chawla, Sarita & Renesch, John Learning Organizations. Productivity Press, Portland, OR.

Assessment dealing with how well values purpose and vision are being reached See 12 points below

Assessment of learning to determine if it is dealing with advancing values purpose and vision see 12 points below

Resources so can do and communicate

A program that makes workers more receptive to innovations imposed on them is not possible. Change is always a threat when it is imposed (done to people), but it is an opportunity when driven by people. Give people the tools (resources) and autonomy to make change happen.

Kanter, Rosabeth Moss (1995). Mastering Change. Edited by Chawla, Sarita & Renesch, John Learning Organizations. Productivity Press, Portland, OR

Organizational Conditions for Building a Learning Organization (LO)

1)Management commitment to making learning a key sustainable competitive advantage

2)Vision of desired learning organization that all buy into and are/remain excited about

3)Clear process of how organization will change to being a LO

4)Milestones that are identifiable, achieved, and celebrated on the quest to being a learning organization.

5)Management modeling learning (most important)

6)Correct those managers and others who do not embrace the change

7)Management releasing resources and recognizing that resource need is ongoing (years, forever; change does not happen fast)

8)System that matches rewards to achievement of milestones

9)Not only is experimentation, collaboration, innovation, and new paradigm thinking accepted, it is encouraged

10)Urgency

11)Multiple feedback structures

12)Multiple learning channels

Thompson, John W. (1995) The Renaissance of Learning. Edited by Chawla, Sarita & Renesch, John Learning Organizations. Productivity Press, Portland, OR

C.Communities of Practice

Learning can be described as a process of social participation in a Community of Practice (add citations). Key assumptions of the Community of Practice perspective as they apply to this research project:

  • Professionals organize their learning through the professional communities to which they belong.
  • Knowledge is integrated in the life of professional communities that share values, beliefs, languages, and ways of doing things.
  • The processes of learning and membership in a community of practice are inseparable.
  • Knowledge is inseparable from practice. It is not possible to know without doing. By doing, we learn.
  • To empower people, provide opportunities for them to contribute to the community and provide opportunities to engage in real action that impacts individuals and the communities. This empowerment creates the most powerful learning environments.

D.Summary Comments on the Literature Review

The literature provides rich and deep insights into the nature of learning as it applies to professors. While there are various theoretical perspectives, there are a number of common themes. These themes and how they connect with this study are.

  • Learning by experience is effective
  • Adult learners chose what they learn based on
  • Learning is social
  • Learning is organizational—

III.DESCRIPTION OF WORKSHOP

Part of ELE grant

Solicitation @ UI, WSU, LCSC. Leads. Marketing

Description of facilitator.

Workshop Objectives

  • Learners will recognize the value of daily reflective practice. The way this will be manifest after the workshop is over is that the learners will be more inclined to, on a regular basis (several times per week), write down strengths of their present performance, ideas for improving their present performance, and insights. These writings will demonstrate processing of “what is important” and “why is this important.”
  • Learners will develop a desire for an academic culture in which there is effective assessment (see intro section for defn of effective assessment). The way this will be manifest after the workshop is over is that the learners
  • During the workshop, learners will improve their abilities to analyze their observations and produce meaning and learning from these observations. That is, learners will become more effective at experiential learning. The way that this will be manifest after the workshop is over is ……

SII—give handout in appendix A

Workshop Methods—use photos

IV.METHODS

We chose case study from the five traditions described by Creswell (1998) because ….. While the case student is largely qualitative, we selected mixed methods ….. We used six instruments to gather data

A.Pre and post survey Instrument

Appendix B contains the pre & post survey. Mixed. Rationale. Purpose of intrument

  • What did participants know about assessement when they came in
  • What were the changes that
  • Did the workshop have any impacts?

B.Workshop interview Instrument

Appendix B contains the workshop interview instrument. Purpose = rich & deep understanding the factors that are mediating in the learning.

C.Factor Survey Instrument

App. C. Purpose = rank order the factors that mediating in learning.

D.Factor Focus Group Instrument

App. D. Purpose = focus group ranking of factors that mediated in learning.

E.Workshop E-Log

V.Results

F.Participant Data

Table 1 presents demographic data for the participants.

VI.CONCUSIONS

This study addressed the question of how professors learn …

Acknowledgements

This work was funded in part by the National Science Foundation: Grant Numbers EEC0212293 This work was funded in part by the University of Idaho Department of Mechanical Engineering, College of Engineering, and Office of the President.

VII.List of References Cited

Beyerlein, S.W., and Apple, D.K. (2004) Faculty guidebook: A comprehensive tool for improving faculty performance, Lisle, IL: Pacific Crest.

Black, P. and Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Kings College London, School of Education. Retrieved December 29, 2004 from

Blythe, T. (1998). The teaching for understanding guide. San Francisco: Jossey-Bass.

Boyer, E. (1995). The basic school: a community for learning. The Carnegie Foundation for the Advancement of Teaching, San Francisco: Jossey-Bass.

Bransford, J.D., Brown, A.L., Cocking, R.R., eds. (2000). How people learn. Washington D.C.: National Academy Press. (also available online at

Boyer, E. (1995). The basic school: a community for learning. The Carnegie Foundation for the Advancement of Teaching, Josey Bass: San Francisco.

Creswell, J.W. (1998). Qualitative inquiry and research design: Choosing among the five traditions. Thousand Oaks, California: Sage.

Creswell, J.W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, California: Sage.

Dewey, J. (1938). Experience and education. New York: Touchstone.

Dick, W., Carey, L., & Carey, J.A. (2005). The systematic design of instruction. New York: Pearson.

Gagne, R.M., Briggs, L.J. & Wager, W.W. (1992). Principles of instructional design. Belmont, CA: Wadsworth/Thompson Learning.

Duncan-Hewitt, W., Mount, D., Beyerlein, S., Elger, D., & Steciak, J. (2001). Using developmental principles to plan design experiences for beginning engineering students, Proceedings of 2001 Frontiers in Education Conference.

King, P. M., & Kitchener, K. S. (1994). Developing reflective judgment: Understanding and promoting intellectual growth and critical thinking in adolescents and adults. San Francisco: Jossey-Bass.

Kolb, D.A. (1984). Experiential Learning: Experience as the source of learning and development. Eaglewood Cliffs, NJ: Prentice Hall.

Leonard, D.C. (2002). Leaning theories: A to Z. Westport, CT: Greenwood Publishing Group.

Perry, W.G. (1970). Forms of intellectual and ethical development in the college years: A scheme. New York: Holt, Rinehart and Winston.

Stiggins, R. (2000). Student involved classroom assessment. NewYork: Prentice Hall.

Strauss A. & Corbin J. (1998). Basics of qualitative research: Techniques and procedures of developing grounded theory. Thousand Oaks, California: Sage.

Wenger, E. (1998). Communities of practice: Learning, meaning, and identity. Cambridge: Cambridge University Press.

Wiske, M.S., Ed. (1998). Teaching for understanding: Linking research with practice. San Francisco: Jossey-Bass.