Institutional Effectiveness Training Manual

ABHES

7777 Leesburg Pike, Suite 314N

Falls Church, Virginia 22043

703.917.9503 /

January 2008

TABLE OF CONTENTS

The Purpose of the IEP...... 3

The Standards for the IEP...... 4

(As published in the ABHES Standards Manual)

Section B, Subsection 1 – Institutional Effectiveness

Standard IV.B.1 (IEP)...... 4

Section B, Subsection 2 – Outcomes Assessment

Standard IV.B.2.a (Student Surveys)...... 5

Standard IV.B.2.b (Graduate Surveys)...... 5

Standard IV.B.2.c (Employer Surveys)...... 5

Standard IV.B.2.d (Advisory Board)...... 5

Standard IV.B.2.e (Credentialing)...... 6

Standard IV.B.2.f (Credentialing Pass Rates)...... 6

Standard IV.B.2.g (Retention Rates)...... 6

Standard IV.B.2.h (Placement Rates)...... 6

Standard IV.B.2.i (Short & Long-Term Goals)...... 7

Developing, Implementing, and Monitoring the IEP...... 8

Guidelines (As published in the ABHES Standards Manual)...... 10

Subsection 1 – Assessment of Education Effectiveness

  1. Program Characteristics...... 10
  2. Student population...... 11
  3. Types of data to be used for assessment...... 12
  4. Expected outcomes...... 13
  5. How continuous improvement will be made to enhance outcomes...... 13
  6. The ten elements to be evaluated for institutional effectiveness...... 14
  7. Annual evaluation...... 23

Subsection 2 – Outcome Assessment

  1. Initial baseline rates & measurements of results...... 23
  2. Historical outcomes used to achieve expected goals...... 23
  3. Adjusting goals...... 23
  4. Data collection and rationale for use of each type of data ...... 24
  5. Summary and analysis of data collected...... 24
  6. Using data to improve the educational process...... 25
  7. Outcome rates, future goals, and the rates for the past two years...... 25
  8. Activities undertaken to meet future goals...... 25

Conclusion...... 27

1

THE PURPOSE OF THE

INSTITUTIONAL EFFECTIVENESS PLAN

Developing and using the Institutional Effectiveness Program (IEP) should fulfill several purposes, including:

1.Assisting the institution in achieving internal effectiveness through establishing goals for both short- and long-term successes. Further, criteria for measuring the accomplishment of these goals can be defined, allowing the institution to focus its plans and activities on the critical processes needed for effectiveness. Once defined, these goals and criteria should then be used to unify administrative and educational activities, which can help achieve a high degree of commitment and common direction among all employees.

2.Assessing progress and the need for change and continuously reviewing the process to help the institution make timely changes based upon valid information to achieve even greater effectiveness.

3.Communicating key information regarding the institution’s goals, its degree of effectiveness, and how it plans to enhance overall quality to external publics, such as graduates, employers, and community leaders. Information, which depicts the most important elements of the institution’s operation, communicates clearly and accurately to external publics how well the institution is meeting the needs of students and providing quality-learning experiences.

  1. Measuring how the IEP meets the expectations and requirements of approving or accrediting organizations, including state boards and ABHES to demonstrate regulatorycompliance. A document which defines institutional goals and educational processes is a primary focus of most accrediting agencies as they measure overall effectiveness and the quality of programs and services provided.

All institutional effectiveness goals and activities the institution selects should link to the mission to demonstrate mission achievement and continuous improvement, as the institution’s mission is the driving force of the institutional effectiveness program. The IEP requires an institution to look at its past, present, and future and to continuously ask:

Where have we been? / This data becomes the baseline for gauging and demonstrating improvements.
Where are we now? / Current data demonstrates how you will measure change from the baseline data using the caparison to identify changes needed.
Where do we want to go? / A look towards the future for goals to improve or enhance processes and/or programs.

THE STANDARDS FOR THE

INSTITUTIONAL EFFECTIVENESS PLAN

The following is the Assessment of Educational Effectiveness standard (Section B, Subsection 1) as published in the ABHES Standards Manual. The standards outline the ABHESrequirements regarding the development, implementation, and maintenance of an IEP, including the outcomes assessment requirements (Section B, Subsection 2), which expounds on many of the required components of an IEP:

SECTION B – Assessment of Educational Effectiveness

An institution documents success based on student achievement in relation to its mission, including, as appropriate, consideration of retention rates, results of licensing and certification examinations, graduation rates, job placement rates, and responses from graduates and employers in the workplace. An institution also reviews as applicable, default rates in the student loan programs under Title IV of the Higher Education Act, based on the most recent data provided by the Secretary of Education. These findings are coupled with student retention and placement rates to determine what correlation, if any, can be determined. Any correlation identified is reviewed for correction and acted upon promptly.

Subsection 1 – Institutional Effectiveness

IV.B.1.An institution has an established documented plan and process for assessing its effectiveness, which meets the requirements of Appendix C (Institutional Effectiveness Plan).

The plan and process is reviewed and updated annually and addresses all delivery methods. The IEP includes:

a.Its relevance to the mission and objectives of an institution and its programs.

  1. Student retention rates by program.

c.Graduate employment rates by program.

d.Results of licensure and credentialing exams, by program, as required for employment by accepted industry standards.

  1. Student and graduate satisfaction by program.
  2. Externship and employer satisfaction by program.
  3. Assessment of faculty participation in professional growth and in-service activities.
  4. Advisory board participation.
  5. Short- and long-term goals for improving its qualitative and quantitative outcomes.
  6. Effectiveness of distance education delivery, if applicable.

An institution evidences continued progress and effort through use of this plan in improving educational effectiveness, operating efficiency, and service.

At a minimum, this plan must include written documentation of an annual review evidencing that all components have been evaluated by appropriate personnel for effectiveness and actions taken as deemed necessary.

Subsection 2 – Outcomes Assessment

IV.B.2.a.Aninstitution maintains and uses documented student comments concerning course content and instruction to improve its programs.

An institution has a systematic plan for regularly surveying students. Survey questions include as appropriate such things as satisfaction with the institution, training, and job preparedness. At a minimum, an annual review of the results of the surveys is conducted, and the results are shared with administration, faculty and advisory boards. Decisions and action plans are based upon the review of the surveys, and any changes made are documented (e.g., meeting minutes, memoranda).

IV.B.2.b.An institution maintains and uses documented evaluations by graduates of the program.

An institution has a systematic plan for regularly surveying graduates. Survey questions include as appropriate such things as satisfaction with the institution, training, job preparedness, and credentialing and employment outcomes. At a minimum, an annual review of the results of the surveys is conducted, and the results are shared with administration, faculty and advisory boards. Decisions and action plans are based upon the review of the surveys, and any changes made are documented (e.g., meeting minutes, memoranda).

IV.B.2.cAn institution maintains and uses documented evaluations by the employers of graduates.

An institution has a systematic plan for regularly surveying employers. Survey questions include as appropriate such things as satisfaction with the institution, training, job preparedness, and graduate effectiveness on the job. At a minimum, an annual review of the results of the surveys is conducted, and the results are shared with administration, faculty and advisory boards. Decisions and action plans are based upon the review of the surveys, and any changes made are documented (e.g., meeting minutes, memoranda).

IV.B.2.d.An institution has an active advisory board of appropriate specialists, representing its communities of interest, to assist administration and faculty in fulfilling stated educational objectives.

An institution's advisory board is composed of a minimum of one community representative (e.g. employers, extern site supervisors) not employed by the institution from each program discipline or group of related programs. The board can include current students, graduates and institutional staff and faculty who actively participate in scheduled meetings. Depending on the size of an institution and number of programs offered, board composition or format (e.g., one board for the entire institution with sub-committees per discipline, separate boards per discipline) will differ. Institutions offering complete or partial programs by distance education have a non-school employee representative specializing in this method of delivery. The board convenes a minimum of twice each year and addresses such topics as an institution's mission and objectives, curriculum, and student, graduate and employer comments. Prepared minutes of meetings and recommendations made by the advisory board members are maintained and used to improve curricula. Meeting minutes must include participation of members, topics discussed, recommendations made, and resulting actions.

IV.B.2.e.An institution regularly reviews the results of its graduates’ performance on credentialing and/or licensure examinations required for employment to improve its educational process and employment opportunities for graduates.

An ongoing review of graduate success on credentialing and/or licensing examinations required for employment is performed to identify curricular areas in need of improvement. An institution maintains documentation of such review and any pertinent curricular changes made as a result.

IV.B.2.f.An institution annually calculates the results of its graduates’ performance on licensure and credentialing examinations required for employment.

The method of calculation, using ABHES’ reporting period July 1 through June 30th, is as follows:

F/G= L%

F=Graduates passing examination (any attempt)

G=Total graduates taking examination

L=Percentage of students passing examination

At a minimum, an institution maintains the names of all graduates by program, actual graduation date, and the credentialing or licensure exam for which they are required to sit for employment.

IV.B.2.g.An institution demonstrates that students complete their programs.

At a minimum, an institution maintains the names of all enrollees by program, start date, delivery method and anticipated or actual graduation date. The method of calculation, using the reporting period July 1 through June 30, is as follows:

(EE + G)/(BE+NS+RE)=R%

EE = Ending Enrollment (as of June 30 of the reporting period)

G =Graduates

BE =Beginning Enrollment (as of July 1 of the new reporting period)

NS =New Starts

RE =Re-Entries

R % = Retention Percentage

IV.B.2.h.An institution demonstrates the successful employment of its graduates in the field, or related field, for which they were trained.

An institution has a system in place to assist with the successful initial employment of its graduates. At a minimum, an institution maintains the names of graduates, place of employment, job title, employer telephone numbers, and employment dates. For any graduates identified as self-employed, an institution maintains appropriate evidence of employment. Documentation in the form of employer or graduate verification forms or other evidence of employment must be retained.

The method of calculation, using the reporting period July 1 through June 30, is as follows:

(F + R)/(G-U)=P%

F = Graduates placed in their field of training

R = Graduates placed in a related field of training

G = Total graduates

U* = Graduates unavailable for placement

P% = Placement percentage

*Unavailable is defined only as documented: health-related issues, military obligations, or continuing education status.

IV.B.2.i.An institution maintains short and long term goals for improving its quantitative and qualitative outcomes.

An institution should not only develop goals, but also implement and review such goals for the purpose of improving outcomes. At a minimum, an institution must annually evaluate its success or failure in achieving its goals.

DEVELOPING, IMPLEMENTING, AND MONITORING

THE INSTITUTIONAL EFFECTIVENESS PLAN

Developing an IEP requires that an institution collect, maintain, and use information reflecting the areas outlined in Chapter IV, Section B of the ABHES Manual, including Appendix C. The data should be analyzed for a specific period of time, as defined by the institution and be used as the foundation for making comparisons across future time periods. Many institutions perform its analysis in conjunction with its fiscal/calendar year-end or, as recommended by ABHES, in conjunction with its annual reporting period (July 1 – June 30), since the majority of the required IEP information is also required on the ABHES Annual Report. Regardless of the selected timeframe, the data should be used as the foundation for making comparisons across future time periods and should be updated annually.

The IEP is unique to the institution and the institution must evidence its efforts made to ensure continuous improvement. A process that involves an institution to: (1) systematically collect data and information on each of the educational outcomes areas and achievement of its occupational objectives at least annually; (2) complete an analysis of the data and information including, but not limited to, performing a comparison with previous findings; and (3) identify what changes in educational operations or activities it will make based on the analysis.

Steps in preparing and managingthe IEP are similar to those suggested for institutions preparing a self-study. Proper organization is essential. Although the exact organizational procedures will vary from institution to institution, the following suggestions may be helpful:

  • The president/director, director of education, a representative from admissions and placement, department chairs, and a student leader are the key individuals acting as a teamto initiate, guide, and direct the development of the IEP. It is their commitment to the IEP and the careful selection and empowerment of the IEP team to oversee the institutional effectiveness activities that will ensure continuous improvement and the ultimate success of the planning process and the institution.
  • The process should involve allfaculty, administrators, staff, and advisory board members. Consideration should also be given to actively recruiting student, graduate, and employer representatives in the process. It is important that all members of the administration, faculty, governing board, and student body understand and appreciate the importance of the IEPand its value to the institution.
  • Establish subcommittees, to prepare specificIEP sections. These subcommittees should be effectively utilized to complete the various tasks in all facets of the IEP, including development, implementation, and evaluation. The consideration of subcommittee members should depend on each member’s responsibilities. The names of those responsible for implementing and monitoring the IEP should be included.
  • Establish baseline rates developed through analyzing the results of past annual retention and placement rates, which will be used in the analysis process. The data collected each year on the ABHES Annual Report includes retention and placement percentages. Therefore, the annual report becomes a valuable part of the IEP. Institutions should maintain these annual reports with supporting documentation, for at least three years in an effort to provide historical data from which goals may be determined. An institution may want to consider entering the data from previous years into the new calculation for retention and placement to develop a baseline rate. Be specific in the data to be collected and collect data that will clearly evidence the level of educational outcomes and satisfaction experienced by current students, graduates, and employers.
  • Institutions arealso toinclude information, which is relevant to improving their overall effectiveness, such as in-service training programs and professional growth opportunities for faculty. The IEP may include any elements determined to be important measures of institutional effectiveness such as a review, as applicable, of default rates in the student loan programs under Title IV of the Higher Education Act, based on the most recent data provided by the Secretary of Education. These findings may be coupled with student retention and placement rates to determine what correlation, if any, can be determined. Any correlation identified should be reviewed for correction and acted upon promptly.
  • Because the IEP focuses on overall institutional improvement and should be considered a work in progress, there are many potential elements of the institution’s daily operations, which are relevant and important to improving institutional effectiveness. The institution is encouraged to collect a variety of statistical data, which will assist it in improving the educational outcomes.
  • It is expected that meetings of the chief executive officer, president/director, director of education, key staff members, committees, and subcommittees will occur throughout the year. Adopt and implement a realistic and enforceable periodic schedule to review the IEP and document progress through minutes of all meetings where the IEP is discussed. A minimum of two meetings per year should include discussion of the IEP. The meeting minutes should show the progress to date, a short summary of the data analyzed, the changes anticipated, and the continuation or new direction the institution is taking to improve the educational processes. Minor revisions to goals may be made during the monitoring of the IEP; however, substantial revisions should only be made at the annual review unless there is a major change in the institution’s leadership and/or mission. These periodic meetings will ensure that the IEP is utilized and evaluated on a continuing basis.

GUIDELINES

GUIDELINES

The following section is Appendix C of the Standards Manual, which was developed to serve as a guide as it contains elements that should be incorporated into the IEP:

Subsection 1 – Institutional Effectiveness

  1. Program Characteristics of each currently offered program should include:
  2. Degree level,
  3. Program descriptions,
  4. Program objectives, and
  5. Description of student outcomes, specifying the competencies students should possess upon conclusion of the program.

EXAMPLE: