Report to Division Chairs on The

Academic Assessment Team

Report to Division Chairs on the

Fall 2003 Division Assessment Reports

May 10, 2004

Dear Colleagues:

The Academic Assessment Team would like to thank you for your leadership in advancing assessment initiatives within your division. At your request, and in a spirit of humility, we offer you the following feedback on the Fall 2003 Division Assessment Reports (DARs).

I say a spirit of humility because the AAT realizes that its assessment of your efforts is far from perfect in its attempt to provide useful, formative feedback, so we beg your indulgence as we all learn together to get better at assessment for the improvement of student learning.

The attached report include comments on all the divisions, not just your own. This report will be posted on the assessment website (

Both the “general comments” and the “category specific comments” represent the unedited responses of individual reviewers, not the consensus of the AAT. Even the several “AAT Recommendations” were not the product of group consensus, but individual comments posted by reviewers. It was decided not to make “official” AAT recommendations to individual divisions on this round, but to allow the individual comments to speak for themselves. Instead, we make some “General Comments to All Divisions” about the process and our institutional progress.

Please bear in mind that the statistical averages are not viewed as reliable by the AAT. Much more important is to determine whether your division is making progress in implementing a genuine, coherent assessment plan that results in improvement of instruction.

I would be happy to chat with any division regarding the DAR evaluations.

Thanks.

Thomas Butler, Chair

PVCC Academic Assessment Team

Attachments:

AAT comments on the Fall 2003 DARs
DAR scoring rubric

/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report

General Comments to All Divisions

Disclaimers:

This is the first time the AAT has attempted to “assess the assessment reports” as a team.

The AAT was fully aware that the rubrics used to assess the division reports were not developed--and thus not available to the divisions to guide them--prior to the submission of the reports. Because a number of divisions requested feedback in the conclusions to their Division Assessment Reports (DARs), the AAT decided to develop the rubrics after the fact to provide that feedback.

The AAT was also aware that this AAT report in May of 2004 could only evaluate DARs on the basis of reports submitted a full semester earlier; the AAT recognizes that in many Divisions there has been progress during the Spring of 2004 that is not accounted for in the DARs.

There was a high degree of variability among the AAT reviewers in the interpretation of the graduated rubrics applied to each category assessed. Some of that variability was due to the reviewer, and in some cases, to the detail and manner in which the individual divisions reported their results.

All of the following “comments” represent the written, verbatim observations of individual reviewers, not the consensus of the AAT. In some cases, reviewers made no comments regarding particular categories on specific reports.

Sometimes the AAT reviewers were looking for information that was not reported, yet there was an awareness that it existed. The AAT rubric for assessing DARs needs revision so as to clarify the type of information most important/relevant to report in a uniform way.

Thus, the numerical averages provided on the ensuing pages are not very reliable, though we have chosen to include them as part of our open process of evaluation. Much more important would be to consider actual comments and general recommendations.

The revised rubric will be distributed to the Divisions in the beginning of the semester, at the same time as the revised DAR template will be distributed.

General Conclusions/Observations by the AAT:

There has clearly been progress in all Divisions in the development of assessment projects and initiatives.
Since many projects are just beginning, there are few “results of implementation” and even fewer “changes from results” reported. It is expected that next year there will be considerably more reports of changes in instruction resulting from analysis of assessment data.
There are many good projects that assess course-specific learning outcomes; however, there need to be more cross-discipline efforts to assess the broader institutional general education learning outcomes. This is not a criticism of the divisions’ efforts, but an observation of the AAT’s recognition that we need to work with divisions in that direction. Divisions should consider this as they plan for the next year.
Although there is a “monitoring-for-compliance” element to the purpose of the DAR, the process needs to be one that is formative and consultative in assisting divisions and departments to generate assessments that are meaningful and useful at all levels of the institution.
In some cases there needs to be a refinement of what genuine, formative assessment is—and that the “end” of any and all assessment is not compliance, but the use of assessment data to improve instruction.
The AAT can be helpful by providing clearer expectations about the depth and breadth of division-, department-, and course-level assessment initiatives and their relation to institutional needs.

/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report
for Division:
Business and IT
General Comments: (none)
Specific
Comment Category /

Average

Category
Rating
(0 to 4 points) / Category Specific Comments of up to five different raters:
1. Level of
Progress / 1.8 /

There was extensive progress from a cold start, but only in one course.
Only CIS105/BPC110 have assessment plans.
Very good progress in CIS105/BPC110; have followed through from last year’s report; need to infuse into more courses in division.
Although quite comprehensive, assessment is confined to just sections of CIS105/BPC110.

2. Specification of
Goals / 2.4 /

Unclear whether planned goals for Fall 03 were actually met.
Realistic goals for the next year, and includes entire division.
Goals are clear and specific, but confined to just three initiatives.

3.1 Extensiveness
of areas / 2.0 /

Very careful plan for BPC 110; template can be followed by other areas.
Good coverage for CIS105/BPC110; lots more to go for the entire division to be involved.
Assessment goals to impact all full and part-time faculty, but courses beyond CIS105/BPC110 are not identified.

3.2Measures of
Outcomes / 3.4 /

What does “identified learning outcomes” refer to?
L.O.s have been identified for IT and BUS and tied to Gen. Ed. L.O.s, though I question some of the connections; BPC 110 good matrix; what is Info Lit. # 4 addressing? (p. 3)
PVCC Gen Ed. Departmental, and CIS105/BPC110 course competencies have been identified and aligned with multiple assessment indicators.

3.3 Alignment of
Measures / 2.6 /

No division goals aligned specifically apart from CIS105/BPC110
L.O.s have been identified for IT and BUS and tied to Gen. Ed. L.O.s, though I question some of the connections; BPC 110 good matrix; what is Info Lit. # 4 addressing? (p. 3)
Well-developed alignment of measures, but for CIS105/BPC110 only; no other courses identified.

3.4 Specification
of steps / 2.6 /

Not yet “closing the loop”; will do so, but not yet specified.
The BPC 110 matrix is very specific, includes well thought out steps.
Very specific, comprehensive and well-organized steps to assess CIS105/BPC110 only; other assessment goals are general and somewhat vague.

3.5 Results of
Implementation / 0.8 /

No results yet.
Just now implementing, so no real results yet.
Specific results from assessing CIS105/BPC110 only are reported; other courses have not been identified.

3.6Changes from
Results / 0.0 /

No results yet.
Will start seeing changes on future reports
Div Report indicates that changes will be reviewed and documented end of Spring 2004.

AAT Recommendations:

The Division needs to involve more faculty and more courses in assessment;
The Division should develop an overall plan.
The CIS105/BPC110 assessment model is highly recommended where appropriate, and the experience and “expertise” of the faculty who implemented it should be drawn upon.

/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report
for Division:
Communication and Humanities
General Comments:

Had the sense that HUM is playing along, but not really, seriously into it, or not really understanding the purpose of assessment.
SPA 101 is trying but unclear how he might make changes as a result.
SPA 101 is clearly a summative evaluation, but not a formative assessment with a plan for change as a result.

Specific
Comment Category /

Average

Category
Rating
(out of 4) / Category Specific Comments of up to five different raters:
1. Level of
Progress / 3.0 /

All F-T faculty are involved in at least one initiative.
Progress in COM 100, 225; SPA 101; Hum205; COM236; PHI 101.
Progress is being documented to assess learning in several courses: COM 100, 225; SPA 101; Hum205; COM236; PHI 101.
Really good work in COM 100, 225; one SPA 101 instructor has begun; some beginning thinking for Hum205

2. Specification of
Goals / 3.0 /

The goals are very clear and direct, but there isn’t much cohesiveness; should there be for this division?
Goals and objectives to assess learning are clear for (all?) areas (courses) in the division.
Unclear: “each member” of division—RFP or adjunct also?

3.1 Extensiveness
of areas / 3.75 /

All F-T faculty are involved in at least one initiative.
Clearly much is happening in most of the courses.

3.2Measures of
Outcomes / 3.0 /

The identified outcomes are indeed being measured; Walt’s second statement isn’t really an outcome.
Three learning outcomes are identified in the division report although many others are implied and undoubtedly being measured.
COM 100, 225, and 230, and Hum205—not clear how the learning outcomes will be measured, just what will be measured.

3.3 Alignment of
Measures / 3.75 /

The identified outcomes are all directly linked to the PVCC Gen Ed L.O.s
Several L.O.s being assess by this division are aligned with MCCCD and PVCC Gen Ed L.O.s.
For those courses actively involved an attempt to identify alignment has been made.

3.4 Specification
of steps / 3.25 /

The three plans each have a concrete, specific set of steps to ensure completion.
COM 100, 225 and 230 assessment steps are quite specific; for SPA as well; quite vague for HUM sections.
Steps clearly identified.

3.5 Results of
Implementation / 1.5 /

Just beginning.
I noticed results and use of results only in COM 225.
Specific results were documented for SPA 101; results for COM and HUM are pending.
All at beginning; no results to report.

3.6Changes from
Results / 1.0 /

I wouldn’t classify the changes as minor, but it appears only one or two have been made to date.
The only changes I noticed were in preparing for the assessment, not a result of the assessment.
Changes are to be reported end of Spring 2004.
Have identified some vague changes which are likely to take place, but hard to determine how the assessment strategies will generate change in instruction.

AAT Recommendations:
/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report
for Division:
Counseling and Personal Development
General Comments:

A comprehensive initiative that is specifically correlated to the PVCC Gen Ed L.O.s.

Specific
Comment Category /

Average

Category
Rating
(out of 4) / Category Specific Comments of up to five different raters:
1. Level of
Progress / 2.8 /

29 sections in schedule; CPD 102AB, 150, 150AB, 103BL and 250 showed some progress.
It looks like a lot of progress has been made, but I don’t know what was in place last year.
CPD has accomplished a lot and has infused across the entire division; L.O.s (B) not really L.O.s, but PTAs include actual L.O.s.

2. Specification of
Goals / 3.2 /

The Division goals do seem clear and comprehensive.
Good, realistic goals for next year.

3.1 Extensiveness
of areas / 3.2 /

Six out of how many?
Six courses listed, don’t know how many sections.
The emerging assessment efforts seem pervasive, although again, I am unsure of how many CPD courses there are in all.
Definitely assessment across the division!

3.2Measures of
Outcomes / 1.8 /

Not listed.
Not answered.
The program outcomes are not really learning outcomes; The competencies on the PTAs are L.O.s for the most part.
Lots of PTAs; Unclear when/how administered.

3.3 Alignment of
Measures / 3.2 /

Most outcomes are aligned, and the alignment is specific.
Some of the PTAs include aligned Gen Ed L.O.s

3.4 Specification
of steps / 1.6 /

Not listed.
Not answered.
There seems to be a plan (in goals), but not a specific sequence of steps beyond “weekly agenda item.”
Steps aren’t clearly outlined.

3.5 Results of
Implementation / 0.0 /

Not listed.
Not answered.
I didn’t see any evidence of results so far, only plans to use these measures.
Just implementing, therefore not a lot to report.

3.6Changes from
Results / 1.2 /

Still new.
Since there are no results, there are no documented changes.
Just implementing, therefore not a lot to report.

AAT Recommendations:

Implement measures and create a positive plan for action (changes)

/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report
for Division:
English
General Comments:
Specific
Comment Category /

Average

Category
Rating
(out of 4) / Category Specific Comments of up to five different raters:
1. Level of
Progress / 2.6 /

I’m not certain, but is appears some courses/sections don’t have assessment plans yet.
Div Assessment goals will come from Mission and Gen Ed L.O.s?
Progress is being made in many areas by several FT faculty.

2. Specification of
Goals / 2.8 /

Despite the above comment, it appears that most areas have well-defined goals.
Goals are listed; Goal B is not very strong (easily attained).

3.1 Extensiveness
of areas / 3.2 /

A wide range of assessment activities, but probably not “all.”

3.2Measures of
Outcomes / 2.4 /

Different for each project.
Some have explicit outcomes, others don’t.

3.3 Alignment of
Measures / 3.4 /

The links to PVCC’s Gen Ed L.O.s are clear and explicit.

3.4 Specification
of steps / 2.8 /

Some better identified for others.
Each plan has mostly explicit steps, but I’m not sure they all lead to assessment results.

3.5 Results of
Implementation / 2.2 /

As indicated in the report, there are no results as yet.
Some results from W.A.G. and P. Osback

3.6Changes from
Results / 1.4 /

Changes to W.A.G. rubric, but no much else yet.

AAT Recommendations:

Revisit Learning Outcomes and be sure assessment measures measure student outcomes, not teacher goals (Hopefully, these will coincide.)

/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report
for Division:
Fine and Performing Arts
General Comments:
Specific
Comment Category /

Average

Category
Rating
(out of 4) / Category Specific Comments of up to five different raters:
1. Level of
Progress / 1.2 /

It appears that the Division Plan addresses most areas/courses.
Since the Division is new, all work to date could be considered progress.
No previous report.

2. Specification of
Goals / 2.2 /

There seems to be goals specified for most areas of the Division.
Goals have been met? Are there new goals to strive for?
Some good first year goals. Don’t quite understand using the same PTAs in each discipline.

3.1 Extensiveness
of areas / 2.6 /

The two Rubrics seem applicable to a wide range of courses. All?
Attempt to cover all disciplines in the FPA division.

3.2Measures of
Outcomes / 2.2 /

The assessment measures do address most of the “objectives.”
Not clear; the PTAs are the measurements but when implement them?

3.3 Alignment of
Measures / 1.4 /

I didn’t see any documentation aligning Div outcomes with PVCC Gen Ed L.O.s.
“Created with express purpose…” but the link is not explicit.
PTA statements are aligned with some of the Gen Ed L.O.s.

3.4 Specification
of steps / 3.2 /

The steps in the plan seem specific and complete.
A specific plan is laid out; will it end with results that inform change?
The steps seem reasonable.

3.5 Results of
Implementation / 0.0 /

As indicated in the report, no results have been obtained to date.
None so far.
N/A

3.6Changes from
Results / 0.0 /

None so far.
N/A

AAT Recommendations:

Carry on with plan; how will results inform change?

/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report
for Division:
Health and Exercise Science
General Comments:

Need to see more specifics
Don’t really seem to be aligned with PVCC’s Gen Ed. L.O.s (that is, these L.O.s predate PVCC’s Gen Ed L.O.s.)
PLUS=Good emphasis in Dietetic Tech program on measuring “program objectives”; NEGATIVE=Need to refocus Div Assessment Report on “learning outcomes” for specific courses or services.

Specific
Comment Category /

Average

Category
Rating
(out of 4) / Category Specific Comments of up to five different raters:
1. Level of
Progress / 2.0 /

Progress made in EMT, PED115, Dietary Tech program; mission statement written.
Report includes one EMT rubric; extensive documentation of dietetic tech program w/ MCCCD and PVCC Mission, goals, but no learning assessment.

2. Specification of
Goals / 0.4 /

No goals specified for Division. Seems that there are goals for specific programs, but they were not reported.
Division Assessment Goals can be tied to mission and Gen Ed L.O.s
No “Division” Goals reported; rather, program objectives. Well-defined goal statements for dietetic tech; no other goal statements in Div report.

3.1 Extensiveness
of areas / 2.0 /

Hard to tell because I am not quite sure that I know the extent of division offerings.
Courses, programs and cohorts include EMT, dietetic tech and PED 115 cohorts. However, learning outcome goal statements are implied, not defined.

3.2Measures of
Outcomes / 1.4 /

Not reported.
Dietetic program: 4; other programs: zero
L.O.s identified and being assessed for EMT; implied for/not documented for dietetic tech and PED 115; no other courses in HES.
Dietetic program: 4; other programs: zero

3.3 Alignment of
Measures / 1.0 /

Not reported.
EMT Program: 3; Dept/Div/other programs: zero
Dietetic Tech program aligns w/ MCCCD and PVCC mission/vision statements, but notPVCC Gen ED L.O.s statement; EMT program is aligned; other PED classes not aligned/written.
EMT Program: 3; other programs: zero

3.4 Specification
of steps / 1.4 /

Not reported.
Division seems to be discussing process, but not documented.
PED 115 continues to use student assessment tool; Dietetic Tech. program continues w/ state agency to assess program outcomes; but beyond “EMT rubric,” no steps described.

3.5 Results of
Implementation / 1.6 /

Not reported.
Results are available for two programs.
EMT rubric assessment results documented; no learning outcomes assessment described in HES Division report.

3.6Changes from
Results / 0.0 /

Not reported.
Changes will be identified as results are tallied.
No documentation included in and PED 115 cohorts. However

AAT Recommendations:
/ Academic Assessment Team comments
on the Fall 2003 Division Assessment Report
for Division:
Library
General Comments:
Specific
Comment Category /

Average