WP6 Piloting and evaluation at partner universities
DEV6.1.Monitoring of university regulatory documents implementation.
Report- draft version
Prepared by
Ivana Ognjanović, PhD – University Mediterranean, Podgorica, Montenegro
Dragica Anđelić – University Mediterranean, Podgorica, Montenegro
Srdjan Jovanovski, PhD – University Mediterranean, Podgorica, Montenegro
June, 2014
This project has been funded with support from the European Commission. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
Tempus Project 516758-TEMPUS-1-2011-1-GR-TEMPUS-JPGR
IIntroduction
This document is intended to provide inputs to SINC@HE WP 6 – ‘Piloting and evaluation at partner universities’, activity 6.1-‘Monitoring of university regulatory documents implementation’. The main aim of this activity is to produce evaluation results about adapted and newly established regulatory documents among HEIs in Montenegro and established SAO. This document will be divided into three major parts: (i) approach used for realisation of 6.1; (ii) quantitative and qualitative measurement of conducted evaluations and (iii) future plans about corrective measures (if needed) and further improvements and development.
Having in mind small number of students with disabilities in ME, heterogeneous structure of HEIs (two private universities, the largest state university which is not included in project consortium, several independent faculties, etc.), our approach includes consideration of several techniques and evaluation tools in order to achieve realistic results and recommendations. This approach draws on: (i) qualitative technique (such as theory of change) and (ii) quantitative techniques (including different surveys conducted on different groups with different aims and objectives). All techniques should be carefully adopted for small groups and above mentioned characteristics of ME environment.
IIMonitoring of university regulatory documents implementation
As the regulatory university framework for studying with disabilities is defined and agreed in WP2, the piloting process should monitor on how new regulatory norms are respected and to what extent they are helpful. Since during the process of accepting and establishing all regulatory documents, the Senate at both universities (UNIM and UDG) and authorities at AYDM had the opportunity to analyze and make all changes in proposed documents, monitoring process should be done among practitioners, i.e. workers (administrative and academic staff) at HEIs and SAO and students. To this end, the following actions are realised:
(i)Theoretical analyses to estimate current position in triangle: needed-planned-achieved
(ii)Surveys among students with disabilities and administrative and academic staff.
University management will carry out monitoring and survey among all students and staff involved, and tracking of newly enrolled disabled students.
IIITheory of change for estimation of current position in triangle: needed-planned-achieved
Theory of Change defines all building blocks required to bring about a given long-term goal. This set of connected building blocks–interchangeably referred to as outcomes, results, accomplishments, or preconditions is depicted on a map known as a pathway of change/change framework, which is a graphic representation of the change process.
Built around the pathway of change, a Theory of Change describes the types of interventions (a single program or a comprehensive community initiative) that bring about the outcomes depicted in the pathway of a change map. Each outcome in the pathway of change is tied to an intervention, revealing the often complex web of activity that is required to bring about change.
A Theory of Change would not be complete without an articulation of the assumptions that stakeholders use to explain the change process represented by the change framework. Assumptions explain both the connections between early, intermediate and long term outcomes and the expectations about how and why proposed interventions will bring them about. Often, assumptions are supported by research, strengthening the case to be made about the plausibility of theory and the likelihood that stated goals will be accomplished.
Stakeholders value theories of change as part of program planning and evaluation because they create a commonly understood vision of the long-term goals, how they will be reached, and what will be used to measure progress along the way.
A Theory of Change is a specific and measurable description of a social change initiative that forms the basis for strategic planning, on-going decision-making and evaluation. The methodology used to create a Theory of Change is also usually referred to a Theory of Change, or the Theory of Change approach or method. So, when you hear or say “Theory of Change”, you may mean either the process or the result.
Like any good planning and evaluation method for social change, it requires participants to be clear on long-term goals, identify measurable indicators of success, and formulate actions to achieve goals.
Itdiffersfrom any other method of describing initiatives in a few ways:
-it shows a causal pathway from here to there by specifying what is needed for goals to be achieved (e.g. you might argue that children attending school a minimum number of days is necessary if they are going to learn).
-it requires you to articulate underlying assumptions which can be tested and measured.
-it changes the way of thinking about initiatives from what you are doing to what you want to achieve and starts there.
A Theory of Change provides a roadmap to get you from here to there. If it is good and complete, your roadmap can be read by others and show that you know how to chart your course. This is helpful with constituents, staff, partners organizations and funders.
More importantly, if it is good and complete, you have the best chance of making the change in the world you set out to make and of demonstrating your successes and your lessons along the way.
Expected result:
-Comprehensive analyses of goals needed to be achieved versus their current achievement during project implementation and piloting phase
// Enes and Srdjan should complete this part
IV Survey analyses
Meeting of Montenegrin partners (see Attachment) was organised as initial step for planning the questionnaires and their conductions. Having in mind that established regulatory documents cover activities at both, HEIs and SAO, with enrolments of students, administrative and academic staff, several actions were agreed. Due to their importance for understanding the content of the rest of this Report, as well as organisation of surveys that is used, we outline them in original form (numberings are reset due to the consistency of Report):
Action 1: Due to expressed and signed support to all activities, University of Montenegro (UoM) should be included in all planned activities if it is possible. AYDM will be responsible for communication and cooperation with UoM.
Action 2: Different kinds of surveys should be conducted in order to provide comprehensive analyses and monitoring of SAO (that is planned in WP6), as follows:
-Surveys among students, academic and administrative staff
-Surveys focused on specific activities during one academic year: lectures, exams and enrolment period
-Surveys aimed on each developed services in SAO separately
Action 3: Having in mind that the same samples of students, administrative and academic staff will be surveyed for all activities during one academic year (as planned with Action 3) and almost each kind of developed services, optimal structure of questionnaires should be used:
-One questionnaire should consists of several sections: the first section focused on general issues of awareness about students with disabilities and their inclusion at HEIs, and other sections focused on developed services in SAO (one section for one service)
-Questionnaires should be prepared separately for students, administrative and academic staff
-Questionnaires should be prepared separately for specific activities during one academic year: lectures, exams and enrolment period
-Surveys should be conducted upon completion of certain activities in the academic calendar
iv.i.Surveys Setup
According to accepted actions, surveys are divided for three periods:
April-May – teaching process
May-June – exams
June-July- enrolment period
and for each period, questionnaires are prepared for students, academic and administration staff separately. Also, in accordance with Action 3, each questionnaire is similarly structured (aimed on general issues, legal framework, and developed services). Thus, this Report (due to its topic: Monitoring of university regulatory documents implementation) is focused and interested on the first two sections of questionnaires conducted among students, academic and administration staff. For some questions that are currently classified for Report DEV6.2 it is hardly to make separation strictly from legal issues, and thus, it is recommended to read this Report together with DEV 6.2 and DEV 6.4.
This section is organised as follows: sub-section iv.i.i. gives an overview of questionnaires by making classification and categorisation of questions, while iv.ii.i. gives summarised results with discussions in iv.iii.
iv.i.i.Surveys aimed on teaching process
Questionnaires that are used for surveys among students are available here and questions aimed on established legal framework are identified and listed in Appendix I.I.
Questionnaires that are used for surveys among administrative and academic staff are available here and questions aimed on established legal framework are identified and listed in Appendix I.II. It can be seen that one questionnaire is used for both staff categories, while several groups of questions are clearly divided for both categories.
The structure of questions and their content is summarised in the following table.
Table 1. Structure of questions in surveys among students and staff (administrative and academic)
Groups of respondents / Question No. / Target / Kind of questionsCurrent students / 1 / awareness / closed
Current students / 2, 3 / exercised rights / closed
Administrative and academic staff / 1, 4 / experience in working with SwD and thus directly/indirectly using legal framework (depends on working position) / closed and opened
Administrative and academic staff / 2 / Experience in working with SAO (problems may be caused by legal acts and established framework ) / closed and opened
Administrative and academic staff / 3 / Awareness of extended working duties (specific legal act at HEIs) / closed and opened
iv.i.ii.Surveys aimed on examinations
iv.i.iii.Surveys aimed on enrolment process
iv.ii.Results
The questionnairs are conducted among 22 students and 27 staff members (administration, IT and service support) and ??? academic staff (see Table 2).
Table 2. Review of respodents in the surveys aimed on teaching process
Institution / Current students with disabilities / ADMINISTRATIVE staff memebrs / ACADEMIC staffNumber / % (of total number of SwDs[1]) / Number / % (of the total number of employed staff[2]) / Number / % (of the total number of academic staff[3])
UNIM / 7 / 100% / 9 / 100% / ?
UDG / 2 / 100% / 10 / 100% / ?
UoM / 13 / ? / 7 / n/a / ? / n/a
Surveys results are summarised in the following sub-sections:
iv.ii.i.Results of surveys aimed on teaching process
The first question for students is related to their awareness of possibilities for using specialized equipment and teaching resources for an unconstrained participation in the education process. Only 14 students (63.63%) gave positive answer (see Figure 1). One student (4.54%) did not answer on this question.
Even the students’ right for using specialised equipment (that is purchased in WP4) is defined by specific legal acts at both universities (UNIM has opened their equipment to all students, from all HEIs in ME, while UDG decided to assign equipment to AYDM thus making them opened to all students), only 16 students (54.54%) answered positively about their awareness about exercising this right (see Figure 2). Two students (9.09%) did not answer on this question.
Also, the students’ right for asking for specific examination environment (e.g. duration of exams, exams form, etc.) is defined by specific legal acts at both universities. Only 13 students (59.09%) answered positively about their awareness about exercising this right (see Figure 3). One student (4.54%) did not answer on this question.
Administrative staff members were asked about experiences in working with SwD, i.e. first question was focused on realisation of this communication via newly established SAO. Only 6 staff members (23.08%) answered that they are in constant communications with SAO, while 12 staff members (46.15%) haven’t experienced any communication and cooperation with SAO. The rest of surveyed staff (8 respondents) have experienced work with SAO (30.77%). Thus, 14 staff members (53.85%) may report a problem and/or inconsistency in established legal framework since they have potentially direct references to legal documents. (see Figure 4)
Those 14 staff members that have experienced in working with SAO were asked about their satisfaction in communication and cooperation. All of them answered positively and thus may be concluded that there were not reported any problem and/or inconsistencies in implementing established legal framework.
Since the responsibilities of administrative staff (students’ support, secretary, IT support) should be defined explicitly in accordance with assigned responsibilities and duties for specific working positions, 10 staff members (38.46%) were not be aware (see Figure 5). It is important to note that only 2 staff members from partner HEIs were not informed about additional duties to their working positions.
Administrative staff members were asked about their experience in working with students with disabilities during current academic year. Only 16 staff members (61.54%) have experienced communication with SwDs during current academic year (see Figure 6).
It is interesting to analyse which working positions were more frequently in contact with SwDs. As might be expected, they are Students’ supports (43.75%) and administration (50.00%) (see Figure 7).
iv.iii.Discussion
Based on presented results of conducted surveys aimed on teaching process the following conclusions may be delivered:
-Students with disabilities are not completely aware of their rights and possibilities at HIEs (including use of specialized equipment, adopted examination conditions, etc.)
Action: make more intensive presentations of established legally framework
-Administrative staff members at partner institutions are mostly informed about their assigned duties in working with SwDs. Staff members at UoM (that is not partner institution) are not informed, since legal framework is not established as one at UNIM and UDG.
Action: suggest to UoM to establish similar legal framework
APPENDIX
I.I.Teaching process - Questions for students
- Did you know that as a student with disability you can use the specialized equipment and teaching resources for an unconstrained participation in the education process?
a)Yes
b)No
Mark the appropriate answer.
- Did you know that you can use following aids for facilitated participation in the teaching process:
- Lap-top computer;
- Desktop computer;
- Network printer and toner;
- Books and periodicals;
- Copy machine;
- Server;
- Web publishing software;
- Digital camera;
- Speech recognition software;
- Screen content magnifying software;
- Specialized software (i.e ClarisWorks, Inspiration, Out line, Powerpoint, Hyper studio)
- Text recognition software (OCR);
- Software for translation and editing of Braille alphabet;
- Alternative keyboard;
- Track ball/track pad/joystick with screen keyboard;
- Large format magnifier;
- CCTV (closed circuit television);
- Braille writer;
- Braille printer;
- Signaling device;
- Portable word processor;
- Personal enhancing system (hearing aid),
- Electronic voice dictionary/lexicon/spell checking;
- keypadwithlarge keysand/orwith a largeLCDdisplay.
a)Yes
b)No
Mark the appropriate answer.
3.Did you knowthatyou have the rightto requestadditional time(outsideschool hours) forthe use ofspecial equipment andteaching aidsfroma list of equipmentand teaching materials?
a)Yes
b)No
Mark the appropriate answer.
I.II.Teaching process - Questions for administrative and academic staff
- Has the Student Advisory Office (Association of Youth with Disabilities of Montenegro) contacted you at any point during the current academic year?
a)Yes;
b)yes, many times;
c)no;
d)I don’t remember.
Mark the correct answer.
- If you are familiar with the work of Student Advisory Office, are you satisfied with cooperating with it?
a)Yes
b)No
If your answer is “No”, please explain: ______
- Are you familiarthat one of the activities within yourjob should betoperform the dutiesrelated to theenforcement of the rightsof students withdisabilities regarding thealternative solutions?
a)Yes
b)No
If your answer is “Yes”, can you please briefly describe what specific duties you should undertake in order to support students with disabilities in exercising rights on alternative solutions:
4. Have the students with disabilities contacted you at any point during the current academic year?
a) yes
b) no
If your answer is yes, can you please write what specific issues students with disabilities have asked you about.
______
[1]The total number of students with disabilities is related to official registration at AYDM
[2]The total number of staff is related to the following working positions: Students’ support, Secretary, General Secretary, Librarist, IT Sector
[3]The total number of academic staff is related to those experienced in working with SwD and Vice Deans for Education