Evaluating Students' Satisfaction: The Case of Informatics Department of TEI Athens[1]
CHRISTOS KOILIAS
Informatics Department
TEI Athens
GREECE
Abstract: The achievement of students’ satisfaction is of vital importance for every higher education institution, because it promotes its internal organization and strengthens its image in order to meet the expectations and needs of students. This paper presents an original study that measures the students' satisfaction in an Informatics Department in Greece. A set of powerful ordinal regression methods has been applied on a survey database. The most important results focus on the determination of the weak and strong points of the department, according to the MUSA methodology. Suggestions for the quality improvement of provided education are also included.
Key-words: service quality, education, statistics
1 Introduction
Several research efforts, which have been carried out by scientific bodies on international level, focus on the evaluation of the quality and effectiveness of educational institutions. Many organizations, both national and international, have been highly activated towards planning as well as upgrading the effectiveness and quality of the educational institutions operation and educational task.
In this context, the evaluation and scheduling of the educational task of an educational institution department has recently become a strong trend internationally. This evaluation may contribute, among other factors, to the broadening of knowledge concerning the accomplished educational task, to the improvement or modification of the adopted educational practices and may lead to rational decisions concerning the future of the unit.
This study is based on a research project aimed at the measurement of satisfaction of the students who attend the Informatics Department of the Technological Educational Institution of Athens. In Greece, higher education consists of two branches: Universities and Technological Educational Institutions (TEI). TEI of Athens is the major TEI of Greece and the 3rd greater educational institution, in respect to students’ population.
2 Research methodology
2.1 Planning of the research
Research planning included the following 4 basic stages which were designed and implemented successively [Grigoroudis and Siskos (2000)]
a) Preliminary stage: This first stage included identification and definition of the problem, followed by a preliminary analysis of students’ behaviour and investigation of the environment. The work team, after exhaustive discussions and taking under consideration the relevant international bibliography, specified the quality dimensions as well as the satisfaction ranges.
b) Development of a questionnaire and gallop poll: In this stage, the research pattern was determined, the coding of the necessary information into questions was prepared and the carrying out of the research took place.
c) Analysis: The third stage included data processing, using tools of descriptive and estimating statistics as well as MUSA (MUlticreteria Satisfaction Analysis) methodology.
d) Results: During the last stage, the results from the processing of the questionnaires were estimated and evaluated.
2.2 Definition of satisfaction criteria
The organization and structure of the Technological Educational Institutions differs substantially from the ones of the respective institutions abroad. Therefore, it was imperative to detect the quality dimensions that correspond to the conditions of the greek higher education, especially those related to the Technological Educational Institutions.
Thus, the following five quality dimensions (criteria) were defined and used.
1. Educational Program: It refers to the entire apprehension and planning of the Educational Program applied to the Informatics Department (variety and quality of courses, content overlap, degree of their correspondence to the labour market and post-graduate studies demands, time table–theory and practice)
2. Teaching Personnel:. It concerns the implementation of the Department’s Educational
Program (training, efficiency and communication skills of the teaching personnel, evaluation methods).
It is obvious that the two previous dimensions (criteria) together cover the issue “Education” (Educational Program, instruction).
3. Tangibles: It refers to the settlements, laboratories’ equipment, air-conditioning, printed material, TEI’s Library and its electronic system for book inquiries
4. Administrative Support: It refers to the working hours of all relevant services as well as their credibility and service efficiency.
5. Image of the Department: It concerns the credibility, prestige and acknowledgment of the Informatics Department as well as the procedures implemented for its promotion and connection with the labour market.
The criteria and sub-criteria used in this research project are represented in figure 1.
Fig. 1. Hierarchical structure of students’ satisfaction criteria and sub-criteria
2.3 Development of the questionnaire
An eight-page detailed questionnaire was developed, based on the previous mentioned satisfaction dimensions. It included the following three parts.
a) Fill in instructions
b) Personal data: This part includes questions of demographic and personal kind (birth date, date of enrollment, sex, semester, working/not working student etc)
c) Questions concerning satisfaction: It is the main part of the questionnaire and includes questions related to the students’ satisfaction from each criterion and sub-criterion. More precisely, the students are asked about the degree of satisfaction from each one of the five criteria (Educational Program, Teaching personnel, Tangibles, Administrative support, Image of the Department) as well as their sub-criteria and the whole framework of their studies in the Department.
For the questions aimed at the evaluation of students’ satisfaction from the 27 sub-criteria it was used a 5-points Likert scale (ranged from “totally agree” to “totally disagree”)
For the questions aimed at the evaluation of students’ satisfaction from the 5 criteria it was used a 5-points Likert scale (ranged from “totally satisfied” to “totally dissatisfied”)
The same 5-points Likert scale was used to measure the global satisfaction.
Graphics representing feelings and opinions (J L K) were used, in order to facilitate the filling in of the questionnaire.
For each one of the 5 criteria and the global satisfaction, the students were asked to give an additional grade, ranged from 0 (worst) to 10 (excellent).
An open question was placed at the end of each criterion, so that students could give remarks and suggestions concerning the improvement of the educational framework provided by the Informatics Department of the TEI of Athens.
2.4 Conduction of the research
The research was carried out within 5 days, during the 2004 spring semester enrolment period (February 2004). The students had been previously informed and most of them had already received the questionnaire.
The students delivered the unnamed filled in questionnaires to the members of the work team, who performed a quick check during the delivery, to ensure that all the questions concerning the satisfaction dimension were answered.
950 questionnaires were collected, corresponding to the 90% of the total number of enrolled students. It is estimated that this percentage corresponds over the 95% of active students, which leads to the conclusion that the sample used corresponds to the entire student population of the Department.
2.5 Preliminary processes
The collected questionnaires were registered in an Excel worksheet. A validation check for typing mistakes was performed in a sample. The outcome showed a very low possibility for typing mistakes and was decided not to perform further checking.
8 questionnaires were withdrawn due to incomplete filling.
Search for missing values: The registered data were compared with the questionnaires in order to ensure that missing data were not filled in by the students. It was ascertained that, in general, the number of missing values was small in the questions concerning satisfaction, while it was a little higher in the questions concerning personal data. The missing values, in the questions concerning satisfaction, were properly added in the following way:
Ø If the answers in all other relevant sub-criteria were identical, the added missing value was the same.
Ø If the answers in all other relevant sub-criteria were positive (perfectly or rather satisfied), the missing value was added as “rather satisfied”.
Ø If the answers in all other relevant sub-criteria were negative (totally or moderately unsatisfied), the missing value was added as “totally unsatisfied”.
Ø If the answers in other relevant sub-criteria were either positive or negative or indifferent, the missing value was added as “indifferent”.
Missing values in the questions concerning personal data was not possible to be added.
Track down inconsistent answers. Inconsistent answers are considered to be those that may include the statement «satisfied» to all criteria concerning partial satisfaction and the statement «dissatisfied» for global satisfaction (or the reverse). The same rationale is applied to the relation between each criterion and its sub-criteria. The percentage of inconsistent answers to at least one criterion was 10%. However, the analysis that followed showed that the rate of the effect of inconsistent answers on various results was 1% (stability and fitting errors not included). Thus, it was decided not to exclude the questionnaires that included inconsistent answers.
Some of the students’ population characteristics are the following:
The sample consisted of 75% male and 25% female students. 40% of the students’ population stated that they are fully or partial employed. Students who enrolled in the 1st semester had not been included in the sample.
3 Results of satisfaction measurement (based on qualitative data)
This chapter presents the most significant research results, deriving from the study of the qualitative data. The methodology used was the MUSA-MUlticreteria Satisfaction Analysis [Grigoroudis, E. and Siskos, Y. (2002)].
MUSA methodology is based on the use of a collective value function that includes an accumulation of individual judgments, under the assumption that students’ global satisfaction depends upon the characteristic dimensions of the provided services, which are represented by a set of criteria (fig. 1). It is a preference desegregation methodology that is implemented through an ordinal regression based approach in the field of multi-criteria analysis. This approach is used for the evaluation of a set of marginal satisfaction functions, which results to the measurement of a global satisfaction representing, as consistently as possible, students’ judgments.
In the survey, students are asked to state their individual satisfaction (global and partial). Taking under consideration the above mentioned assumptions, the problem can be approached as a qualitative regression problem and is solved through the use of special linear programming formulations, where the sum of deviations between the student's global satisfaction evaluation and its multicriteria satisfaction evaluations is minimized.
3.1 Global and partial (per criterion) satisfaction
The level of global satisfaction of students, from the quality of services provided by the Department of Informatics, is rather high (Table 1). 63% of the students are rather or perfectly satisfied, 12% are dissatisfied, while ¼ of the students are indifferent.
The average satisfaction index, as it is calculated by the MUSA method, has a value of 77% and it is
Table 1. Relative frequencies of global and per criterion satisfaction
Total-ly Un-satisfied / Mode-rately unsatisfied / Neith. Satisfied / unsatisfied / Mode-rately satisfied / Perfe-ctly satisfiedGLOBAL SATISFAC. / 2% / 10% / 25% / 55% / 8%
EDUCAT. PROGRAM / 4% / 16% / 33% / 43% / 4%
TEACHING PERSON. / 5% / 22% / 34% / 35% / 4%
TANGIBL. / 7% / 26% / 33% / 31% / 3%
ADMIN. SUPPORT / 12% / 23% / 31% / 29% / 6%
IMAGE / 1% / 4% / 20% / 55% / 20%
considered to be moderate to high. The value of the satisfaction indices for all criteria is higher than 50%, besides the one concerning Administrative Support which lies to 45.7%. This result is in accordance with the results of other research project [Kotler and Fox (1995)], which supported the opinion that the majority of students are satisfied by the provided education but not by the administrative support.
The criterion that appears to have the higher satisfaction index (92.3%) is the one of the Image of the Department (Table 2).
Table 2. Results of global and per criterion satisfaction
Weight / Satisfa-ction / Deman-ding / ImpactGLOBAL SATISFAC. / 0,77 / -0,47
EDUCAT. PROGRAM / 0,20 / 0,77 / -0,50 / 0,05
TEACHING PERSON. / 0,14 / 0,65 / -0,30 / 0,05
TANGIBL. / 0,13 / 0,60 / -0,23 / 0,05
ADMIN. SUPPORT / 0,12 / 0,46 / 0,06 / 0,06
IMAGE / 0,41 / 0,92 / -0,76 / 0,03
Students seem to be neutral or not demanding, so that the demanding index is – 47%
The demanding index expresses the average deviation between satisfaction functions and a normal or neutral (linear) function of values. Thus:
· Neutral students are those that the more satisfied they declare, the higher percentage of their expectations is fulfilled.
· Demanding are the students who declare satisfied only when they get the highest level of services.
· Not-demanding students are those who declare satisfied even if the percentage of their fulfilled expectations is low.
Table 2 displays the demanding indices for each criterion. For the criteria Educational Program and Image of the Department the demanding indices are low (not demanding students), while for the remaining criteria indices are moderate. The Administrative Support criterion is the only one that shows a positive value of the demanding index.
Table 2 presents also the weights of the criteria. The higher weight (41.1%) corresponds to the criterion Image of the Department followed by the Edu-cational Program criterion (20%), while the weights of the remaining criteria vary from 11% to 14.2%.
Students consider the criterion Image of the Department as the most important and simultaneously this criterion has the higher satisfaction value (92.3). This result can be explained by the fact that there is competition among other similar Departments in Universities and TEIs and this competition leads the students to focus on this specific criterion.
It must be noted, however, that the value of the demanding index for the Image of the Department criterion is extremely low (-75.6%). This leads to the conclusion that the students feel the need to declare “satisfied” even if they are not completely.
The sum of the weights of the criteria Educational Program and Teaching Personnel is 34.2%. It can be attributed, as a whole, to the dimension Education and raise it to the second top place in the hierarchy of satisfaction dimensions.
Combining weights and satisfaction indices, a series of action diagrams can be developed, which can designate the weak and strong points of students’ satisfaction and facilitate the efforts towards improvement.