Introduction to Clinical Information Management for A&E Specialist Registrars Evaluation Questionnaire Results
Course Evaluation Results
Produced by: Robin Beaumont
Date: 25/09/2000 10:18
Version: 1.0
E-mail:
Contents
1 Executive summary 2
2 Introduction 3
3 Results 3
3.1 Personal assessment of course 3
3.2 Electronic support for course 5
3.3 Possible changes to the content 5
3.4 Possible changes to teaching methods 7
3.5 Knowledge of concepts introduced during course 8
3.6 Course preparation 9
3.7 Textual comments concerning expectations 10
3.8 Additional free text comments 10
1 Executive summary
Background
This report provides details of the data collected from an evaluation questionnaire assessing a Clinical Information Management course for Specialist Registrars (SpRs) in Accident & Emergency (A&E). Full details of the course can be found at http://www.ieg-net/resources/hi/kings/
Recommendations based upon the information provided in this report along with other supporting information can also be found at the above Web site.
Results
Three quarter of respondents considered that the course had at least adequately provided what they wanted
Over 50% of respondents had attended all the days
Two Thirds of respondents felt that the knowledge and skills gained were useful to them
Two thirds of respondents did not use the course web site or the e-mail facilities
Nine of out ten respondents wanted some hands on experience during the course the majority suggesting that just under half the time should be devoted to such aspects
Over half of the respondents suggested that future sources should include:
1. Statistics using computers (hands on)
2. Audit techniques facilitated by computers
3. Developing your own databases
4. Methods of developing clinical information systems
5. Software packages e.g. Excel etc (hands on)
It is interesting to note that all the above choices, except '4', are concerned with hands on skills.
Suggestions for changing the teaching methods used indicated that:
· more "computer mediated learning" approaches (e.g. web based) were unpopular
· methods to increase face to face communication such as "Additional small group sessions" and "additional days" for specific areas were popular
· Open learning techniques and distribution of the course material on CDRom were also popular
Eight out of ten respondents would recommend such an improved course
Format and distribution of pre-course information was poor
2 Introduction
This report provides details of the data collected from an evaluation questionniare assessing a Clinical Information Management course for Specialist Registrars (SpRs) in Accident & Emergency (A&E). . Full details of the course, including the questionnaire, can be found at http://www.ieg-net/resources/hi/kings/
Recommendations based upon the information provided in this report along with other supporting information can also be found at the above Web site.
3 Results
The following sections work through each section of the questionnaire and make comparisons between relevent variables.
3.1 Personal assessment of course
Respondents were asked "For you, how far would you say the course had given you what you wanted?" (n=19):
not at all / very poor / inadequate / adequate / good / excellent1 (5%) / 0 / 4 (21%) / 10 (53%) / 4 (21%)
Respondents were asked "How much do you think you have learnt from the course?"
nothing / far too little / too little / enough / significant amount / vast amount7 (41%) / 4 (24%) / 6 (35%)
The course was divided into several parts, respondents were asked how adequate each area was covered and also if they would have liked less or more. The results are given below. It should be noted that day 4 did not take place for details see the final report at http://www.ieg-net/resources/hi/kings/
course aspect / Unable to remember or did not attend / very poor / poor / adequate / good / excellent / n / less / correct amount / more / nDay 1- looking at the information you collect (questionnaires etc) / 5 (26%) / 7 (37%) / 5 (26%) / 2 (11%) / 19 / 2 (17%) / 6 (50%) / 4 (33%) / 12
Day 2 - Project management & BPR / 3 (16%) / 6 (37%) / 10 (63%) / 16 / 1 (10%) / 8 (80%) / 1 (10%) / 10
Day 3- Care pathways / 6 (32%) / 4 (31%) / 6 (46%) / 3 (23%) / 13 / 7 (88%) / 1 (12%) / 8
Day 3 - Confidentiality & security / 7 (37%) / 1 (9%) / 3 (27%) / 7 (64%) / 11 / 1 (17%) / 4 (66%) / 1 (17%) / 6
Day 4 - Analysing information / 9 (47%) / 2 / 1
Based upon various responses in the questionnaire it was possible to calculate the number of days each respondent attedned.
number of days attended / number of paticipants4 / 10
3 / 2
2 / 4
1 / 2
Respondents were asked "How useful do you think the knowledge and skills you have gained from the course will be to you when you become a consultant?" (n=19)
don't know / no use / little use / useful / extremely useful /1 (5%) / 4 (21%) / 13 (69%) / 1 (5%) /
Respondents were asked "Do you think the present course should be offered to other SpRs?" (n=19)
yes / no / don't know11 (58%) / 2 (10%) / 6 (32%)
Respondents were asked to provide a textual reason why they had answered the above question the way they did:
Those who answered 'Yes':
The course informs you of 'management speak' and enables you to understand management and therefore communicate better.
Data is'nt going to go away. Data can be enpowering and future consultants need to know this.
Gave me a new way of thinking about things we do everyday i.e. processes and data collection.
I think this course should be helpful in framing & understanding project management & care pathways particularly.
Care pthways are particularly useful. Making us think about information sources and how to use them BEFORE we start a research project is very beneficial.
Provides a structured basis and framework for all information gathering, which in turn is the foundation for all audit, research and clinical goverance work.
Important in our day to day & future work. A course tailored to our needs helpful.
The little I went to was extremely [sentence incomplete]
Those who answered 'Maybe' or 'No':
Different SpRs have different levels of understanding of clinical information management.
I hoped the course would help in using various systems in A&E work.
I only attended day one. The course was NOT what I expected. Therefores I did not attend the others. I was expecting a course advising us how to use a pc and their applications.
With some modifications perhaps
Not in current format. If objectives and clinical relevance are stated + continued throughout the course it could be very good.
3.2 Electronic support for course
The course had both a web site and the facility to e mail the course co-ordinators.
Respondents were asked "The course had a web site. How useful was this?" (n=19)
did not use it / no use / little use / useful / extremely useful /13 (68%) / 1 (5%) / 3 (16%) / 2 (11%) /
Respondents were asked "The course provided you with the ability to e-mail the course facilitators. How useful was this?" (n=19)
did not use it / no use / little use / useful / extremely useful /13 (69%) / 0 / 1 (5%) / 5 (26%) /
3.3 Possible changes to the content
The course was designed as a pilot and the co-ordinators were keen to discover what the participations felt could be improved with the course. The course co-ordinators made the conscious decision to not have any hands on sessions during it as it was felt that those undertaking the course would both possess the relevent IT skills and have access to basic IT facilities. Is soon became clear that this was not the case.
Respondents were asked "The present course provided very little computer hands on experience. Would you have liked?":
1 (5%) / same format / 17 (90%) / Some hands on experience"Please state the percentage of hands on experience you consider appropriate"
Mean / Std Dev / Range / Minimum / Maximum / N42.19% / 17.70% / 60.00% / 20% / 80% / 16
During the course many possible topics were not included and others only touched on. Respondents were asked "Which of the following would you like included in any improved course?" (n=19) Values above 50% highlighted:
6 (32%) / 2.Coding & classifications / 3 (16%) / 11. Decision support with the help of a computer
2 (11%) / 3.Casemix & HRGs / 3 (16%) / 12. Consultation techniques involving a computer
11 (58%) / 4.Developing your own databases / 12 (63%) / 13. Audit techniques facilitated by computers
3 (16%) / 5.Liaising with the computer dept. / 7 (37%) / 14.Library searches
4 (21%) / 6.Specifying requirements to the computer dept. / 7 (37%) / 15.The Internet
10 (53%) / 7.Methods of developing clinical information systems / 13 (68%) / 16.Statistics using computers (hands on)
6 (32%) / 8.Querying databases / 10 (53%) / 17. Software packages e.g. Excel etc (hands on)
0 / 9. Other (Please specify) / 1 (5%) / 18. General updating of topics
A graphical version of the above results is provided below
From the above it can be seen that five topic areas were considered to be important by over 50% of respondents:
- Statistics using computers (hands on)
- Audit techniques facilitated by computers
- Developing your own databases
- Methods of developing clinical information systems
- Software packages e.g. Excel etc (hands on)
It is interesting to note that all the above choices, except '4', are concerned with hands on skills.
3.4 Possible changes to teaching methods
Section three of the questionnaire was concerned with investigating any possible changes that respondents felt that would be advantagous to any future course.
Respondents were asked "The course delivery could be changed in a variety of ways. What do you think of each of the following"
no use / little use / some use / useful / invaluable / n1.Additional personal tutorials / 3 (18%) / 3 (18%) / 6 (35%) / 3 (29%) / 17
2.Additional small group sessions for specific areas / 3 (18%) / 5 (29%) / 9 (53%) / 17
3.Residential weekend / 8 (44%) / 6 (33%) / 1 (6%) / 2 (11%) / 1 (6%) / 18
4.Visits to centres of excellence / 4 (21%) / 7 (37%) / 4 (21%) / 4 (21%) / 19
5.Additional days / 5 (28%) / 3 (16.5%) / 3 (16.5%) / 6 (33%) / 1 (6%) / 18
6.More emphasis on self/open learning with exercises between sessions / 3 (17%) / 6 (33%) / 9 (50%) / 18
7.Laptop computers for candidates / 1 (6%) / 6 (33%) / 4 (22%) / 6 (33%) / 1 (6%) / 18
8.Provide handouts on a CD Rom / 1 (5%) / 5 (26%) / 3 (16%) / 9 (47%) / 1 (5%) / 19
9.Newsgroup / bulletin board via e-mail / 2 (11%) / 7 (39%) / 4 (22%) / 5 (28%) / 18
10.Convert the course to a web based open learning format / 5 (28%) / 6 (33%) / 5 (28%) / 2 (11%) / 18
11.All four sessions together / 6 (33%) / 4 (22%) / 1 (6%) / 7 (39%) / 18
12.Other (please specify): / 1 / 1
One respondent selected the 'other' option providing the following details "All sessions together with hands-on experience with laptops".
It is interesting to note that options concerned with re-developing the course into a more "computer mediated learning" approach (options 9 and 10) were unpopular. In contrast, methods to increase face to face communication such as "Additional small group sessions" and "additional days" for specific areas were popular. Open learning techniques and distribution of the course material on CD-ROM were also popular.
Respondents were asked "Would you recommend the course to others if it incorporated the above changes you have suggested?"
yes / no / don't know / Probably ( category added by respondent)15 (83%) / 0 / 2 (11%) / 1 (6%)
3.5 Knowledge of concepts introduced during course
To access how much the respondents had assimulated during the course they were asked how much they understood several terms introduced during the course. It should be noted that item 5 "report generator" was intended to assess the material to be presented on day four which did not take place.
term / Unawareof / Aware
of / Understand
Partially Fully / n
1.Caldicott Guardian / 13 (72%) / 2 (11%) / 2 (11%) / 1 (6%) / 18
2. EPR / EHR (Electronic Health Record) / 9 (50%) / 7 (39%) / 2 (11%) / 18
3. Clinical Information Management / 1 (6%) / 8 (44%) / 7 (39%) / 2 (11%) / 18
4. Care pathways / 7 (39%) / 7 (39%) / 4 (22%) / 18
5.Report generator / 13 (72%) / 3 (17%) / 2 (11%) / 18
6. Coding frame / 5 (28%) / 6 (33%) / 4 (22%) / 3 (17%) / 19
7. PRINCE or any other project management method / 9 (53%) / 4 (23.5%) / 4 (23.5%) / 17
8. Critical path / 5 (29%) / 5 (29%) / 5 (29%) / 2 (13%) / 17
9. Post coding / 9 (50%) / 3 (17%) / 6 (33%) / 18
10. Health Informatics / 8 (44%) / 10 (56%) / 18
11. Questionnaire development stages / 1 (6%) / 6 (33%) / 9 (50%) / 2 (11%) / 18
12.Data dictionary / 7 (39%) / 4 (22%) / 6 (33%) / 1 (6%) / 18
13.Business Process Re-engineering (BPR) / 14 (78%) / 3 (17%) / 1 (5%) / 18
14. Data quality / 4 (22%) / 6 (33%) / 5 (28%) / 3 (17%) / 18
15. Information for Health document / 8 (47%) / 7 (41%) / 1 (6%) / 1 (6%) / 17
The scores for each respondent for the above question set were summed to create a total learning score. Considering the total learning score against days attended. there did not appear to be any difference ( Kruskal-Wallis 1-Way Anova c2=4.8181, df=3, p=0.1856).