Deciding Which Method(S) to Use in Evaluation

Deciding which method(s) to use in evaluation

Work through the columns A to E to decide which methods would be most useful for collecting the data you require. There are pros and cons to using different methods and you will find that you will have to make decisions considering their overall feasibility.

A : Objectives / B : Audience / C : Sources / D: Nature of this
information / E : Additional data
What are the intended outcomes of this CPD activity?
Rephrase these as questions you would like to answer. / For whom is this data intended?
In what form/level of detail would it be best received? / Think about the primary question you are trying to answer.
Who is best placed to provide you with this information? (select more than one if required)
Participants/Students/Peers
/Staff involved in CPD/
Line managers/ Institutional leaders / What would this information look like?
Consider your responses to columns B and C to select a method with guidance from the next table / What other information would it be useful to consider?
Is there any data you could use to explore the results obtained in more detail? Reconsider columns B,C and D with this in mind.
For the participants
1.
2.
3.
For students
1.
2.
3.
For the institution
1.
2.
3.
Prompts to consider / Possible methods
Is this information already available? / Existing data sets.

Institutional/departmental/programme-level data sets

Is this information you can elicit from analysing material which already exists? / Participant or student reflections, journals, assessment design, modifications to curriculum
Is this information fairly straightforward, which an individual could provide quite easily by selecting from different options or responding in a few sentences?
Is it important that you reach a large number of people for this information?
Would you like to give individuals a chance to remain anonymous? / Questionnaires

Can include a mix of closed and open questions to generate both qualitative and quantitative data
Design of the questionnaire to be considered carefully, keeping in mind analysis.
Piloting is important

Is this a question that can be better answered by discussion between individuals involved in the CPD activity – bringing together different opinions and ideas about a subject? / Focus groups

Important for individuals to feel comfortable in the group
Useful to have two facilitators - one to encourage discussion and the other to take notes
Recording, transcription and ethics around this

Does the information necessitate detailed explanations, understanding of thought processes, attitudes or a range of influencing factors?
Would you rather have in-depth information from a small group of people rather than short, factual responses from a larger group? / Interviews

Structured, semi-structured or unstructured.
Interview schedule
Recording, transcription and ethics around this

Are you interested in seeing what people do rather than listening to what they say they do? / Observations
Could you collect this information using other innovative methods? /

Role-play
Using Lego to express and explain concepts,
Expression through artefacts – paintings, sculptures.