1

RESOURCE: COMPARISON OF DIFFERENT EVALUATION METHODS

Method / Strengths / Things toconsider / Maybeuseful for…
Outputs/ statistical
data /
  • Complementsoutcomeevidenceby showingthat themuseumisworking with relevant groupsand individuals.
  • Canbeusedtosupport more qualitativestatements.
  • Straightforward tocollect.
/
  • Notevidenceofanoutcomeinitself. Forexample, datashowing that familiesaretakingpartinactivitiesis notevidence that familial tiesare beingstrengthened.
/
  • Recordingvisitorstoasite, participants inanactivity, membersof agroup.

Responsecards
Commentscards
Commentsbooks /
  • Areflexibleandaccessiblewaysof engagingwithusers.
  • Canbeusedaspart ofadisplaythat will stimulateother people toread and respond.
  • Need minimal administrationasusers cancompletethecardor writeina commentsbook themselves.
  • Can targetpeopleengagedin particular activitiesor indifferent partsofasitedependinguponwhere theresponsecardsor comment booksareplaced.
/
  • Posingquestionsinacommentsbook oronacardwill improve thequalityof thecommentsrelating tospecific GSO outcomes.
  • Placecommentscardsso thatall userscanseethem andare encouragedtosharetheirviews.
/
  • Visitorstoatemporary exhibition.
  • Participants inadropinactivity at theorganisation.

Graffitiwalls /
  • Areinteractiveascommentscanbe madetolookattractiveaspartofa display-peoplecanreadothers' commentsandmaybeencouragedto add their own.
/
  • Commentsneedtobefixedstrongly tothewall ortheymight belost. Provideapostingboxfor respondentswanting tokeep their commentsanonymous.
/
  • Visitorsataspecial event, festival oropenday.
  • Publicconsultationatavenue outside yourorganisation e.g. standorstall intowncentre.

  • Canincludeasinglequestionbased on theGSOssothat people's responsesarefocused.
/
  • Canbedoneusingdigital technologies/ interactivescreen ina galleryor exhibitionspace.
  • POSTITnotescanbeusedasa colourful/cheapwayof getting people tosharetheir comments.
/
  • Workswellwithyoungpeople.

Observation /
  • Observationisoneof themost powerful waysofunderstandingwhat isgoingon.Watchingadultsor childreninteractingwithanobject, exploringabuildingor takingpart ina creativeactivity will giveyouan insight intotheirexperiences.
  • Observationcanworkwell if participants ‘talkoutloud’about their experienceswhile theobserverwalks aroundwith them (walking tour).
  • Anaccompaniedwalking tourallows theobservertoclarifypointsmadeor askspecificquestionstoparticipants.
/
  • Setuptheobservationso that participantsdon't feel likeyouare judging/ assessingvisitors. Observationwithoutexplanation showswhatpeopledobutnotwhy theydoit soevidenceof GSO outcomesmaybelimited.
  • Mayneed tobecombinedwith interviewsor questionnaires.
  • Ifpeopleknowtheyarebeing observed their behaviourmaybe affected.
  • Iftheobserver ishidden itmaybe complextorecogniselearningfrom people'sactions/speech.
  • Thereareethical implications to hiddenobservations.
  • Thinkabout: how youwill recordthe observations, timingandfocusof
  • theobservations,whowill makethe observations–amemberof the
  • team or anindependent observer?
  • Considerusinganobservation checklist torecordwhat yousawor
/
  • Maybe moreusefulfor longer-termworkwithgroups. Recordingvisitorstoasite, participants inanactivity, membersof agroup.

guidetocodeand identifycertain behaviour.
  • YoucoulduseResource:blank observation templatetostructurethe recordingofyour observations.
  • You might alsowant todrawa map ofthevenue, torecordwhere participantswentandwhat theydid.

Letters/emails /
  • Lettersandemails tomuseums, archives and libraries can show evidenceofGSOoutcomes.
  • Will providearichsourceofdata.
/
  • Unpredictableandadhocsourceof collecting GSOoutcomes.
  • Uselessunlessproperlyanalysed.
/
  • Supportingevidence touse alongsideamoreformaldata collection method.
  • Goodwayofevidencing feedback from usersor staff frompartnerorganisationsat theendofaproject or pieceof partnershipwork.

Questionnaires /
  • Aregoodfor large-scalecollectionof evidenceofoutcomeswherebroad informationisrequiredrather than in- depthexploration.
  • Canbeusedon-site, byemail or post.
  • Couldbeself-completion,or completedbymembers ofstaff.
  • Canincludeopenandclosed questions, or multiplechoice,
focusingonspecificGSOoutcomes
youhope tocapture.
  • Canalsocollectdemographic
/
  • Questionsneedcareful phrasing to reflect age, languageandability levels ofthetarget group.
  • The'look' of thequestionnaireis importantandgooddesign iscrucial – thinkabout layout, colour,lengthof questionnaire, possiblyproviding questionnairesonapostcardformat, whichlooksmoreinviting tocomplete thanan A4sheetofwhitepaper.
  • Beawareof questionnaire 'overload' - is thismethodsuitablefor theusers youaretrying toreach?
/
  • Endoflonger-termprojectwork. Oneoffworkshopsoractivity sessions.

1

informationtoprovideevidenceof outputs(seeabove). / Make them manageableso thatusers arenotputoff by(perceived)lengthor difficultyof questions.
Journals/Reflective diaries /
  • Yourownexperiences,ideasand observationsoftheprojectarealsoa validformofevaluationdata. Your reflectivediarycanrecordcomments fromparticipants,anecdotes, inform your futureplans,andsupport other evidenceyouhavecollected.
  • Canprovideindepth, qualitativedata about theexperiencesof participants inlonger-termprojects.
  • May includediagrams,drawings, imagesor video (ifdoneusingdigital technologies) that canalsobe analysed.
/
  • Whowill keepthejournals,whenwill theycomplete it, andhow will you support them tofill itin? Forexample, specificquestionsor promptsto answeratspecificpoints in theproject or timemadeavailable tocompleteat theendofmeetings?
  • Youcouldkeeparecordofeachtime youworkwithagroupor are involved inaproject, usingthefollowing headingsasaguide:
-What wentwell
-Issuesor problems
-Howdoyoufeel?
-Ownevaluationofhowtheproject isgoing
-Action/ whatwill youdonext
  • Somemayfindkeepinga journalquite challenging,eitherbecauseoftheir literacyskills, or becausetheyare unused toreflectivewriting.
  • Thinkabouthow youwillanalyseany diagrams, drawingsandimagesyou mayfindinjournals. Thiscanbea verytimeconsumingprocess.
/
  • Long term projectwithadultsor youngpeople, working towards a joint projectorsharedgoal.
  • Canbeusedbystaff delivering sessionsasaself-reflective processandtocapture commentsoftenmissed with other methods.

Onetooneorgroup
Interviews /
  • It ishelpfultothinkofinterviewsas conversationswithapurpose. Interviewsgiveyouuseful data, butareconstructedthroughinteraction– boththeinterviewerand interviewee affect theinterviewoutcome. Forexample,childreninterviewingeach other will gatherverydifferent sortsofdatathanyouwouldinterviewing children,evenifyouusethesame questions.Therefore itis important totriangulate–see Resource6.2for more informationabout triangulation.
  • Interviewscanprovideindepth, qualitativeinformationonoutcomes– goodforpeoplemorecomfortable with talking thanwithwriting.
  • Can takeplacefacetofaceor onthe telephone.
  • Usedonetoone,interviewscan providegoodinformationfeelings, experiences,changes inperception orattitudesetc.
/
  • Findwaysofsettingparticipantsat
ease- theyneedtofeelcomfortable aboutsharingtheirexperienceswiththeinterviewer.
  • Iflanguage isan issueusea translatororpeerinterviewswhere onepersoncouldtranslatefor the other.
  • How will youstructuretheinterview andwhat questionswill youask?
  • Will you makenotesonpaperduring theinterview,or record theinterview toanalyselater?
  • Planhow youwill analyse thedata in advance–aninterviewmayproduce alargeamount ofevidence that may be time-consuming toanalyseunless theinterviewisstructured(then answers maybemorepredictable).
  • Youcouldconsider asking participantstocomeupwith the questions(tell them whatyouwantto findout, thenask for suggestionsfor thequestions) andtoconduct the interviews.
/
  • Longer termworkwith communitygroups, familiesor youngpeople.
  • Wouldworkwell for recording theoutcomesandbenefitsof beinginvolvedinlonger-term groups, suchasfriendsgroups, volunteeringor readersgroups.

Focusgroups /
  • Canelicitin-depth informationfrom participantsabout theirviewsand experiencesofmuseums,libraries andarchives.
  • Mayencouragepeopletosharetheir attitudes, beliefsandexperiences
/
  • Thefacilitatorshouldbeskilledin leadingthediscussionandkeeping thegroupfocused. Theyshould ensurethat everybodyfeels comfortableabout sharing their opinionsequally.
/
  • Longer termworkwith communitygroups, familiesor youngpeople.
  • Wouldworkwell for recording theoutcomesandbenefitsof beinginvolvedinlonger-term

1

moreopenlythroughgroup interaction.
  • Canbestructuredaround theGSOs tofocusdiscussion.
/
  • Makepractical arrangementsclear for all involvedincludinglocation, maps, furnitureandrefreshments.
  • You mayneedtopayfor participants toattendor provideanincentive.
  • Will you makenotesonpaperduring theinterview,or record theinterview toanalyselater?
  • Planhow youwill analyse thedata in advance–afocusgroupmayproduce alargeamount ofevidence that may be time-consuming toanalyseunless theinterviewisstructured.
/ groups, suchasfriendsgroups, volunteeringor readersgroups.
  • Canworkwell withyoung people.

Drawings /
  • Areuseful whenwritingskillsare limitedandmaybemore"fun" or engaging.
  • Canbeused incombinationwith writtencommentstoaidanalysis.
/
  • Thesearechallenging to interpret without questioningandmediation
  • The important aspectof thisapproach is that you thendiscusswith the participantsabout the imagesthey havecreated, whytheychose to
maketheminthisway,andwhatthey wanted toportray.
Samplequestionsfor this:
-Canyoudescribefor mewhatyou havedrawn?
-Why?
-Whatgaveyou theidea?
- Trytoavoidasking toomany questions, but let theparticipants
choosewhattotell you.
Youcanmakepositivecommentsto /
  • Workswell asawayforchildren tofeedbackonwhat theymost enjoyedabout theexperience etc–ask thechildrentodraw theiranswer toaspecific question, thentoexplain the drawingtoyou.

1

encourageparticipantstocarryon talking, rather thanasking them
anotherquestione.g. 'that'sgood', 'I
love that','uh-huh', 'wow'.
Video /
  • A videoboxcouldbe madeavailable for people toansweraspecific question,or toact asavideodiaryfor groupwork.
  • Isanappealingalternative to traditional commentscards- maybe morefunandengagingforsome usersthanwritingcomments.
  • Ispotentiallyapowerfultoolfor gatheringevidenceforadvocacy purposes.
/
  • Analysisofvideo maybetime consumingwith toomuch material.
  • Considerhow youwill codeand analyse that footageandthetime needed todothat.
/
  • Groupworkandprojects.
  • Generalconsultationwith the publicaboutyourservice.
  • Goodfor youngpeople.

Photographs/
Images /
  • Useful supportingevidence,when usedwithotherformsof data collection,to illustratehowaproject workedandwhathappened.
/
  • Maybedifficult toanalyseifthe context for thephotographsis unknown
  • Unlikelytoprovideconvincing evidencefor GSO outcomesby itself. The important aspectof thisapproach is that you thendiscusswith the participantsabout the imagesthey havecreated, whytheychose to
  • Maketheminthisway,andwhatthey wanted toportray.
Samplequestionsfor this:
-Canyoudescribefor mewhatyou havedrawn? /
  • Groupvisits, trips, projects, events.
  • Canbeusedbyparticipantsto conveytheirexperiences creativelye.g. recording their visittoamuseumondisposable cameras, thenturningit into artworkwithcaptionsetc.
  • Workswellwithfamily audiences–get thechildrento recordtheadultsusinga camera,orviceversa.

1

-Why?
-Whatgaveyou theidea?
- Trytoavoidasking toomany questions, but let theparticipants choosewhattotell you.
  • Youcanmakepositivecommentsto encourageparticipantstocarryon talking, rather thanasking them anotherquestione.g. 'that'sgood', 'I love that','uh-huh', 'wow'
  • Thinkaboutwhether it wouldbeworth investinginaprofessional photographer, andalsoabouthow
youwill gather permissionfrom people tobephotographed,andcopyright tousethephotosagain.
Artworks/
Sculptures/ things producedasaresult ofthevisit/ portfolio ofwork /
  • Canbeused inconjunctionwithother methodse.g. interviewsandfocus groupswithparticipants toobtainthe contextandhelparticulate the experience.
/
  • Asforimages, it maybedifficult to interpretandanalyseifthecontext is not known.
  • The important aspectof thisapproach is that you thendiscusswith the participantsabout the imagesthey havecreated, whytheychose tomaketheminthisway,andwhatthey wanted toportray.
Samplequestionsfor this:
-Canyoudescribefor mewhatyou havedrawn/made?
-Whatgaveyou theidea?
- Trytoavoidasking toomany /
  • Groupwork,projects,art and craftsessions, asappropriate. Couldbeusedasaprompt or startingpointforcapturing the voiceofparticipantse.g. throughonetoone interviews, focusgroups,annotatingand captioning thework.

1

questions, but let theparticipants choosewhattotell you.
  • Youcanmakepositivecommentsto encourageparticipantstocarryon talking, rather thanasking them anotherquestione.g. 'that'sgood', 'I love that','uh-huh', 'wow'.

WalkingTours /
  • Anincreasinglypopular wayof understandingspaceand communities.
  • Directedbytheparticipants–they decidewhere togoandwhatto includeon thetour, andyoufollow.
/
  • Forexample, youcouldwalkarounda museum or libraryor local areawitha childoradult.Asktheparticipantsto guideyouanddecidewheretogo, talkingaboutfavouritespaces,places they likeanddon'tlike, and talking about theobjectsthat interest and meansomething tothem.
  • A walking tour couldalso includea camera.
/
  • Linkingyour museum, libraryor archive moreclosely with the local community.
  • Understandinghowtheservices youoffer operateinawider context.

Secondarydata. For
exampleschool test results(SATS) /
  • Againuseful inbackingup more qualitativeevidence
  • Oftenprovidesapowerful message forfundingbodies.
/
  • Availabilityandinterpretationofdata–
therewill beconfidentiality issues here.
  • Establishingcausality-avery wide rangeof factorswill affect these longer-termindicators.
  • Avoid makingclaimsthat are unsubstantiated.
/
  • Formal educationalgroupswith whomyouhavelongterm contact.

Authors - CHE Associates

Published originally by - Museums, Libraries, and Archives Council, 2010