Resource: Comparison of Different Evaluation Methods

1

RESOURCE: COMPARISON OF DIFFERENT EVALUATION METHODS

• Complementsoutcomeevidenceby showingthat themuseumisworking with relevant groupsand individuals.
• Canbeusedtosupport more qualitativestatements.
• Straightforward tocollect.

• Notevidenceofanoutcomeinitself. Forexample, datashowing that familiesaretakingpartinactivitiesis notevidence that familial tiesare beingstrengthened.

• Recordingvisitorstoasite, participants inanactivity, membersof agroup.

• Areflexibleandaccessiblewaysof engagingwithusers.
• Canbeusedaspart ofadisplaythat will stimulateother people toread and respond.
• Need minimal administrationasusers cancompletethecardor writeina commentsbook themselves.
• Can targetpeopleengagedin particular activitiesor indifferent partsofasitedependinguponwhere theresponsecardsor comment booksareplaced.

• Posingquestionsinacommentsbook oronacardwill improve thequalityof thecommentsrelating tospecific GSO outcomes.
• Placecommentscardsso thatall userscanseethem andare encouragedtosharetheirviews.

• Visitorstoatemporary exhibition.
• Participants inadropinactivity at theorganisation.

• Areinteractiveascommentscanbe madetolookattractiveaspartofa display-peoplecanreadothers' commentsandmaybeencouragedto add their own.

• Commentsneedtobefixedstrongly tothewall ortheymight belost. Provideapostingboxfor respondentswanting tokeep their commentsanonymous.

• Visitorsataspecial event, festival oropenday.
• Publicconsultationatavenue outside yourorganisation e.g. standorstall intowncentre.

• Canincludeasinglequestionbased on theGSOssothat people's responsesarefocused.

• Canbedoneusingdigital technologies/ interactivescreen ina galleryor exhibitionspace.
• POSTITnotescanbeusedasa colourful/cheapwayof getting people tosharetheir comments.

• Workswellwithyoungpeople.

• Observationisoneof themost powerful waysofunderstandingwhat isgoingon.Watchingadultsor childreninteractingwithanobject, exploringabuildingor takingpart ina creativeactivity will giveyouan insight intotheirexperiences.
• Observationcanworkwell if participants ‘talkoutloud’about their experienceswhile theobserverwalks aroundwith them (walking tour).
• Anaccompaniedwalking tourallows theobservertoclarifypointsmadeor askspecificquestionstoparticipants.

• Setuptheobservationso that participantsdon't feel likeyouare judging/ assessingvisitors. Observationwithoutexplanation showswhatpeopledobutnotwhy theydoit soevidenceof GSO outcomesmaybelimited.
• Mayneed tobecombinedwith interviewsor questionnaires.
• Ifpeopleknowtheyarebeing observed their behaviourmaybe affected.
• Iftheobserver ishidden itmaybe complextorecogniselearningfrom people'sactions/speech.
• Thereareethical implications to hiddenobservations.
• Thinkabout: how youwill recordthe observations, timingandfocusof
• theobservations,whowill makethe observations–amemberof the
• team or anindependent observer?
• Considerusinganobservation checklist torecordwhat yousawor

• Maybe moreusefulfor longer-termworkwithgroups. Recordingvisitorstoasite, participants inanactivity, membersof agroup.

• YoucoulduseResource:blank observation templatetostructurethe recordingofyour observations.
• You might alsowant todrawa map ofthevenue, torecordwhere participantswentandwhat theydid.

• Lettersandemails tomuseums, archives and libraries can show evidenceofGSOoutcomes.
• Will providearichsourceofdata.

• Unpredictableandadhocsourceof collecting GSOoutcomes.
• Uselessunlessproperlyanalysed.

• Supportingevidence touse alongsideamoreformaldata collection method.
• Goodwayofevidencing feedback from usersor staff frompartnerorganisationsat theendofaproject or pieceof partnershipwork.

• Aregoodfor large-scalecollectionof evidenceofoutcomeswherebroad informationisrequiredrather than in- depthexploration.
• Canbeusedon-site, byemail or post.
• Couldbeself-completion,or completedbymembers ofstaff.
• Canincludeopenandclosed questions, or multiplechoice,

• Canalsocollectdemographic

• Questionsneedcareful phrasing to reflect age, languageandability levels ofthetarget group.
• The'look' of thequestionnaireis importantandgooddesign iscrucial – thinkabout layout, colour,lengthof questionnaire, possiblyproviding questionnairesonapostcardformat, whichlooksmoreinviting tocomplete thanan A4sheetofwhitepaper.
• Beawareof questionnaire 'overload' - is thismethodsuitablefor theusers youaretrying toreach?

• Endoflonger-termprojectwork. Oneoffworkshopsoractivity sessions.

1

• Yourownexperiences,ideasand observationsoftheprojectarealsoa validformofevaluationdata. Your reflectivediarycanrecordcomments fromparticipants,anecdotes, inform your futureplans,andsupport other evidenceyouhavecollected.
• Canprovideindepth, qualitativedata about theexperiencesof participants inlonger-termprojects.
• May includediagrams,drawings, imagesor video (ifdoneusingdigital technologies) that canalsobe analysed.

• Whowill keepthejournals,whenwill theycomplete it, andhow will you support them tofill itin? Forexample, specificquestionsor promptsto answeratspecificpoints in theproject or timemadeavailable tocompleteat theendofmeetings?
• Youcouldkeeparecordofeachtime youworkwithagroupor are involved inaproject, usingthefollowing headingsasaguide:

• Somemayfindkeepinga journalquite challenging,eitherbecauseoftheir literacyskills, or becausetheyare unused toreflectivewriting.
• Thinkabouthow youwillanalyseany diagrams, drawingsandimagesyou mayfindinjournals. Thiscanbea verytimeconsumingprocess.

• Long term projectwithadultsor youngpeople, working towards a joint projectorsharedgoal.
• Canbeusedbystaff delivering sessionsasaself-reflective processandtocapture commentsoftenmissed with other methods.

• It ishelpfultothinkofinterviewsas conversationswithapurpose. Interviewsgiveyouuseful data, butareconstructedthroughinteraction– boththeinterviewerand interviewee affect theinterviewoutcome. Forexample,childreninterviewingeach other will gatherverydifferent sortsofdatathanyouwouldinterviewing children,evenifyouusethesame questions.Therefore itis important totriangulate–see Resource6.2for more informationabout triangulation.

• Interviewscanprovideindepth, qualitativeinformationonoutcomes– goodforpeoplemorecomfortable with talking thanwithwriting.
• Can takeplacefacetofaceor onthe telephone.
• Usedonetoone,interviewscan providegoodinformationfeelings, experiences,changes inperception orattitudesetc.

• Findwaysofsettingparticipantsat

• Iflanguage isan issueusea translatororpeerinterviewswhere onepersoncouldtranslatefor the other.
• How will youstructuretheinterview andwhat questionswill youask?
• Will you makenotesonpaperduring theinterview,or record theinterview toanalyselater?
• Planhow youwill analyse thedata in advance–aninterviewmayproduce alargeamount ofevidence that may be time-consuming toanalyseunless theinterviewisstructured(then answers maybemorepredictable).
• Youcouldconsider asking participantstocomeupwith the questions(tell them whatyouwantto findout, thenask for suggestionsfor thequestions) andtoconduct the interviews.

• Longer termworkwith communitygroups, familiesor youngpeople.

• Wouldworkwell for recording theoutcomesandbenefitsof beinginvolvedinlonger-term groups, suchasfriendsgroups, volunteeringor readersgroups.

• Canelicitin-depth informationfrom participantsabout theirviewsand experiencesofmuseums,libraries andarchives.
• Mayencouragepeopletosharetheir attitudes, beliefsandexperiences

• Thefacilitatorshouldbeskilledin leadingthediscussionandkeeping thegroupfocused. Theyshould ensurethat everybodyfeels comfortableabout sharing their opinionsequally.

• Longer termworkwith communitygroups, familiesor youngpeople.
• Wouldworkwell for recording theoutcomesandbenefitsof beinginvolvedinlonger-term

1

• Canbestructuredaround theGSOs tofocusdiscussion.

• Makepractical arrangementsclear for all involvedincludinglocation, maps, furnitureandrefreshments.
• You mayneedtopayfor participants toattendor provideanincentive.
• Will you makenotesonpaperduring theinterview,or record theinterview toanalyselater?
• Planhow youwill analyse thedata in advance–afocusgroupmayproduce alargeamount ofevidence that may be time-consuming toanalyseunless theinterviewisstructured.

• Canworkwell withyoung people.

• Areuseful whenwritingskillsare limitedandmaybemore"fun" or engaging.
• Canbeused incombinationwith writtencommentstoaidanalysis.

• Thesearechallenging to interpret without questioningandmediation
• The important aspectof thisapproach is that you thendiscusswith the participantsabout the imagesthey havecreated, whytheychose to

• Workswell asawayforchildren tofeedbackonwhat theymost enjoyedabout theexperience etc–ask thechildrentodraw theiranswer toaspecific question, thentoexplain the drawingtoyou.

1

• A videoboxcouldbe madeavailable for people toansweraspecific question,or toact asavideodiaryfor groupwork.
• Isanappealingalternative to traditional commentscards- maybe morefunandengagingforsome usersthanwritingcomments.
• Ispotentiallyapowerfultoolfor gatheringevidenceforadvocacy purposes.

• Analysisofvideo maybetime consumingwith toomuch material.
• Considerhow youwill codeand analyse that footageandthetime needed todothat.

• Groupworkandprojects.
• Generalconsultationwith the publicaboutyourservice.
• Goodfor youngpeople.

• Useful supportingevidence,when usedwithotherformsof data collection,to illustratehowaproject workedandwhathappened.

• Maybedifficult toanalyseifthe context for thephotographsis unknown
• Unlikelytoprovideconvincing evidencefor GSO outcomesby itself. The important aspectof thisapproach is that you thendiscusswith the participantsabout the imagesthey havecreated, whytheychose to
• Maketheminthisway,andwhatthey wanted toportray.

• Groupvisits, trips, projects, events.
• Canbeusedbyparticipantsto conveytheirexperiences creativelye.g. recording their visittoamuseumondisposable cameras, thenturningit into artworkwithcaptionsetc.
• Workswellwithfamily audiences–get thechildrento recordtheadultsusinga camera,orviceversa.

1

• Youcanmakepositivecommentsto encourageparticipantstocarryon talking, rather thanasking them anotherquestione.g. 'that'sgood', 'I love that','uh-huh', 'wow'
• Thinkaboutwhether it wouldbeworth investinginaprofessional photographer, andalsoabouthow

• Canbeused inconjunctionwithother methodse.g. interviewsandfocus groupswithparticipants toobtainthe contextandhelparticulate the experience.

• Asforimages, it maybedifficult to interpretandanalyseifthecontext is not known.
• The important aspectof thisapproach is that you thendiscusswith the participantsabout the imagesthey havecreated, whytheychose tomaketheminthisway,andwhatthey wanted toportray.

• Groupwork,projects,art and craftsessions, asappropriate. Couldbeusedasaprompt or startingpointforcapturing the voiceofparticipantse.g. throughonetoone interviews, focusgroups,annotatingand captioning thework.

1

• Youcanmakepositivecommentsto encourageparticipantstocarryon talking, rather thanasking them anotherquestione.g. 'that'sgood', 'I love that','uh-huh', 'wow'.

• Anincreasinglypopular wayof understandingspaceand communities.
• Directedbytheparticipants–they decidewhere togoandwhatto includeon thetour, andyoufollow.

• Forexample, youcouldwalkarounda museum or libraryor local areawitha childoradult.Asktheparticipantsto guideyouanddecidewheretogo, talkingaboutfavouritespaces,places they likeanddon'tlike, and talking about theobjectsthat interest and meansomething tothem.
• A walking tour couldalso includea camera.

• Linkingyour museum, libraryor archive moreclosely with the local community.
• Understandinghowtheservices youoffer operateinawider context.

• Againuseful inbackingup more qualitativeevidence
• Oftenprovidesapowerful message forfundingbodies.

• Availabilityandinterpretationofdata–

• Establishingcausality-avery wide rangeof factorswill affect these longer-termindicators.
• Avoid makingclaimsthat are unsubstantiated.

• Formal educationalgroupswith whomyouhavelongterm contact.

Authors - CHE Associates

Published originally by - Museums, Libraries, and Archives Council, 2010