Module 7: Analyzing Data

Overview

STEPS Module 7: Analyzing Data Page 24

· Prepare and test your analysis plan- with a focus on understanding your impact

· Ensure adequate analysis of the data

· Interpret your findings

STEPS Module 7: Analyzing Data Page 24

Key Concepts

Step 1: Creating your Analysis Plan

Task 1: Develop your analysis plan

Task 2: Practice run--test your analysis plan

Task 3: Adjust your Logical Framework as necessary

Worksheet: Analysis Planning

Step 2: Ensuring Adequate Analysis of the Data

Task 1: Managing Data Collection

Worksheet: Checklist for Supervising Data Collection

Task 2: Process your data carefully

Task 3: Use descriptive statistics to summarize group responses and make the comparisons that answer your evaluation questions

Worksheet: Checklist for Good Analysis Practice

Step 3: Interpreting your Findings

Task 1: Understand aspects within your findings that are particularly surprising to you and your M&E Team

Task 2: Undertake analyses that can help you know if the findings are the result of your program

Summary

Tips

References

STEPS Module 7: Analyzing Data Page 24

Key Concepts

STEPS Module 7: Analyzing Data Page 24

Analysis plan – the steps you intend to carry out in order to make the comparisons needed for your evaluation

Basic comparisons – the overall comparisons you will make between pre- and post- program data and between program and comparison group data.

Code - a rule for converting a piece of information, for example the data you collect, into another form. For example, all women will be coded as “1,” or all women who attended all of the workshop sessions will be coded as “intensive participants”. You will decide on codes and all of the team members should apply the codes in the same way.

Coding – the process of applying your codes- organizing, categorizing, and giving meaning to quantitative and qualitative data.

Cross-tabulation - a way to analyze and present 2 or more variables and how they intersect, in a matrix table format. For example, male and female might be on one side of the table and 3 levels of program outcomes might be on the other. This way you can easily see subcategories of variables by gender and program outcome.

Data analysis – the process of examining data to summarize information about particular groups at particular points of time in ways that will allow you to determine if the changes that you are seeking actually occurred, what other changes your program might have produced and whether, how and why the program did and/or did not achieve its intended objectives.

Data processing – careful and systematic procedures to prepare, code, and organize your data for analysis.

Descriptive statistics are used to describe a collection of data in quantitative terms. Descriptive statistics quantitatively summarize a data set, rather than being used to support statements about the population that the data are thought to represent (which is what inductive statistics do).

Frequency - the number of occurrences or observations for a variable. For example, if the variable you are examining is ‘respondent has one child’, and you found that of the 135 teenage girls you interviewed who gave valid responses, 52 had one child, you are stating a frequency.

Inferential Statistics - using statistics so that you can infer, or learn, something about the larger population from the smaller one you have sampled and studied.

Intra-group comparisons – comparisons done by separating out different subgroups within your basic comparison groups such as boys who had highly gender equitable attitudes at the beginning of the program vs. boys who did not.

Mean - the value that is the average of all of the data combined. For example, 9.4 is the mean for this data set: 1,3,3,4,5,10,12,12,16,17,20. The mean can be skewed if there are too many very low or very high values.

Median - the value that is halfway through your data set, at which point an equal number of data points are above and below. For example, 10 is the median for this data set: 1,3,3,4,5,10,12,12,16,17,20

Mode - the most common answer in a data set. You can have more than one mode in each data set. For example, 3 and 12 are the modes for this data set: 1,3,3,4,5,10,12,12,16,17,20.

Statistical Significance - a finding is unlikely to have occurred by chance if it is statistically significant. You can only test for significance with the right amount (usually a sample of at least 200) of properly collected data. Statistical significance does not in and of itself make the finding meaningful for your evaluation.

Typology - a way of analyzing and understanding your findings, it is a type or characterization of a sub-group of experience, person, or outcome.

STEPS Module 7: Analyzing Data Page 24

Step 1: Creating your Analysis Plan

Rationale

STEPS Module 7: Analyzing Data Page 24

Even if you have planned the evaluation well and have designed excellent instruments and data collection plan, you need to pay close attention to planning your data analysis:

· Before you start collecting the data and

· During data collection

¬ Never start data collection until you know how you will analyze your data and tested out the data collection instruments and analysis plan.

Task 1: Develop your analysis plan.

If you have access to someone with experience in data analysis, hopefully you already have invited that person to be part of the M&E Team (see Module 1) even if she or he has not been involved at every moment. If you are going to contact an evaluation expert, do not wait until you already have the data collected. Make sure you carefully plan the analyses yourselves, or do it with an expert before you start to collect the data.

There are two types of data analysis questions: one type is focused on understanding how your program was delivered and the other is focused on your program’s impact. You will be focusing your data analysis on understanding, or evaluating, the impact you’ve had- the change you have brought about in your target population. In order to understand your program’s impact you will also need to pay attention to how the program was delivered- your monitoring data.

ð Review the evaluation questions and the program objectives. Focus on the comparisons that will answer your evaluation questions and tell you whether or not the program achieved its objectives.

The most basic comparisons are:

· Pre- vs. Mid,- and/or Post-program

· Post-program vs. Comparison group

Within each of the “rows” on this chart there may be sub-groups you need to explore according to dimensions that are relevant to the program; particularly you may want to find out if certain people benefited more from the program than others.

Within each “column” there may also be variations you will want to explore. Did people who attended the program more times, were exposed to its messages more intensely, who perceived it more positively or had more skills initially end up having better scores on the immediate or intermediate results measures?

STEPS Module 7: Analyzing Data Page 24

Groups of indicators / Sub-groups / Pre-program / Mid and/or Post-program / Comparison Group
High attendance / Low attendance / High
attendance / Low attendance
Objective 1 / With certain history
Without certain history
Objective 2 / High on some aspect
Medium on some aspect
Low on some aspect
Objective 3 / High on some demographic aspect
Low on some demographic aspect
Other evaluation question

STEPS Module 7: Analyzing Data Page 24

List all the comparisons that are linked to your evaluation questions and objectives.

ð Check the kind of data you have and decide which analyses you can and should use

Remember that your indicators will either be numerical or non-numerical.

· You can transform qualitative data into numerical or non-numerical indicators

· Your quantitative data will be analyzed as numerical indicators

If you are analyzing non-numerical indicators, you will

be able to describe your findings in terms of themes and typologies and demonstrate your findings with quotations and testimonies. For more details, Tips: Forms of Qualitative Analysis

· need to indicate how many data sources (e.g. people or documents) you assessed showed the findings you report out of the entire group you studied, e.g. 5 out of 7 newspapers reported stories about domestic violence in negative ways that denigrated or criticized the woman and defended the man.

If you are analyzing numerical indicators, you will be able to calculate differences in means, medians, or percents.

To choose the kind of analysis you can do according to the level of indicator or type of numerical data you have, see Tips: Kind of Analysis According to Level of Numerical Indicators

For more details about the principle kinds of descriptive statistics you will use with numerical indicators see Tips: Descriptive Statistics for Analysis of Numerical Indicators

ð Decide what intra-group comparisons are needed to address your evaluation questions

· Using your monitoring data explore how aspects of the program and attendance or exposure to the program may have affected the evaluation results

· Using socio-demographic data you collected and evidence you used to construct your Theory of Change (Module 2) and Causal Pathway (Module 3), explore if there were different results within each of your groups, such as by age, sex, marital status, economic level, sexual and/or reproductive history, level of motivation, certain pre-dispositional factors, etc. Are there other comparisons needed that better reflect a rights-based social justice perspective? Tips: Rights-based Social Justice Considerations for Both Quantitative and Qualitative Analyses

STEPS Module 7: Analyzing Data Page 24

ð For each set of indicators, fill in the Worksheet: Analysis Planning

Worksheet: Analysis Planning

Objective or Evaluation question:
Indicator:
Type of Data / Type of Indicator / Indicators reported as: / Basic comparisons to be made / Intra-group comparisons
Qualitative / Non-numerical /  categories
 quotes
 themes
 typologies
Numerical /  percent
 mean
 median
 cross tabulation
Quantitative / Numerical:
nominal
ordinal
interval / percent
mean
median
cross tabulation
inferential statistics

STEPS Module 7: Analyzing Data Page 24

Task 2: Practice run- Test your analysis plan

Why is this so important?

· To help you improve your data collection tools before it is too late

· To help you know whether or not you are collecting all the information you need, or gauge if you are collecting information that won’t be useful

· To help you find out if you are collecting information from the correct data sources

ð Tabulate the data you obtained in the practice run-through (pilot) of your data collection instruments and data collection plan using the same system you will use for data analysis.

Tips: What If You Have No Pilot Test Data?

· See if there are problems with:

o missing data

o instruments filled out incorrectly

o answers that are drastically different than you expected which might indicate

§ the questions were not understood as intended

§ they were not coded correctly

§ you are asking the wrong people

Ask the data collectors for their impressions about this.

You may need to

· rephrase the questions

· clarify the instructions

· refresh the training of your data collectors

ð As you test out your analysis plan, check off each piece of information you use in your data collection instrument. Are there any unchecked pieces of information left after you have specified analyses for each objective? If so, review whether they are necessary and if so, why they are not addressed in your objectives.

Example: In a needs assessment about sexual rights conducted among young people, interviews were carried out to see which rights were most important. One of the choices was: “this right is indispensable”. After the pilot test, the team saw that very few people checked this option. The interviewers had observed that many respondents seemed confused by the word ‘indispensable.’ The questions had been initially tested with the peer educators, but not those similar to the interviewees.

ð For each of your evaluation questions, use the data from your practice run to try out the comparisons that you plan to use according to your analysis plan. Check off each piece of information that you use in your data collection instrument.

Ask yourselves:

· Do you have the data you need?

· Are they in the form needed for the analyses you plan?

· Are there any unchecked pieces of information left after you specified analyses for each objective or evaluation question? If so, are they necessary? Find out why they were not addressed in your comparisons and correct the mistake. If they are not necessary, eliminate them if possible.

Example: If you expect that the impact of your program may be different for women who have had more than one abortion, as compared to those who have had only one or none, but you only asked the question if she ever had an abortion (yes vs. no) you would not be able to make the comparison. Now is the time to adjust the question and answer options needed for your analysis. Tips: What If You Have No Pilot Test Data?

¬ After considering how you would use the data, cut out items that are not useful.

ð Make the modifications as needed.

· Consult your data collectors to base changes on their insights and observations.

· Adjust:

· the instruments

· how data are collected and coded, and/or

· how data are tabulated

· When adjustments are significant enough to change the way people might respond, exclude pre-revision responses from the analysis.

Module 7: Analyzing Data