The Research Design Checklist: A Guide for Developing a Research Project

Dr. Ken Mease

This document will help guide you through the development of your research design (RD). It outlines what is necessary for a proper research design and summarizes many of the important concepts. It does not replace critical thinking. There is no one size fits all when it comes to solving problems and answering burning questions. Please read it very carefully; I suggest at least twice. For more in-depth information, look at Russ Bernard's Social Research Methods, Sage Publications, 2000. This is a good site for basic information on research methods and statistics:

  • There are many other sources available to help you design a research project. However, if you want my help, you must use this guide as a template; use the exact same section headings in the same order in your RD.It is best to begin writing your RD in a copy of this file by removing most of the explanatory text and just keep the headings and the Section Checklists. Start byconcentrating on Sections 1 through 10.
  • The Section Checklists,located at the end of most sections,remind you what you need to address in each section. A research design is not complete until every item inthe Section Checklists is addressed.Please examine the FINAL CHECKLIST before sending me your RD.

What’s Inside

1

  • Helpful Hints
  • Problem Statement
  • Literature Review
  • Hypotheses
  • Variables and Data
  • The Model
  • Surveys
  • The TABLE
  • Sampling
  • Methodology and Data Collection
  • Data Analysis
  • Potential Problems
  • Important Contacts
  • Research Schedule
  • Budget
  • The FINAL CHECKLIST

1

Appendix: - Some Examples and Helpful Information.

  • Example of Hypotheses (Section 3)
  • Example of a Model (Section 5)l
  • Example of Survey Questions (Section 6)
  • Example of the TABLE (Section 7)
  • The RD Continuity Crosscheck Table (not required in your RD)
  • Writing better Questions and Items –Read this before working on your survey. (Section 6)
  • Quick Guide to Bivariate and Multivariate Statistical Testing (Section 9)

Helpful Hints

Types of RDs.

  1. Traditional RDs – where you are trying to explain some behavior or outcome. Here there are usually a set of independent variables (IV) and usually one dependent variable (DV), a classic Model (Section 5) with IVs on the left and a single DV on the right. This type of RD also fits nicely into TABLE (Section 7).
  2. Evaluation – this type of RD seeks to evaluateor determine if there is a need and or support for a policy, program, process or approach. Often Best Practices (BP)are used to compare what exists to what is considered Best Practice. In cases where you use a BP approach, you may end up developing hypotheses based on various elements of what is included in BPs. Where needs or support are involved, your RD may end up looking similar to a Traditional RD.
  3. If you are examining more than one population – say regular staff and managers - you have two populations, and may need two sets of hypotheses, models, tables, surveys, etc.

Section 1. Problem Statement and background to the problem – 1 to 3 pages.

Section 2. Literature Review (LR). Just show me the short versions (summaries) of the major research you will use. I do not need the full length version, save that for your final paper (2 to 3 pages). Most important: summarize the research reviewed; tell me which research will guide your project with specifics; and then introduce your own research project (1 to 3 pages).Combined total: 6 pages max.

Section 3. Hypotheses. Whenever possible they should be short and with only one independent (IV) and dependent variable (DV). Use the same words every time you mention your DV. Sometimes there is a need from more, but not that often. I will talk about how you structure an evaluation and how it differs from RDs with a single dependent variable and a traditional model. (1 page; Maybe 2 pages with more than one set of hyps)

Section 4. Variables, Data and Surveys. This is the most overlooked section, even though it is one of the most important. The use of headershere helps me. Make sure you have addressed all the items in the Section 4 Checklist. If I don’t have a good understanding of what your variables are and how they are measured, it really limits what I can do. (1 to 2 pages)

Section 5. The Model. This is graphic representation of your research. Whether you need one depends on what type of research you are carrying out. (1 page; 2 pages with two models)

Section 6 . The Survey. Some RDs use semi-structured survey, others structured and some both types.

Section 7. The TABLE. This shows you and me that what you have in your hypotheses, variables, survey and model are all in sync. (1 page; 2 pages if you need a second TABLE)

Section 8. Sampling. Follow the checklist for this section. You may want to do a census. I don’t need a long discussion about sampling here. Please just answer the items on the checklist (1 – 2 pages).

Section 9. Methodology and Data collection. Many students do a Case Study; some use the Statistical Method with a Case Study; and a few do Quasi-experimental designs. (1 to 2 pages)

Sections 10. Data Analysis. Shows you the options you will have in analyzing your quantitative data will be determined by the level of measurement of your variables. Text data (words) must also be analyzed rigorously. (1 to 2 pages)

Sections 11& 12 are important to think about now. Sections 13 & 14 can wait.

Section 1. The Problem Statement. The first step is to clearly state the problem or objective of your study. Discuss the problem and provide some background statistics and/or history with sources (Citations Please!). Sources can include newspapers, studies, surveys, previous research, books, company documents or reports, the internet, etc. A general discussion comes first, followed by one or more (not too many) research questions aimed at addressing the problem in a meaningful way.

Section 2. Literature Review. It is always important to place your intended research within existing research. With a little effort you will be able to find scholars and practitionerswho have written about and/or examined the same or similar topic, problem or research question.Theory from previous research on most topics is not impossible to find - go see the Reference Librarian. While you will need to spend some serious time completing your LR, you can often get started using Google. Do not re-invent the wheel.

  • In this section, you review the literature, theory and research associated with your topic; pick the literature that will guide your own work; and briefly introduce your research project.Think of it as a funnel; from a broad discussion narrowed down to your topic and setting.Few research questions have not already been asked.
  • Theory Defined - a set of interrelated propositions that suggest why events occur in the manner in which they do. Think of theory as a conversation you want to join and add something to. This is a conversation among social scientists or practitioners who share an interest in your topic.

Literature Review (LR) Checklist

  1. Keep things consistent and in context is critically important. Make sure you use the LR in the rest of your research design - Sections 3&4&6 – Hypotheses, Variables and Sampling
  1. The Last Section of the LR:Please use the three headers (in bold) below.Depending on the amount of literature, this section should be between 1 and 3 pages.
  2. Summary of all Literature Reviewed: Synthesize what you have introduced and explain the relevance to your own research.
  3. Literature that Guides this Research: Narrow the LR down andtell the reader which literature/researchwill guide your research. Provide specifics about the authors, years, main points, variables, hypotheses, etc.
  4. This Research:Introduce your research project. Tell the reader a about where it will take place, who will be involved.Restate your major research question and goals.

Section 3. Hypotheses. Now that you have placed your work in a theoretical context, you need to develop some clearly stated hypotheses.

  • Hypothesis Defined. A hypothesis is used to organize a study; a hypothesis proposes a relationship between two or more variables. Start building your hypotheses with just two variables – an independent (IV) and dependent (DV) variable. Keep it Simple Please (KP). See examples on Page 13. Cite the theory or previous research like this: (Mease, 2013, p. 1)
  • While many RDs have a single dependent variable and a number of hypothesized independent variables, some RDs don’t. Sometimes, the evaluation of a program, policy or practice is the focus of the research and may have several dependent variables. In this case, there may be several hypotheses linked to best practices.Here are some examples of approaches used in evaluations: comparing the policy/program to established best practices; investigating the level of support for or need for a program/policy (a needs assessment.)

Hypotheses Checklist

  1. Hypotheses should be linked to theory or an existing body of knowledge, and be testable. It is OK to develop your own hypotheses, especially in the beginning.
  2. Whenever you use information from another source, you must cite it: (author/source, year, page#). Please cite the page number/sin all citations. Please, no footnotes.
  3. A hypothesis is always statement, never a question!
  4. It must clearly identify both an independent and dependent variable and make a prediction about their relationship, as shown in the example above. The independent variable is mentioned first, and then its predicted impact on the dependent variable
  5. Keep your language simple and similar in each hypothesis. This is not a time for using different words to describe your DV. Keep it boring and repetitive
  6. Very Important If you have more than one population (group fo people to interview) , you will likely multiple sets of hypotheses,models, surveys and sampling strategies. Think about this now!

Section 4. Variables and Data. This is a long and very important section. Read it carefully and take your time. Be sure to double-check the Section Checklist at the end of this section. Please use the headersbelow: these are the only areas I need to see in this section. No tables please.

  • Variables – type, definition, level of measurement, significance to the study
  • Data Type/s - short explanation

Variables- A variable is something that takes on different values that can be measured or counted. Variables represent the operationalized equivalent of the concepts in your hypotheses.

  • Independent variable (IV) causes or is associated with a change in the dependent variable.
  • Dependent variable (DV) – sometimes the toughest to measure. It is usually the focus of your research - the process or behavior you are trying to explain. The best way to start is with one dependent variable. Sometimes the DV is made up of several related elements.

Levels of Measurement. Always measure at the Highest Level Possible

  • Nominal–Yes/No; True False and other categories, such as male/female, religion, race,marital status, occupation, location, etc. Lowest level of measurement
  • Ordinal - the attributes can be rank-ordered, but the distances between attributes do not have a precise meaning. For example, the distance between Agree and Strongly agree in a Likert scale may not be the same for everyone. The same can be said for other ordinal scales.
  • Interval - the distance between attributes has meaning – for instance,distance can be measure in miles and the distance between each mile year is known and equal.
  • Ratio - there is always an absolute zero that is meaningful. This means that you can constructa meaningful fraction (or ratio) with a ratio variable. Distance traveled, which starts at 0 is a good example of a ratio variable. Highest level of measurement.
  • Measurement Hierarchy–It is always best to use the highest level of measurement possible. Example: Think of using a scale (ordinal) rather than a YES/NO question (nominal).The lowest level is Nominal, highest is Ratio. As you move up, the current level includes all of the qualities of the one below it and adds something new. Generally you can move from a higher level of measurement to a lower one, but not the other way around. Example - you can recode age measured intervally in years into ordinal categories: 18-25; 26-40; and over 40.

What is an Indicator?

  • An indicator is a device for providing specific information on the state or condition of something. An indicator is also a measure, gauge, barometer, index, sign, signal, standard, touchstone, yardstick, benchmark, criterion and point of reference. Source: Oxford Dictionary
  • An indicator can be numerical or text based (qualitative). It measures the quality of life in a country, such as governance, democracy or human rights. Indicators can be used to illustrate the progress of a country or an organization in meeting a range of economic, social, political or environmental goals.

Types of Indicators

  • Objective indicators can be developed from archival or secondary data sources. Some de jure examples include the existence of an integrity commission, existence of a law protecting human rights, freedom of speech or some other civil liberty. De facto (practice) examples include the number of corruption cases prosecuted or the number of human rights violations reported.
  • Reported behavior or events-based indicators are often used in surveys, but can also be developed using data from other sources. These indicators can come from sources such as government statistics, company documents, etc. Example: A citizen survey might ask respondents if they have ever been asked to pay a bribe to a public official (de facto).
  • Subjective or Perception based indicators are often found in surveys of typical citizens or smaller surveys of workers or key stakeholders (experts). These indicators rely on opinions or perceptions of how things are (de facto). Even though people often say one thing and do another, perception based data have proven to be very reliable over the years in many different contexts and cultures. Moreover, in many cases the only data available are based on people’s perceptions.
  • Proxy indicators measure the subject of interest indirectly, rather than directly. Sometimes issues of time and money influence the need to use proxy indicators. Other times, researchers use proxies for subjects that are difficult to measure directly such as income in developing countries, so researchers often use proxies, such as the roofing material of the house, ownership of livestock, ownership of land, etc.

The De facto and De Jure States (not critical, but important to consider)

  • The de facto state refers to what happens in practice. An example here you might want to know - the degree to which there is freedom of the press in a country
  • The de jure state refers to the existence of formal rules. These formal written rules are often found in policies, laws, regulations andconstitutions. For instance, is there a law or constitutional provision protecting freedom of the press?
  • A well designed project examines both the de jure and de facto states. This is because sometimes rights or laws may exist on paper (de jure), but not in practice (de facto).

Using Multiple Indicators to Measure a Concept: An index or a scale?

Index:An index is a numerical measure that allows the user to compare results overtime.A well-known scale is the UNDP Human Development Index (HDI). The HDI is a combination of many different indicators from different sources, such as GDP, literacy rates, educational enrollment, life expectancy, number of telephone lines, etc. Weighting of the individual indicators is usually required and best left to experts. It is important to consider the sources and quality of information used when interpreting an existing index.

Scales:A scale is a set of numerical values assigned to subjects, objects, attitudes or behaviors for the purpose of quantifying the qualities. Scales measure the degree to which an individual or object possesses the characteristic of interest. Scales usually rely on original data, but it is possible to use secondary data to develop a scale. Scales are easier to understand and construct than an index. Example: a job satisfaction scale. There are many tried and tested scales available. Get started with a simple Google search.

Understanding the Quality of Indicators

  • Reliability:Can the results you get be reproduced with a similar sample? Even if the data and indicators are not survey based, reliability is still important. For instance, in a desk study one would expect to get similar results from a different expert examining the same information.
  • Validity:Does the indicator measure what it is supposed to measure? Sometimes, we think (assume) an indicator is measures a construct accurately, but we find out later (not good) it does so poorly or is measuring something completely different. Validity issues can arise in both survey and non-survey-based indicators. Like reliability, there are tests available for validity.

Types of Data - Regardless of which type or types of data you choose to collect, you cannot escape developing a complete RD. Sources of data include- surveys, administrative data, documents, laws, policies, manifestos, national statistics, newspapers, etc.