Research Design Self Assessment

Recently a group of SCS faculty met to identify the properties of high quality research and research communication we wanted to impart to our students as an essential and central part of their career preparation. As a result of that meeting and surrounding reflection and discussions, we identified four core values that are prized within our research community, including selecting research problems that are important and likely to have significant impact, using creativity in formulating solutions, evaluating work with stringent methodology, and producing results that are able to be replicated and achieve a significant improvement over the state-of-the-art. From that list, we formulated this self-assessment document.

I. Importance of the Research Problem

  1. Can you explain clearly who your audience is? Do you know what area of language technologies this problem fits into?
  2. Who are the key players in this area? If one of them asked you why he or she should be interested in your work, do you think you could convince them?
  3. If someone asked you, why does the world need a solution to this problem, what would you say?

II. Creativity and rigor of the Solution

  1. Can you explain clearly what is different about your approach than other approaches that have been taken to solve this problem in the past?
  2. Can you explain clearly what new insight(s) you have brought to bear on this problem?
  3. Is your solution robust enough to be of practical use? In what contexts could it be used?

III. Rigor of the evaluation

  1. Can you formally state what hypothesis you were testing?
  2. Can you provide a rationale for the design of your evaluation in terms of internal validity, external validity, and face validity?
  3. Did you use the evaluation metrics that are standard for your research area?
  4. Did you use a publically available data set, if there was an appropriate one available? And did you ensure that your gold standard was reliable and valid?
  5. Have you evaluated the generality of your result?
  6. Have you done an error analysis? What new insights into the problem did you gain from it? And what new directions does it suggest for ongoing work?

IV. Significance of the Results

  1. Did you evaluate your solution in comparison to the best baseline you could find from previously published work related to this problem?
  2. Did you evaluate the statistical significance of the difference between your performance and the baseline? What was the effect size?
  3. To what extent does the result of your evaluation answer the research question you started out with? And what new questions does it raise?