Guide to Literature study
By Bo Nørregaard Jørgensen, MMMI)
(! Please note that the guide is meant for inspirational purposes and is subject to ongoing revision)
Purpose
The purpose of a literature study is to analyze critically a segment of a published body of knowledge through summary, classification, and comparison of prior research studies, reviews of literature, and theoretical articles.
Elements of the literature study include:
1. Literature search
2. Detailed review of selected research papers
3. Writing up state-of-the-art
4. Putting your own research into perspective of state-of-the-art
ECTS points
The literature study as described here is scheduled as 6 ECTS points (I.e., 150 work hours).
Getting started
Do a comprehensive literature search based on topics relating to your thesis subject. Search on keywords etc. The final outcome of your literature search should be roughly 15 research papers (corresponding to approximately 300 pages.) You should only include papers that are essential to your field of research in your literature study report. That is, you have to discard all papers that are not relevant to your research subject as part of the search process. During the search process, you should expect to read the abstract of at least 3-4 times as many papers as you will include in your final literature report. You should schedule a least 20 hours to do the literature search and initial skimming of papers for relevance. The literature study report is expected to contain a detailed review for each of the selected papers; that is, you have to review approximately 15 research papers. You should schedule 7 hours for each research paper. This leaves approx. 25 hours for write the state of the art section based on the selected research papers.
How to skim a research paper
When you are going to find relevant candidate papers for your literature study you will browse through a large amount of papers. Not all papers are relevant, so how do you quickly read a paper to decide whether it is relevant? Here are some steps that will help you to survive:
· Read the title. (What is the paper about?)
· Read the abstract. (Should give you a concise overview of the paper.)
· Read the introduction. (Look for motivations, relation to other work, and a more detailed overview.)
· Look at the structure of the paper. (What do the remaining sections address? How do they fit together?)
· (Read the previous/related work section. (How does this work relate? What is new or different about this work?))
· Read the conclusions. (What were their results?)
· Read the body of the paper. You may want to skip over all the equations the first time through.
As soon as you find the paper irrelevant to your research subject you simply skip the rest of the steps and continue with the next paper in your stack of papers.
The references won’t mean much to you if you’re not familiar with the area. Sometimes important parts of the work may be contained in the references, particularly in conference papers since space is limited. The references are very important when you are researching a topic—they point you to related research as well as the research upon which the current paper builds upon.
Reviewing a research paper
When you have to review the work of others as part of your literature study you will have to address many of the same elements as you do when writing an abstract about your own work. That is, do the paper’s authors tell you what god researchers are expected to tell their readers? You should fill out the following form for each paper that is part of your literature study.
Title of the paperName(s) of author / authors / What’s the name of the dude?
Motivation for doing the research. Claimed relevance or benefits! / What is the motivation of the research work? Is there a crisis in the research field that the paper attempts to resolve? Is the research work attempting to overcome the weaknesses of existing approaches? Is an existing research paradigm challenged? In short, what is the niche of the paper?
The research problem / objective / What is the argument of the paper?
Is this research original? / What is new in this paper? A new question is asked? A new understanding of the research problem? A new methodology for solving problems? A new algorithm? A new breed of software tools or systems? A new experimental method? A new proof technique? A new formalism or notation? A new evidence to substantiate or disprove a previously published claim? A new research area? In short, what is original about this paper?
Chosen research approach / methodology. / What have they done to investigate / find a solution to the research problem? What is the methodology adopted to substantiate the claims? What are the major theorems? What experiments are conducted? Data analyses? Simulations? Benchmarks? User studies? Case studies? Examples? In short, what makes the claims scientific (as opposed to being mere opinions)?
Reported results / products / effects. / What is the outcome of their research? What did they find or create?
What are the contributions / conclusions? / What do they claim to have achieved? Summaries their contributions in a clear and concise fashion!
Your evaluation of the reported results with respect to the research objective. / 1. Are the results significant? Is the work scratching minor itches? Are the authors solving artificial problems (aka strawman)? Does the work enable practical applications, deepen understanding, or explore new design space?
2. Are the contributions significant? Are the authors simply repeating the state of the art? Are there real surprises? Are the authors aware of the relation of their work to existing literature? Is the paper addressing a well-known open problem?
3. Are the claims valid? Have the authors been cutting corners (intentionally or unintentionally)? Has the right theorem been proven? Errors in proofs? Problematic experimental setup? Confounding factors? Unrealistic, artificial benchmarks? Comparing apples and oranges? Methodological misunderstanding? Do the numbers add up? Are the generalizations valid? Are the claims modest enough? That is does the paper contain evidence for the claims?
Your conclusion on the relevance of this research with respect to the field in general. / What have we learned from the paper? Is this a key paper? Do the reported results contribute significantly to state of the art? Shall the standard practice of the field be changed as a result of the new findings? Is the result generalizable? Can the result be applied to other areas of the field? What are the open problems? In short, what are the lessons one can learn from the paper?
Your conclusion on the relevance of this research with respect to your own research. / How does this work related to yours? Are there common elements? Are they addressing the same research problem? Can you use their results to solve your problem, or part of it? Can you improve on their approach? Do their ideas have other applications or extensions that the authors might not have thought of?
Review a research paper implies careful reading. That is, when you read a research paper, your goal is to understand the scientific contributions the authors are making. This is not an easy task. It may require going over the paper several times. Expect to spend several hours to read a paper.
Here are some initial guidelines for how to read a research paper carefully:
Read critically: Reading a research paper must be a critical process. You should not assume that the authors are always correct. Instead, be suspicious. Critical reading involves asking appropriate questions. If the authors attempt to solve a problem, are they solving the right problem? Are there simple solutions the authors do not seem to have considered? What are the limitations of the solution (including limitations the authors might not have noticed or clearly admitted)? Are the assumptions the authors make reasonable? Is the logic of the paper clear and justifiable, given the assumptions, or is there a flaw in the reasoning?
If the authors present data, did they gather the right data to substantiate their argument, and did they appear to gather it in the correct manner? Did they interpret the data in a reasonable manner? Would other data be more compelling?
Read creatively: Reading a paper critically is easy, in that it is always easier to tear something down than to build it up. Reading creatively involves harder, more positive thinking. What are the good ideas in this paper? Do these ideas have other applications or extensions that the authors might not have thought of? Can they be generalized further? Are there possible improvements that might make important practical differences? If you were going to start doing research from this paper, what would be the next thing you would do?
Make notes as you read the paper: Many people cover the margins of their copies of papers with notes. Use whatever style you prefer. If you have questions or criticisms, write them down so you do not forget them. Underline key points the authors make. Mark the data that is most important or that appears questionable. Such efforts help the first time you read a paper and pay big dividends when you have to re-read a paper after several months.
After the first read-through, try to summarize the paper in one or two sentences. Almost all good research papers try to provide an answer a specific question. (Sometimes the question is a natural one that people specifically set out to answer; sometimes a good idea just ends up answering a worthwhile question.) If you can succinctly describe a paper, you have probably recognized the question the authors started with with and the answer they provide. Once you have focused on the main idea, you can go back and try to outline the paper to gain insight into more specific details. Indeed, if summarizing the paper in one or two sentences is easy, go back and try to deepen your outline by summarizing the three or four most important subpoints of the main idea.
If possible, compare the paper to other works. Summarizing the paper is one way to try to determine the scientific contribution of a paper. But to really guage the scientific merit, you must compare the paper to other works in the area. Are the ideas really novel, or have they appeared before? (Of course we do not expect you to be experts and know the areas ahead of time in this class!) It is worth mentioning that scientific contributions can take on many forms. Some papers offer new ideas; others implement ideas, and show how they work; others bring previous ideas together and unite them under a novel framework. Knowing other work in the area can help you to determine which sort of contribution a paper is actually making.
Sooner or later, you will come across something that you don’t understand. What can you do? You should try to figure out what it is and how it is being used (even though you still don’t understand it). For further reading, see the references!
Writing state of the art
This section reviews the state of the art and related work relevant to your research. Here you list the current knowledge in the field of your research and the respective references. You base this review on your collection of paper reviews.
The idea is to present (critical analysis comes a little bit later) the major ideas in the state of the art right up to, but not including, your own personal brilliant ideas. You organize this section by ideas, and not by author or by publication. For example, if there have been three important main approaches to Software Evolution Analysis to date, you might organize subsections around these three approaches.
In the first paragraph of the section you have to outline the contribution up to what is considered to be the present state of the art. In the next paragraph you describe state of the art. The final paragraph summaries gaps in research, e.g., open questions, unresolved problems or new perspective of immediate interest in the field, some of which should relate to your own research question.
Composing the literature study report
You should organize our report as follows:
Preface
Describe the context for the literature report.
Introduction
Here you introduce the reader to your field of research. The idea is to give the reader a birds- eye view on the subject of your PhD work before you zoom in on the particular topics that you are about to investigate.That is, define or identify the general topic, issue, or area of concern, thus providing an appropriate context for reviewing the literature.
State-of-the-art
If your literature study includes more than one research field you will have to create a subsection for each field of research. You write this section after you have written all of the literature reviews!
Literature reviews
Create a subsection for each paper.
References
This section contains references for all the papers you have reviewed in the section ‘Literature reviews’ . You are not to include all the papers that you have discard during the literature search process. However, you should maintain an additional list over all the papers that seemed interesting, but did not make it to the literature review. This is important, as you will properly encounter these papers again in later literature searches.
2 State of the art (or "Literature review" or else):•
”State-of-the-art” afsnit er bygget op således: Først en kort opsumering af de vigtigst historiske bidrag fra begyndelse op til i dag, der efterfølger en kort gennemgang af hvad der er de bedste løsninger, dvs. state-of-the-art i dag til at løse de problemer som der er beskrevet i afsnittet ”introduction / el. Motivation”. Så kommer en kort analyse af hvorfor disse ’fantanstiske’ løsninger ikke er nok – manglerne ved dem identificeres. Endelig beskrives hvordan vi (artiklens forfattere) vil råde bod på disse mangler og derved lukke det hul i forskningen som vi lige har identificeret i vores analyse af state-of-the-art.