Like Primary Research Projects, Source Evaluation Requires a Mixture of Methods in Order
Evans 1
Laurie Evans
Dr. Sherry Southard
English 7730 (Project 2B)
October 29, 2007
The Search for Credibility
Like primary research, source evaluation requires a variety of methods to achieve a comprehensive summation of validity and accuracy. Much of my prior knowledge related to determining source credibility was based on topic relevance rather than author reputation and publication history. This assignment, involving determining the credibility of a website and an article, broadened my scope of the various elements that require scrutiny in order to estimate a source’s integrity. Verifying source credibility requires the same level of investigative skill needed to conduct successful literature searches for the scholarly examination of a topic. Often, the researcher must think tangentially in order to locate information to identify an author’s previous publications and areas of study. This investigation is not always as simple as conducting an author search on the Internet because many scholarly activities, such as presentations and computer-based publications, are not identified through traditional search strategies. In reflection, after critically assessing Kairos and "Teaching Technical Communication in an Era of Distributed Work: A Case Study of Collaboration Between U.S. and Swedish Students," by Marie C. Paretti, Lisa McNair, and Lissa Holloway-Attaway, in Technical Communication Quarterly, I became aware of several concerns and considerations to keep in mind as I prepare for future activities that will require searching and critically evaluating secondary, existing literature. These issues certainly reflect the naivety of a beginning graduate student and punctuate the importance of self-reflection to frame, organize and synthesize future secondary research endeavors.
Frustrations
Underlying all of my work to evaluate the scholarly merit of those two sources is recognition that part of the evaluation process is quantitative (quality—typos, link accuracy, misspellings; and currency) and other aspects of evaluation, such as depth and author credibility, are qualitative in nature. For example, a search for the academic rank of Marie C. Paretti revealed her status as an assistant professor at Virginia Tech; however, this fact has the potential to be misleading because she has held this rank since 1997. That length of time in rank represents a long pause at that stage of her career, but several plausible justifications exist for this timing: for example, has her administrative appointment in the Virginia Tech Engineering Communications Center substantially increased her knowledge of the praxis of educating technical-oriented students? Has the fact that she is doctorally prepared in a humanities field negatively impacted her promotion in a technically-oriented academic discipline (engineering)? One can only surmise that an English PhD is underrated in a technical program such as engineering at a premiere technical institute. So, is rank a completely definitive measure of Dr. Paretti’s eminence in the pedagogy of communication within technical academic programs? Consideration of similar ironic questions will certainly permeate my future source evaluations, require me to give pause to look beyond the traditional classifications of assistant professor, associate professor and professor and spur me to question apparent gaps of time between academic promotions.
Another frustration encountered during these evaluations was the occasional inability to locate information about an author or editor. This problem typically occurred in attempting to verify the credibility of Kairos staff because many section editors are graduate students lacking publication beyond a single poster presentation. This information was often difficult to assemble because universities rarely list graduate students on websites. While Kairos identifies the home institutions of the publication’s staff, rank and position are not delineated on the journal’s website. Using university websites to verify affiliation of graduate students usually turned up inconclusive documentation and created scenarios where individuals may have (1) departed the university or (2) simply be a graduate student and, thereby, not listed on departmental websites. The concern with such conclusions is the lack of definitive facts and worry of making inaccurate assumptions regarding professional status.
A consequence of this search experience related to my future scholarly activity is the creation of a “people” search process that will be replicable when searching for faculty, graduate students, and other professionals. My search method evolved to follow a pattern with alternate beginnings and endings based on the information presented at the search outset and information discovered throughout the investigation process. See the following flow chart.
Finally, this activity challenged my developing skills in the area of research evaluation. My primary research comfort zone is quite narrow and includes only basic knowledge related to research method and statistical terminology. Generally, my skill set is limited to identifying whether a study uses quantitative or qualitative methods; however, after reviewing the class materials for research methods, I gained slightly more confidence in my ability to recognize patterns, statistical implications and controls that are used in study design and interpretation. In approaching this type of evaluation in the future, I will need to develop a schema such as the one above to guide me through the intricacies of statistical measurement.
Lessons Learned
Chief among the lessons learned through this assignment are the necessity of dogmatic pursuit in order to locate information needed to verify source credibility and the confidence required to assert an opinion on the usefulness of this information. At several points in the source verification process, I encountered roadblocks or, seemingly, dead ends which stop my research and caused me to rethink the approach I was using to locate information. In these circumstances, stepping away from the search for information related to the critical assessment for a bit was useful to clarify perspective and think through the resources used thus far. In almost every situation, subsequent searches yielded the information I was seeking. The source verification aspect of this project reinforced the need to be determined and unrelenting in seeking details about authors and media through diligence and downright nosiness. Discovering Dr. Cheryl Ball’s electronic tenure portfolio yielded valuable information about the acceptance rate of Kairos and the underlying mission of the journal’s editors to increase the perceived respect and scholarly value of the online journal. Certainly, both of these details figured into my evaluation of the online journal’s content, validity and credibility.
Evaluating the scholarly merit of media requires the reviewer to form an opinion, articulate that opinion and be willing to defend it. My experience with this project indicates that I can assert an opinion more easily if I have thoroughly investigated a topic (media or author). The old marketing adage that “I can sell or talk about anything as long as I have a strong working knowledge of the product” is relevant for the evaluation portion of a source analysis. My hunch is that students performing scant searches to verify credibility will find it proportionally difficult to state an opinion as to the source’s merit. Further, in order for me to render a definitive opinion about a source or author, I have to be confident that I have exhausted all avenues of possible related information. As a beginning graduate student, it is difficult (even uncomfortable) to state an opinion due to fear of casting a judgment that dissents from classmates or, worse, a professor—a reluctance characteristic of my classmates as well; I strive to increase my research abilities to be able to satisfactorily critically evaluate existing, secondary research literature.
Conclusions
Not surprising, my two most significant take-away bits of wisdom discovered in the search and evaluation activities are both related to work organization. Most of my school work is completed in brief stints of evening time or extended stretches of time on the weekends which require frequent stops and starts as well as various shifting study environments. Several times during this project, I stole a few minutes between other tasks to research an author or publication. While my searches were typically successful, my fleeting study sessions prevented me from accurately keeping track of where I found information. This situation was problematic when preparing the narrative text to describe my search processes because I had details in my head, but I did not have a blueprint of the websites visited during the research stage of the project. For future projects, I must keep a running chronological list of websites/sources reviewed during the completion of an activity. This list will be most useful and accurate if it resides on a thumb drive that I keep with me in all of my work/study environments.
Second, I must work to stay focused on the goal of my research when reviewing sources and author credentials. I am developing a time draining habit of becoming engrossed in an author’s “story” and surfing from link to link in order to satisfy my curiosity to answer all of the adverbial questions related to personnel, cause, time, place and process. Often, I lose sight of the original research purpose and waste valuable time that was necessarily allocated to other parts of a project. The same tendency also occurs when I am reviewing a journal whereby I scan the table of contents to discern a sense of the scope of topics covered in an issue, and, inevitably, find myself reading several articles out of curiosity. One solution may be to develop a flagging system where I note items that catch my eye that I want to return to read at a later date. The amount of time expended on this type of “wandering” or “sidebar” reading is significant and worthy of consideration of a way to stay focused on research objectives at hand.
As I head into the annotated bibliography and literature review exercises for the course, I am confident that I have the skills to complete the secondary research, but I am anxious about my ability to analyze and disseminate findings for these upcoming exercises. Much of my plan of action will revolve around using multiple strategies to perform searches while staying focused and keeping an open mind during the critical review of these materials. Finally, I must shed any reticence with regard to expressing opinions as to the merit or credibility of a source or author so that I can make informed decisions about the inclusion or exclusion of items in a fashion that allows me the latitude needed to critically evaluate and justify my assessments. My search for credible sources continues as I begin the next phase of this course.