BSAD 6311 Experimental Design

Spring 2010

Instructor: Dr. Kenneth H. Price

Office: 226 Business Bldg.

Hours: Wednesday 9:00-12:00 (schedule) or by appointment

Phone: 817- 273-3863 (Office)

817-451-2693 (Home)

E-Mail:

Webpage:

Course Objectives

The objectives of this course are to provide the theoreticalfoundation that will enable students to (a) evaluate research that appears in the literature, (b) discuss research strategies with their colleagues, and (c) develop a set of research skills that they can apply to conducting studies. To facilitate these learning objectives, students will be asked to complete weekly assignments to demonstrate their ability to explain, comprehend, and analyze readings and lecture material. To facilitate synthesis of the material, students will complete a semester-long independent research project. The distal learning objective is for the student to apply these skills to the conduct of dissertation research and as future scholars in their disciplines. Doing research involves a willingness to consider and reconsider central issues (persistence, persistence, persistence), a desire to look for patterns and relationships, a measure of creativity, a genuine joy from generating new information, and a lot of trial-and-error learning.

My role is to help and guide you through the research process. My door is always open. Come in or call if you have any questions. The desire to work with Ph.D. students is a major reason why many faculty members are at UTA. Take advantage of this.

Learning Objectives

  1. Students will be able to identify and discuss the critical elements of research design.
  2. Students will be able to identify and discuss critical ethical issues in the conduct of research with human participants.
  3. Students will be able to identify and discuss critical factors that confound experimental and survey research.
  4. Students will be able to identify a theoretical framework, describe hypotheses that need to be examined, and execute a research design to examine these questions.

Format of the Class

Most class periods will involve a lecture portion and a class discussion portion. The class discussion portion will be based on discussion of class readings and presentation of class assignments.

Assignments

There will be assignments throughout the semester based on the lectures and readings. These assignments are designed to clarify specific issues based on the material covered in class, to move you forward on your research proposal, which is discussed in the next section, and to prepare you for the comprehensive examination based on the courses that you take in the research field. These assignments are due on the day the class meets or no points will be awarded.

Research Project

A major focus of this class is to assist you with developing a formal research proposal. For many in class, this will be the first try at designing a research project from the starting point (theoretical foundation) to operationalization of the study (methods). This project, for some, may evolve into a research study or may provide a step along the way towards a dissertation. In this project, you will go through the same steps that you will use for your dissertation, with the exception of the data analysis and discussion sections.

Your topic should be from your major field, of interest to you, and of sufficient importance to people in your field to justify spending your time doing the research. Models to use are research articles from your major field that are not "just" review articles but empirical articles that collect and analyze data. That said, research is in many ways incremental, so you don't have to find a cure for incurable diseases your first go-around. Save it for the second project.The research project does not have to be an experiment. It can be an empirical study where you used archival data and/or collected data through self-report measures. However, all projects must use a strategy consistent with the course material.

The proposal should contain the following sections and follow the outline of the reading by Schroeder on Research Reports. The best was to sample material in each section is to read articles in your discipline. You mimic the framework they use in your paper. You must also read the top academic journals in your field. I will provide a list of the journals that you department suggests should be used in evaluating the research efforts of their faculty. Sections Include:

  1. Literature Review –A comprehensive review of the literature leads up to the specific research questions that you are examining. You need to lead us from what we know to what we need to know. What questions are not answered by these studies and need to be answered?Even when you are charting a new direction, you build your theory section based on theoretical models and on related research, although the context for theoretical foundations and related research may be related disciplines. For example, in looking at team programming, there is little literature in information systems, but you can look at the team problem-solving literature in the behavioral science to build a case for using teams in programming.
  1. Hypotheses Derivation and Hypotheses Statements – The theoretical justification for the hypotheses must be clearly presented. In doing this, you must illustrate the model that you are testing and the specific hypotheses that you are examining. Clear directional statements of the hypotheses follow the theoretical justification.
  1. Methods Section– Thisis a methods course, so this section must include all of the details about how you are going to conduct your research. This covers everything from subjects to dependent measures. The level of detail should allow others to know exactly what you did in order to replicate your research. The methods section includes a number of subsections such as Subjects,Design, Procedures, Measures, and Preliminary Analyses. What is described in each section can vary according to the specific type of study that you are conducting.
  1. Appendix –Please include all materials, such as consent forms, measures (scales), instructions, debriefing sheets, and instructions to the participants. The appendix should contain acompletedUTA IRB prospectus. The IRB prospectus should not be submitted but instead be completed as if you were to submit it for approval by the University.The text of the proposal excluding the appendix should not exceed 20 pages. The appendix can be any length as appropriate.

Preliminary Research Proposals

At minimum, a three-page, single-spaceddescription of your proposal is due on March2. This proposal should briefly describe the theoretical framework for your study, the hypotheses, and the methods you intend to use in this research. This will allow me to give you some early feedback on your project, to assist you with your project,and to try to avoid major problems which can be difficult to correct at a later time.In class, we will discuss proposals on March 23. Prepare a 1-2page description of your research for distribution to the other members of the class. This description must be distributed by March 22 at 9am (via e-mail) to the other class members. It should describe the overall research question, why it is important, the hypotheses, and how your proposed research adds to the literature. We will have to work out how we will proceed with a class discussion of the proposals on March 23.

Project Presentations

Each student will present his or her completed research project to the other members of the class and invited faculty at the end of the semester. This is an extensive research manuscript modeled again on one that appears in the Academy of Management Journal or Journal of Applied Psychology. Reference style and formatting is available at the journal websites under Instructions to Authors. Your research project (an executive summary) must be copied and distributed to the class on the day of your presentation, along with any slides that you use in your presentation.

At the presentation, one class member will be assigned to be a discussant. We will pair people up so that a discussant and presenter will not be performing both roles on the same day. That means your paper must go to your discussant at least three days before your presentation. They must read it to be able to provide appropriate comments. We will allocate times for the presenter and discussant in class based on enrollment.

Examinations

There will be one in-class examination that will be administered prior to the start of class presentations. The test will mirror the comprehensive examination you will take in the research field but it will only cover material from this course. It is likely that this essay examination will ask you to complete 4 questions within a 3-hour time frame. I will provide study questions prior to the exam to guide your preparation.

Class Grades

Assignments...... 20%

Participation…………….10%

Examination...... 35%

Paper...... 35%

Course Outline

WeekDateTopic

1January 19Orientation

2January 26Overview and Methods

3February 2Variables and Relationships in Research

4February 9Research Strategies

5February 16Internal Validity and Factorial Designs

6February 23Research Design: Quasi-Experimental Designs

7March 2Conducting Experiments(Proposals Due)

8March 9Conducting Experiments

9March 16Spring Break

10March23Proposal Discussions

11March 30Ethical Issues In Conducting Research

12April6Survey Research and Measurement Issue

13April13Special Issues

14April20Multilevel Theory and Publishing

15April 27Examination

16May 4Presentations

17May 11Presentations

Readings

WEEK 2. OVERVIEW, INTERPLAY OF THEORY, HYPOTHESES, AND METHODS

  1. Schroeder, D. A., Johnson, D. E., & Jensen, T. D. 1985. Reading research reports: A brief introduction.In D. A. Schroeder, D. E. Johnson, & T. D. Jensen (Eds.), Contemporary Readings in Social Psychology, 35-42. Nelson-Hall: Chicago.
  2. Daft, R. L.1984. Antecedents of significant and not so significant organizational research.In T. Bateman and G. Ferris (Eds.), Method and analysis in organizational research, 3-14. Reston, VA: Reston Publishing.
  3. Edmondson, A. C., & McManus, S. E. 2007. Methodological fit in management field research. Academy of Management Review, 32: 1155-1179.
  4. Alvesson, M., & Karreman, D. 2007. Constructing mystery: Empirical matters in theory development.Academy of Management Review, 32: 1265-1281.
  5. Buchanan, D.A. & Bryman, A. (2007). Contextualizing methods choice in organizational research. Organizational Research Method, 10, 483-501.

WEEK 3. Hypotheses and Variables in Research (Moderation and Mediation)

  1. McGuire, W. J. 1997. Creative hypothesis generating in psychology: Some useful heuristics.Annual Review of Psychology, 48: 1-30.
  2. Baron, R. M., & Kenny, D.A. 1986. The mediator-moderator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51: 1173-1182.
  3. Howell, J. P., Dorfman, P. W., & Kerr, S. 1986. Moderator variables in leadership research. Academy of Management Review, 11: 88-102.
  4. Johns, Gary. (2006). The essential impact of context on organizational behavior. Academy of Management Review, 31, 2, 386-408.
  5. Wood, R.E., Goodman, J.S. Beckman, N., & Cook, A. (2008). Mediation testing in management research: A review and proposals. Organizational Research Methods, 11, 270-295.

Reference

James, L. & Mulaik, and Brett, A tale of Two Methods. (2006). Organizational Research Methods, 9 (2), 233-244.

WEEK 4. RESEARCH STRATEGIES

  1. Sackett, P. R., & Larson, J.R., Jr.1990. Research strategies and tactics in industrial and organizational psychology research. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology: 2nded., 1: 419-489. Palo Alto, CA: Consulting Psychologists Press.
  2. Ilgen, D. R. 1986. Laboratory research: A question of when, not if. In E. A. Locke (Ed.), Generalizing from laboratory to field settings: Research findings from industrial-organizational psychology, organizational behavior, and human resource management, 257-267. Lexington, MA: Lexington Books.
  3. Campbell, J. P. 1986. Labs, fields, and straw issues. In E. A. Locke (Ed.), Generalizing fromlaboratory to field settings: Research findings from industrial-organizational psychology, organizational behavior, and human resource management, 269-279. Lexington, MA: Lexington Books.
  4. Highhouse, S. (2009). Designing experiments that generalize. Organizational Research methods, 12, 554-566.

WEEK 5. INTERNAL VALIDITY and FACTORIAL DESIGNS

  1. Cook, T. D.,& Campbell, D. T. 1979. Chapter 2, Validity. In Quasi-experimentation: Design and Analysis issues for field settings, 37-51. Boston: Houghton Mifflin.
  2. Trochim, W. M. K. 2005. Experimental design.Research methods: The concise knowledge base, 150-170. Atomic Dog Publishing.
  3. Folger, R., & Turillo, C. J.1999. Theorizing as the thickness of thin abstraction.Academy of Management Review, 24: 742-758.
  4. Price, K.H., Lavelle, J. J., Henley, A.B., Cocchiara, F.K., & Buchanan, F.R. 2006. Judging the fairness of voice-based participation across multiple and interrelated stages of decision making. Organizational Behavior and Human Decision Processes, 99: 212-226. (and handouts: IRB Forms 1 and 2, Response to the Editor)

WEEK 6. RESEARCH DESIGN

  1. Cook, T. D., Campbell, D. T., & Peracchio, L. 1990. Quasi experimentation. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology: 2nded., 1: 491-576. Palo Alto, CA: Consulting Psychologists Press.
  2. Truxillo, D. M., Bauer, T. N., Campion, M. A., & Pronto, M. E. 2002. Selection fairness information and applicant reactions: A longitudinal field study.Journal of Applied Psychology, 87: 1020-1031.
  3. Greenberg, J.2006. Losing sleep over organizational injustice: Attenuating insomniac reactions to underpayment inequity with supervisory training in interactional justice.Journal of Applied Psychology, 91: 58-69.
  4. Grant, A.M. & Wall, T.D. (2009). The neglected science and art of quasi-experimentation: Why-to, when-to, and how-to advice for organizational researchers. Organizational Research Methods, 12, 653-686.

WEEKS 7 and 8 CONDUCTING EXPERIMENTS and Paper People Experiments (Some Redundancy)

  1. Aronson, E., & Carlsmith, J. M. 1968.Chapter 9. Experimentation in social psychology.In G. Lindzey & E. Aronson (Eds.), The handbook of social psychology: 2nd ed., 2: 1–79. Reading, MA: Addison-Wesley.
  2. Aronson, E., Ellsworth, P. C., Carlsmith, J.M., & Gonzales, M. H. 1976. Chapter 5. The independent variable: Types of independent variables in social-psychological research; Methods of research in social psychology, 134-169. New York: McGraw-Hill.
  3. Aronson, E., Ellsworth, P. C., Carlsmith, J. M., & Gonzales, M. H. 1976. Chapter 6. The dependent variable.Methods of research in social psychology, 170-219. New York: McGraw-Hill.

Aronson, E., Ellsworth, P. C., Carlsmith, J. M., & Gonzales, M. H. 1976. Chapter 7. Setting the stage.Methods of research in social psychology, 220-233. New York: McGraw-Hill.

25a.Karren, R.J. & Woodward-Barringer, M. (2002). A review and analysis of the policy-capturing methodology in organizational research: Guidelines for research and practice. Organizational Research Methods, 5, 337-361.

WEEK 10. PROPOSAL DISCUSSIONS

WEEK 11. ETHICAL ISSUES IN CONDUCTING RESEARCH

  1. Miller, R. L. 2003. Ethical issues in psychological research with human participants. In S. F. Davis (Ed.), Handbook of research methods in experimental psychology, 8: 127-150. Malden, MA: Blackwell Publishing.
  2. Chapter 2: Informed Consent.
  3. Chapter 8: Debriefing.
  4. Lowman, R. L., Ed. 1998. Cases 18, 20, 23, 28, 29, 30, 35, 36, 39, 40. The ethical practice of psychology in organizations. Washington, DC: American Psychological Association.
  5. UTA IRB Policy Manual and IRB Prospectus
  6. Seashore, S. E. 1987. Surveys in organizations. In J. W. Lorsch (Ed.), Handbook of organizational behavior, 140-153. Englewood Cliffs, NJ: Prentice-Hall.

Assignment: Completion of on-line training, and Completion of IRB Form #1

CITI Training for Human Subjects:

(This can take several hours; I would suggest you not wait until the last minute. You can print off a copy of the certificate as proof of completion.)

Week 12.SURVEY RESEARCH AND MEASUREMENT ISSUES

  1. Crano, W. D., & Brewer, M. B. 2002. Chapter 8. Correlational design and causal analysis. In Principles and methods of social research. 125-144. New York: McGraw-Hill.
  2. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. 2003. Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88: 879-903.
  3. Crano, W. D., & Brewer, M. B. 2002. Chapter 3, Measuring concepts: reliability and validity. In Principles and methods of social research. 125-144. New York: McGraw-Hill.
  4. Trochim, W. M. K. 2005. Survey research. Research methods:The concise knowledge base, 76-99. Atomic Dog Publishing.
  5. . Becker, T.E. (2005). Potential problems in the statistical control of variables in organizational research: A qualitative analysis with recommendation. Organizational Research Methods, 8, 274-289.

Reference

Richardson, H.A., Simmering, M.J., & Sturman, M.C. (2009). A tale of three perspectives: Examining post hoc statistical techniques for detection and correction of common method variance. Organizational Research Methods, 12, 762-800.

Week 13. SPECIAL ISSUES

  1. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P.2004. Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires.American Psychologist, 59: 93-104.
  2. Kraut, R., Olson, J., Banaji, M., Bruckman, A., Cohen, J., & Couper, M.2004. Psychological research online: Report of board of scientific affairs’ advisory group on the conduct of research on the Internet.American Psychologist, 59: 105-117.
  3. Gelfand, M. J., Raver, J. L., & Ehrhart, K. H. 2002. Methodological issues in cross-cultural organizational research. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology, 11: 216-246. Malden, MA: Blackwell Publishing.
  4. Schaffer, B.S. & Riordan, C.M. (2003). A review of cross-cultural methodologies for organizational research: A best practice approach. Organizational Research Methods, 6, 169-215.
  5. Rogelberg, S.G. & Stanton, J.M. (2007). Introduction: Understanding and dealing with organizational survey non-response. Organizational Research Methods, 10, 195-209.

WEEK 14. MULTILEVEL THEORY AND PUBLISHING

Klein, K.J. & Kozlowski, W.J. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3, 211-236.van Mierlo, H., Vermunt, J.K., Rutte, C.G. (2009). Composing group-level constructs from individual-level survey data. Organizational Research Methods, 12,368-392.

  1. Linden, R. C., Wayne, S. J., Jaworski, R.A., & Bennett, N. 2004. Social loafing: A field investigation. Journal of Management, 30: 285-304.
  2. Hofmann, D. A. 2002. Issues in multilevel research: Theory, development, measurement, and analysis. In S. G. Rogelberg (Ed.), Handbook of research methods in industrial and organizational psychology, 11: 247-274. Malden, MA: Blackwell Publishing.
  3. Agarwal, R. 2006. Reap rewards: Maximizing benefits from reviewer comments. Academy of Management Journal, 49: 191–196.
  4. Harrison, D. 2002. Obligations and obfuscations in the review process. Academy of Management Journal, 46: 1079–1084.
  5. Jauch, L. R., & Wall, J. L. (1989). What they do when they get your manuscript: A survey of Academy of Management reviewer practices. Academy of Management Journal, 32, 157-173.

Drop Date: The last date to drop a class is April 2.

Americans with Disabilities Act (ADA): The University of Texas at Arlington is on record as being committed to both the spirit and letter of federal equal opportunity legislation; reference Public Law 92-112 - The Rehabilitation Act of 1973 as amended. With the passage of federal legislation entitled Americans with Disabilities Act (ADA), pursuant to section 504 of the Rehabilitation Act, there is renewed focus on providing this population with the same opportunities enjoyed by all citizens. As a faculty member, I am required by law to provide “reasonable accommodations” to students with disabilities, so as not to discriminate on the basis of that disability. Student responsibility primarily rests with informing faculty of their need for accommodation and in providing authorized documentation through designated administrative channels. Information regarding specific diagnostic criteria and policies for obtaining academic accommodations can be found at you may visit the Office for Students with Disabilities in room 102 of University Hall or call them at 817-272-3364.

Academic Dishonesty Policy: It is the philosophy of The University of Texas at Arlington that academic dishonesty is a completely unacceptable mode of conduct and will not be tolerated in any form. All persons involved in academic dishonesty will be disciplined in accordance with University regulations and procedures. Discipline may include suspension or expulsion from the University. “Scholastic dishonesty includes but is not limited to cheating, plagiarism, collusion, the submission for credit of any work or materials that are attributable in whole or in part to another person, taking an examination for another person, any act designed to give unfair advantage to a student or the attempt to commit such acts.” (Regents’ Rules and Regulations, Series 50101, Section 2.2)