Course Design Guide (Tables Version)
[Insert Course Title Here]
[insert Faculty Name Here]
[Insert Contact Information Here]
The primary objective of this course is not to teach you about individual psychological tests, but to prepare you to be informed consumers—either test users or test takers—of psychological tests and measurements. Through discussion, as well as individual and group activities, you will learn about the basic concepts, issues, and tools used in psychological testing and measurement and how these concepts, issues, and tools are relevant to you everyday. You will also learn how tests and measurements are used in educational, clinical, and organizational settings. While these concepts, issues, and tools may not be as exciting to you as the psychological tests themselves, an understanding of them is vital to the proper use of the tests. At the conclusion of this course you will be more knowledgeable about psychological tests and will be a more informed consumer.
Text
Miller, L. A., & Lovler, R. L. (2016). Foundations of psychological testing: A practical approach (5th ed.). Thousand Oaks, CA: Sage.
Topic 1: Overview of Tests and MeasurementsDetails / Due / Points
Text readings / Read Ch. 1, 2, 13, 14, 15
Participation / Participate in class discussion.
Discussion / Post an answer to one of two posted discussion question.
Individual
Psychological Test Worksheet / Complete the Psychological Test Worksheet and submit for instructor review. / [insert date/time here]
Preparation
Psychological Test Critique / Select one test from your Psychological Test Worksheet to critique for the Psychological Test Critique, due in The final week.
Review Appendix B of Foundations of Psychological Testing in preparation for the Psychological Test Critique, due in The final week. Each week, complete a different section of the critique, using Appendix A as a guide.
Find and document answers to questions in Part One of the Psychological Test Critique.
Topic 2: Test Reliability and Validity
Details / Due / Points
Text readings / Read Ch. 6–9
Participation / Participate in class discussion.
Discussion / Post an answer to one of two discussion questions.
Preparation
Psychological Test Critique / Continue working on the Psychology Test Critique, due in The final week.
Find and document answers to questions in Part Two of the Psychological Test Critique.
Individual
Reliability Worksheet / Complete Part One of the Reliability Worksheet:
Complete Part Two of the Reliability Worksheet.
Submit The Reliability Worksheet for instructor review.
Topic 3: Test and Measurement Development
Details / Due / Points
Text readings / Read Ch. 10 & 11
Participation / Participate in class discussion.
Discussion / Post an answer to one of two discussion questions.
Preparation
Psychological Test Critique / Continue working on the Psychology Test Critique, due in The final week.
Find and document answers to questions in Part Three of the Psychological Test Critique.
Individual
Test Development, Part One / Imagine you are asked to develop a 10-item multiple-choice test to measure knowledge of reliability and validity.
Create a test specification table to guide your development.
Include the following information in the test specification table:
· Type of test, item format, and test length
· Testing universe
· General content areas and subareas
· Content area or subarea weight
· Number of items per content and subarea
Write 10 multiple-choice test items based on the test specification table and leveraging best practices in item writing. Bold the correct response for each item.
Submit the Test Plan and Test Items for instructor review
Topic 4: Piloting Tests
Details / Due / Points
Text readings / Read Ch. 12
Participation / Participate in class discussion.
Discussion / Post an answer to one of two discussion questions.
Preparation
Psychological Test Critique / Continue working on the Psychology Test Critique, due in The final week.
Find and document answers to questions in Part Three of the Psychological Test Critique.
Individual
Test Development, Part Two / Resource: Test Development, Part One
Imagine you piloted the test items developed last week with 10 individuals.
Create a mock data set, based on the pilot. When creating the mock data set, ensure there are some test questions that most individuals answer correctly, some that most individuals answer incorrectly, and some items where there are a varied number of correct and incorrect answers. Indicate whether each individual answered each question correctly (indicated by a 1) or incorrectly (indicated by a 0) (see sample below).
Question 1 / Question 2 / Question 3
Individual 1 / 0 / 1 / 1
Individual 2 / 0 / 1 / 0
Individual 3 / 1 / 1 / 0
Conduct a quantitative item analysis using Microsoft® Excel® or SPSS.
Include the (a) item difficulty, (b) item discrimination, and (c) a measure of internal consistency
Write a brief summary describing which items you would keep and not keep based on the results.
Submit your data file, calculations, and summary for instructor review
Topic 5: Interpreting Test Scores
Details / Due / Points
Text readings / Read Ch. 5
Participation / Participate in class discussion.
Discussion / Post an answer to one of two discussion questions.
Preparation
Psychological Test Critique / Continue working on the Psychology Test Critique, due in The final week.
Find and document answers to questions in Part Four of the Psychological Test Critique.
Individual
Test Development, Part Three / Resources: Test Development, Parts One and Two
Imagine the item analysis resulted in you keeping all 10 original multiple-choice items and that you moved forward and administered your test to 10 individuals.
Create a mock data set that reflects the test results for the 10 individuals (see sample below). For each individual, indicate whether they answered each question correctly (indicated by a 1) or incorrectly (indicated by a 0).
Q1 / Q2 / Q3 / Q4 / Total correct
Individual 1 / 0 / 1 / 1 / 1 / 3
Individual 2 / 1 / 1 / 0 / 1 / 3
Use a Microsoft® Excel® spreadsheet or SPSS to calculate the (a)
the appropriate measures of central tendency reflecting average test performance, (b) the appropriate measures of variability, (c) standard scores, for linear transformations only
Prepare a 5- to 7-slide Microsoft® PowerPoint® presentation, displaying and summarizing the results, and including a written interpretation of the results in the speaker notes of each slide.
Submit your presentation for instructor review.
Topic 6: Ethical Use of Tests and Measurements
Details / Due / Points
Text readings / Read Ch. 3
Review the following websites:
· http://www.apa.org/ethics/homepage.html: Click Ethics Code – HTML format.
· http://www.apa.org/science/: Click Testing and Assessment in the bottom-left corner.
· http://www.uniformguidelines.com/: Click uniform guidelines on employee selection procedures.
Participation / Participate in class discussion.
Discussion / Post an answer to one of two discussion questions.
Individual
Psychological Test Critique / Write a 5-10-page paper, presenting the results of your test critique. Organize the paper according to the Parts completed in previous weeks.
Discuss the test critique process, addressing your experience and why it might be important that a testing processional be able to evaluate a test.
Format your paper consistent with APA guidelines.
Submit your paper for instructor review
Letter to the Editor
Teams / Search the Internet for information on Ricci v. DeStefano (2009), the U.S. Supreme Court case.
Imagine you are contacted by the editor of a local newspaper to comment on the ethical issues associated with Ricci v. DeStefano.
In teams, Write a 1-page letter to the editor of the newspaper, including a brief summary of the case and the implications of the case for ethical and proper use of tests in organizational settings.
Submit your letter for instructor review.
Potential Discussion Questions by Topic
Topic 1· How might psychological tests be similar to and different from one another? Select two tests, and identify two similarities and two differences.
· What are four assumptions we make when using psychological tests and why these assumptions are important?
· Identify a psychological assessment, a psychological test, a measurement, and a survey, all measuring a similar construct. Discuss how the instruments are similar and how they are different.
· Discuss two recent trends in the use of tests in organizations.
· Identify two print and online resources individuals might use to research, and select an appropriate test for a specific circumstance. Explain the type of information communicated about each test and its value to the person using the test. / Topic 2
· What are two factors or sources of error that may affect a test’s reliability/precision? Explain each factor in terms of how it increases or decreases reliability/precision.
· What is the difference between random and systematic error? Provide an example of each.
· What is the best way to establish a test’s reliability/precision? Explain your rationale.
· How do testing professionals use confidence intervals?
· Why is it important to estimate the amount of error associated with any measure?
· What is the goal of test validation?
· How do you determine which source of evidence of validity is most appropriate?
Topic 3
· Explain why understanding the steps associated with the scientific method is important to test and measurement development.
· What steps would you take to design an employee attitude survey?
· Discuss three most important issues that must be considered prior to item writing that may affect the quality of the outcome.
· What must you consider when determining the best test format?
· What must you consider when identifying the most appropriate scoring method for a test? / Topic 4
· What steps could you take to determine effective and ineffective test items?
· What are the implications of not pilot testing test items?
· How does a quantitative item analysis compare to a quantitative item analysis?
· How might the criteria for evaluating item difficulty and discrimination indexes change if a test is norm-referenced?
Topic 5
· How can individuals use descriptive statistics to interpret a set of test scores? Provide an example.
· What is the value of standard scores? Provide an example.
· What type of statistics could you report to help others understand a distribution of test scores and one individual’s test performance?
· Do test users need to set cutoff scores? What are the pros? The cons? Defend your answer. / Topic 6
· How do the rights to privacy, anonymity, and informed consent relate to the use of tests by educational administrators?
· To properly use tests, what qualifications do you need?
· What are some of society’s concerns about the use of psychological tests? What is an example of one concern?
Psychological Test Worksheet
Based on your Text readings, online resources, and a review of test publisher catalogues, identify 9 specific psychological tests professionals in educational, clinical, or organizational settings might use (3 tests for each setting). Include the test name, what the test measures, the setting the test is commonly used in, and how the test results are commonly used (i.e., what types of decisions are made using the test results).
Test Name / What does the test measure? / Setting where test is commonly used / How is the test typically used? /1.
2.
3.
4.
5.
6.
7.
8.
9.
Reliability Worksheet
Part 1
Read each of the three scenarios below. For each scenario, identify the one or more strategies you could use, given the situation, to establish evidence of test reliability/precision (e.g., test-retest method). Provide specific rationale for why each chosen strategy is appropriate. Describe the steps you would take to assess reliability for each reported strategy.
1. An I-O psychology practitioner has designed a test to assess employees’ knowledge of areas generally accepted as best practice within project management. The test is made up of multiple-choice questions that measure the following management knowledge areas: integration, scope, time, cost, quality, human resource, communications, risk, and procurement. The practitioner may only administer the test once, but he or she must know how reliable the scores are.
2. A teacher wants to use an existing test that measures student attitudes about community involvement. She wants to be sure the instrument is reliable. The instrument contains 20 statements individuals rate from 1 to 5. The test was designed to be homogeneous.
3. An HR professional has designed a test her organization can use as a part of the selection process for maintenance staff. The test requires applicants to perform maintenance activities and be rated by two experts. How may the raters scores be checked for reliability?
Scenario / ReliabilityStrategy / Rationale / Steps
Part 2
A corporate trainer administered a test to eight employees prior to their participation in a training program. The trainer administered the test again after participation. Raw test scores from each administration are as follows:
Test-Taker / First Administration / Second AdministrationJohn / 6 / 6
Doug / 5 / 4
Christina / 8 / 7
Sammy / 4 / 2
Todd / 4 / 8
Terri / 2 / 1
Abel / 6 / 7
Threase / 3 / 3
Answer the following questions.
Question / AnswerWhat is the test-retest reliability coefficient for the test?
What is the standard error of measurement for the first and second administration?
What is the confidence interval at which we may be 95% certain of Abel’s true score on the first administration? Given the confidence interval, could we feel confident that Abel’s true score on the first administration is higher than Todd’s true score on the first administration?