Validation Exercise

The purpose of this exercise is to practice computing correlation coefficients in order to assess the reliability and criterion-related validity of predictor and criterion measures.

ACME Insurance is in the business of selling life insurance policies. They do a lot of entry-level hiring of agents, and then train them about the life insurance business and how to effectively sell life insurance policies. They have experienced substantial turnover and have determined that the reasons are two-fold. One cause of turnover is that some of the new hires fail out of the very intensive and demanding training program (or drop out due to frustration). Another common cause is employees who simply aren’t sufficiently “smooth” with customers—even after the training program—and whose sales are mediocre as a result.

Bob Schmidtenhunter, the new HR manager, feels that a better selection system could help cut turnover by hiring more qualified employees. He decides to make two changes in the hiring system. The first is to experiment with the use of the Wonderlic Personnel Test, a popular measure of general mental ability (i.e., intelligence), figuring that applicants who are more intelligent will do better in the training program and will also learn and adapt faster on the job. He also develops a more structured approach to interviews that focuses on sales aptitude. With this, he feels ready to do some really effective hiring at the next job fair.

At the job fair, 50 people expressed interest and were given the Wonderlic and then interviewed by both Bob and one of the sales managers. They were told that they would be contacted within a week with information about whether or not they would be hired. However, amidst the confusion of closing down the job fair and transporting the materials back to his office, Bob misplaced all the Wonderlic tests and the interview rating forms. Bob realized he now had a big problem. Since the next training class would be starting in two weeks, he didn’t have time to call all the applicants back in (nor did he really want them to know what a mistake he’d made). On the other hand, he didn’t want to make random choices among the applicants and risk being sued for hiring in a capricious or discriminatory fashion. So, with a deep sigh, he decided the safest thing was to simply hire all 50 applicants. But since he still wanted to try out the Wonderlic, he had all the newly hired employees re-take the test on the first day of training.

As luck would have it, the first set of Wonderlic tests and the interview ratings showed up a few weeks later. It turned out that another company at the job fair had accidentally mixed up Acme’s box with their own materials and transported it back to their offices. When they finally figured out who the material belonged to they returned it, but now, of course, it was too late to make use of the information. Or was it, Bob wondered? He instructed his college intern to create a spreadsheet that included both sets of Wonderlic test scores, both interview ratings (his and the sales managers’), and a measure of sales performance that was collected after the new employees had been working for six months. But the student’s internship was over before he had a chance to do any analyses of the data. The spreadsheet is available on the course website ( ), and you are now assigned to complete the analyses. Be sure to include a printout of your Excel analyses as you answer the following questions. (Don’t forget to type your report.)

  1. Compute the correlation between Bob’s interview ratings and those of the sales manager. What does this correlation tell you about reliability, validity, or both?
  2. Calculate the test-retest reliability of the Wonderlic Personnel Test. Explain how you did so, and what you conclude.
  3. How valid are the interview ratings for predicting sales performance?
  4. How valid is the Wonderlic test for predicting sales performance?

Hint: If you looked in a statistics book, you would find that for a sample of 50, correlations of .28 or greater are statistically significant.

For bonus points: Use the regression analysis tool (within Data Analysis) to determine the relative effectiveness of the Wonderlic test and both sets of interview ratings for predicting job performance.

If you wish, you may work with others to plan the Excel analyses, but each person should turn in their own interpretation of what the results mean and their own printouts of the Excel analyses.

Hints For Those Not Comfortable With Statistical Analysis

(revised 9/26/2011)

You can use any statistical package to compute the required correlations (you should have learned to do this in BCOR1020.) For EXCEL, you have two options:

  1. The basic approach is to use function CORREL to compute the correlations among the variables. For example, you could place the formula =CORREL (A2:A51, E2:E51) in a cell to report the correlation between Bob’s interview ratings and job performance ratings. [In Excel 2010, click on the Formulas tab, then More Functions, then Statistical.] You may find it easiest to first create a correlation matrix of all the variables, and then to use the matrix to answer the questions above.
  2. A more sophisticated and more powerful approach is to use the Data Analysis add-in to Excel. Click on the Tools toolbar [Data tab with Excel 2010] and see if “Data Analysis” shows up. If not, click on “Add-Ins” (also under the Tools toolbar), and then click on “Analysis Tool Pak,”[For Excel 2010, click on the File tab, then on Options, then on Add-ins, then choose the Analysis Tool Pack and click on “Go”. In the next window, choose “Analysis ToolPak” and “Analysis ToolPak – VBA” and then OK.]

Once you have the Data Analysis add-in installed, click on Data Analysis under the Tools [or Data] toolbar/tab. Then click on Correlation. The only problem with this approach is that you can only compute correlations between adjacent columns, an unfortunate and rather inane limitation. However, there’s an easy workaround: specify ALL the columns of data, and what you’ll get is a matrix of the correlations among all the variables. That matrix will provide all the information you need to answer questions 1-4 in the homework.

  1. The best option is to use the MegaStat add-in for Excel that you received in the BCOR 2010 course (remember the disk that came with Essential Statistics in Business and Economics)? It works about the same as Option 2, but provides much more information in a more user-friendly format. Unfortunately, this add-on is only available with the book, although it’s also on the computers in the computer labs in Leeds. I’ve learned that the MegaStat add-on was not available for Macs, and apparently wasn’t distributed at all to some classes. If you have access to it—including by using one of the computer labs--it makes things easier, but you can still complete the assignment without it.