Module 8 - New Evaluation Model Final Exam
Please select the one BEST answer for each question.
1. Name the five tiers of the new training evaluation model?
a. Tier 1: Reactionnaire
Tier 2: Attitude, Knowledge, and Subjective Norms
Tier 3: Perceived Behavioral Control Factors
Tier 4: Behavioral Intentions and Actual Behavior
Tier 5: Changes in Results/Return On Investment
b. Tier 1: Reactionnaire
Tier 2: Attitude, Knowledge, and Subjective Norms
Tier 3: Self-Efficacy
Tier 4: Behavioral Intentions and Actual Behavior
Tier 5: Changes in Results/Return On Investment
c. Tier 1: Reactionnaire
Tier 2: Attitude and Subjective Norms
Tier 3: Self-Efficacy
Tier 4: Behavioral Intentions and Actual Behavior
Tier 5: Changes in Results/Return On Investment
d. Tier 1: Reactionnaire
Tier 2: Attitude and Subjective Norms
Tier 3: Self-Efficacy and External Control Factors
Tier 4: Behavioral Intentions and Actual Behavior
Tier 5: Changes in Results/Return On Investment
2. Explain the benefit of using the new model's instruments.
a. Creating self-efficacy instruments is easier than creating reliable and valid
knowledge exam or skills tests. Additionally, evaluators can predict
behavior transfer.
b. The benefit is that evaluators have an easier time creating the instruments
as well as an easier time proving reliability and validity because they can focus on
proving content validity.
c. Evaluators can predict behavior transfer and have an easier time creating
the instruments as well as an easier time proving reliability and validity
because they only need to prove content validity and internal consistency.
d. Evaluators can predict Tier 4 outcomes by using evaluation data
collected from Tiers 1, 2 and 3. They need to illustrate predictive
validity of the actual behavior instrument. Whereas, they need to
prove content validity and internal consistency for Tiers 1, 2 and 3 instruments.
3. Given the performance objectives below, create a reactionnaire using the new model’s guidelines and include a built-in pre/post self-efficacy section.
Performance Objectives Deliver Times
1. Name the four elements. 15 minutes
2. Determine which elements are compatible. 30 minutes
3. Name the three modalities. 15 minutes
4. Indicate how many houses are in an astrological chart. 15 minutes
5. Name the four basic aspects. 15 minutes
6. Assess what aspect would appear between two astrological signs. 30 minutes
a.
b.
c.
d.
4. Explain how the self-efficacy section of a reactionnaire aids evaluators in
gathering more information than a traditional reactionnaire.
a. The self-efficacy section of a reactionnaire aids evaluators in gathering
more information than a traditional reactionnaire because the items
focus on the attitudinal components of the trainees which is important information to gather in order to determine if they will transfer their newly learned skills to the job.
b. The answers from the self-efficacy section can aid evaluators in
gathering more information about the training event due to the specific self-efficacy items which mirror the training performance objectives. Moreover, these items can aid evaluators in supporting content validity as well as internal consistency reliability.
c. The answers from the self-efficacy section can aid evaluators in
gathering more information about the training event due to the specific
section on "ease of understanding" as illustrated in the Basic HTML Coding training. For example:
BASIC HTML CODING
The section on...
1. setting font type 3 2 1
2. setting font size 3 2 1
3. bolding font 3 2 1
4. aligning font 3 2 1
d. The self-efficacy section of a reactionnaire aids evaluators in gathering
more information than a traditional reactionnaire because the items
focus on the trainer and not the trainees.
For Questions 5 through 10, use the scenario below to create the respective instrument for the New Training Evaluation Model.
Scenario: You are interested in evaluating a two hour training on The Basics of
Astrology. The performance objectives and delivery times are below.
Performance Objectives Deliver Times
1. Name the four elements. 15 minutes
2. Determine which elements are compatible. 30 minutes
3. Name the three modalities. 15 minutes
4. Indicate how many houses are in an astrological chart. 15 minutes
5. Name the four basic aspects. 15 minutes
6. Assess what aspect would appear between two astrological signs. 30 minutes
5. Create an attitudinal instrument.
a.
b.
c.
d.
6. Create a subjective norms instrument.
a.
b.
c.
d.
7. Create a self-efficacy instrument.
a.
b.
c.
d.
8. Create an external control instrument.
a.
b.
c.
d.
9. Create a behavioral intentions instrument.
a.
b.
c.
d.
10. Create an actual behavior instrument.
a.
b.
c.
d.
11. Prove content validity for the self-efficacy instrument used in the scenario below.
Scenario: You have been hired to evaluate an Exercise Physiology class. The class
consisted of the following performance objectives and delivery times. You created the
self-efficacy instrument below, and you need to prove that the instrument is content valid.
Performance Objectives Delivery Times
1. Name the two dimensions that comprise an individual's
weight on a scale. 10 minutes
2. Comprehend the two main purposes of blood. 10 minutes
3. Apply the maximum oxygen consumption formula to
calculate an individual's aerobic capacity 30 minutes
The self-efficacy instrument is content valid because
a. all the instrument items parallel the training performance objectives, the
instrument items are in proper balance to the delivery times and the instrument
is free from pre-requisite information, that is, we are not testing anything we did
not teach.
b. all the instrument items parallel the training performance objectives, the
instrument items are in proper balance to the delivery times and the instrument
includes pre-requisite information, that is, we are testing things we did
not teach in order to challenge the trainees.
c. all the instrument items parallel the training performance objectives, the
instrument items are in proper balance to the delivery times if we multiply item
"3" by three, and the instrument is free from pre-requisite information, that is,
we are not testing anything we did not teach.
d. all the instrument items parallel the training performance objectives, the
instrument items are in proper balance to the delivery times if we multiply item
"3" by two, and the instrument is free from pre-requisite information, that is, we
are not testing anything we did not teach.
12. Explain what is a multiple regression analysis.
A multiple regression analysis is
a. a statistical procedure used on evaluation instruments to illustrate that the
instrument can regress back to multiple analyses.
b. the relationship between one dependent variable to one independent variable.
c. the relationship between one dependent variable to multiple independent
variables.
d. a statistical procedure used on an evaluation instrument to illustrate that the
instrument possesses validity and reliability in order to predict behavior.
13. Prove predictive validity.
Scenario: You have been hired to evaluate an Exercise Physiology class. You
created seven instruments (i.e., a reactionnaire, attitudinal, subjective norms, self-
efficacy, external control, behavioral intentions and actual behavior instruments) and you
need to prove that the instruments can predict actual behavior. Which answer is correct?
a. The dependent variable is self-efficacy, and the independent variables are
reactionnaire, attitudinal, subjective norms, external control, behavioral
intentions and actual behavior.
b. The dependent variable is actual behavior, and the independent variables are
reactionnaire, attitudinal, subjective norms, self-efficacy, external control,
and behavioral intentions.
c. The dependent variable is actual behavior, and the independent variables are
attitudinal, subjective norms, self-efficacy, external control, and behavioral
intentions.
d. The dependent variable is self-efficacy, and the independent variables are
reactionnaire, attitudinal, subjective norms, self-efficacy, external control,
behavioral intentions and actual behavior.
14. Prove predictive validity.
In a different (i.e., this question does not relate to question 13) training session for retail
clerks, you are asked to prove predictive validity of your evaluation instruments.
You are given the following two statistical outputs. What do the outputs indicate about
the predictive validity of your evaluation instruments?
Also see the output on the next page.
The outputs indicate that:
a. the reactionnaire, subjective norms, attitudinal instrument, self-efficacy and behavioral
intentions instrument can predict actual behavior as indicated by model 3 with a
statistical significance level of .001 and common variance at 52%.
b. the subjective norms and attitudinal instruments alone can predict actual behavior as well
as if we add self-efficacy to the prediction (as indicated in model 2), those instruments
collectively can predict actual behavior with 43% common variance. And, if we add
behavioral intentions to the other evaluation instruments, the prediction is even greater
with 52% common variance and a statistical significance level to .001.
c. the subjective norms and attitudinal instruments alone cannot predict actual behavior.
However, if you add the self-efficacy instrument to the analyses, then the three
instruments collectively can predict actual behavior with 43% common variance.
Moreover, by adding behavioral intentions to the analyses, we enhance common variance to 52% and increase the statistical significance level to .001.
d. the subjective norms and attitudinal instruments alone cannot predict actual behavior.
However, if you add the self-efficacy instrument to the analyses, then the three
instruments collectively can predict actual behavior with .049 common variance.
Moreover, by adding behavioral intentions to the analyses, we enhance common variance to .001 and increase the statistical significance level.
15. Support internal consistency reliability for the self-efficacy instrument used in the scenario below. Scenario: You have been hired to evaluate an Exercise Physiology class. The class consisted of
the following performance objectives and delivery times. You created the self-efficacy instrument
below, and you need to support the instrument in terms of internal consistency reliability.
Performance Objectives Delivery Times
1. Name the two dimensions that comprise an individual's
weight on a scale. 10 minutes
2. Comprehend the two main purposes of blood. 10 minutes
3. Apply the maximum oxygen consumption formula to
calculate an individual's aerobic capacity 30 minutes
a. You would run a regression analysis and show support for reliability by indicating that
each significance level is at .001 which is less than .05 as illustrated below.
b. You would run a reliability analysis, indicating scale equals Alpha. The output below
shows Alpha equals .8712 which shows support that the instrument is reliable.
c. You would show support for reliability by illustrating that all the instrument items
parallel the training performance objectives, the instrument items are in proper balance to
the delivery times if we multiply item "3" by two and the instrument is free from pre-
requisite information, that is, we are not testing anything we did not teach.
d. You would run a correlation to show support for reliability. You would illustrate the
support with a high correlation as shown below as well as a statistically significant two
tailed test.
16. Support test re-test reliability.
Using the same scenario as number 15 and the same answers, what is the correct
answer to support test re-test reliability?
17. Analyze training effectiveness via pre and post training. Based on the output below,
how effective is training?
Paired Sample Statistics
Mean / N / Std. Deviation / Std. Error MeanPre Self-Efficacy / 28.1232 / 150 / 5.1232 / 1.234
Post Self-Efficacy / 31.2343 / 150 / 5.6543 / 1.345
a. Training is effective. The post self-efficacy mean is higher than the pre mean score and
the significance level is greater than .05.
b. Training is effective. The post self-efficacy mean is higher than the pre mean score and
the "t" equals 3.601.
c. Training is ineffective. The post self-efficacy mean is higher than pre mean score and the
significance level is less than .05.
d. Training is ineffective. The post self-efficacy mean is higher than pre mean score and
the significance level is greater than .05.
18. Analyze training effectiveness via meeting post training benchmarks. Which data below would help you in analyzing training effectiveness via meeting post training benchmarks?
a.
b.
c.
d.
Module 8 Final Exam Copyright © 2012 Third House Inc. 19