ESECT: assessment analysis tool
Notes for users
This coaching tool can be used directly by undergraduate programme leaders.
Subject centres are encouraged to customise it for different disciplines, associating it more closely with benchmark statements and other points of reference. Educational developers may also customise for better fit with university missions and priorities.
PVCs may call in a series of reviews to get a sense of the consistency of assessment practice across the institution.
Instructions
This is a coaching tool, designed to help appraisal of the quality of undergraduate programme or department assessment arrangements.
Since the complexity of programmes varies, it asks you to target the six most important or most popular modules in the programme. ‘Important’ modules are core or compulsory ones. If there are more than six core or compulsory modules, use your judgement to reduce the number to six modules.
This is why the tool is a coaching tool. It illustrates a process of analysis that can then be adapted to give a more comprehensive picture of the ways in which an undergraduate programme’s assessment arrangements are conducive to the development of student employability.
Step 1. Goodness of fit between programme specification and assessment practices
It is widely believed that students and teachers take seriously things that are assessed. An implication is that all outcomes in a programme specification should be assessed.
Begin by identifying six target modules. Choose those that are ‘important’ because they are core or compulsory modules in the programme. If you do not identify six, add the most heavily subscribed non-core modulesto make up the figure. Details of these target modules head the six empty rows in the form below/
Use the rows to display to twenty programme learning outcomes. If you have more than twenty programme learning outcomes, concentrate those most likely to foster the general achievements valued by employers. Click here for a list of them[1].
For each of the programme learning outcomes the target modules in which the outcome is clearly and directly assessed. Submit the form and await feedback.
Target module codeTarget module level (1-3) / 1 / 1 / 2 / 2 / 3 / 3
Learning outcomes
The software needs to perform the following functions to give the following feedback:
For each row:
If number of cells checked is equal to or less than three, response = You might ask whether this outcome is getting enough sustained attention from your assessment arrangements.
If number of cells checked is four or greater and there is at least one cell at levels 1, 2 and 3, (row 3) response = It looks as if this outcome is getting enough sustained attention from your assessment arrangements.
If number of cells checked is four or greater and there is no entry for level 1 or 2 or 3, (row 3) response = It looks as if this outcome is getting attention from your assessment arrangements but you might want to be clear about why this attention is so unevenly distributed.
For any column
If the number of checks for the column is greater than half the number of learning outcomes, response = This module seems to engage with a lot of learning outcomes – perhaps too many.
If the number of checks for the column is less than one third the number of learning outcomes, response = This module seems to focus on a few learning outcomes – are you sure that others will receive sufficient attention elsewhere?
Step 2: range of assessment methods
The completed grid used in step 1 should be redisplayed but the check boxes will need to be able at this stage to accommodate numbers from 1 to 4. There needs to be a new eighth column, headed ‘mean row score’. There will also need to be a new, final row, headed ‘mean column score’
Please download a list of 50 common assessment methods[2]. Each is numbered.
Now replace the checked cells with the number of the assessment method used to assess that outcome in that module. For example, this display from step 1
Target module code / 101 / 102 / 221 / 231 / 332 / 333Target module level (1-3) / 1 / 1 / 2 / 2 / 3 / 3
Learning outcomes
- Developing autonomy
Should be edited to produce this
Target module code / 101 / 102 / 221 / 231 / 332 / 333 / MeanTarget module level (1-3) / 1 / 1 / 2 / 2 / 3 / 3
Learning outcomes
- Developing autonomy
Note two things. It is common for a learning outcome to be addressed by more than one assessment method in any one module. That is why there are three entries for module 221.
You will see that a new column has appeared. You need to calculate the mean (average) of the numbers you have used in that row. In this case it’s 13/5. There is also a new row. Again, please calculate the mean for each of the six target modules.
When you have completed the form, submit and await feedback
The software needs to perform the following functions and give the following feedback:
For each row:
Where the mean is between 1 and 1.99, response = The assessment methods that address this outcome tend to be most suited to assessing recall and the ability to perform routine operations accurately. They are not suited to judging the complex achievements that employers value[3].
Where the mean is between 2 and 2.99, response = The assessment methods that address this outcome tend to be best suited to assessing performances on fairly well-structured tasks, often of quite a familiar sort. Although they may be stimulating, they will not provide evidence that students can bring the complex achievements that employers value to bear in unfamiliar and ill-structured situations.
Where the mean is between 3 and 3.99, response = The assessment methods that address this outcome are well-suited to generating evidence that students can bring the complex achievements that employers value to bear in unfamiliar and ill-structured situations.
For any column
Where the mean is between 1 and 1.99, response = The assessment methods used in this module tend to be most suited to assessing recall and the ability to perform routine operations accurately. They are not suited to judging the complex achievements that employers value[4].
Where the mean is between 2 and 2.99, response = The assessment methods used in this module tend to be best suited to assessing performances on fairly well-structured tasks, often of quite a familiar sort. Although they may be stimulating, they will not provide evidence that students can bring the complex achievements that employers value to bear in unfamiliar and ill-structured situations.
Where the mean is between 3 and 3.99, response = The assessment methods used in this module are well-suited to generating evidence that students can bring the complex achievements that employers value to bear in unfamiliar and ill-structured situations.
Step 3. Balancing formative and summative assessment
This step needs the same grid as for step 2 (i.e. with mean columns). It needs to be able to accept new data and give new feedback.
There is a common distinction between summative assessment, which gives marks for that count towards module of degree grades, and formative assessment, which is concerned to create feedback for improvement.
Formative judgements seldom contribute to module or degree grades. Summative, or high-stakes judgements, are expected to be more finely graded and to have high reliability. This can make them unsuitable or unaffordable for many of the ‘soft’ learning outcomes in benchmark statements and in employers’ ‘wish lists’[5].
To learn more about this distinction, visit
The form below shows your responses at step 2. Pleae put new numbers in each of the cells that you completed at step 2. Use the following codes:
If an outcome is wholly assessed in that module for summative purposes, enter the number 1.
If an outcome is mainly assessed in that module for summative purposes, enter the number 2.
If an outcome is mainly assessed in that module for formative purposes, enter the number 3.
If an outcome is wholly assessed in that module for formative purposes, enter the number 4.
Please then calculate the means for each row and each column and then submit the form.
The software needs to perform the following functions and give the following feedback:
For each row:
Where the mean is between 1 and 1.99, response = The approach to assessing this outcome relies on high-stakes summative approaches. These are sometimes not suitable for assessing the complex achievements that employers value[6]. Complex achievements can sometimes be satisfactorily addressed by summative assessment but only when a great deal is invested in ensuring adequate reliability and validity[7]
Where the mean is between 2 and 2.99, response = There is a blended approach to assessing this outcome: there are formative and summative elements that should make it possible to reach the complex achievements that employers value[8].
Where the mean is between 3 and 3.99, response = The approach to assessing this outcome relies on low-stakes formative approaches. There is a view that formative assessment does improve student learning. They can also often reach complex achievements that employers value[9]. However, there are concerns that students do not take low-stakes tasks seriously and that there is a need to make sure they have a sound command of relevant information and procedures – summative assessment is well-suited to doing that.
For any column
Where the mean is between 1 and 1.99, response = The approach to assessing this module relies on high-stakes summative approaches. These are sometimes not suitable for assessing the complex achievements that employers value[10]. Complex achievements can sometimes be satisfactorily addressed by summative assessment but only when a great deal is invested in ensuring adequate reliability and validity[11]
Where the mean is between 2 and 2.99, response = There is a blended approach to assessing this module: there are formative and summative elements that should make it possible to reach the complex achievements that employers value[12].
Where the mean is between 3 and 3.99, response = The approach to assessing this module relies on low-stakes formative approaches. There is a view that formative assessment does improve student learning. They can also often reach complex achievements that employers value[13]. However, there are concerns that students do not take low-stakes tasks seriously and that there is a need to make sure they have a sound command of relevant information and procedures – summative assessment is well-suited to doing that.
Step 4. An efficiency check.
We have finished with the forms used in steps 1-3. No feedback loop is required.
This step involves thinking about the whole set of programme assessment arrangements. The outcome-by-outcome, module-by-module approach of steps 1-3 is not appropriate.
Please complete the following form, which invites you to study a list of seven ways of reducing the time staff spend on assessment. For each, and thinking of the assessment arrangements for the programme as a whole, respond to two questions:
- Is this time-saving device widely used in the programme/department?
- Could its use be extended?
Efficiency method[14] / Widely used? / Possible to extend its use?
Step 5: Assessment and employability.
No feedback loop is required.
We started with the assumption that your programme specification contained outcomes that promote the achievements employers value. Steps 1-3 checked whether the assessment arrangements supported those learning outcomes.
It is time to check that assumption.
In the form below is a composite account of the achievements employers value. It does not cover achievements that are valued in particular professions or fields, so you may wish to extend this activity to cover achievements not listed here.
For each of these valued achievements, note the number of the programme learning outcome that should promote it. For example:
Valued achievement / Promoted by programme learning outcome …willingness to learn / A1, B2, 3, C2, D3
imagination/creativity
adaptability/flexibility / B2, 3
Achievement 1 is well-handled, as long as steps 1-3 showed that its assessment procedures were adequate.
Something needs to be done about achievement 2, which looks like aspirational candy-floss. It is possible that you have shied away from this, believing that the assessment difficulties are insurmountable. Low-stakes, formative techniques may allow you to get a grip on it – find out more about this at
Achievement 1 gets some attention. Taking into account the results of steps 1-3 and your wider knowledge of the programme, you need to judge whether that is enough.
Valued achievements / Promoted by programme learning outcome …willingness to learn
imagination/creativity
adaptability/flexibility
independent working/autonomy
working in a team;
managing others
working under pressure
good oral communication
communicating in writing for varied purposes/audiences
attention to detail
time management
taking responsibility and decisions
planning, coordinating and organizing
Some general achievementsvalued by employers. (based on work by John Brennan and Lee Harvey)
- imagination/creativity
- adaptability/flexibility
- willingness to learn
- independent working/autonomy
- working in a team;
- managing others
- working under pressure
- good oral communication
- communicating in writing for varied purposes/audiences
- attention to detail
- time management
- taking responsibility and decisions
- planning, coordinating and organizing
Page 1 of 9
[1] Page 9, below
[2] To follow
[3] Option to access menu of the achievements employers value, Page 9, below.
[4] Option to access menu of the achievements employers value, Page 9, below.
[5] Option to access menu of the achievements employers value, Page 9, below.
[6] Option to access menu of the achievements employers value, Page 9, below.
[7] These terms are discussed in
[8] Option to access menu of the achievements employers value, Page 9, below.
[9] Option to access menu of the achievements employers value, Page 9, below.
[10] Option to access menu of the achievements employers value, Page 9, below.
[11] These terms are discussed in
[12] Option to access menu of the achievements employers value, Page 9, below.
[13] Option to access menu of the achievements employers value, Page 9, below.
[14] The seven items are to follow