Sift Feedback Report

Qualifying Test Feedback Report

Fee-paid Judge of the First Tier Tribunal, Social Entitlement Chamber(Social Security and Child Support), Scotland 2013

April 2014

Fee-paid Judge First-tier Tribunal Social Entitlement Chamber

(Social Security and Child Support)

QUALIFYING TEST: 24 October 2013

Feedback Report

Purpose

The purpose of this report is to provide general feedback on candidate performance in the qualifying test for this selection exercise. 143 candidates applied for the competition; 137 candidates took the test. The results were taken into account in the sift where 45 candidates were selected for interview and 20 were recommended for appointment. A copy of the Sift Feedback Report for this exercise is available on the JAC website.

The first part of this report describes how the Judicial Appointments Commission (JAC) developed the test and marking schedule and how the test was structured. The second provides information on the overall performance of candidates in the test, identifying areas where they performed well and where they performed poorly.

Qualities and Abilities

As this appointment required high level interpersonal skills and at least competent legal abilities, the test was set to assess:

Intellectual Capacity, specifically:

• Ability quickly to absorb and analyse information.
• Appropriate knowledge of the law and its underlying principles, or the ability to acquire this knowledge.

Personal Qualities’ specifically:

• Ability to apply objectivity.
• Ability to be decisive.

Efficiency, specifically:

• Ability to work at speed and under pressure.
• Ability to organise time effectively.

Development of the Test

The test and marking schedule were designed by a team of one District Tribunal Judge assisted by one Regional Tribunal Judge for the multiple choice part of the test and a scenario question devised by a District Tribunal Judge.

The JAC Advisory Group composed of senior judiciary and representatives of the legal profession and the Assigned Commissioner, reviewed the test to ensure that it was fit for purpose.

In common with all qualifying tests used by the JAC, both the test and the marking schedule were subject to a quality and equality assurance process. Its suitability was assessed by means of two dry runs with a range of volunteers who had no prior experience in the jurisdiction. The range of marks established that the test was good at assessing the generic legal skills indicated above and that candidates with no previous experience in the jurisdiction could score highly on the test. Performance in the test was poor but the candidates had generally done little preparation and were not familiar with the jurisdiction. In the second dry run, candidates had been better prepared and there was some amendment of the questions. There were no complaints from those participating that there was any inherent unfairness in the test. The time allowed for the test was 90 minutes. Although some of the participants regarded this as quite a challenging timescale almost all finished the test within the time allowed.

The innovative aspect of this qualifying test was that candidates were not required to attend a test centre in order to undertake the test. The procedure followed was that the qualifying test was e-mailed to all the candidates, supported by two JAC staff in case of problems, on Thursday 24 October and they were required to complete and return the completed test within a 2 hour time window from 19:30 to 21:30 to suit the availability of candidates who might otherwise have work commitments. This also meant that some candidates who were on holiday could equally well undertake the test from locations abroad. Only one candidate failed, due to a computer system failure, to return a completed script within this time frame. The 120 minute time frame was perhaps more generous than in previous exercises but notwithstanding about one quarter of candidates failed to complete all the questions, apparently due to lack of time.

Unlike previous selection exercises for the Social Entitlement Chamber, the test was devised to provide supportive and objective evidence for the quality of intellectual capacity. The sift, which scored on all qualities, was based primarily on the self assessment applications with cross-reference to the score from the qualifying test and the marks were aggregated.

Structure of the Test

The preparation materials for the test were circulated in advance and could be printed off by the candidates for ease of access.

The test, as with the ones from previous selection exercises in this Chamber, in 2011 and 2012, consisted of two parts, Part A and Part B, each consisting of five questions each. Each question was worth 10 marks, thus 100 marks in total. Candidates were advised that the test should take 90 minutes to complete but were given a two hour time frame between 19:30 and 21:30 to complete and return the test as a word document attachment by e-mail.

Part A

• The questions in Part A related to Tribunal procedure. The purpose was to test the candidate’s ability to conduct Tribunal hearings fairly; to recognise and apply relevant procedure rules; and to explain why the candidate made that choice.

• The questions in Part A were part multiple choice and part explanatory answers. Points were awarded ranging from 3 or 4 for the most correct answer to no points for an incorrect answer. Candidates were asked to state the answer (for example a, b, c, or d) which best represented the decision they would make in the scenario outlined. Candidates were advised that they should not state more than one answer as this would result in no marks being awarded. None of the candidates failed to observe that instruction.

• They were also asked to explain the reasons for their choice of answer in no more than 150 words. Use of note form and bullet points was permissible. The explanation made up the remainder of the 10 points available for the five questions in Part A. A maximum of 6 or 7 marks was awarded for the quality of their explanation and identification of the applicable statutory provision. This element was significantly less well completed.Although candidates were told that their explanation should contain reference to any relevant rule or rules in the Tribunal Procedure (First-tier Tribunal) (Social Entitlement Chamber) Rules 2008 this element was not consistently satisfied.

Part B

• The questions in Part B related to a particular area of substantive social security law. The area chosen related to the assessment of visual impairment in terms of the provisions applying to claimants in respect of Disability Living Allowance primarily in respect of the conditions of entitlement to the higher rate of the mobility component. This is addressed in very specific statutory provisions which apply a stringently prescribed test capable of objective ophthalmic assessment. The purpose was to test the candidate’s ability to identify, understand and apply complex statutory provisions such as those commonly found in the jurisdiction.

• The questions consisted of scenarios followed by a number of focussed questions to ensure consistency of marking. Candidates were instructed that all questions should be answered as concisely as possible and in no more than 150 words with use of note form and bullet points permissible.

• Again, candidates were advised that answers should make reference to the relevant provisions of the statutory materials where appropriate. There was a clear division between candidates who were able to satisfy these instructions and those who were not.

Marking Schedule

All scripts were identified by number only. Two markers marked all the scripts which could be sent by secure e-mail and double marked between markers also by e-mail.

At the outset of the marking a selection of scripts were all double marked and this ensured consistency in marking according to the marking schedule.

Distribution of Marks

The multiple choice question, Part A, attracted 50 marks and the scenario question, Part B, 50 marks.

Part A Part B

Average marks 2219

Highest 3543

Lowest 100

The multiple choice Part A scored more highly proportionately than the scenarios in Part B. This was to be expected in view of:

(1) the issue of the core material in advance of the test and the generic nature of tribunal rules,

(2) the fact that the question was placed first and on the whole it appeared that candidates had taken more time to answer this question which was almost always completed whereas some candidates ran out of time for Part B,

(3) it was easier to complete at least partially by ticking a given box than completing an open ended answer.

Part B attracted the same number of marks and there was an expectation that candidates would allocate their time proportionately between the two parts. Candidates clearly found the substantive law scenario questions more challenging and the marks reflected this. There was no threshold and all candidates had both parts of the script marked.

General Comments on Candidate Performance

In order to assess the effectiveness of any test it is necessary to consider the extent to which the test was successful in assessing the relevant competence of intellectual ability.

Since all candidates produced typed answers there was not a problem in relation to legibility. Keyboard skills varied but no marks were deducted for typographical errors, poor spelling, grammar or syntax.

It is evident that the quality of the test has been effective in distinguishing those who were able to work at speed and under pressure in the test environment distinguishing those who were able to produce very creditable answers. Some candidates struggled with basic aspects of the test, such as:

(1) the ability to understand the question correctly,

(2) identifying the relevant legal tests to be applied despite the fact these were provided in the copies of the statutory provisions supplied at the test,

(3) relating the relevant factual information supplied in the appropriate parts of the question, and

(4) incorporation of wholly irrelevant aspects of the law

It is inherently improbable that only those with a prior knowledge of this field of law could have performed well in the qualifying test since the substantive question related to a test not commonly encountered. However, it is clear that those who performed well demonstrated core legal skills and were able to cite legislation correctly, identify findings in fact from information provided and explain how they applied the applicable statutory tests.

Had the test been less demanding it is unlikely that it would have been as effective in identifying and testing the specific intellectual ability relevant to the jurisdiction. Given the perception of multiple choice questions being easier to score marks, it was important to have a more exacting element to Part B. However, there was no clear correlation between performance in the multiple choice section and the scenario question. Some candidates scored highly in Part A and poorly in part B but for a smaller number of others the reverse was the case. It is probable that there were a number of candidates who already hold some form of judicial office and they were more adept at applying the procedural rules which are similar across chambers but less familiar with the substantive law.