/

POSITION DESCRIPTION

Ministry of Education

POSITION TITLE: / Chief Psychometrician / POSITION
NUMBER(S): / 58568
DIVISION:
(e.g., Division, Region, Department) / Learning Modernization Project
UNIT:
(e.g., Branch, Area, District) / Curriculum and Assessment Branch /

LOCATION:

/ Victoria
APPROVED CLASSIFICATION / CLASS CODE / EO30
SUPERVISOR’S TITLE: / Director / POSITION
NUMBER / 103700
SUPERVISOR’S CLASSIFICATION: / PHONE NUMBER: / 250-589-0277

Page 2 of 5

PROGRAM

The Curriculum and Assessment Branch develops and manages programs that systematically gather information from a variety of sources to determine the extent to which B.C. students are achieving the knowledge, skills and understanding prescribed in the provincial curricula. B.C. students participate in three types of large-scale assessments:
·  Provincial Foundation Skills Assessment (FSA) – administered annually to all B.C. students in grades 4 and 7 to test student abilities in reading comprehension, writing and numeracy.
·  National and International Assessments – tests a sample of B.C. students of various ages in a variety of subject areas to compare student achievement relative to other provinces and countries.
·  Provincial Graduation Assessments – certifies the performance of B.C. students graduating from secondary school.
The Curriculum and Assessment Branch is responsible for the design and development of provincial assessments, the management of national and international assessments, and the psychometric analysis of assessment results. The Branch also sets policies and procedures to allow students with special needs the accommodations necessary to write provincial assessments and assists in the development of classroom assessment and student reporting policy, guidelines and tools for teachers.

PURPOSE OF POSITION

The Chief Psychometrician is responsible for planning, implementing, managing and monitoring all psychometric and technical requirements of BC’s large-scale assessment programs; ensuring the accuracy of data; collaborating with program managers and test developers to support the program with routine psychometric services; suggesting and leading additional research studies to strengthen the program; working closely with content experts to improve and expand current assessments; participating in the design and development of new assessments; writing technical reports and other documentation; participating in design of internal research and test development processes/systems; writing research reports, and making presentations at various sessions.


NATURE OF WORK AND POSITION LINKS

The province has one of the most robust assessment programs in Canada. With the renewal of curricula, a completely revised assessment program is being designed; one which focuses on students’ abilities in literacy and numeracy in the graduation years. These new assessments are being designed for electronic delivery, and with elements of student collaboration and student choice in order to align with the pace of individual learning and students' needs. It is paramount that the assessment used throughout the year, across sessions, and within the choices on any individual assessment is comparable and fair to students. With statistical techniques, assessments are equated to one another, through the linking of common items. This has the effect of minimizing differences in assessment difficulty that may exist. Graduation assessment processing and equating occurs several times each year.
The Chief Psychometrician must employ equating techniques to ensure accurate student assessment scores and provincial year-to-year performance trends, using item response theory and other equating procedures. The Chief Psychometrician mentors and provides advice to other staff and acts as the key statistical and psychometric resource for the Department. The Chief Psychometrician serves as a national expert in education measurement and psychometrics.
Position Links:
Branch Management: Provides information to the branch and senior ministry executive on the interpretation and calculation of statistics utilized in assessment and examination score reporting.
Measurement experts from Universities and Ministries of Education across Canada and in other countries: Consults and liaises on psychometrics, testing and measurement topics with peers and colleagues.
Contracted Educational Professionals: Manages the work of contracted measurement experts.
Learning Modernization Project: Works closely with content experts to ensure that test development meets criteria for high quality measurement.
Student Certification Branch: Works closely with Branch staff to ensure data is transferred into and out of TRAX for processing student results in a timely manner.
Knowledge Management and Accountability Division: Ensures smooth flow of data from the FSA and Exam TRAX systems to data warehouse

SPECIFIC ACCOUNTABILITIES / DELIVERABLES

The Chief Psychometrician manages the production of accurate student assessment scores and provincial year-to-year performance trends, using a range of equating and statistical processes and professional analysis.
·  Plans, coordinates and conducts the statistical work required for score reporting and data interpretation for the operation and maintenance of complex testing programs.
·  Incorporates scaled items using Item Response Theory (IRT) and Classical Test Theory
·  Accountable for complex equating decisions.
·  Processes and analyses test data from both field tests and from live tests.
·  Identifies program and measurement-related problems and opportunities and their psychometric implications and works to resolve problems and pursue opportunities.
·  Confers with the test developers to ensure that test development meets criteria for high quality measurement.
·  Confers with staff members responsible for production, administration and business development to ensure a smooth and efficient workflow.
·  Provides information to the branch and senior ministry executive on the interpretation and calculation of the various statistics included in reports.
·  Oversees and verifies contractor(s)’ data analyses and reports.
·  Evaluates software for test development and test analyses.
·  Helps maintain the examination item bank by updating the item parameters after each exam session and in the presence of item parameter drifting.
The Chief Psychometrician provides technical expertise on educational measurement, evaluation and testing theory.
·  Mentors psychometrics staff and serves as a resource person for the organization and department in reference to statistical analysis.
·  Explains theoretical and technical issues associated with provincial assessments to ministry executive, colleagues and other educators.
·  Conduct seminars on psychometric issues for staff and clients.
·  Advises on formats for results and reports related to examinations and FSA.
·  Consults with professional colleagues on issues related to psychometrics, statistics and testing programs.
·  Meets with university researchers and leaders in educational measurement in order to share and gain additional technical expertise regarding procedures for the analysis of the results of provincial assessments.
·  Establishes contact and makes regular communication with counterparts in other Canadian and international jurisdictions.
·  Maintains currency and expertise in the fields of educational measurement, testing theory, and educational evaluation
·  Contributes to research, development, and implementation of new testing modes, item types, psychometric methods and technology.
·  Initiates discussions of the potential effects of changes in programs with appropriate staff; work to develop and implement long-range goals and plans for programs.
·  Makes recommendations to the ministry for further research and analysis that should be conducted as follow-up to provincial, national and international assessments
·  Identifies research opportunities to improve the quality of the test instruments
The Chief Psychometrician assumes primary responsibilities for designing, developing, and documenting technical information and statistical guidelines for Department assessment programs.
·  Produces technical reports and information detailing the development, results and interpretation of results of the provincial assessments.
·  Writes technical reports on test instruments.
·  Publishes through relevant publication channels on test development and test evaluation.
·  Prepares statistical reports for client programs.
·  Develops and reviews score interpretation materials and relevant portions of testing program publications.
·  Ensures accurate and consistent use of testing and assessment procedures and terminology throughout branch activities, documents and reports.
·  Responds to requests for data and data interpretation from educational researchers.

DIRECT SUPERVISION (i.e., responsibility for signing the employee appraisal form)

Role / # of Regular FTE’s / # of Auxiliary FTE’s
Directly supervises staff
Supervises staff through subordinate supervisors

PROJECT /TEAM LEADERSHIP OR TRAINING (Check the appropriate boxes)

Role / # of FTE’s / Role / # of FTE’s
Supervises students or volunteers / Provides formal training to other staff (mentors and advises) / 8-9
Lead project teams / Assigns, monitors and examines the work of staff (mentors and advises) / 1

Page 2 of 5

SPECIAL REQUIREMENTS

This position requires the Chief Psychometrician to have specialized statistical expertise in educational measurement and testing theories.

TOOLS / EQUIPMENT

·  Specialized analysis software—Parscale, Bilog, Iteman, SPSS and Excel.

WORKING CONDITIONS

·  Office environment.
·  Some BC travel required; may involve occasional national and international travel.
·  Some possible weekend and evening work required.

Page 2 of 5

PREPARED BY

NAME:
Markus Baer / DATE:
January 3, 2017

EXCLUDED SUPERVISOR AUTHORIZATION

I confirm that:
1. the accountabilities / deliverables were assigned to this position effective: January 3, 2017
2.  the information in this position description reflects the actual work performed.
3.  a copy has / will be provided to the incumbent(s).
NAME:
Markus Baer / SIGNATURE:

STAFFING CRITERIA

Education
·  Graduate degree in educational measurement, psychometrics, statistics or a related field, plus a minimum of three years experience working on large-scale testing programs.
Experience
·  Experience with analysis, interpretation and results reporting of large scale testing programs.
·  Must have experience in IRT-based scoring methods, test form equating, standard setting and validity studies.
·  Experience in supporting content specialists in their work of test development.
·  Must have experience in handling last data sets and a demonstrable experience in large scale data analysis.
·  Experience utilizing psychometric methods, statistical procedures and application of classical test theory and item response theory.
·  Proficient psychometric software experience (i.e., PARSCALE, Winsteps, Biolog, etc.) and awareness of software available for test and item analysis.
·  Experience with SAS or SPSS statistical computer packages.
Knowledge
·  Must have a thorough knowledge of classical and modern test theory equating methodologies.
·  Thorough knowledge of large-scale assessments.
·  Knowledge of large-scale assessment programs in BC.
·  Detailed knowledge of item development and test construction standards.
·  Knowledge of educational assessment, evaluation and measurement practices.
·  Thorough knowledge of the various concepts and measures of validity and reliability.
·  Thorough knowledge of the applications of test statistics used to measure reliability, item difficulty, item discrimination and internal consistency.
·  Knowledge of the applications of test statistics used to measure reliability, item discrimination, internal consistency
Skills and Abilities
·  Ability to work collaboratively with subject area and testing/measurement experts.
·  Ability to coach and mentor psychometrics staff, and serve as a resource person for the organization and department in reference to statistical analysis.
·  Ability to work collegially with co-workers and education professionals.
·  Ability to work within a team environment or independently while taking into consideration changing priorities, tight deadlines, volume of work and available resources.
·  Ability to apply measurement concepts of validity and reliability to program design and implementation.
·  Ability to explain psychometric requirements to content staff.
·  Ability to interpret and analyze statistical test data.
·  Ability to write technical reports.
·  Must have ability to explain complex technical concepts to non-specialists.

COMPETENCIES

Teamwork and Co-operation is the ability to work co-operatively within diverse teams, work groups and across the organization to achieve group and organizational goals. It includes the desire and ability to understand and respond effectively to other people from diverse backgrounds with diverse views.
Analytical Thinking is the ability to comprehend a situation by breaking it down into its components and identifying key or underlying complex issues. It implies the ability to systematically organize and compare the various aspects of a problem or situation, and determine cause-and-effect relationships ("if...then…") to resolve problems in a sound, decisive manner. Checks to ensure the validity or accuracy of all information.
Conceptual Thinking is the ability to identify patterns or connections between situations that are not obviously related, and to identify key or underlying issues in complex situations. It includes using creative, conceptual or inductive reasoning or thought processes that are not necessarily categorized by linear thinking.
Planning, Organizing and Co-ordinating involves proactively planning, establishing priorities and allocating resources. It is expressed by developing and implementing increasingly complex plans. It also involves monitoring and adjusting work to accomplish goals and deliver to the organization’s mandate.
Problem Solving/Judgement is the ability to analyze problems systematically, organize information, identify key factors, identify underlying causes and generate solutions.
Expertise includes the motivation to expand and use technical knowledge or to distribute work-related knowledge to others.
Concern for Order reflects an underlying drive to reduce uncertainty in the surrounding environment. It is expressed as monitoring and checking work or information, insisting on clarity of roles and functions, etc.
Commitment to Continuous Learning involves a commitment to think about the ongoing and evolving needs of the organization and to learn how new and different solutions can be utilized to ensure success and move the organization forward.

Page 2 of 5