Consultant Performance and Selection System

Consultant Performance and Selection System

Procedures Guide

MTO Qualification Committee

September 2015

Table of Contents

Page

Summary 1

Consultant Performance and Selection System (CPSS) 2

Corporate Performance Rating (CPR) 2

Starter CPR 3

Performance Appraisals 4

Consultant Joint Ventures 5

CPSS Application at the Expression of Interest (EOI)/Phase I Stage 6

CPSS Application at the Request for Proposal (RFP)/Phase II Stage 7

CPSS Application at the Request for Quotation (RFQ) Stage 8

Consultant Performance Appraisal Reviews 9

Consultant Infraction Reports 11

Summary

·  Effective January 1, 2001, the Ministry of Transportation (MTO) modified its consultant selection procedures to take past performance into account at the Expression of Interest and Request for Proposals / Quotation (RFP / RFQ) stages. The objective is to improve the quality of engineering and construction administration services and reduce infrastructure life cycle costs. The revised consultant selection system is referred to as the Consultant Performance and Selection System (CPSS).

·  Past performance is measured by a consultant's Corporate Performance Rating (CPR) that is the weighted-average of a consultant's appraisals over the last three years.

·  A consultant’s Corporate Performance Rating (CPR) is available for Planning, Engineering, Construction Administration, Area Materials Testing and Small Value Assignments. A Small Value Assignment is an assignment with an Acquisition Value of $25,000 or less.

·  Effective October 15, 2000, a Consultant Appraisal Review process is available, as:

-  If a consultant firm disagrees with a performance appraisal received, they may request a review by the appropriate ministry regional manager, within 21-calendar days (unless 30-calendar days specified in the signed Legal Agreement).

-  If the matter is not resolved at the regional manager level, the consultant may request a review of the appraisal by Qualification Committee (QC).

·  Effective September 1st, 2015 the Qualification Committee Procedures and the Procedures for Processing the Consultant’s Infraction Report have been revised.

1

Consultant Performance and Selection System (CPSS)

·  Effective January 1, 2001, the ministry modified its consultant selection procedures to take past performance into account at the Expression of Interest (EOI)/Phase I and at the Request for Proposals / Quotation (RFP/RFQ) stages. The objective is to improve quality of Engineering and Construction Administration services received and thereby reduce road user costs and infrastructure life cycle costs. The revised consultant selection system is referred to as the Consultant Performance and Selection System (CPSS).

Corporate Performance Rating (CPR)

·  Past performance is measured by a consultant's Corporate Performance Rating (CPR), which is the weighted-average of a consultant's appraisals over the last three years.

·  The CPR of a consultant firm is calculated, by the following equation:

CPR = 3(Avg.Yr. 1) + 2(Avg. Yr. 2) + 1(Avg. Yr.3)

6

Avg Yr.1 = Average of all appraisals within the most recent 12 months

Avg Yr.2 = Average of all appraisals in 12 months prior to Year 1

Avg Yr.3 = Average of all appraisals in 12 months prior to Year 2

·  A consultant’s Corporate Performance Rating (CPR) is available for Planning, Engineering, Construction Administration (CA), Area Materials Testing and Small Value Assignments appraisals. A Small Value Assignment is an assignment with Acquisition Value of $25,000 or less.

·  If a firm does not have any performance appraisals in one or two of the three years, the formula will self-adjust accordingly, to calculate the CPR based on the year(s) for which performance appraisals are available for the firm.

·  The ministry's Registration, Appraisal and Qualification System (RAQS) automatically calculates CPR on a quarterly basis, for each consultant, using past performance appraisals available (at the end of the day, on the first Business Day of January, April, July and October of a year).

·  Only “approved” performance appraisals are included in the CPR calculation. An appraisal is "approved" if the consultant signs off the Performance Appraisal Form or does not respond within the 21-day time limit (to request a formal review). In case of a request by a consultant for a formal review, the appraisal is not considered approved until the completion of the regional manager review stage or the Qualification Committee review stage, depending on how far the consultant chooses to proceed with the review.

·  At each CPR calculation, performance appraisals with the approval date older than the past thirty-six months are dropped from the calculation.

·  For calculating one of the above CPRs, the corresponding appraisals are applied. For example, only CA appraisals are be used to calculate the CA CPR of a firm.

·  A consultant’s CPR based on Planning, Engineering, CA, Area Materials Testing or Small Value Assignments appraisals is applied for selection for the corresponding assignment type. The ministry staff is responsible to upfront identify the assignment type. Where an assignment includes more than one Grouping, a joint CPR is applied for consultant selection. For example, for a Total Project Management (TPM) assignment, which includes Engineering and CA, a joint CPR based on both Engineering and CA appraisals will be used for selection. To allow CPR application for both the stand-alone and joint assignments, the following quarterly CPRs are made available in RAQS, as:

-  Separate CPR’s for Planning, Engineering and CA.

-  Joint CPR for Planning and Engineering.

-  Joint CPR for Engineering and CA.

-  Joint CPR for Planning, Engineering and CA.

-  Separate CPR for Area Materials Testing Assignments.

-  Separate CPR for Small Value Assignments.

Starter CPR

·  If a firm does not have any past appraisals in the Planning, Engineering or CA Groupings, the firm is assigned a “Starter” CPR for that Grouping. A “Starter” CPR for a Grouping is equal to the ministry wide average of all approved appraisals in that Grouping. A Starter CPR is calculated every quarter and may change from quarter to quarter.

·  A Starter CPR is available to a new registered firm or to a registered firm venturing into a new area. A Starter CPR allows a registered firm with no past appraisals to compete for assignments.

·  A Starter CPR is no longer available once a firm has one approved appraisal in RAQS in that Grouping. The firm’s CPR will then be based on their approved appraisal(s) in the Grouping.

Performance Appraisals

·  When a consultant assignment is completed, an appraisal will be completed for the prime consultant / consultant who has signed the legal agreement with the ministry. Appraisals will not apply to sub-consultants.

·  In the case of consortiums or legal partnerships, one overall performance appraisal rating for the assignment will be completed and the rating will apply to each member of the consortium or partnership.

·  When joint assignments involving Planning and Engineering are completed, a consultant will receive two performance appraisals, one for the Planning and the second for Engineering. For a Total Project Management assignment involving Detailed Design & Contract Administration (CA), a consultant will receive two separate performance appraisals, one for the Detailed Design and the second for the Contract Administration. Those appraisals, when approved, will apply to calculate the corresponding CPRs for a firm.

·  For multi-year assignments, an annual appraisal will be completed near the end of each year, to provide feedback to the consultants until the final appraisal is completed.

·  Where both interim / annual and final appraisals are approved and available in RAQS for the same assignment, only the final appraisal will be used to calculate CPR.

·  For assignments of duration two years and greater where RFP or RFQ is issued on/after October 1, 2007, Interim, Annual and Final Appraisals for assignments will apply in CPR and will be subject to the Formal Review/Appeal and Approval process, if requested by a consultant.

-  For an assignment of duration two years or greater but less than two and half years, an Interim Appraisal will be completed at 12 months and Final Appraisal within sixty days of completion of the assignment.

-  For a multi-year assignment, Annual appraisal will be completed near the end of each calendar year until the Final appraisal is completed. Annual and Final Appraisals issued will be subject to “approval” and included in CPR calculation.

-  For assignments of duration less than two years, only Final Appraisal will apply in CPR and will be subject to Formal Review/Appeal process.

-  For all assignments issued prior to October 1, 2007, only Final Appraisal will apply in CPR and will be subject to Formal Review/Appeal process.

·  Where any “approved” interim and final appraisals are available in RAQS for the same assignment, the final appraisal will supersede any earlier interim appraisal and used to calculate CPR. In the case of annual and final appraisals approved in RAQS, the later annual or final approved appraisal will supersede any earlier annual appraisal(s).

·  An approved Post-Construction Engineering Appraisal will apply in the calculation of Engineering CPR of a consultant as one of the appraisals together with other approved Engineering appraisals. A Post-Construction Engineering Appraisal will apply for the period of three years in the above CPR formula and will be dropped after three years.

Consultant Joint Ventures

·  A Consultant Joint Venture is treated as an Equal Partnership of two or more consultant firms.

·  The consultants wishing to undertake an assignment as a Joint Venture are to provide a Letter of Intent jointly signed by Senior Partner / CEO of each firm, as part of their EOI/Phase II submission. The Letter of Intent states:

-  The firms will be working as Joint Venture for the purposes of the assignment.

-  The Specialties / areas of work that each individual firm will be responsible for.

-  At least one of the firms meets the RAQS Registration Requirements for Prime Specialty.

-  Identifies the lead firm and Ministry’s contact for the purposes of an assignment.

·  Each firm must submit the declarations at EOI/Phase I Stage that have no conflict of interest with the assignment.

·  At the RFP/Phase II Stage, each firm must declare that they have no conflict of interest with the assignment, are tax compliant with the Ministry of Finance. Also each firm must provide the proof of insurance.

·  For the purposes of an assignment, the ministry will deal with the Lead Firm as the ministry’s contact.

·  For the purposes of an assignment, CPR for a Joint Venture is calculated using the approved appraisals of all firms in a joint venture and applying the formula.

·  At the award of an assignment, Agreement will be signed by and will be binding on all the firms in a joint venture. All Provisions and obligations of Legal Agreement will apply equally to each joint venture firm for the purposes of the assignment.

CPSS Application at the Expression of Interest (EOI)/Phase I Stage

·  The EOI/Phase I submissions and past performance (CPR) each are scored out of 100 points.

·  The highest EOI/Phase I score and the highest CPR are each assigned a score of 100 and the other values are pro-rated lower

·  Weights of 50% each are applied to both the EOI/Phase I and past performance scores.

·  The weighted EOI/Phase I and past performance (CPR) scores are added, and the total is used to rank the applicants. Based on the ranking, the ministry will short-list three to five consultants for the next step.

An example of application of CPSS at the EOI/Phase I stage is illustrated below:

EOI/Phase I Submission Scores

Firm / A / B / C
Project Organisation Score (Max. Score 20) / 10 / 14 / 17
Project Team Experience & Performance Score
(Max. Score 80) / 55 / 48 / 58
Total / 63 / 62 / 75
Score out of 100 / 84.00 / 82.67 / 100.00
Weighted Score (50%) / 42.00 / 41.34 / 50.00

Past Performance Scores

Corporate Performance Rating (CPR) / 3.70 / 3.20 / 2.80
Score out of 100 / 100 / 86.49 / 75.68
Weighted Score (50%) / 50.00 / 43.25 / 37.84

Total Weighted Scores

Total Weighted Score / 92.00 / 84.59 / 87.84

Ranking

/ 1 / 3 / 2

11

CPSS Application at the Request for Proposals (RFP)/Phase II Stage

·  The RFP/Phase II submissions, CPR and prices are scored out of 100 points max for each.

·  The highest RFP/Phase II score, the highest CPR and the lowest price each are assigned a score of 100 and the other values are pro-rated lower.

·  Effective July 28, 2006, the following weights of technical scores, past performance (CPR) and price apply for selection:

EOI: Technical Score = 50%, Past Performance (CPR) = 50%

RFP: Technical Score = 40%, Past Performance (CPR) = 50%,

Price = 10%

RFQ: Past Performance (CPR) = 50%, Price = 50%

·  The weighted RFP, past performance (CPR) and price scores are added, and the total is used to rank the applicants. The highest Total Weighted Score wins the assignment.

An example of the application of CPSS with the revised weights at the RFP/Phase II stage is illustrated below:

Firm / RFP
Tech.
Score / Score out of 100 / Wght
Score
(40%) / Perf.
Rating
(CPR) / Score out of 100 / Wght.
Score
(50%) / Price / Score out of 100 / Wght
Score
(10%) / Total Score / Rank
A / 635 / 100.00 / 40.00 / 3.60 / 94.74 / 47.37 / 78000 / 73.08 / 7.31 / 94.68 / 1
B / 505 / 79.53 / 31.81 / 3.00 / 78.95 / 39.48 / 57000 / 100.00 / 10.00 / 81.29 / 4
C / 552 / 86.93 / 34.77 / 3.20 / 84.21 / 42.11 / 69250 / 82.31 / 8.23 / 85.11 / 3
D / 575 / 90.55 / 36.22 / 2.90 / 76.32 / 38.16 / 99130 / 57.50 / 5.75 / 80.13 / 5
E / 545 / 85.83 / 34.33 / 3.80 / 100.00 / 50.00 / 94000 / 60.64 / 6.06 / 90.39 / 2

If the Total Scores for two or more consultants are tied, the consultant with the lowest price wins the assignment.

CPSS Application at the Request for Quotation (RFQ) Stage

·  The CPR and price are scored out of 100 points maximum for each.

·  The highest CPR and the lowest fee each are assigned a weight of 100 and the other values are pro-rated lower.

·  Weights of 50% each are applied to the CPR and price scores and the scores are converted.

·  The weighted past performance (CPR) and price scores are added, and the total is used to rank the applicants. The highest Total Weighted Score wins the assignment.