A group decision support system for RFP evaluation
A strategic solution for purchasing organizations
Complete eRFP Tutorial Available at:
Bringing RFP Evaluations Into The 21st Century
Paul J. Brennan, CPPB, C.P.M.
For years, those in the procurement field, both public and private, have heard the expectations of how our profession would dramatically change from the implementation of e-procurement technology. While most of those in the procurement profession will certainly agree that our jobs today have changed dramatically over the past five years; it remains a fact that many of the processes involved in source selection have seen little change or have remained basically the same.
While most of the Enterprise Resource Planning Software and Procurement Software that is available today has embraced “e-procurement” by offering E-Bidding, Bid Tabulation, Automatic Supplier Notification, and other new features, most if not all of the software providers have not provided an automated solution to the evaluation of Request for Proposals.
As more and more government organizations move away from the “low bid or low cost approach” to contracting, the Request for Proposal process is becoming the preferred method of source selection to identify the “best qualified” or “best value” proposal.
Rockland County, New York, like most other government agencies had seen their use of the Request for Proposal increase significantly over the past five years. Already issuing over 225 Competitive Bids on an annual basis, Rockland began to see the number of Requests for Proposals issued each year reach an average of 50 – 60. Along with the increased use of the RFP came many operational issues including, how to manage multiple evaluations being conducted at the same time, how to ensure thorough, independent and objective evaluations by multi-disciplinary teams, how to take evaluation data from paper rating forms and roll that information up to meaningful reports that can assist the evaluation teams in making informed decisions, and how to document the decision making process so that the source selection decision can be justified to administrators and elected officials.
Rockland County determined that it needed an automated group decision support system to help manage the RFP evaluation process. With no knowledge of such software currently available, the County issued a Request for Information, which was advertised nationally to attempt to locate such a software product. The County received many responses with offers to build such a system from scratch; and only two responses from software providers that had similar software available or in development.
One of the responses came from a software provider named Sysoft for a product called eRFP. eRFP was designed as a group decision support system for RFP evaluations. At the time of Sysoft’s response to the County’s Request for Information, the design of the eRFP software was ninety-five percent complete but had yet to be beta tested. The timing of our RFI could not have been better as the County identified a new software product as well as a software provider that was looking for a government agency to partner with to help beta test the software and identify any design changes needed based on using the system to evaluate actual government projects. The result was a contract that benefited both the County as well as Sysoft.
Since implementing the eRFP evaluation system the County has come to rely on the use of this on-line evaluation tool to make better sourcing decisions. The system is database driven and uses the most current web-based technologies. The software is deployable through an organization’s Intranet or through the Internet. Internet access allows evaluators in many different locations to easily evaluate the same proposals. Since the software is web-based, it also allows evaluators access from their home computers. Allowing evaluators to work from home while evaluating proposals can lead to more in-depth evaluations, usually in a shorter timeframe.
Rockland County’s experience has been very positive from the beginning of this project. There have been the normal bumps in the road or “bugs” that are normally experienced when one is participating in a software development project. The use of a web-based software product has enabled Sysoft to respond quickly to any software issues that arose during this project.
From a procurement management standpoint, the eRFP software has permitted us to track the progress of each evaluator, capture evaluator’s comments for use at caucus meetings, identify strengths and weaknesses of each proposal, score proposals in a more objective and less subjective manner, and produce detailed reports that assist in the decision making process.
Using the eRFP software has given the County the ability to easily identify the “best qualified” firm or proposal. By ensuring the selection the most qualified firms, the County anticipates that it will realize cost savings by having projects completed on time and within budget.
The eRFP software has highlighted the various effects of using “Cost” as an evaluation category and how the use of cost as an evaluation category may skew evaluation results; resulting in lesser qualified firms or proposals being selected. This software has also enabled the County to begin including “Risk Assessments” as part of the evaluation process. Proposals are now evaluated in relation to Risks associated with the proposed approach or Statement of Work; Risks associated with the proposed schedule, Risks associated with the proposed personnel; and Risks associated with the financial capability of the firm to complete the project.
Using the software has also helped the County assess the strengths and weaknesses of evaluators, and has helped identify evaluators who require additional training as well as those who may need to be removed from future consideration as an evaluator.
From the evaluator’s point of view the eRFP software gives them the ability to enter their evaluation on-line. The software guides each evaluator through an organized set of on-line forms with pre-defined evaluation criteria and standards. Evaluators have access to the relevant sections of the RFP that was issued for the project without having to revert to a paper copy of the RFP. Evaluators can record strengths, weaknesses, deficiencies, and statements regarding their rating of each evaluation criteria.
The use of this software has helped provide a more objective evaluation process by having the evaluators complete the evaluation with no knowledge of the scoring formulas, and by asking for comments in relation to each rating they assign. When all of the evaluators assigned to a project have completed their evaluations, rating scores and comments are automatically consolidated into a detailed summary report that is used during a caucus meeting to develop a short list of the best proposals for further consideration. The short listed firms are normally invited to make formal oral presentations. After each oral presentation is completed, evaluators log back onto the eRFP system to score and provide comments on the oral presentation. At this time the evaluators also complete the Risk Assessment of each of the short-listed proposals.
The final evaluation report combines the scores of the written proposal evaluation, the oral presentation evaluation, and the risk assessment evaluation.
The software documents the entire evaluation process, from project set-up and the selection of evaluation criteria, to the final award recommendation. The software captures all notes generated during the evaluation process. This documentation helps justify the selection of the most qualified firm or proposal, especially when the cost is higher than other proposals.
In summary, the use of software to facilitate the RFP evaluation process has enabled Rockland County to reduce the time and cost of proposal evaluations, reduce the subjectiveness of evaluations, provide a complete audit trail, eliminate paper intensive processes, and increase evaluator efficiency.
Sysoft eRFP proposal evaluation system automates Request for Proposal evaluations. Although a seemingly straightforward management responsibility, the task of managing the evaluation process for request for proposals requires careful administration and the tracking of a substantial amount of information. If the evaluation for request for proposals is mainly a paper driven process, it makes the collection of evaluation data and the development of reports a very time consuming task. Sysoft eRFP automates the source selection process for various types of contracts including but not limited to: Service Contracts, Rental and Lease Contracts, Professional Service Contracts and Software and Technology Contracts. The software assists in providing quality evaluations, complete and traceable documents and timely contract award decisions.
- Database driven and Web based
- Deployable within Internet, Intranet and Extranet
- The ability for evaluators to enter their evaluation online
- The ability to view progress and change task assignments of evaluators
- The ability to assign risk assessments
- Provides on-line access to relevant acquisition documents
- Provides electronic evaluation forms with pre-loaded evaluation criteria and standards
- Guides the evaluator through an organized set of on-line forms to create clarifications, deficiencies, ratings, rationale and impact statements
- Ability to store proposal strengths and weaknesses in a notepad area
- On-line access to all evaluations allows source selection officials to monitor progress and provide feedback
- Ratings are automatically consolidated
- During caucus sessions, individual ratings and evaluations can be viewed on a projector screen
- Consensus ratings are generated on-line
- Standard and customized reports can be printed throughout the process
- Easy evaluator assignment
- Email notification for evaluation deadlines and automated email/Fax capability
- Reduce the time and cost of proposal evaluations
- Helps reduce subjectiveness of evaluations
- Availability of complete audit trail
- Eliminates paper intensive processes
- Increases evaluator efficiency
- Instant on-line monitoring of the evaluation process
- Facilitates more precise evaluations
- Ensures complete traceability of evaluations
- Availability of real-time interim and final reports
- Availability of various cost and other analytical models
Experience of Rockland County - New York
Management
The most significant benefit of using eRFP is that it provides the ability to easily identify the “best qualified” firm or proposal. By selecting the most qualified firms or proposals, agencies should realize cost savings by having projects completed on time and within budget.
eRFP helps show the various effects of using “Cost” as an evaluation category and helps an agency to determine if cost should be a weighted evaluation category. The system shows how the use of cost as an evaluation category can skew evaluation results; resulting in lesser qualified firms or proposals being selected.
Management gains the ability to track the progress of the evaluators; which helps meet target dates to complete the evaluations. In general the use of the system assists an organization in keeping the evaluation process on schedule.
The system can also help management assess the strengths and weaknesses of the evaluators, and helps identify evaluators who require additional training, and those who should be removed from evaluation teams.
Better Evaluations
The eRFP system provides a more objective evaluation process by having the evaluators complete the evaluation with no knowledge of the scoring formulas, by asking for comments in relation to each evaluation criteria, and by providing detailed summary reports for use at evaluation team meetings.
The system allows each evaluator to easily score and enter comments for each proposal.
Since the system is web-based, it allows access from multiple locations, including giving evaluators access from their home computers. Allowing evaluators to work from home while evaluating proposals can lead to more in-depth evaluations; usually in a shorter timeframe.
Documentation
The eRFP system documents the entire evaluation process, from project set-up and the selection of evaluation criteria, to the final award recommendation. It gives the evaluation team the ability to enter notes regarding a specific proposal, or the evaluation process in general.
This documentation helps justify the selection of the most qualified firm or proposal, especially when the cost is higher than other proposals.
This documentation also easily identifies the area in which a proposal is weak, and helps eliminate proposals that may be very weak in one key area.
Process Customization
The eRFP system allows customization of the RFP evaluation processes. A library of templates (managed and maintained by the eRFP configurator) makes the eRFP system flexible to handle any RFP process. It also allows evaluation of any type of RFP.
eRFP Process
eRFP Architecture
eRFP – Evaluator Task Assignment
You can assign tasks to evaluators and change the task assignment, if required at anytime.
eRFP – View Progress
You can view progress of the RFP evaluation process at anytime to ensure completion before the caucus meeting.
/ County of USADepartment of General Services - Purchasing Division
eRFP Evaluation System /
Written Proposal Evaluation Report
Date: 1/8/2004 Time: 3:26:17 PM
Project Name: RFP-CR-003 Technical Assistance Project
Project Number: 0000000144
Upset Level Imposed: Yes
Selected Cost Model: Competitive Range Analysis
Proposal: ATI / Number of evaluators who completed initial evaluations: 5 out of 5 assigned
Proposal: KRL and Associates / Number of evaluators who completed initial evaluations: 5 out of 5 assigned
Proposal: MultipleSystems Inc. / Number of evaluators who completed initial evaluations: 5 out of 5 assigned
Proposal: Reliable Systems Inc. / Number of evaluators who completed initial evaluations: 5 out of 5 assigned
Initial Written Proposal Evaluation Ranked by Overall Score
Proposal: MultipleSystems Inc.
Overall Score=3.35Proposal: ATI
Overall Score=3.32Proposal: Reliable Systems Inc.
Overall Score=2.72Proposal: EAK Inc.
Overall Score=2.24Proposal: KRL and Associates
Overall Score=2.21Overall Caucus Comments:
Proposal: MultipleSystems Inc.
Shortlisted / Proposal / Initial Evaluation of Written ProposalYes
/ MultipleSystems Inc. / Technical CriteriaWeight=40%
Weighted Score:
/ Personnel Criteria
Weight=30%
Weighted Score:
/ Past Performance
Weight=30%
Weighted Score:
/ DBE/MBE Criteria
Weight=0%
Weighted Score:
/ Cost Criteria
Competitive Range Analysis
Caucus Comments:
Technical Criteria
- Criteria: The Proposal showed a full and complete understanding of the need(s) or problem the proposed solution will address
Evaluator: Sam Adams
Comment: Solution anticipates turning over to County at some point for maintenance. County has little resident expertise in this area.
Evaluator: Kara Blasco
Comment: MultipleSystems Inc. and their partner companies are apparently very aware of the needs of this type of project. However, there is one area in which their approach differs from what we envision. Repeatedly, there is reference to a single or "the" technology vendor. We see the very strong likelihood that the best final product will be realized by acquiring technology from a variety of vendors, picking and choosing the items that are best for us and the vendor that seems best suited for our needs. I would see this as one of the most important functions and strangely out of place in an otherwise amazingly detailed and comprehensive proposal.
- Criteria: The proposal showed knowledge of potential technical, regulatory, economic, environmental, or other issues that will be addressed by the technical approach
Evaluator: Kara Blasco
Comment: Obviously the Multiplesystems Inc. team, especially the Principle in Charge John Poloski, is on the leading edge of the industry edge of the industry's effort to establish technical and regulatory guidelines that this rapidly developing application of ITS technology to transit requires. Their proposal provides a extensive reference to the process of regulatory development, the base from which it is developing and the progress to date. John Poloski has been and still is a prominent figure in this effort.
This report consists of 30 more pages. For the purpose of this brochure the report is truncated here.
/ County of USADepartment of General Services - Purchasing Division
eRFP Evaluation System /
Oral and Risk Summary Report
Project Name: RFP-CR-003 Technical Assistance Project
Project Number: 0000000144
Proposal: ATI
Number of evaluators who completed Oral Evaluations: 4 out of 5 and
number of evaluators who completed Risk Evaluations: 5 out of 5
Proposal: MultipleSystems Inc.
Number of evaluators who completed Oral Evaluations: 4 out of 5 and
number of evaluators who completed Risk Evaluations: 5 out of 5
Ranking Based on Oral Evaluation Score
Bottom of Form
Proposer's Name: ATI
Oral Evaluation Score:12.50 Risk Evaluation Score:8.25 Less Risk Score (MAX Risk - Risk EVAL Score): 27.75
Proposer's Name: MultipleSystems Inc.
Oral Evaluation Score: 10.00 Risk Evaluation Score: 12.25 Less Risk Score (MAX Risk - Risk EVAL Score): 23.75
Top of Form
Overall Caucus Comments:
Proposer's Name: ATI
Oral Evaluation Score: 12.50 Risk Evaluation Score: 8.25