TEMPO Guidelines for Reporting Evaluation Results

European Commission - DG-TREN Evaluation of VMS

TEMPO Secretariat

Evaluation of VMS deployment on Slovenian part of corridor V

Evaluation_VMS V1 Page i Prepared: 20.3.2009

European Commission - DG-TREN Evaluation of VMS

TEMPO Secretariat

Document details:

Document Control Record:

Table Of Contents

Evaluation_VMS V1 Page iii Prepared: 20.3.2009

1. Key Evaluation Results 2

1.1 Impact on Traffic Flow 2

1.2 Impact on Increased Safety 2

1.3 Impact on Environment 3

1.4 Other Key Results 3

2. Description of the Problem 5

2.1 Site 5

2.2 Issues Addressed 6

3. Description of the ITS Project 7

3.1 Project Domain 7

3.2 Key Words 7

3.3 Objectives 7

3.4 Systems and Technologies Applied 8

3.5 Status of the Project 11

4. Evaluation planned 12

4.1 Evaluation planned 12

4.2 Objectives for the Evaluation 12

4.3 Research Questions 12

4.4 Study Area for the Evaluation 12

4.5 Expected Impacts 12

4.6 Expected Methods 13

5. The Impact of the Project - Results 14

5.1 Technical Performance 14

5.2 Results 14

5.3 Reliability of Results 16

5.4 Research Questions Answered 16

5.5 Overall Assessment 16

6. European Dimension: Transferability of the Results 19

Annex 1: Technical Annex 20

Data Collection Methods 20

Results of collected data 23

List of Figures

Figure 1: Highway sections Vransko – Blagovica and Blagovica – Šentjakob 5

Figure 2: Highway section Ljubljana Kozarje - Razdrto 6

Figure 3: VMS portal 8

Figure 4: VMS semi-portal 8

Figure 5: VMS traffic sign 9

Figure 6: VMS variable signpost signalization 9

Figure 7: TMS 10

Figure 8: SCADA system 10

Figure 9: SCADA system 11

List of Tables and Graphs

Table 1: Key words used to describe the nature of the project and the applications used 7

Table 2: The results of the survey performed on the treated sections 23

Table 2a: The results of the survey performed on the treated sections 24

Table 3: Overview of the number of traffic accidents per year and by „Classification of the accident according to the consequence“ (Database of traffic accidents-PN, Road section Ljubljana Kozarje – Razdrto) 26

Table 4: Overview of the number of traffic accidents per period and by „Classification of the accident according to the consequence“ (Database of traffic accidents-PN, Road section Ljubljana Kozarje – Razdrto) 26

Graph 1: Overview of the number of traffic accidents per year and by „Classification of the accident according to the consequence“ (Database of traffic accidents-PN, Road section Ljubljana Kozarje – Razdrto) 26

Table 5: Overview of the number of injured persons per year and by „Classification of the person hurt according to the consequence“ (Database of persons, involved in traffic accidents-PNO, Road section Ljubljana Kozarje – Razdrto) 27

Table 6: Overview of the number of injured persons per period and by „Classification of the person hurt according to the consequence“ (Database of persons, involved in traffic accidents-PNO, Road section Ljubljana Kozarje – Razdrto) 27

Graph 2: Overview of the number of injured persons per year and by „Classification of the person hurt according to the consequence“ (Database of persons, involved in traffic accidents-PNO, Road section Ljubljana Kozarje – Razdrto) 27

Table 7: Overview of the number of traffic accidents per year and by „Classification of the accident according to the consequence“ (Database of traffic accidents-PN, Road section Vransko – Blagovica) 28

Table 8: Overview of the number of traffic accidents per period and by „Classification of the accident according to the consequence“ (Database of traffic accidents-PN Road section Vransko – Blagovica) 28

Graph 3: Overview of the number of traffic accidents per year and by „Classification of the accident according to the consequence“ (Database of traffic accidents-PN, Road section Vransko – Blagovica) 28

Table 9: Overview of the number of injured persons per year and by „Classification of the person hurt according to the consequence“ (Database of persons, involved in traffic accidents-PNO, Road section Vransko – Blagovica) 29

Table 10: Overview of the number of injured persons per period and by „Classification of the person hurt according to the consequence“ (Database of persons, involved in traffic accidents-PNO, Road section Vransko – Blagovica) 29

Graph 4: Overview of the number of injured persons per year and by „Classification of the person hurt according to the consequence“ (Database of persons, involved in traffic accidents-PNO, Road section Vransko – Blagovica) 29

Table 11: Overview of the number of traffic accidents per period and by „Classification of the accident according to the consequence“ (Database of traffic accidents-PN, Road sections together) 30

Table 12: Overview of the number of injured persons per period and by „Classification of the person hurt according to the consequence“ (Database of persons, involved in traffic accidents-PNO, Road sections together) 30

Glossary

The STREETWISE project has developed a common glossary, based on previous work in the CONVERGE project. The Evaluation Expert Group has agreed that this glossary should be used by projects to assist comparisons between them by ensuring that common terminology is used when summarising projects and presenting evaluation results.

Telematics (or ITS) Application

A telematics tool designed to solve a specific problem or problems.

Appraise

To appraise is to determine the performance of a telematics system X by assessing individual telematics applications at regular intervals.

Assessment

The process of determining the performance and/or impacts of a candidate telematics application, usually in comparison to a reference Case. This would include assessment of individual aspects of an application’s performance, for example it environmental performance or technical performance.

Assessment Objectives

A precise statement of the objectives, which a telematics application aims to achieve. The performance on a telematics application can be reviewed against these objectives, which will be associated with performance indicator(s) and definitions of success.

Assessment Categories

Assessment Objectives are classified under different Assessment Categories. These categories relate to general National or International goals on which transportation projects can be assessed, for example the impact on the environment or on safety and danger to people. ‘CONVERGE’ suggests 6 different Assessment Categories, while the UK Government judges performance of transport initiatives against 5 broad Assessment Categories.

AST

The Assessment Summary Table records the degree to which the five UK Central Government Assessment Objectives for transport are achieved and provides a summary of the impacts of the scheme or initiative in question. A separate AST must be prepared for each specific transportation scheme option or initiative considered, including options later rejected.

Business Case

A Business Case (BC) is a structured approach to obtaining financial approval for a project. It defines the project, it presents the current situation and the anticipated situation after project implementation. It looks to justify the investment, based on experience elsewhere with similar telematics applications and modelling. The business case usually constitutes the financial assessment for a telematics application.

Depending on the level of knowledge available about a telematics application, the BC will be more or less detailed. For a new telematics application or strategy not yet employed the BC is likely to be detailed, whereas for further deployment of a proven application the BC will be much less detailed referring to previous work. Similarly the level of detail will depend on the scale of the project with justification of a €1M scheme requiring a smaller business case than the justification for a €200M scheme.

Cost-Benefit Analysis

This is a mechanism widely used within business cases for financial assessment of the net benefit of implementing a telematics application. It determines the ratio (or difference) between the costs and benefits of an application that can be expressed in monetary units.

Evaluation

The process of determining the value of a telematics system or individual telematics application in comparison with alternative s and/or with reference to a "base case".

Pre-Implementation Evaluation (often known as ex-ante evaluation) concerns the evaluation of a telematics application prior to its implementation. In this instance, expected benefits are anticipated, based on modelling techniques and possibly previous experience of similar applications in similar locations.

Post-Implementation Evaluation (often known as ex-post evaluation) is carried out after a telematics system or application has been implemented, to compare observed results following implementation with expected results of the pre-implementation evaluation.

Feasibility Study

This is a study usually carried out at an early stage of any project to determine the practicability of a implementing a telematics application or system. They tend to be small-scale studies to determine whether or not to proceed to a pilot study or to put together a full business case. The outcomes of the study are assessed against reference figures and a recommendation on the best way to proceed is made.

Evaluation Framework

This is an outline plan applicable to all projects that can form the basis for the appraisal of a telematics system or the assessment of individual applications. It will contain assessment objectives.

Evaluation Guidelines

Set of advice highlighting how to use an evaluation framework in order to appraise a telematics system or assess a telematics application. They may relate to pre and/or post evaluation.

Impacts

Changes or effects brought about by the installation and/or use of a telematics application in an experimental or real situation.

Implementation

The installation and commissioning of a telematics application in a real life environment.

Indicators

Parameters, directly measured or derived from measurements or modelling, indicating the performance and/or impacts of a candidate system.

Monitoring

This refers to the process of collecting data aimed at assessing the performance of a particular aspect or aspects of a telematics application. Monitoring is often referred to as “pre implementation” or “post implementation” monitoring. Pre-implementation monitoring is carried out prior to implementation of a telematics application, to establish the existing situation. Post implementation monitoring is use to assess the impacts of the telematics application in question.

Multi-Criteria Analysis

Different criteria are combined to determine a single deciding value. The weighting factor applied to each item of analysis is a matter of decision and agreement and is particular to the specific case and circumstances in question and results are not readily transferable.

Off Road Trial

These are trials where a telematics application or system is installed in a situation off the national road network to assess the technical performance of the application/system. Off road trials are also used to assess various different strategies and assess the driver response to them. These trials are often used where there is a possible danger to motorists on the main road network with an untried and tested application.

Pilot Study

This is an on-road trial of a new application which may or may not have been initially tested in an off-road trial. Pilot studies are often used to test the technical feasibility of a telematics application, and/or to carry out a preliminary assessment of the impacts of a new telematics application. The focus of a pilot study is on the assessment of the technical performance and impacts of the application.

Roll Out

This term refers to the further implementation of a telematics application, which has been studied, researched, and is already implemented on some other sites.

Testing

Before a telematics application or system is fully implemented, testing to prove the efficacy of the system is required. Testing will be required to ensure the system operates according to the design specification.

Validation

This is the process of verifying that an application performs as expected. Validation can occur throughout the evaluation process with, for example, post-implementation evaluation results used to validate expected benefits generated at the pre-implementation stage, or, at the pre-implementation evaluation stage where a model used to predict likely benefits would need to be validated to ensure that the results produced were realistic.

Abbreviations

European Commission - DG-TREN Evaluation of VMS

TEMPO Secretariat

Aim of the Guidelines (version 6), based on which this evaluation was prepared

These guidelines are designed to help those evaluating projects in EasyWay to prepare and present the results of their work on evaluating the implementation of ITS projects in a common format.

It is important to recognise that projects in EasyWay are funded by Member States as well as by the European Commission, and that some Member States will have their own requirements for presenting evaluation results. However, adoption of this common format for presenting results will help to ensure that results of similar implementations can be compared, assisting in the transfer of results between sites and providing European Added Value. It will also ensure that a periodic report can be prepared by the EasyWay Evaluation Expert Group (EEG) of progress towards achieving EasyWay’s objective of reducing congestion and emissions from transport and improving safety.