Personal travel planning projects in England
/ This is a pre-publication version of the following article:Chatterjee, K. (2009). A comparative evaluation of large-scale personal travel planning projects in England. Transport Policy, 16(6), 293-305. /
A comparative evaluation of large-scale personal travel planning projects in England
KironChatterjee
Centre for Transport & Society
University of the West of England, Bristol, UK
WORD COUNT: 8,150 (not including abstract, acknowledgement, references, tables and figures)
Abstract
Findings are presented from a study assessing the effectiveness oflarge-scale, residential-based personal travel planning (PTP) projects in eight areas in England. The project evaluation results show consistent reductions in car driver trips with anaverage reduction of eleven per cent. The mode of travel which experiences the most substantial increase is walking with modest increases reported for cycling and public transport. Results have not been disaggregated according to the type of participation that individuals have had in projects, therefore it is not possible to identify how project designinfluences outcomes. Despite the consistency of outcomes reported and many aspects of good practice in project evaluations, there are some concerns about evaluation methodology, notably that the estimation of outcomes might be systematically biased. The main concerns relate to independence of evaluators, sample sizes and survey response biases. A priority in future project evaluations is to use independent evaluators and to collect aggregate-level travel data with which to corroborate survey-based results and enable monitoring of outcomes over longer time-scales. Another priority is to increase understanding of how design elements of PTP projects influence behavioural outcomes and to develop appropriate research methods to investigate this. Improved evaluations will better enable the value for money of PTP to be assessed relative to other investment options.
Keywords: voluntary travel behaviour change, personal travel planning, evaluation, travel survey
Acknowledgments
This paper draws largely on the evaluation findings from a research project conducted on behalf of the Department for Transport (DfT), UK. As such, the author would like to thank Jacqui Wilkinson and Daniel Barrett of the DfT, who were responsible for managing the project. The authoris grateful to his colleagues Jon Parker and, Lynsey Harris of Integrated Transport Planning, Richard Armitage of Richard Armitage Transport Consultancy, Jo Cleary of Cleary Stevens Consulting and Phil Goodwin of the University of the West of England, Bristol, for contributions to the project. Thanks are also due to the large number of individuals at case study organisations and on the project steering board and expert panel who provided input to the research. However, the views expressed in this paper are those of the author and do not necessarily represent the individual views of members of the research team, the organisations they represent, or the Department for Transport. Finally, the author would like to thank the anonymous referees for their valuable comments on the paper.
1. Introduction
Personaltravel planning (PTP) involves directly contacting individuals with the offer of information, assistance, incentives and motivation to enable them to voluntarily alter their travel choices. PTP is an example of a voluntary travel behaviour change (VTBC) measure and has also been referred to as personalised travel planning and individualised travel marketing. In Japan, PTP projects have been referred to astravel feedback programs.The concept of PTP has been deployed worldwide since the 1980s. Its proponents claim that PTP projects can reduce travel problems (e.g. congestion) without the need for expensive infrastructure projects or unpopular restraint measures while detractors claim that the impacts of PTP projects are modest and short-lived.
Small-scale trials of PTP took placein the UK between 1997 and 2002 before theDepartment for Transport (DfT) part-funded a set of PTP pilot projectsthat took place in 2003 and 2004. Seven of the pilots targeted householdsin residential areas, while six targeted employees in workplaces and two targeted pupils in schools. In addition, Transport for London (TfL), the transport delivery authority for Greater London,also funded four pilot projects at this time. In 2004 a DfT commissioned review was published on the experiences and potential for VTBC measures (Cairnset al., 2004). PTP was one of ten VTBC measures assessed. Drawing on UK and international experience, including interim results from the DfT and TfL pilot projects, the reviewfoundthat “personalised travel planning typically reduces car driver trips amongst targeted populations by 7-15% in urban areas”. After the DfT pilot projects were completed,it was concluded that the residential-based projects provided more consistent results, larger impacts and better value for money than the projects based on workplaces and schools,but that it is “…difficult to evaluate how easily the results from these pilots could be replicated in other areas across the country” (DfT, 2005).
Cairnset al.(2004) had notedconcerns about evaluation methodology and recommended that future opportunities for evaluation should be seized. They noted that ‘…monitoring of planned large-scale projects…as part of the Department for Transport’s Sustainable Travel Demonstration Towns project may help to provide more convincing evidence about the effects of such schemes.’(Cairns et al., 2004, p. 130). A contrasting opinion was put forward in a report to the Australian Government (AGO, 2006) which concluded that there is ‘…little further need to undertake major evaluations of household projects, as…data is in broad agreement’ and in Australia ‘larger household projects routinely show decreases in car use of 4 - 15%’.
A review of Japanese PTP projects (Taniguchiet al.,2007) found an average reduction in car use of 19% amongst participating individuals in residential PTP projects. However, the authors noted that projects in Japan have been at a smaller scale thanthose in Australia and UK – too small to assess their suitability as a practical transport policy tool. Möser and Bamberg (2008) recently re-examined international evidence on the impact of VTBC measures and, although their meta-analysis of 72 projects (including PTP, travel awareness campaigns and public transport marketing) yielded a mean increase of 11% in non-car trips (representing a mean change in the proportion of non-car trips from 34% to 39%), the authors were concerned about the methodological quality of the evaluations from which the data had come.
The possibility of examining emerging evidence from new PTP projects arose in 2007 whenDfT appointed a team, including the current author, to research the effectiveness of residential-based PTP in the UK and worldwide. Themain aims of the study were to identify the factors influencing effectiveness of PTP and to produce guidance for clients and practitioners on how to deliver effective PTP projects. The study involved detailed examination of eight case study areasin England withinvestigation of PTP project delivery processes and outcomes. This was achieved through interviews with project stakeholders and documentary analysis. Formal meta-analysis was not attempted due to the relatively small number of projects and the known heterogeneity of the projects(and their evaluations); rather, the intention was to use differences between project resultsto deduce the factors affecting PTP effectiveness. This paper summarises the findings of the study which relate to the evaluation evidence and processes. Wider research findings can be found in Parker et al. (2007) and practitioner guidance based on the study can be found in DfT (2008).
2. Theevaluation of PTP projects
PTP projects have generally been funded by public authoritiesand the evaluation of public projectsrequires systematic investigation of their effectiveness in achieving benefits for society and whether they represent good use of limited resources. The main benefits from PTP projects arise from changes in travel behaviour and the impact that these have on travel conditions, therefore a central part of the evaluation of PTP projects is the assessment of changes in travel behaviour. Although assessment of the costs of delivery of PTP projects is a major component of evaluation, this paper focuses on project benefits rather than the costs.
Impact evaluation is concerned with a project’s inputs (resources used), its outputs (directly attributable effects) and its outcomes (the short-, medium- and long-term final impacts that arise for society).An impact evaluation assesses whether intended outputs and outcomes have been achieved and whether a project provides good value for money. Since PTP projects have no physical outputs of the kind produced by infrastructure projects, the main emphasis is on evaluating behavioural outcomes.
Process evaluation is concerned with the way in which a project is implemented,and received by its target population, and is intended tohelp understand how and why certain outcomes were achieved. It involves examining whether the steps (or preconditions) that are needed for a project to reach its goals are achieved, or not, and the reasons for this. Process evaluation supports adaptive learning from experience and enables understanding to be gained which can assist the design of future projects. Previous comparative evaluations of PTP projects (Cairnset al., 2004; DfT, 2005; AGO, 2006; Taniguchiet al.,2007; Möser and Bamberg, 2008) have concentrated on project impacts with less analysis of the project implementation process and howthis affects impacts.
The evaluation of any PTP project needs to reflect its specific objectives. There may be a variety of motivations to commission a PTP project. For example, projects might be aimed at outcomes such as increasing the number of trips by public transport or decreasing the distance travelled by motorised vehicles.Local authorities in England are required to monitor a large number of performance indicators (DfT, 2004; CLG, 2008)of which a number of these could be conceived to be potentially subject to influence by PTP projects. Table 1 identifies these.
Table 1
Project sponsors mayalso have the objective of engaging as many members of the public in PTP projects,or achieving a high level of satisfaction of the public with the services offered to them in PTP projects.These can be considered as intermediate outcomes that arelikely to influence final impacts of a project.
It is clear that the motivations for PTP projects,and hence the performance indicators that might be usedin evaluations,are diverse. Comparative evaluations of PTP projects have focused onthe small number of indicators which arecommonly available from individual projectevaluations.The two indicatorsproviding the main focus have been mode share (proportion of trips made by different modes) and the number of car driver trips. The measurement of these indicators has generally been achieved through travel surveys of individuals in PTP project areas. The distance travelled by car has also been the subject of analysis in some PTP projects. There has been little attempt (in comparative evaluations) to assess aggregate-level outcomes such as traffic levels, congestion and public transport revenue. One exception has been attempts to estimate aggregate carbon emissions reductions from individually reported changes in distance travelled by car (AGO, 2006). Estimating aggregate-level outcomes draws attention to the need to take account of all changes in travel behaviour attributable to the PTP project – not just those by participants.
In any evaluation it is important that robust results are obtained. A number of criteria for achieving robust results are now introduced. Validityis a fundamental requirement of any evaluation. To achieve validity attention needs to be paid to reliability (ensuring that results would be repeatable if conducted again in similar circumstances), construct validity (ensuring that measurements are assessing what they are intended to measure),internal validity (ensuring that measured impacts are caused by intervention) and conceptual validity (ensuring theoretical explanation for impact is supported by data).Also important is the longevityof impacts.
Forevaluation results to apply to other persons, contexts, situations or times it is necessary for them to have external validity(be generalisable). It is also important that there isindependence of evaluators from project delivery to avoid the risk that results are biased towards a particular outcome.A final evaluation consideration is consistency of evaluation designsbetween projects. The adoption of consistent designs allows comparison and synthesis of results.
Even though PTP projects have been subject to substantial evaluation efforts (much more than most other transport measures), the robustness of PTP project evaluations has been subject of considerable debatewith a number of criticisms raised and rebuttals made. The reliability of project results has been questioned by, amongst others, O’Fallon and Sullivan (2003) who identify inadequate sample sizes and failure to report the sampling errors of results in past PTP evaluations.It has been suggested that use of self-report travel surveys to obtain travel behaviour data threatens construct validity due to response biases such as the under-reporting of trips in surveys (identified by Stopher and Bullock, 2003) and social desirability bias (identified by Bonsall, 2009a, as likely to be a particular problem in PTP evaluations).
The research designs usedin PTP project evaluations have been claimed to decrease internal validity. Möser and Bamberg (2008) suggest that single-group pre- and post- test designs(before and after surveys in project area without surveys incontrol areas) haveaffected the ability to draw causal inferences. Gärling and Fujii (2006)note the absence of the collection in travel surveys of ‘process’measures (referring to social-psychological measures such asbeliefs and attitudes)to supplement ‘effects’ measures (behaviour) and suggest that this limits the ability to explain any measured changes in behaviour, thus threatening conceptual validity.It is often noted that the longevity of PTP project impacts has not been demonstrated, althoughpublic transport boarding data obtained in projects in Australia and Germany has suggested that increased public transport use may be sustained up to four years after the PTP projects (Cairnset al., 2004).
Stopher and Bullock (2003) identify a threat to external validity arising from survey samples that are not representative of the population from which they are drawn.The survey samples they examined contained a lower proportion of large households than in the wider population. Their concerns about survey design and sampling led them toestimatethat the reduction in car use associated with IndiMarkPTP projects in Australiawas likely to be have been over-estimated by a factor of two. They also raise the concern that areas selected for PTP projects in the past are not typical of the population of areas which will need to be considered for future projects – furtherthreatening theexternal validity of results.Both of these assertions have been countered by those involved with the IndiMark projects (Roth et al., 2003).
The lack of independence of evaluators has been noted by several authors. Cairns et al.(2004) note that “the fact that those advocating the initiatives are sometimes also responsible for monitoring them... has led to a lack of confidence in conclusions amongst some professionals”.Möser and Bamberg (2008) raise an additional issue in their meta-analysis of VTBC measures. They identify the risk of reporting and retrievalbiases where projects for which results are available are systematically biased (potentially towards those that had favourable results).
The methodological quality of the project evaluations reviewed in this paper will be assessed based on the above considerations so that a view can be taken on the robustness of results and so that recommendations can be made for future evaluation practice.
3. The English case studies
The research study investigatedthe experience of local authorities that were making a significant commitment to PTPat the time of the study. The eight selected local authority areas were Brighton, Bristol, Darlington, Lancashire, London, Nottingham, Peterborough and Worcester. Table 2givesdetails of PTP projects conducted in the case study areas. It can be seen that the scale of the projects, measured by the number of households targeted, ranged from about 2,000 to 30,000 (with the projects being delivered in 2005 and 2006 being considerably larger than the earlier projects) and that, although most projects were associated with major towns and cities, they varied in terms of their locations within the urban area (inner, outer and edge locations), average income levels and other socio-economic characteristics. In some cases the PTP project was the only significant intervention in the locality whereas in others they were accompanied by physical or economic improvements to non-car options.
The objectives of the projects involve both intermediate outcomes (such as number of project participants) and final outcomes and impacts (such as modal share) – although outcome objectives were rarely identified in terms of specific targets.Many of the local authorities contracted Sustrans or Steer Davies Gleave (SDG)to deliver their projects.In Brighton and London the projects were managed in-house but with project support from SDG. There are a number of commonalities between theTravelSmart®[1]approach used by Sustrans and that used by SDG,but akey difference is that, in TravelSmart, participants are provided with a menu of information from which to select what they feel will be useful, whereasSDGuse travel advisorsto engage potential participants in a short door-step conversation during which they listen out for characteristics of travel needs and behaviour and key motivators in order to determine what types of message and information are likely to be relevant to that participant. The Nottingham Wollaton project, managed by the city council, was unique in not involving personal travel advisors or a menu of potentially useful items; letters were sent to residents with personalised travel information including bus stop-based timetables and a smartcard prepaid with one day’s free bus travel.
Projects varied in pre-project publicity, wider travel awareness campaign work and the amount of working with community groups and events to promote PTP. A particular feature of the projects in Darlington and Peterborough was the establishment of loyalty clubs designed to engage participants and encourage them to make pledges. These clubs also acted as a forum for motivational messages referring to the health benefits of walking and cycling.