PROJECT PERIODIC REPORT
Grant Agreement number: FP7-ICT-2007-1-215847
Project acronym: EU-ADR
Project title: Exploring and Understanding Adverse Drug Reactions by Integrative Mining of
Clinical Records and Biomedical Knowledge
Funding Scheme:Collaborative Project – Small or medium-scale focused research project
Date of latest version of Annex I against which the assessment will be made: 25/05/2011
Periodic report: 1st□ 2nd□ 3rd 4th □
Period covered: from01/02/2010 to 31/07/2011
Name, title and organisation of the scientific representative of the project's coordinator:
Prof.dr.Johan van der Lei, Erasmus Universitair Medisch Centrum Rotterdam
Tel: +31 10 704 3050
Fax: +31 10 704 4722
E-mail:
Project website address:
1
© Copyright 2011 EU-ADR Consortium
Declaration by the scientific representative of the project coordinator
I, as scientific representative of the coordinator of this project and in line with the obligations as stated in Article II.2.3 of the Grant Agreement declare that:- The attached periodic report represents an accurate description of the work carried out in this project for this reporting period;
- The project (tick as appropriate)[1]:
□has achieved most of its objectives and technical goals for the period with relatively minor deviations.
□has failed to achieve critical objectives and/or is not at all on schedule.
- The public website, if applicable
□is not up to date
- To my best knowledge, the financial statements which are being submitted as part of this report are in line with the actual work carried out and are consistent with the report on the resources used for the project (section 3.4) and if applicable with the certificate on financial statement.
- All beneficiaries, in particular non-profit public bodies, secondary and higher education establishments, research organisations and SMEs, have declared to have verified their legal status. Any changes have been reported under section 3.2.3 (Project Management) in accordance with Article II.3.f of the Grant Agreement.
Name of scientific representative of the Coordinator: Johan van der Lei
Date: 13/12/2011
For most of the projects, the signature of this declaration could be done directly via the IT reporting tool through an adapted IT mechanism.
1Publishable summary
During its third period, the EU-ADR project has continued its progress towards the production of the main results expected from the core work packages.
The project did successfully pass its second review with the help of external independent experts during the period. It has reinforced its visibility and the contacts established with key stakeholders (including regulatory authorities and other projects), and has developed plans for long term sustainability. EU-ADR has actually become a seminal project, spinning out several other projects (mainly funded via FP7-Health) that apply EU-ADR technologies and methods to the study of specific drug and vaccine safety problems.
On the scientific and technical front, the most important advances in the project have been the nearly completion of work in core WPs, and a progressive shift towards a ‘system view’ of the development efforts, by which the different pieces (signal detection, substantiation, evidence combination, validation, exploitation) have been integrated and the focus has moved to the challenges that such integration presented. This process has been helped by a ‘cycle-based’ configuration of the workplan that, superimposed to the original workplan, has helped steer efforts through a common, ‘pipeline’ framework addressed to reproduce the sequential steps of the system, revealing input-output relationships and dependencies.
Notably, the project has also made great progress in devising a strategy for long term sustainability.
Particular outputs that deserve to be highlighted include:
- Reappraisal and rework of the set of true-positive and true negative signals, now named ‘reference sets’, so as to maximise their usefulness, especially for the signal detection method performance asessment and the validation activities.
- Addition of three new events to the priority list of ten: Hip Fracture, Pancreatitis and Progressive Multifocal Leukoencephalopathy. Event definition for these events, and completion of terminology mapping, data extraction, and query harmonisation tasks related to all events.
- Development and testing of algorithms and methods for signal detection. Tackling of studies on how parametrisation of methods affect their performance.
- Development and completion of signal substantiation methods, implemented as web services.
- Development of evidence combination methods based on Dempster-Shafer theory. Implementation of tools allowing visualisation, exploration and user interaction with such methods.
- Development of the integrated EU-ADR Web Platform, which allows drug-event dataset management and sharing, visualisation, comparison and exploration of signal detection methods results, of signal substantiation results, and of evidence combination results, including custom user weightings given to the different methods based on belief and plausability.
- Progress in system validation using both retrospective and prospective approaches, comparing results with those of spontaneous reporting systems. Tackling of event validation activities, addressed to verify the reliability of the events extracted from the databases.
Overall, the project progress has been substantial, and the Consortium believes that it has allowed EU-ADR to become a breakthrough, reference project not only in the application of IT to pharmacoepidemiology, but also more generally in relation to demonstrating the feasibility of a federated model for re-using massive amounts of existing EHR data for research purposes; a model that, while respecting local and national ethico-legal limitations, allows to unleash the power of available medical information on millions of patients.
2Core of the report for the period: Project objectives, work progress and achievements, project management
2.1Project objectives for the period
The overall objective of the EU-ADR project is the design, development and validation of a computerized system that exploits data from electronic healthcare records and biomedical databases for the early detection of adverse drug reactions. The EU-ADR system intends to generate signals using data mining, epidemiological, computational and text mining techniques, and subsequently substantiate these signals in the light of current knowledge and understanding of biological mechanisms. The system should be able to detect signals better and faster than spontaneous reporting systems and should allow for identification of subpopulations at higher risk for ADRs. For the system to operate as an adjunct to safety reviewers during signal evaluation and follow-up, it should also enable easy access to the underlying data sources, allowing to quickly focus on information that is pertinent to a suspected ADR.
In this project, electronic healthcare records (EHRs) comprising demographics, drug use and clinical data of over 30 million patients from several European countries are available. These EHR databases form the foundation of the project, insofar as they supply the patient data on top of which the system is built.
A number of designs and techniques are used to process these electronic medical records. One of the objectives of this project is to study and compare a number of different techniques that, in essence, all aim to detect unexpected or disproportional rates of events.
Once generated, the signals are substantiated to place them in the context of the current biomedical knowledge. Essentially, this means searching for evidence that supports causal inference of the signal. The list of signals will be assessed by automatically investigating feasible paths that connect the drug and the adverse reaction involved in the signal. The general strategy is the automatic linkage of biomedical entities (drugs, proteins and their genetic variants, biological pathways, and clinical events) by means of data mining approaches and in silico predictions based on biomolecular structures.
The signal detection and substantiation algorithms are integrated in a computerized ADR detection and monitoring system. This involves the development of an evidence weighting scheme to combine the various pieces of information and present the user with a final list of ranked signals.
The system is expected to be tested retrospectively using test sets that are based on recent literature, including both known side effects and spurious signals. The system’s ability to rediscover drug-event combinations from the test set with known side effects provides an indication of the sensitivity of the system. The system’s ability not to signal drug-event combinations from the test set with spurious signals provides an indication of the specificity of the system.
After the system has been validated retrospectively, a prospective evaluation is also among the original objectives, centred on further investigating the top-ranking signals generated by the system.
The ultimate objective of the project is to demonstrate that an earlier detection of adverse side effects of drugs is possible using EHRs.
There is no definition of project objectives per period in the EU-ADR Description of Work. For our second report, submitted last year, we distilled from the work plan what the project was expected to deliver during its secondperiod. Activities in this third project period were obviously expected to extend and deepen the tasks of the previous periods, and result in the bulk of outputs expected from the project.
For what refers to WP1, the third period of the project was expected to continue the maintenance of appropriate communication and work dynamics among partners and with external stakeholders. It was also expected to entail continuing ethical surveillance, and, following the experience of the previous period, repeat the project assessment exercise and compare the results obtained.
WP2 was officially finished in previous periods; however, the previous period already showed that a re-assessment of the true negative signals in the test set was needed. In this period, it was also planned to re-assess as needed the test sets, in case the results obtained from other WPs showed any important shortcoming in those sets. WP2 was also expected to cover as needed terminology mapping exercises derived from new events of interest beyond the 10 selected by the project, time and resource permitting.
The WP2/WP3 task force created in the previous period was also expected to continue its work in the third period, harmonising queries among databases by monitoring and optimising the actual codes used in each case to detect a specific event.
For WP3, and once the use of Jerboa as platform to extract, aggregate and export data was well established, work was expected to advance in signal detection decisively, so that a ‘final’ set of methodologies and parameters could be obtained in this third period.
Similarly, WP4 was expected in this period to complete the development of the different web services used to filter and substantiate signals output by WP3.
Additionally, this period was central for WP5, including the development of both evidence combination tools and the EU-ADR platform itself, including re-evaluation of design decisions as needed to enable full development of the system.
Although not expected to be completed in the period, activities in WP6 (System Validation) were expected to quickly unfold and show results as the full system ‘pipeline’ became progressively available.
For what refers to WP7, and aside from deepening the communication activities, with specific emphasis on liaison with relevant stakeholders and elaboration of relevant scientific publications, the focus was on developing the exploitation strategies, defining post-project use scenarios for the long term sustainability of the EU-ADR system.
Finally, management activities in WP8 were expected to consistently support all other activities in the project, ensuring efficiency and compliance with contractual requirements.
2.2Work progress and achievements during the period
WP1: Scientific Coordination
WP leader: Johan van der Lei – EMC
Work for WP1 has continued as expected during the third project period, providing overall scientific leadership to the initiative, steering the project and balancing scientific excellence with pragmatic approaches. As in previous periods, cross-talking between WPs has been a specific concern, moreso as the different results in the central technical WPs (3, 4, 5) were progressively evolving towards their ‘almost-final’ shape – accordingly, system integration has been a continuing focus in the period.
A noteworthy activity in the period, developed in collaboration with WP8, was the devising of development ‘cycles’ to promote alignment of activities across WPs. This was necessary because, as WPs 3, 4 and 5 progressed, it was noted that there could be mismatches in the input-output relationships between them, derived from the different stages of development of each WP, a situation that could hardly be avoided taking into account the significant overlapping in the time schedules of these WPs as set out in the original plan. Therefore, three development cycles were devised (“silver”, “gold” and “bonus”), so that all WPs could followthe intended system ‘sequence’ from terminology mapping, to query harmonisation, signal detection, substantiation, evidence combination, system integration and validation. This helped clarify the dependencies among WPs, highlighted the iterative nature of the project developments, and served to test the system pipeline in order to detect bottlenecks or specific risks that could feed the exploitation plan. As an example, the cycles time schedule, which was updated during the period as needed, was originally devised as depicted below:
The different cycles were defined according to the events that were included, the data coverage used in the process, and the Jerboa version used to extract and pool the data. Overall, the definition of development cycles has proven invaluable to focus the efforts of partners and achieving the expected results.
It is to be remarked thatthe last cycle (“bonus”) was explicitly devised as an optional activity, as it resulted from interactions with relevant stakeholders. Indeed, the attendance of representatives of the Dutch national regulatory authorities to one EU-ADR meeting resulted in a ‘request’ to explore what the project could contribute to two events of recent interest regarding drug safety (Progressive Multifocal Leukoencephalopathy and Acute Pancreatitis). The project agreed to look into these events after the bulk of the work with the ‘committed’ ten main events was complete. As the period elapsed, and given some of the difficulties encountered with these events, it was agreed in the Consortium to also look at Hip Fracture, as additional event of interest.
Aside from strengthening the relationships with regulators, the project has also continued to ‘spin-out’ other projects in the FP7 Health and other programmes, in which subsets of partners participate. This is the case of VAESCO and SAFEGUARD, who together with the ongoing SOS and ARITMO projects, make use of EU-ADR technology to implement specific safety studies. At the end of the period, the project has also established solid bonds with the very relevant EHR4CR IMI project; this has resulted in a joint application to the recent IMI call. Relationships are also ongoing with the EMA-coordinated EnCePP initiative. Importantly, collaboration with USinitiatives (FDA Sentinel, OMOP) has been reinforced during the reporting period (including mutual visits), and the lead developer of Jerboa won the international “OMOP Cup” competition thanks to one of the algorithms for signal detection developed in EU-ADR.
In the previous period, an interim assessment of the progress of the project (Deliverable D1.3) was elaborated. For this, two complementary methods were used. Firstly, an ‘objective’ assessment was carried out following standard project management practices for progress evaluation (earned value),considering both cost and schedule performance. Secondly, a survey was carried out among partners to provide a ‘subjective’ assessment counterpart, by scoring the degree of achievement of each original objective on a 0-10 scale. Both exercises yielded a progress estimation of around 64% of the project completed at the end of month 24.
The exercise was repeated at the end of the period currently being reported, in order to compare the results and evaluate the interim progress achieved. Using the same earning rule (50/50), the ‘objective’ estimation at the end of July 2011 yields a “progress” (Budgeted Cost of Work Performed, BCWP, or Earned Value) of 89% of the project’s budget. For comparison, taking into account that the project was granted during the period a 6-month extension (see Management section of this report for details), a linear time reference at month 42 over 48 months of total duration means that by July 2011 87.5% of the project had elapsed. However, the bulk of activities were expected to be completed by that date, and the estimated BCWS (Budgeted Cost of Work Scheduled) is actually 96.7%. Therefore, it could be concluded that the project was behind schedule (Schedule Performance Index, SPI, of 0.92, where unity means perfectly on track). However, it is noteworthy that both WP1 and WP8 (and, to a lesser extent, WP7) are continuing WPs throughout the project, and in these cases the 50/50 earning rule artificially reduces the earned value estimation at the late stages of the project. Taking into account only WPs 2 to 7, the BCWP is 94%, while the BCWS is 98% - for an improved SPI of 0.95. Looking at the costs incurred in to achieve this progress, the calculations (pending approval of the period costs by the EC) show an Actual Cost of Work Performed (ACWP) of 101% of the budget, which clearly indicates that the project is over budget (Cost Performance Index, CPI, of 0.88). Taking into account WPs 2 to 7 only, the CPI improves to 0.92, which indicates only a slight overspending compared to the original budget. The ‘objective’ analysis, therefore, shows the project slightly behind schedule and over budget, but reasonably on track.
When looking at the ‘subjective’ assessment, partners (n=18) score the degree of achievement of the objectives with a mean 8.5 (which can be interpreted as 85% of the project completed):