Table 1: Revisionsand responses to the comments from Reviewer:Jesus Bisbal

Comments / Corrections
Major Compulsory Revisions
I miss a discussion on how the questionnaires were created, and how werethe different questionnaires for different groups developed. The rationale behindthis process. / Thank you for pointing this out to us. A factor was included in the questionnaire if mentioned by at least three studies. In addition, two bilingual members of the research team validated the translation.
Major Compulsory Revisions
It is stated that "the questionnaires listed key factors from the literature". Suchstatement should be backed up by some references (or is it essentially [16], thesystematic review?). Was there any filtering of such key factors performed? if so,on what basis? What factors where used for each of the different groups (ifdifferent)? / The questionnaire lists key factors from the literature, identified by the systematic review. We have changed this sentence for “The questionnaires listed key factors (barriers and facilitators) concerning EHR implementation identified by the systematic review (McGinn et al. 2011 BMC Medicine) and specific to each user group.”
Major Compulsory Revisions
When the paper states that "a factor was included in the questionnaire if
mentioned by at least three studies [...]", what there any rationale behind this
"three" value? / In order to ensure that the questionnaire took less than 30 minutes to complete (Okoli 2004) we choose to includebetween 10 and 20 questions representing the most frequent factorsfound in the literature. Thus, factors that were mentioned in at leastthree studies in our systematic review (McGinn et al. 2011 BMC Medicine) were kept as they represented between 10 and 18 individual factors,depending on the user groups. We also made an exception and retained factors mentioned by two studies if one of them was conducted in Canada.
Minor Essential Revisions
Search and replace "HER" by "EHR". That's a common hard-coded Microsoft
Word substitution rule, very disturbing when writing papers about electronichealthcare records. / Thank you! We have done a search and replace.
Minor Essential Revisions
Tables 2-5 would benefit from a description of the values in its cells,
particularly those that appear under "criteria" and "consensus" / We have put the numerical values in the cells.
Discretionary Revisions
In the Analysis section, the paragraph describing the "tenth" and "twentieth"percentiles and how they are used. It could be rephrased, as it is rather dense,required a significant effort from the reader to follow. it could include someexamples referring to Table 2. / We have clarified this section and added an example We have changed this paragraph for: “Percentile scores and interquartile range were used to calculate the level and strength of the consensus, respectively. To determine the level of consensus, tenth and twenty-fifth percentile scores were calculated. Tenth percentile scores indicate the lowest number on the Likert scale upon which at least 90% of participants agreed and 25th percentile scores indicate 75% agreement. Interquartile range, a measure of statistical dispersion, indicates the strength of the consensus, where 0 specifies a strong group consensus and 2 indicates dispersed responses. For example, in Table 2 physician consensus, the factor “confidence in EHR developer or vendor” (applicability) shows that according to the 10th percentile scores 90% of respondents responded either 2, 3,4 or 5 on the 5 point Likert scale, while 75% of respondents responded either 4 or 5. The interquartile range of 0 indicates strong group consensus.”

Table 2: Revisions and responses to the comments from Reviewer:Anne Holbrook

Corrections
Major Compulsory Revisions
The authors over-estimate the importance of this type of work in terms of eHealthand clinical decision-making, and do not present useable results in themanuscript. I was not able to access the Figure or any of the Tables. / We have removed discussion aspects that over-estimate the importance of our work and acknowledge that the results have very focused implications that mainly concern EHR implementers in Canada, but could also be transferred to other settings where the aim is to understand what is important for eHealth users.
As for Tables and Figure, we hope that it will be resolved. Please contact the journal’s editors if the problem persists.
Abstract:
a)The factors listed are too general to be meaningful; need to separate into facilitators vs barriers.
b)Also this sentence could not be gleaned from the results. The abstract should only reflect what is in results. / a)Thank you for this comments but it is not possible to address this as in our systematic review, factor could be either a facilitator or a barrier to EHR implementation. A studycould report the absence or presence of a given factor as having a positive or a negative influence on the EHR implementation. Additional files 2 to 5 show the link between factors in each questionnaireand the barriers or the facilitators identified in the literature.
b)We changed the sentence in the abstract in order to only reflect the results.
Abstract:
This conclusion is very speculative. The more pessimistic and more probable is that few of your participants have sufficient experience, expertise and familiarity with the evidence to adequately comment on your questions. There are too few completed EHR implementations in Canada. / The conclusion section of the abstract has been revised according to your suggestions.We changed this section for: «Amongst all factors influencing EHR implementation identified in a previous systematic review, ten were prioritized through this Delphi study. The varying levels of agreement between and within user groups could meanthat users’ perspectives of each factors are complex and that each user group has unique professional priorities and roles in the EHR implementation process. As a more completed EHR implementations in Canada will occur it will be possible to validate this preliminary result with a larger population of users.»
Background:
a)Define EHR (as opposed to EMR, PHR, etc).
b)Why were patients considered? / a)We have included description and differentiation between of EHRs, EMRs and PMRs.
b)Patients were considered in this Delphi study because the systematic review identified patients as an important user group and, as such, significant recruitment efforts were made to include patients in this Delphi study.
Background:
What is the evidence that anything connecting clinics hospitals, pharmacies is functional now? This is currently primarily a Canadian pipe dream, not a reality. / We agree with this comment that interoperable EHRs in Canada are still far from reality. However, the purpose of this study was to explore the perceptions of partners of the EHR implementation process that use (or are expected to use) EHRs in order to inform decision makers on possible implementation strategies. Canada Health Infoway was the decision maker partner in this CIHR Knowledge synthesis grant so the study was designed with their input.
Background:
This group of sentences should be identified as highly speculative e. There is no evidence currently that EMRs, EHRs, CDSS, etc improves patient outcomes (and there have been multiple studies). Hyperbole should be avoided. The best we can hope for is perhaps manageable storage, more reliable chart access, and more sharing of data. Your paragraph, however, ignores the harms that come automatically with EHRs, harms which may neutralize many of the benefits. / This section has been revised according to your suggestions. We modified this sectionin order to reflect the lack of evidence on the benefits of EHR and other systems. We also acknowledge potential harms. But it is important to bear in mind that this study is not a review of EHR benefits but takes the position that EHR implementation in Canada is a priority with enormous amounts of money already invested without knowing very much about the conditions that could enable the integration of this technology into the healthcare system.
Background:
Again, no evidence / We have removed this sentence from the background section.
Background:
Poorly described rationale – too vague. Understanding how international successful implementations might be successfully adapted to Canadian healthcare might well be more meaningful… / We have replaced this section with:«Understanding users’ perspectives of EHR international successful implementation by asking them through the Delphi technique to achieve consensus in a given area which lacks empirical evidence is key to informed decision-making, effective implementation strategies, and successful adaptation of EHRs into the Canadian healthcare system.»
Background:
Your comments below suggested that these are not users, as does much of your participant list. / We followed your suggestion. Wehave explained what “users” mean in our study.
Background:
Objective disjointed. Should introduce SR early / We have introduced the SR earlier in the paper.
Additional comments attached to the manuscript
Background:
This background on Delphi method, should be in Background section / The description of the Delphi method has been revised according to your suggestions and moved to the Background section.
Additional comments attached to the manuscript
Background:
Redundant sentence.
Unfortunately homogeneity of the panel, while perhaps increasing similarity of feedback (inter-rater reliability), makes the results much less generalizable. / This section has been revised according to your suggestions. We changed this section for: “As shown by Akins, Tolson & Cole (2005), reliable outcomes could be obtained with a relatively small Delphi panel of experts with similar training and general understanding in the field of interest [20], However generalizability of the results may consequently be reduced.”
Additional comments attached to the manuscript
Background:
The appropriate use and application of consensus opinion development deserves more justification. How does consensus of opinion, in the absence of empirical evidence, actually improve outcomes? / We agree that consensus opinion is not “hard evidence” but given the lack of experimental studies on EHR implementation factors, the Delphi method appeared as the most appropriate to answer our research question.
Additional comments attached to the manuscript
Background:
This also is not Methods, belongs in Background. / The description of the systematic review has been revised according to your suggestions and moved to the “background” section.
Additional comments attached to the manuscript
Background:
Need short sentence on what databases and what languages were used, what the key question was and what type of studies were included / This section has been revised according to your suggestion.
Additional comments attached to the manuscript
Background:
Are these really professionals, or technicians/technologists/programmers? / This section has been revised .We defined non-physician healthcare professionals and health information professionals in subdivision «Participants.» of the methods section.
Health information professionals (HIP) are considered as “information professionals” and include distinct professionals. In Canada, they have a professional association:
Additional comments attached to the manuscript
Methods:
Would add a few examples here / We followed your suggestion and included an example and refer to additional file 1.
Additional comments attached to the manuscript
Methods:
See above – people who ‘participated in known CND EHR implementations’ are not necessarily users. Need to be precise. / The description of experts has been revised according to your suggestions. We changed this section for: “Experts who were known users of Canadian EHR implementation projects were purposefully selected and also invited by telephone and email to join the study.”
Additional comments attached to the manuscript
Methods:
These are? / We clarified the five types of user groups. We changed this sentence for: “Participants also needed to belong to one of the five EHR user groups (physicians, non-physician healthcare professionals, health information professionals, healthcare managers and patients), have a valid email address and access to the internet, and speak English or French.
Additional comments attached to the manuscript
Methods:
Why? This would make them very unusual patients / We clarified therationale behindthe recruitment of patients’ representativeparticipants. As per request by the Research Ethics Board of the Centre Hospitalier Universitaire de Québec, patients were required to hold official positions because otherwise, we would have needed to go through as many ethics committees as individual patients…
Additional comments attached to the manuscript
Methods:
This sentence has poor face validity – healthcare professionals and HIT technical people have very different backgrounds, perspectives, and approaches to EHR. So again, main question is why administer different questionnaires? This only decreases generalizability. / The aim of the study was not generalizability in the sense that a single vision of EHR implementation factors is achievable, but rather the representation of various visions from different users and the exploration of the similarities and differences between them in the Canadian context.
Additional comments attached to the manuscript
Methods:
Why were different questionnaires used? Would this not ablate any possibility of collation across the whole group (which is already a small sample size)? / Based on identified factors of each users group found in the systematic review (McGinn et al. 2011 BMC Medicine) and pertaining to this study the most frequent implementation factors of each user groups were used to design a specific questionnaire for each user group because the findings of the SR indicated that they held different views.
Additional comments attached to the manuscript
Methods:
Questionnaires listed in additional files – are these complete? / Yes they are complete. We revised according to your suggestions and added “see complete questionnaires as presented in additional files” at the beginning of section: Design of the Delphi study.
Additional comments attached to the manuscript
Methods:
Why does good consensus have to agree on both applicability and importance? / The rationale has been provided in the text with a reference to Okoli & Pawlowski 2004.
Additional comments attached to the manuscript
Methods:.
Need to improve clarity – discussion mixture of IQR, likert scores, percentile scores is confusing. / We clarified this section. We changed this sentence for: “Priority for decision-making concerning EHR implementation in Canada was determined among the questionnaire items upon which 90% of participants considered both applicable and important (that is, factors scoring 4 or 5 for applicability and importance on the 5-point Likert scale). Tenth percentile scores were used to determine if the questionnaire item reached sufficient consensus. The analyses were performed using SAS software version 9.1.”
Additional comments attached to the manuscript
Results.
What was the possible denominator; this looks to be a very small responder % / We have indicated that the number of people contacted was 106.
Additional comments attached to the manuscript
Results.
How did you deal with this considerable amount of missing data? / We chose to only include participants who completed the three rounds. Although the response rate was lower, especially among physicians, we considered that the number of participants was sufficient to provide a first picture of the factors that were important in a given user group because people who participated were considered as knowledgeable informants.
Additional comments attached to the manuscript
Results.
Suggests biased participation – most physicians and HIT people are male, no? / Most participants were female and there is a possible participation bias, but this is common in a Delphi study where participants have to invest a significant amount of their time. The gender bias is perhaps only an indication that females are more committed to such an exercise…
Additional comments attached to the manuscript
Results.
Too many parts to this sentence; very confusing as written / We have clarified this section. We changed this section to: “Table 2 shows that nearly half (8/18) of the factors on the physicians’ questionnaire reached consensus on the applicability or importance of the factors presented in the questionnaire: one item reached a strong consensus (≥ 75%),two items moderate consensus (≥ 60-74%), and five items partial consensus (≥ 60%).”
Additional comments attached to the manuscript
Results.
Thus my comments above; this is a serious limitation to these results. It is difficult to believe that such small subgroup samples with much missing data and inconsistent agreement should be taken seriously. This is a key problem to address…. / We have addressed the limitation of this study in the Discussion. However, it is the first study to take this approach in order to explore how applicable and relevant can be the results of a systematic review for knowledge users in the field. The fact that there is inconsistent agreement is not a problem in itself because it only shows that certain factors are not as important as it may appear in the literature.
The amount of missing data has a limited influence on the interpretation of the findings because one can expect that with more respondents strong consensus would have been easier to achieve.
Additional comments attached to the manuscript
Results.
EHR / We changed HER for EHR everywhere in the document.
Additional comments attached to the manuscript