2012 Inaugural “Digital Pathology Showcase”
Draft – Version 3 for Public Comment
Objective
The primary goals of this event are to provide the audience with a good sense of the state-of-the-art in WSI-based digital pathology and provide visibility into the viability and readiness of digital WSI technology for routineuse in pathology: “is it ready for prime time?”To accomplish that, a set of pathology caseswill be provided by the organizers and reviewed live by practicing pathologists. Participating vendors will provide their equipment to the pathologists to showcase the capabilities of their WSI technology.Systems will be subjectively and informally judged by the audience, observing the time it takes and the quality of the diagnostic recommendation based on the WSI reviews.
The inaugural event will exclude issues such as workflow optimizations via LIS integration and advanced Computer Assisted Diagnostics (CAD). The case reviews will be done based on manual review of the WSI and using basic annotation capabilities. In future years, we do expect this to grow and include some of the excluded aspects.
Logistics
We are anticipating 3 vendors participating in the 2012 event with duration ofapproximately 3 hours. The order of presentation will be pre-determined so that each vendor will present first, second and third of the three scenarios proposed. We plan to interleave the review of the three types of pathology cases so that a subjective side-by-side comparison will be possible for the audience. We are considering some overlap of the scanning for the second case scenario (multi slide biopsy with a focal cancer)to ensure the even can be conducted in a timely manner. We will make that determination once we do have confirmation of the three participating vendors.
Cases
Each vendor will be given a set of representative cases in advance of the event so that it is known what type of cases will be expected to be reviewed during the event. Except for the first case, those cases will be replaced prior to the event to ensure the pathologistsdo not know the diagnostic outcome of the case but will have to perform a “live” review. The three sets of cases will be:
- Frozen section: 1 slide
Slides will be dry and scanned as well as reviewed “live”. The total time to scan, transfer, and diagnose the slides “live” will be observed by the audience. - Biopsy with a focal cancer:6 to 9 slides
Case will consist of 2 or 3 blocks and 3 slides per block, H&E only, various cases of infiltrating breast. Prostate or coloncarcinomas with the cancer being present it at least one of the blocks. The case shared ahead of the event and the one being reviewed during the event will not be the same. New slides will be provided shortly prior to the session so that there is enough time to scan but no access to the images by the pathologist, thus ensuring a “live” review of the case.The time it takes the pathologist to come to theirdiagnosis will be observed by the audience.
- Immunohistochemical Quantitative Interpretation:1 slide
Case will consist of one immunohistochemical stain of a breast biopsy for ER, PR or Her2Neu interpretation. The case shared ahead of the event and the one being reviewed during the event will not be the same. New slides will be provided shortly prior to the session so that there is enough time to scan but no access to the images by the pathologists, thus ensuring a “live” review to determine the finding in terms of the biomarker examined..
Rules
The organizers of the event are setting forth a set of rules to make this a valuable comparison. In the following we list the guidance for the vendors on what is expected as well as permitted:
- Demonstration setup is required to include scanner as well as needed workstation and server(s)
- Demonstration to be conducted by practicing pathologist(s) and a pathology technician (histotechnologist, IT or scanner personnel or other non-vendor personnel. No vendor employeesare allowed to take place in the actual demonstrations.
- Software needs to be a commercially released version, no Alpha or Beta software allowed.
- Hardware equipment needs to reflect a common commercial deployment (e.g. do not replace standard hard disks with solid state drives to boost performance beyond what customers can afford in a real deployment)
- Prefetching or pre-caching of cases will not be allowed and network utilization will be monitored
Additional rules might need to be put in place as we refine the draft of this document.
Setup
The setup will include scanners, client workstations, and possibly server infrastructure, depending on the deployment needs of each vendor. Vendors are asked to setup the equipment the evening before the event and will be assisted with respect to the network setup.
The network bandwidth between the scanners and the respective storage unit (server) will vary based on each case review. For the Frozen section, a 2Mbit/sec network bandwidth and 100ms network latency will be applied while for the other two cases 20Mbit/sec network bandwidth and 30ms network latency will be applied. This is to reflect reasonable network conditions for a remote read of a frozen section as well as regular reads throughout a hospital network. If selected vendor(s) requires a higher degree of network performance that will be considered and all vendors will have access to any upgrades from this specification that occur.
A total of three projection screens will be available. Screen #1 will display a view of the respective scanner during the “live” scanning of the frozen section scanning and subsequently a view of the keyboard + mouse of the pathologist reviewing each case. Screen #2 and #3 will show a view of the two screens of the workstation the pathologist will use to read a case. This setup will allow the audience to fully comprehend the effort and assess ease of use of reviewing each case.
Appendix
Suggested room layout: (This can be altered to fit the available space)
Suggested network setup:
Comments from Vendor Interviews (Draft from MB – some not approved/verified for release)
Interviews with Industry Partners
Mark Newberger (MN), Apollo Telemedicine, Inc
June 6th, 2011
Interviewer – Mike Becich (MB)
Summary of Mark’s Newberger’s comments:
1)Overall MN thought the idea of a digital pathology showcase as proposed in 4/20 Draft is a good idea
2)He also thought the 18-20 month lead time for the showcase in fall 2012 was a good idea as well
3)He did admit that he initially thought we were planning to have the showcase this fall with a 4 to 6 month lead time
4)He had issues with companies like Apollo (which does workflow software and is scanner agnostic) would have to partner with one or two other companies to compete in the showcase.
5)He didn’t feel that the current draft would yield more than two or three vendors who could do end to end (scanning, processing/display and diagnostic workflow) digital pathology
6)He favored breaking the showcase into segments (scanning, processing/display and diagnostic workflow) and having different vendors for each of the segments
7)MN suggested that having a “training” program for the vendors on the technical set up a day before the 2012 meeting was a good idea
8)MN and MB suggested that the technical specification for the showcase set up be made available for testing at the company sites. MB suggested that Pittsburgh would offer a “test” setup in Pittsburgh at least 11 months before the 2012 meeting to let companies decide if they want to participate (test cases and technical set up).
9)MN agreed that offering the first six vendors who committed to the showcase was a good idea but was concerned you might not get six the first year. Vendors might be concerned how they would “show” in this type of open showcase.
10)MN also thought the showcase would provide good marketing potential to their customers as well as new customers.
11)MN agreed the “open” sharing of the whitepaper and the forum discussion as an industry panel at Pathology Informatics 2011 was a good idea.
12)MB closed the meeting by asking MN if he would mind sharing his comments on the showcase with the broad pathology and vendor community and he agreed.
Interviews with Industry Partners
Ole Eichhorn (OE), Aperio
June 10th, 2011
Interviewer – Mike Becich (MB)
Summary of Ole Eichhorn’s comments:
1)Overall OE thought the timing and planning for the industry panel in 2011 and the digital pathology showcase in 2012 was a good plan
2)OE stated that Aperio considered being involved in the European Scanner Contest in 2010 but decided not to participate because of the “closed” nature of the planning and evaluation
3)OE felt the current plan for the 2012 Digital Pathology Showcase was a starting point but suggested more complex activities (e.g. quantitative analysis and algorithms) might be considered as well.
4)OE stated that Aperio (as leader in the field) was interested in further dialogue on the format of the showcase in 2012 and would circulate the document to others in Aperio.
.
Interviews with Industry Partners
John Wellbank (JW) and Jose Costanon (JC), Phillips
June 17th, 2011
Interviewer – Mike Becich (MB)
Summary of John Wellbank’s and Jose Costanon’s comments:
1)Both JW and JC liked the idea of the industry panel in 2011 and the digital pathology showcase in 2012 but raised some concerns.
2)JC felt that Phillip’s would like to see higher volume of slides scanned included in the showcase. When MB told them that there were plans for 10 to 20 slides for the second (biopsy) and third (resection) case this did not fit the “higher volume” scenario. MB assured JC that this would be considered and discussed in the industry panel in 2011 and as part of the “open” comments from the community. JW added that Phillips would need scanning in 20 and 40x as well as more volume if possible. Phillips does not scan in 20x equivalency so that test is not possible. Therefore we need at least some of this test to scan resolutions equivalent to 40x for true time comparisons
3)JW was concerned that the network bandwidth and latency listed in the current (attached) draft of the plan were not sufficient from Phillip’s perspective. MB reassured both JC and JW that network bandwidth and latency will be provided to the highest specification requested by the industry partners. MB asked JW/JC to provide a network specification and that MB would make sure this was part of the plan.
4)JW felt the bar might be set “too low” for the first meeting and would provide additional feedback from others at Phillips on the case scenarios.
5)JW was concerned that there would be no time for vendors to test the setup for the Industry Showcase. MB reassured JW that a “pilot setup” (simulating the setup for 2012) would be available in Pittsburgh for at least 11 months in advance of the showcase and made available for scheduling to industry partners. MB also stated that the specification for the Industry Showcase would be made available to all interested industry partners in November 2011 should an industry partner want to replicate the environment in the company headquarters.
6)Overall JW and JC thought that the timing (over 18 months of lead time) and consensus building with industry partners would ensure a successful showcase.
Interviews with Industry Partners
Philippe Nore (PN), Ventana
July 5th, 2011
Interviewers – Mike Becich (MB) and Bruce Friedman (BF)
Summary of PhilipeNore’s comments:
1)Don’t want a contest that “compares” vendor on only one dimension (perhaps speed of scanning or image quality).
2)Don’t want to promote instruments that are “skewed” on the basis of any one factor (e.g. a “contest” specific set of criteria).
3)Don’t want to end up pushing company production of instruments and software to “win the contest”. The focus should be on how to meet the needs of customers.
4)Question – Will image analysis be a part of the first showcase? Ventana would like to see a IHC or ISH image analysis as part of the first showcase. PN felt many of the initial digital pathology solutions use this today, so why not showcase this to customers today.
5)Will the event will have time for six vendors demos (may be too many).
6)PN thought that even having 4 vendors would be hard to keep audience’s attention through the entire event and should limit time for each exercise.
7)Ideally for any event you would want to be vendor #1 or 2 but not 3 or 4 (much less 5 or 6) in the current setup due to “repetitive/redundant” nature of the tasks
8)Commentary – reduce the time for each serial demo by vendor.
9)PN thought by 2012 everyone would have a image analysis algorithm.
10)PN thought 20 minutes per vendor should be considered – based on his experience with customers.
11)PN asked what “live mode” means. Does this mean scanning occurs on stage. In the Ventana system the slide viewing is dynamic robotic so scanning a frozen section is not required once mounted on their hardware. Ventana would like the flexibility to view the slide without scanning for the frozen section exercise.
12)No FDA approval for primary diagnostics so exercise # 2 and # 3 may be out of question. They need to check with their regulatory lawyers to make sure this event is misperceived as promoting non-FDA approved use of devices.
13)User training is an issue and need this to be addressed. Who will run the demos? How will Pathology Informatics deal with this – told PN about the 3 solutions we propose for this including:
- Availability of a “mock” setup in Pittsburgh
- Availability of a “specification” for the setup and encouraging replication of the setup at the vendors headquarters or at a key clients location.
- Training experience on the “mock” system in Pittsburgh will be shared with users/vendors.
14)Ventana would require their own scanner, image server and application server loaded in advance. Can’t share with others. Will follow up with PN on the specs.
15)Separate fee for vendor participation in the Industry Showcase was PN’s recommendation.
Rob Liepse (RL) and StephaneWillaert (SW), Barco
July 20th, 2012
Interviewer – Mike Becich (MB)
Summary of Rob Liepse’s and Stephane’sWillaert (SW) comments:
1)SW commented that the European contest was only focused on the scanner. SW felt that the “end to end” solution including software, servers, etc... to be included in the showcase was a good focus for the 2012 event.
2)SW asked if there would be opportunities to have multiple vendors partner for the various partners for the component technologies.
3)SW questioned what role did I envision for Barco in the Industry Showcase and it seemed Barco’s role would be in the display of scanned slides.
4)SW stated in the Berlin scanner contest their role was as the display partner for all the vendors that participated.
5)SW stated that Barco is working with all the vendors in the digital pathology space.
6)RL mentioned that they have a very large footprints in digital mammography and PACS displays. They have been a repeat vendor at RSNA for years and participated in IHE (Integrating Healthcare Enterprise). They have extensive experience in primary diagnosis in the mammography space.
7)RL mentioned that Barco, worlds only approved display for TOMO (tomosynthesis – mammography tomography – have 510K approval) and have been very involved in standards efforts and moving this technology to the masses.
8)SW thought that the focus of the proposed Industry Showcase was appropriate but that it would make a number of the vendors nervous due to the lack of industry focus on “end to end” solutions. SW felt this would move the industry forward but it may be ambitious.
9)SW thought the technical focus on bandwidth support for the demo was a good one and that not enough studies are looking at the effect of the network on the viewing experience of the diagnosing pathologist.
Interviews with Industry Partners
Doug Giszczynski (DG) and Donal O’Shea (DO), Leica
July 26th, 2011
Interviewer – Mike Becich (MB) and Bruce Friedman (BF)
Summary of Leica’s comments:
1)DO - Leica participated in the European event and was please that they won awards for their products at the event.
2)DO and DG thought the European event was well communicated and went well but questioned the value of some metrics.
3)DO thought the document was clear and the scenarios and technical setup were useful.
4)DO thought the document needed more details on the metrics and quantification of the measures of the event.
5)DO wanted clarification on the scenarios and the timing (wasn’t clear). MB explained the scenarios and order of the vendors would be shuffled. BF very much liked MB’s idea of having the companies work different scenarios sequentially: vendor #1 works scenario #2; followed by vendor #2 works scenario #1, etc. No vendor then is in an unfavorable position in terms of the order of presentation. However, this causes problems if only one vendor is on the stage at any time with the rest in the bullpen. Lot's of confusing set-up time on the stage. Could get tedious. It seems to me that the only way this would work if all three (or four) vendors are continuously positioned on the stage at the same time with the "spotlight" on each as they go through each scenario. Then, only the switches need to be reset for each vendor.
6)DG what is the output of the showcase....need to finalize. MB stated that this would be proposed in the next draft of the document. Frozen section scenario would have 3 metrics -
7)DO queried on who would run the showcase and MB stated that a pathologist and a technical assistant who used the system would be running the company’s system and not vendor staff.
8)BF asked about the second event was canceled in Europe. DO did now know why Helsinki event was canceled.
9)DO thought the “end to end” scenarios and focus on the workflow was an appropriate one. The focus on how the digital pathology wholistically fits into their laboratory is a good and clean goal.
10)DO thought the technical setup was robust and adequate. Leica wanted gigabit between the image server and the viewing software. Include the make, model and specification of .