Encrypted | Login

SPIE-AAPM-NCI Prostate MR Gleason Grade Group Challenge

The American Association of Physicists in Medicine (AAPM), along with the SPIE (the international society for optics and photonics) and the National Cancer Institute (NCI), will conduct a part 2 “Grand Challenge” on the development of quantitative multi-parametric magnetic resonance imaging (MRI) biomarkers for the determination of Gleason Grade Group in prostate cancer. As part of the 2017 AAPM Annual Meeting, the PROSTATEx-2 Challenge will provide a unique opportunity for participants to compare their algorithms with those of others from academia, industry, and government in a structured, direct way using the same data sets.

A session at the 2017 AAPM Annual Meeting will focus on the PROSTATEx-2 Challenge; an individual from each of the two top-performing teams will receive a waiver of the meeting registration fee in order to present their methods during this session. Challenge participants are encouraged to submit their work for peer review to the AAPM’s scientific journal Medical Physics.

Release date of training set cases with truth: May 15, 2017
Release date of test set cases without truth: June 5, 2017
Submission date for participants’ test set Gleason Grade Group output: June 23, 2017 Extended to 3 PM EDT Monday, June 26, 2017
Challenge results released to participants: June 30, 2017
AAPM Annual Meeting: July 30 – August 3, 2017 (PROSTATEx-2 session: August 1)

Although the PROSTATEx-2 Challenge is now complete, data sets, supplemental data and instructions are available here.

PROSTATEx-2 Challenge Format

The database for this challenge contains a total of 162 MRI cases, each from a single examination from a distinct patient, with each case consisting of four sets of MRI scan data: two sets of T2-weighted images (transaxial and sagittal; DICOM format), Ktrans images (computed from dynamic contrast-enhanced (DCE) images; mhd format), and apparent diffusion coefficient (ADC) images (computed from diffusion-weighted (DWI) imaging; DICOM format). These cases contain a total of 182 findings (lesions): the training set contains 112 findings, and the test set contains 70 findings. Location and a reference thumbnail image will be provided for each lesion, and each lesion will have known pathology-defined Gleason Grade Group:

Grade Group 1 (Gleason score < 6): Only individual discrete well-formed glands

Grade Group 2 (Gleason score 3+4 = 7): Predominantly well-formed glands with lesser component of poorly-formed/fused/cribriform glands

Grade Group 3 (Gleason score 4+3 = 7): Predominantly poorly formed/fused/cribriform glands with lesser component of well-formed glands

Grade Group 4 (Gleason score 4+4 = 8; 3+5 = 8; 5+3 = 8): (1) Only poorly-formed/fused/cribriform glands or (2) predominantly well-formed glands and lesser component lacking glands or (3) predominantly lacking glands and lesser component of well-formed glands

Grade Group 5 (Gleason scores 9-10): Lacks gland formation (or with necrosis) with or without poorly formed/fused/cribriform glands

Epstein JI, Egevad L, Amin MB, Delahunt B, Srigley JR, Humphrey PA, the Grading Committee. The 2014 International Society of Urologic Pathology (ISUP) Consensus Conference on Gleason Grading of Prostatic Carcinoma: Definition of grading patterns and proposal for a new grading system. Am J Surg Pathol, (40)244-252, 2016

Participants may use the training set cases in any manner they would like for the purpose of training their systems; there will be no restrictions on the use of the data or the advice sought from local experts for training purposes. The test set cases, however, are to be manipulated, processed, and analyzed without human intervention (although human-supervised delineation of the prostate gland border or the gross lesion margin will be allowed, if the operation of a computerized system requires such input).

Participants are free to download the training set and, subsequently, the test set when these datasets become available. It is important to note that once participants submit their test set Gleason Grade Group output to the challenge organizers, they will be considered fully vested in the Challenge, so that their performance results (without links to the identity of the participant) will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers. The truth associated with the test set cases is expected to be made publicly available after publication of the PROSTATEx-2 Challenge and its predecessor PROSTATEx Challenge.

Participation in the PROSTATEx-2 Challenge acknowledges the educational, friendly competition, and community-building nature of this challenge and commits to conduct consistent with this spirit for the advancement of the medical imaging research community. See this article for a discussion of lessons learned from the 2015 LUNGx Challenge, also sponsored by SPIE, AAPM, and NCI.

Data Availability

Release of the training set (with truth): May 15, 2017 (tentative)
The training set will consist of 112 findings. This dataset will be representative of the technical properties (scanner type, acquisition parameters, file format) and the nature of the prostate lesions of the test set. An associated Excel file will include case name, the coordinates of the centroid of all lesions, and the lesion ‘truth’ label (Gleason Grade Group). Reference thumbnail images of the lesions will also be provided.

Release of the test set (without truth): June 5, 2017
The training set will consist of 70 findings. The locations of the lesions will be specified in the accompanying Excel file that will follow the same format as for the training set with the omission of the ‘truth’ labels. Reference thumbnail images of the lesions will also be provided.

Deadline for participants to submit test set Gleason Grade Group output: June 23, 2017
Participants should submit their test set Gleason Grade Group output through the online challenge interface by 11:59 PM Pacific Standard Time on June 23, 2017.
An acknowledgment will be sent within 2 business days of receipt of results. In case no confirmation of receipt is received, please contact the challenge organizers at s-armato@uchicago.edu.

Output format for the PROSTATEx-2 Challenge test set results:

Method performance will be assessed using a quadratic-weighted Cohen’s kappa coefficient of agreement with the reference standard Gleason Grade Group for each lesion. If the kappa coefficient across multiple methods is the same, the positive predictive value of each method’s ability to identify Gleason Grade Group greater than 1 will be used to break the tie. To facilitate performance assessment across participants for the test cases in the PROSTATEx-2 Challenge, please follow these guidelines:

  1. Using your method, calculate a Gleason Grade Group for each lesion as an integer value on the range [1, 5]. The submitted output must be an integer between 1 and 5, inclusive.

  2. Record your scores for all cases in a CSV file containing three columns with the contents of the first two columns of the Prostatex-2-Findings-Test.csv file (case number (ProxID) and lesion number (fid)) followed by an extra column containing your method’s assigned Gleason Grade Group.

    Example:         Prostatex-0999, 1, 4
                             Prostatex-0999, 2, 3

  3. Please report your results in the same order as lesions appear in the Prostatex-2-Findings-Test.csv file.

Conflict of Interest:

All participants must attest that they are not directly affiliated with the labs of any of the PROSTATEx-2 organizers or major contributors.

Acknowledgment:

Anyone wishing to use these datasets for research, presentation, or publication purposes outside of the PROSTATEx-2 Challenge should acknowledge the SPIE, the AAPM, the NCI, and Radboud University (“Data used in this research were obtained from The Cancer Imaging Archive (TCIA) sponsored by the SPIE, NCI/NIH, AAPM, and Radboud University.”) and cite the following article:

Litjens G, Debats O, Barentsz J, Karssemeijer N, Huisman H. Computer-aided detection of prostate cancer in MRI. IEEE Transactions on Medical Imaging, (33)1083-92, 2014.

Organizers and Major Contributors: