Details

Purpose

Cardiovascular diseases are the global leading cause of death. The amount of calcified plaque deposited in the wall of the coronary arteries has been shown to be a strong and independent predictor of cardiovascular events. In clinical routine, CT based quantification of coronary artery calcium (calcium scoring) is performed and reported with increasing frequency. In current practice, coronary calcifications are manually identified, which is prohibitive in large scale clinical trials and impractical in daily practice. A number of algorithms describing automatic detection and quantification of calcified lesions in coronary arteries have been proposed. However, these algorithms have not been evaluated for robustness in large and diverse sets of images.

The purpose of this challenge is to compare methods for automatic coronary artery calcium scoring in cardiac CT examinations originating from different CT scanners on a whole-heart and a per-vessel basis.

Task

The task in this challenge is to automatically identify coronary calcifications in non-contrast enhanced ECG-triggered cardiac CT examinations that are acquired as part of standard clinical routine. For this, both non-contrast and contrast enhanced CT images are provided. Results are assessed on a lesion-level in sensitivity and positive predictive value. Additionally, teams can choose to identify the coronary artery in which each calcification is located. In that case, performance of the algorithm is also be evaluated per artery. Finally, the total amount of coronary calcium per patient and assignment of patients to cardiovascular risk categories based on this amount is assessed. Details about the evaluation procedure can be found here.

Terms of participation

The orCaScore challenge is organized in the spirit of cooperative scientific progress. We do not claim any ownership or right to the algorithms but we require anyone to respect the rules below. The following rules apply to those who register a team and/or download the data:

  • The downloaded data sets, associated reference standard, or any data derived from these data sets, may not be given or redistributed under any circumstances to persons not belonging to the registered team.
  • All information entered when registering a team, including the name of the contact person, the affiliation (institute, organization or company the team’s contact person works for) and the e-mail address must be complete and correct. Anonymous or incomplete registration is not allowed.
  • The data provided may only be used for preparing an entry to be submitted to this challenge. The data may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products, without prior participation in the challenge and approval by the organizers.
  • Results of your submission will only be published on the website when a document (e.g. a journal paper) describing the method is provided. Please send this document by e-mail (or a link to it) to the organizers if you want to have your results published.
  • If a commercial system is evaluated no method description is necessary, but the system has to be publicly available and the exact name and version number have to be provided.
  • The organizers of the challenge will check the method description before your results will be published on the website.
  • If the results of algorithms in this challenge are to be used in scientific publications (e.g. journal publications, conference papers, technical reports, presentations at conferences and meetings) you must make an appropriate citation. This citation will refer to the publication describing the results of the challenge (J.M. Wolterink et al. An evaluation of automatic coronary artery calcium scoring methods with cardiac CT using the orCaScore framework, Medical Physics, 2016).
  • Evaluation of registration results uploaded to this website will be made publicly available on the orCaScore website (Results section), and by submitting results, you grant us permission to publish our evaluation. Participating teams maintain full ownership and rights to their method.
  • Teams must notify the organizers of orCaScore about any publication that is (partly) based on the results data published on this website, in order for us to maintain a list of publications associated with the challenge.