Results

Results

Segmentations of the test set submitted through this website are evaluated by the organizers, results are reported to the team and subsequently, published on this page.

Participants

Team name Images used Interaction Last update CPU time Description
BIGR_CaSc CT+CTA Automatic 2014-06-05 80m Download
FAU CT Automatic 2017-09-17 10s Download
ISI* CT Automatic 2014-04-01 30m Download
LIST CT+CTA Semi-automatic 2014-06-25 8m Download
SIECOR CT+CTA Automatic 2014-06-18 3m Download
SK CT+CTA Semi-automatic 2014-06-21 3m Download

* The authors of this method are involved in the organization of the challenge.

In addition to results of the submitted methods, performance of the two observers with respect to the reference standard (consensus reading) is listed. Observer 1 is a radiologist with 12 years experience in calcium scoring CT (>5000 scans), observer 2 is a research physician with 5 years experience in calcium scoring CT (>1000 scans).

Note that you may rank results according to one of the criteria by clicking on the corresponding header cell.

Detection of calcified lesions

Teamname Sensitivity (#) Sensitivity (vol) PPV (#) PPV (vol) F1 score (vol)
Observer 1 0.943 0.986 0.993 0.986 0.986
Observer 2 0.990 0.999 0.990 0.955 0.976
BIGR_CaSc 0.621 0.853 0.878 0.937 0.893
FAU 0.835 0.933 0.974 0.969 0.951
ISI 0.845 0.935 0.950 0.959 0.947
LIST 0.940 0.989 0.954 0.948 0.968
SIECOR 0.838 0.948 0.953 0.938 0.943
SK 0.513 0.573 0.644 0.685 0.623

Calcification volume per coronary artery

Teamname ICC LAD (vol) ICC LCX (vol) ICC RCA (vol) Average ICC (vol)
Observer 1 0.999 0.987 0.985 0.990
Observer 2 1.000 0.998 0.903 0.967
BIGR_CaSc 0.981 0.954 0.897 0.944
FAU 0.983 0.881 0.983 0.949
ISI 0.984 0.981 0.964 0.976
LIST 0.999 0.998 0.961 0.986
SIECOR 0.934 0.869 0.959 0.921
SK 0.333 0.568 0.514 0.472

Calcification volume and Agatston score per patient

Teamname ICC patient (vol) ICC patient (Agatston) Accuracy CVD risk Kappa CVD risk
Observer 1 0.998 0.998 1.000 1.000
Observer 2 0.984 0.979 1.000 1.000
BIGR_CaSc 0.966 0.971 0.850 0.875
FAU 0.986 0.989 1.000 1.000
ISI 0.982 0.986 1.000 1.000
LIST 0.994 0.992 0.975 0.980
SIECOR 0.982 0.980 0.950 0.960
SK 0.597 0.621 0.800 0.801

Abbreviations used in the tables:

  • Vol: Calcification volume
  • #: Number of calcifications
  • PPV: Positive predictive value
  • ICC: Two-way intraclass correlation coefficient for absolute agreement