Tasks for AI๐ฉโ๐ป๐จโ๐ป
The PANORAMA: AI Study (grand challenge) aims to evaluate the performance of modern AI algorithms at patient-level diagnosis and lesion-level detection of PDAC in abdominal CECT scans. Similar to radiologists, the objective of AI developed in this study is to read CECT exa,s and produce an overall patient-level score for PDAC diagnosis and the lesion location, as depicted in the Figure below. 
Figure. (top) Patient-level PDAC diagnosis (modeled by 'AI'): For a given patient case, using the CECT exam, compute a single floating point value between 0-1, representing that patientโs overall likelihood of PDAC. (bottom) Lesion location (modeled by 'AI'): For a given patient case, using the CECT exam (and optionally all clinical/acquisition variables), predict a 3D detection map of the PDAC lesion (with the same dimensions and resolution as the CECT image). For the predicted lesion, all voxels must comprise a single floating point value between 0-1, representing that lesionโs likelihood of harboring PDAC.
We require detection maps as the model output (rather than softmax predictions), so that we can definitively evaluate object/lesion-level detection performance using precision-recall (PR) and free-response receiver operating characteristic (FROC) curves. With volumes of softmax predictions, there's a lot of ambiguity on how this can be handled โe.g. what is the overall single likelihood of PDAC in the predicted lesion, what constitutes as the spatial boundaries of each predicted lesion, and in turn, what constitutes as object-level hits (TP) or misses (FN) as per any given hit criterion?
Similar to clinical practice, PANORAMA mandates coupling the tasks of lesion detection and patient diagnosis to promote interpretability and disincentivize AI solutions that produce inconsistent outputs (e.g. a high patient-level csPCa likelihood score without any significant csPCa detections, and vice versa). Organizers will provide end-to-end baseline solutions, adapted from for example the nnU-Net (Isensee et al., 2021) model in a GitHub repo.
Baseline AI models for 3D PDAC detection/diagnosis in CECT:
(link to GitHub repo PANORAMA baseline)
Evaluation ๐
Performance Metrics
Patient-level diagnosis performance is evaluated using the Area Under Receiver Operating Characteristic (AUROC) metric. Lesion-level detection performance is evaluated using the Average Precision (AP) metric. Overall score used to rank each AI algorithm is the average of both task-specific rankings.
Overall Ranking Score = [Rank(AP) + Rank(AUROC)]/ 2
Free-Response Receiver Operating Characteristic (FROC) curves are used for secondary analysis of AI detection sensitivity at 0.01, 0.001, and 0.0001 false positives per patient, assessing the potential for opportunistic screening. Intersection over Union (IoU) is also used for secondary (spatial congruence) analysis of positive AI detections, but not for the evaluation of detection or diagnosis performance (given that IoU is ill-posed to accurately validate these tasks (Reinke et al., 2022)).
Hit Criterion for Lesion Detection
A โhit criterionโ is a condition that must be satisfied for each predicted lesion to count as a hit or true positive. Hit criteria have been typically fulfilled by achieving a minimum degree of prediction-ground truth overlap, by localizing predictions within a maximum distance from the ground-truth, or on the basis of localizing predictions to a specific region (as defined by sector maps).
For the 3D detections predicted by AI, we opt for a hit criterion based on object overlap: