AI: Phases & Rules🌱

The PANORAMA: AI Study (grand challenge) takes place in two phases:

• Open Development Phase (Duration: 4-6 months)
  Anyone can participate in this phase of the challenge. Interested teams must join the PANORAMA24 challenge at https://panorama.grand-challenge.org/. Afterwards, they will be provided access to download the Public Training and Development Dataset, and in turn, they can start developing and training AI models using their private or public compute resources (e.g. Google Colaboratory, Kaggle). Each team can submit a single trained algorithm (in a Docker container) for evaluation every week (similar to the AIROGS, MIDOG2021 and CoNIC2022 challenges). During evaluation, algorithms are executed on the grand-challenge.org platform, their performance is estimated on the Hidden Tuning Cohort, and team rankings are updated accordingly on a live, public leaderboard. Facilitating validation in such a manner, ensures that any image used for evaluation remains truly unseen, and that AI predictions cannot be tampered with. At the end of this phase, each team can choose to submit a single AI algorithm (presumably their top-performing model) for evaluation on the Hidden Testing Cohort. Based on their performance on this cohort, all-new rankings will be drawn and the top 5 AI algorithms will be determined.
  

• Open Development Phase Dummy (Duration: 4-6 months)
  This phase will be open for as long as the development phase and will have an identical structure regarding scoring metrics, but it consists of only 6 sample cases (from the training dataset). The goal of this phase is to facilitate algorithm debugging, as participants will have access to their submission logs. This phase will not be used to rank teams based on performance or to select challenge winners in any way.

• Closed Testing Phase (Duration: 1 month)
  At the end of this phase, each team can choose to submit a single AI algorithm (presumably their top-performing model) for evaluation on the Hidden Testing Cohort. Based on their performance on this cohort, all-new rankings will be drawn and the top 5 AI algorithms of the PANORAMA24 challenge will be determined. To qualify as one of these top teams, participants must also submit a short paper on their methodology (2-3 pages) and a public/private URL of their source code on GitHub - to ensure fairness, traceability and reproducibility of all proposed solutions