Submission of Inference Containers to the Open Development Phase 👩‍💻👨‍💻¶

Grand Challenge Algorithm Template¶

Algorithm submissions to the Development Phase of the PANORAMA challenge, are to be made in the form of inference containers. Inference containers are equivalent to Grand Challenge (GC) algorithms. They are Docker containers that encapsulate trained AI algorithms (architecture + model weights) and all of the components needed to load a given case (e.g. abdominal CECT + clinical information), generate predictions (e.g. PDAC detection map + patient-level likelihood score), and store the corresponding output file(s) for evaluation.¶

The following code repository contains such example submissions for the Development Phase.  All training cases belong to the Public Training and Development Dataset, and all models were trained via 5-fold cross-validation using the exact same splits. You can use any of this repo as a template for your submission to the Development Phase.¶

Before implementing your own algorithm using any of these templates, we recommend that you first upload a GC algorithm based on an unaltered template. In the following sections, we will provide a walkthrough of how to do this.¶

Installing Docker on Local System 🐳¶

You must have Docker installed and running on your system for the following steps to work. If you are using Windows, we recommend installing Windows Subsystem for Linux (WSL). For more details, please watch the official tutorial by Microsoft for installing WSL 2 with GPU support.¶

Building, Testing and Exporting Algorithm Container on Local System¶

Please use the shell scripts (.sh files) stated in all following commands. If you're using a Windows system, we highly recommend using Windows Linux Subsystem (WLS) with an Ubuntu distribution. We also highly recommend using *Docker Desktop *to manage your Docker containers and images. Please note that in order to build the baseline algorithm image using Ubuntu in WLS, you first need to enable integration with additional distros in Docker Desktop for Windows ( go to settings -> resources -> WSL integration).¶

Clone the template repo (see above) to your local system, with Git¶

LFS initialized (e.g. this can be done directly using GitHub Desktop). Build and test your inference Docker container by running test_run.sh. Testing will build the container and run it on images provided in the ./test/ folder. Export your container using save.sh. Once complete, your container image should be ready as a single .tar.gz file.

For more details, we highly recommend completing this general tutorial on creating GC algorithms: https://grand-challenge.org/documentation/create-your-own-algorithm/¶

Uploading Algorithm Container to Grand Challenge¶

Navigate to Submit > Open Development Phase - Validation and Tuning and complete the form "on this page":¶

¶

Once complete, click the "Save" button. At this point, your GC algorithm has been created and you are on its homepage. Now, click the "Containers" tab on the left panel, and then the "Upload a Container" button. Select your container image (.tar.gz file) for upload. After the upload is complete, click "Save". It typically takes 20-60 minutes till your container has been activated (depending on the size of your container). After its status is "Active", test out your container with a sample, unprocessed training case from the Public Training and Development¶

Dataset. You can also use the sample case provided in the 'test' directory of the baseline algorithm template (see above).

To learn more about GC input/output interfaces, visit: https://grand-challenge.org/components/interfaces/algorithms/ ¶

Create Containers for Your Own AI Algorithm¶

Following these same steps, you can easily encapsulate your own AI algorithm in an inference container by altering one of our provided baseline algorithm templates. You can implement your own solution by editing the functions in ./process.py. Any additional imported packages should be added to ./requirements.txt, and any additional files and folders should be explicitly copied through commands in ./Dockerfile. To update your algorithm, you can simply test and export your new Docker container, after which you can update the container image for the GC algorithm that you created on grand-challenge.org with the new one. Please note, that your container will not have access to the internet when they are executed on grand-challenge.org, so all necessary model weights and resources must be encapsulated in your container image apriori. You can test whether this is true locally using the --network=none option of docker run.¶

If something doesn't work for you, please feel free to add a post regarding the same in the forum.¶

Submission to Open Development Phase - Validation and Tuning Leaderboard¶

Once you have your trained AI model uploaded as a fully-functional GC algorithm, you're now ready to make submissions to the Open Development Phase of the challenge! Navigate to the "Open Development Phase - Tuning Submission" page, select your algorithm, and click "Save" to submit. If there are no errors and evaluation has completed successfully, your score will be up on the leaderboard (typically in less than 24 hours).¶

⚠️ Please double-check all rules to make sure that your submission is compliant. Invalid submissions will be removed and teams repeatedly violating any/multiple rules will be disqualified. Also, please try-out your uploaded GC algorithm with a sample case before making a full submission to the leaderboard.¶

Submission to Open Development Phase - Testing Leaderboard¶

Once you have gone through your whole AI model development + tuning cycle, and are now certain that you have your final algorithm ready and trained for independent testing, please complete the following steps:¶

• Make one final submission to the Development Phase - Tuning Leaderboard following all the steps stated above (only if you haven't already). This is to ensure that your final model is bug-free and performs just as expected.
• Optionally, you can prepare a document (in PDF format; minimum 1 page; no restrictions on template or maximum length), briefly covering the model design and training strategy of your AI algorithm. Please include the names, designations and contact details of your team members in this document. You can upload this document using the "Supplementary File" field in the submission form, or using the "Publication" field to share a URL to the same. If you're submitting an AI algorithm that has already been presented in one of your team's prior publications, then you can directly share a URL to that publication. Please note, this step is mandatory if you want to qualify as one of the top 5 teams of the challenge and move on to the Testing Phase. If you have been selected as one of the top 5 algorithms and have not provided this document in your submission, we will contact you for the same later down the line.
• Navigate to the "Development Phase -Tuning Submission" page, select your final AI algorithm, optionally upload your supplementary file and/or enter your publication URL, and then click "Save" to submit. Please note, each team (not each participant per team) can only submit a single AI algorithm to the Development Phase - Tuning Leaderboard. If multiple submissions are made, only the last submission from that team at the end of the submission deadline will be used to infer performance on the Hidden Testing Cohort.
• Please send us an e-mail at natalia.alves@radboudumc.nl and megan.schuurmans@radboudumc.nl, with your team name, list of team members, and a URL to your submitted algorithm (here is an example). We will use this e-mail thread to share updates regarding your submission and address issues (if any). Additionally, if you make it to the top 5, the designated team will receive the prizes listed here.

Results of all submissions (that are error-free and compliant with all participation rules) will be pre-computed in a hidden setting. Subsequentely, the results will also be made publicly available on the leaderboard.¶

We look forward to your submissions to the PANORAMA challenge. All the best! 🍀¶