Running Docker images with the Swiss Army Knife GUI on the UKB-RAP

Introduction

This article explains how to use Docker images within the Swiss Army Knife (SAK) using the graphical user interface (GUI) on the UKB-RAP. The process is illustrated using the OxWearables stepcount tool, which is designed to process wrist-worn accelerometer data. While the example focusses on the stepcount tool, the same workflow applies to any custom or existing Docker image.

Note: Full documentation of the stepcount tool can be found in the OxWearables GitHub repository and associated publication. The Dockerfile, Docker image, and related scripts are available on the UK Biobank GitHub.

Prerequisites 

(Not required if working wholly through the RAP)

• Docker installed on your local machine
• DNAnexus dx-toolkit installed if uploading via the command line

Step 1: Prepare the Docker image for upload to the RAP

Before running Docker in the SAK app, ensure your Docker image is available in your RAP project space. You can do this in two ways:

• Option A: Use the pre-built Docker image (e.g., from a public registry like DockerHub or GitHub Container Registry).
  • Note: A Docker image with a working version of the stepcount tool is available here.
• Option B: Build your own Docker image using a Dockerfile.

Option A: Using a pre-built Docker image:

If a Docker image already exists, either publicly available or previously built by you, you can use it directly by pulling it into your local machine or into an instance on the RAP. You will then save and upload it as a compressed .tar.gz file so it can be used in the SAK app. 

Note: This UK Biobank GitHub notebook provides guidance and code for using a pre-built Docker image. In summary, the steps are:

1. Pull the image from a registry to your local machine or an instance on the RAP.
2. Save the image to a tarball.
3. Upload the image to the RAP.

Option B: Building the Docker image from a Dockerfile:

If a pre-built Docker image is not available, you can create one yourself using a Dockerfile. This file acts as a blueprint for your image, defining the environment, dependencies, tools, scripts, and configuration settings it should include. This is especially useful for custom workflows or tools.

There are two common use cases:

1. A Dockerfile is already provided (e.g., from a GitHub repository).
2. You need to write your own Dockerfile from scratch.

Case 1: Building from a provided Dockerfile

In some cases, such as the stepcount tool, the Dockerfile is already written and available for download. You can build the image either:

• Locally on your own machine (requires dx login)
• Within the RAP environment

Steps (detailed guidance available here):

1. Download the Dockerfile to your working directory.
2. Build the Docker image.
3. Save and compress the image.
4. Upload to your RAP project space.

Case 2: Creating your own Dockerfile from scratch

If no Dockerfile is available, you will need to write one yourself. This gives you complete control over your analysis environment, allowing you to define the dependencies, configurations, and scripts for your tool. The guides provided by Depot and Docker Docs offer useful explanations and instructions for creating a Dockerfile. 

Once your Dockerfile is created, you can follow the same steps outlined in Case 1: build the image, save and compress it, and upload it to your RAP project space. 

Step 2: Prepare input files

When setting up your job in the SAK app, you will need to provide the necessary input files. For the stepcount example, these are typically .cwa files from Field 90001.

You can prepare these files in two ways:

1. Interactively via the SAK GUI: You can browse and select files directly when launching your job (demonstrated later in this article).
2. Using a RAP instance: You can use the command-line interface to collate the files of interest (this method is detailed here).

Step 3: Configure and run the analysis with SAK

Once your Docker image and input files are uploaded to your RAP project space, you can run your analysis using the SAK app via the GUI.

Launch SAK

• Open your RAP project.
• Click “Start Analysis”.
• Select the SAK app and click “Run Selected”.

Set job details

• Job name: Give your SAK run a relevant name.
• Output to:  Select output directory for your results.

Configure inputs

• "Input files": Select one or more required files. For the stepcount example, you can either:
  1. Use pre-selected files: Choose the .cwa files prepared in Step 2.
  2. Select files interactively: Click the (+) icon and “Enable Batch Run”. Then, navigate to the files of interest and you click the boxes on the left to select them.

• "Platform file containing Docker image accepted by ‘docker load’": If the Docker image is a snapshot uploaded from Step 1, then select this file here.
• "Public Docker image identifier": If instead you want to pull the Docker image directly from a public registry (e.g., DockerHub or GHCR), then you can enter the image identifier here. Note: While pulling from public registries is supported, it is often subject to daily pull limits. To avoid issues and to scale batch processing, it is recommended to upload the image to the RAP.
• "Command line": Specify what to run inside the Docker container. You can configure this in one of two ways:
  1. Single job (all files processed together)
  2. Batch processing (one job per file)

1. Single job:

Enter a command that loops over all input files in the command line. This approach runs all processing within a single job, which will only be marked as complete when all files have finished. In the “Monitor” tab, you will see one job entry.

Using the stepcount tool:

• Single file example:

stepcount your_input_file.cwa 

• Multiple file example:

for f in *.cwa; do stepcount "$f"; done

2. Batch processing:

Click the (+) icon and select “Enable Batch Run”. In the pop-up window, use the command line column to specify the command to run, making sure to reference the relevant file. This creates a separate job for each input file. In the “Monitor” tab, you will see individual jobs, making it easier to track which ones completed successfully or failed.

Configure outputs

• "Output Folder": Choose where results will be saved to in your RAP project.

Review & start

After clicking “Start Analysis”, a pop-up will appear where you can review job name and output location, and finalise your job settings. You’ll have the option to choose the job’s priority, spending limit and appropriate instance type. Your choice depends on what you need for your job. For more details, please refer to the job priority page, instance types page and the UKB-RAP rate card.