Long waiting_on_output state after unzipping DXA files using parallel scripts

Hi UKB RAP Community,

I’m currently working on unzipping all DXA files in bulk using two scripts: unzip_single.sh and unzip_parallel.sh. When I run the following command:

dx run swiss-army-knife -iin="/scripts/unzip_parallel.sh" -iin="/scripts/unzip_single.sh" -icmd="bash unzip_parallel.sh" --instance-type "mem1_ssd1_v2_x2" --destination "unzip" --yes

The jobs are successfully submitted and executed. However, I noticed that even though the unzipping process seems to complete (as seen in the job logs), the jobs remain in a waiting_on_output state for a long time. For example:

2025-01-29 11:01:12 Swiss Army Knife STDOUT job-GyGgqf0JV8ygq8b6F6j46vg2

2025-01-29 11:01:12 Swiss Army Knife STDOUT Finished processing /Bulk/Whole Body DXA/DXA/11.

2025-01-29 11:01:12 Swiss Army Knife STDERR + set +x

*Swiss Army Knife (swiss-army-knife:main) (waiting_on_output) job-GyGg21jJV8yjzBpB71fj3GB8  

2025-01-29 10:06:31 (runtime 0:53:45)

As a result, although I can see the files being unzipped in the job logs, they do not appear in the actual destination directory (/unzip/) for hours. I suspect this may be related to resource limitations, because when the first 5 dicom files were successfully unzipped and located in the right path and after that there were no other files and the jobs were still running.

Questions:

1. Why are the jobs stuck in the waiting_on_output state for so long? Is this related to resource limitations, file system delays, or something else?
2. Are there any best practices for handling bulk unzipping tasks like this on the UKB RAP platform?

Any suggestions or insights would be greatly appreciated!

Thank you in advance for your help!

Below are the two scripts I’m using:

Script 1: unzip_single.sh

#!/bin/bash

num=$1

project="project-ID"

destination_dir="unzip_${num}"

dx find data --name "*.zip" --path "${project}:/Bulk/Whole Body DXA/DXA/${num}" --brief > files_to_manipulate.txt

if [ ! -s files_to_manipulate.txt ]; then

    echo "No .zip files found in /Bulk/Whole Body DXA/DXA/${num}. Skipping..."

    exit 0

fi

echo "Creating folder: ~/unzip/${destination_dir}"

dx mkdir -p "~/unzip/${destination_dir}"

while IFS="" read -r LINE || [ -n "$LINE" ]; do

    folder=$(dx describe "$LINE" | grep "Folder" | awk '{print $2}')

    filename=$(dx describe "$LINE" | grep "Name" | awk '{print $2}')

    destination_subdir=$(basename "$filename" .zip)

    echo "Creating folder: ~/unzip/${destination_dir}/${destination_subdir}"

    dx mkdir -p "~/unzip/${destination_dir}/${destination_subdir}"

    unzip_cmd="unzip ${filename} -d ${destination_subdir}"

    dx run swiss-army-knife \

        -iin="$LINE" \

        -icmd="$unzip_cmd" \

        --tag="unzip_${num}" \

        --destination="${project}:/unzip/${destination_dir}" \

        --instance-type "mem2_ssd1_v2_x2" --brief --yes

done < files_to_manipulate.txt

rm files_to_manipulate.txt

echo "Finished processing /Bulk/Whole Body DXA/DXA/${num}."

Script 2: unzip_parallel.sh

#!/bin/bash

project="project-ID"

batch_size=5  

nums=({11..60})  

total_nums=${#nums[@]}

for ((i = 0; i < total_nums; i += batch_size)); do

    echo “Starting batch from ${nums[i]} to ${nums[i+batch_size-1]}”

    for ((j = 0; j < batch_size && (i + j) < total_nums; j++)); do

        num=${nums[i + j]}

        dx run swiss-army-knife \

            -iin="unzip_single.sh" \

            -icmd="bash unzip_single.sh ${num}" \

            --tag="submit_single_unzip_${num}" \

            --destination="unzip" \

            --instance-type "mem1_ssd1_v2_x2" \

            --brief \

            --yes &

    done

    wait 

    echo "Batch completed! from ${nums[i]} to ${nums[i+batch_size-1]}!"

done

echo "All folders processed!"