Hello, I have a question regarding launching scripts from the CLI. I submitted a batch script from the CLI. All of my scripts were completed successfully, but the output hasn't been saved. I'm not sure where I'm going wrong. Thanks for your help!

• 

  Rachael W UKB Community team Data Analyst

  • 16 August 2022 15:12

  Hi Monika, I took your question to our BioInformatician, and he suggests a few ideas:

   

   

  All looks good in these scripts.

   

  swiss-army-knife  utility should upload all artefacts (i.e. files created in local storage) to --destination

   

  The little complication is that in this case the swiss-army-knife  itself is used to bootstrap these jobs, but AFIK it should not interfere with upload process

   

  The only reason I can think is that plink job fails and no files are created

   

  To debug it: just add && ls -lah

   

  to the end of plink_make_bgen.sh

   

  it will list all files crated by plink2, and the output can be inspected in job log

   

  Also, I would not use the fs mounted at /mnt/project, as it's very slow and unreliable

   

  Instead, the PLINK files should be specified as additional -iin  parameters - this will download the files to local fs ,

  And then modify the plink script to use them

   

  `plink2 --bim "ukb23155_c${1}_b0_v1.bim" \

  --fam "ukb23155_c${1}_b0_v1.fam" \

  --bed "ukb23155_c${1}_b0_v1.bed" \

  --chr "${1}" \

  --no-psam-pheno \

  --keep "/mnt/project/Analysis/input_files/List_of_IDs.txt" \

  --export bgen-1.2 bits=8 \

  --out UKBF-Ex200k-Chr"${1}" && ls -lah`

   

  For the -iin , the path should be project related, that is `Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${1}_b0_v1.bim`

   

   

  Finally, I would run batch_RUN_plink.sh on the local machine or interactive session on RAP, instead of in `swiss-army-knife`

   

   

  0
• 

  Former User of DNAx Community_72

  • 12 September 2022 14:32

  Hello,

   

  Thank you for taking the time to respond to my query.

   

  I have spotted a typo in my code (now corrected). I added the suggestions but I am still getting an error..

  •  just add && ls -lah - this, unfortunately, did not work

  I checked the log files but they are not very informative

  • I would run batch_RUN_plink.sh

  do you mean type "bash  batch_RUN_plink.sh" or " "dx run bash batch_RUN_plink.sh" none of these seem to work

  the only way to launch the script from my command line is to type

  dx run swiss-army-knife -iin="AUGMENT:/Analysis/scripts/batch_RUN_plink.sh" -icmd="bash batch_RUN_plink.sh"

   

   

  Here's my code

   

  batch_RUN_plink.sh

  for i in {1..2}; do

   dx run swiss-army-knife \

     -iin="AUGMENT:/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.bim" \

   -iin="AUGMENT:/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.fam" \

     -iin="AUGMENT:/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.bed" \

     -icmd="bash plink_make_bgen.sh ${i}" \

     --name="plink_make_bgen ${i}" \

     --instance-type "mem1_hdd1_v2_x8" \

     --destination="AUGMENT:/Analysis/outputs/Ex200k/" \

     -y

  done

   

   

  plink_make_bgen.sh

  "plink2 --bim "/mnt/project/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.bim" \

  --fam "/mnt/project/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.fam" \

  --bed "/mnt/project/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.bed" \

  --chr "${i}" \

  --no-psam-pheno \

  --keep "AUGMENT:/Analysis/input_files/List_of_IDs.txt" \

  --export bgen-1.2 bits=8 \

  --out test_UKBF-Ex200k-Chr"${i}" && ls -lah"

   

   

   

   

   

  0
• 

  Former User of DNAx Community_17

  • 15 February 2023 18:15

  @Monika Monika? Could you able to resolve the issue and save the output?

  0
• 

  Former User of DNAx Community_72

  • 30 March 2023 15:17
  Hello, Yes I managed to get the script running with a help of someone from DNA nexus. Here's my test code #This script will run make_bgen scripts from CLI on UKB RAP #unset workspace unset DX_WORKSPACE_ID dx cd $DX_PROJECT_CONTEXT_ID: # wipe all dx env variables out source ~/.dnanexus_config/unsetenv dx clearenv dx login --noprojects --token tcsAUJeR5uodPfJZdWqds4G4Q485740fbwW dx select project-GBkKp50JBffBK69x4fxxj67lk for i in {21..21}; do dx run swiss-army-knife \ -icmd="plink2 --bim \"/mnt/project/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.bim\" \ --fam \"/mnt/project/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.fam\" \ --bed \"/mnt/project/Bulk/Exome sequences_Previous exome releases/Population level exome OQFE variants, PLINK format - interim 200k release/ukb23155_c${i}_b0_v1.bed\" \ --chr \"${i}\" \ --no-psam-pheno \ --keep \"/mnt/project/Analysis/input_files/List_of_IDs.txt\" \ --export bgen-1.2 bits=8 \ --out test_UKBF-Ex200k-Chr\"${i}\"" \ --name="plink_make_bgen${i}" \ --instance-type "mem1_hdd1_v2_x8" \ --destination="project-GBkKp50JBffBK69x4fxxj67lk:/Analysis/outputs/Ex200k/" --detach \ -y \ done I hope this helps BW Monika
  1

Please sign in to leave a comment.