using dx upload and dx download on the worker
Hi,
i'm trying to run the following bash script by swiss army knife:
vcfpref="/Bulk/GATK and GraphTyper WGS/GraphTyper population level WGS variants, pVCF format [500k release]/chr22/"
for i in {0..11};do
dx download "project:${vcfpref}/ukb23374_c22_b${i}_v1.vcf.gz" -o int.vcf.gz
dx download "project:${vcfpref}/ukb23374_c22_b${i}_v1.vcf.gz.tbi" -o int.vcf.gz.tbi
wc -l int.vcf.gz
bcftools query -l int.vcf.gz > samples.txt && bcftools query -i 'AC==1' -f '%CHROM\t%POS\t%REF\t%ALT[\t%GT]\n' int.vcf.gz | awk -v OFS='\t' 'BEGIN {while((getline < "samples.txt")>0) s[++n]=$0} {c=$1;p=$2;r=$3;a=$4;id="";for(i=5;i<=NF;i++) if(substr($i,1,3) == "0/1" || substr($i,1,3) == "1/0") {id=s[i-4];break} if(id!="") print c,p,r,a,id}' > chr22.b$i.singleton
wc -l chr22.b$i.singleton
dx upload chr22.b$i.singleton
dx upload samples.txt
done
with
dx run swiss-army-knife -iin="run.sh" -icmd="bash run.sh" --instance-type mem1_ssd1_v2_x2 --priority low --destination /songweichen
It can be confirmed that the input file path is correct by dx ls. It turns out that:
1) dx download get empty file without error message;
2)dx upload got dxpy.exceptions.PermissionDenied: UPLOAD permission required in project-Gy0745QJg8fFQby0Y2Gb6yqv to perform this action,
It seems that using dx upload and dx download on the worker is different from using on local machine. Is there any suggestion on these issues? Thanks in advance for your help.
Comments
2 comments
Hi Yongyong,
has data been dispensed to the UKB-RAP project ? See “Dispensing Data” in this page of the documentation .
How many Data Bundles can be seen in “Data Dispensed” in the Settings tab ?
If you log in to the UKB-RAP, and start a jupyterlab session from the Tools tab, and open a $_ terminal, can you use dx download to copy a single small file into the jupyterlab storage?
Data dispense is currently switched off temporarily, see this announcement thread .
Data dispense is switched on, see this thread .
Please sign in to leave a comment.