Are there simpler ways of doing things?

Former User of DNAx Community_20

I started using the RAP for UKB analyses two weeks ago, and I have noticed that simple tasks seem to be taking me a very long time to do, and are much more complicated than UKB analyses I have done using our university cluster. If possible, I'd appreciate a bit of input from experienced users on how I can speed up my working to get things done faster.

?

Here are two examples, both of which took me multiple days each to figure out:

?

Task 1: Creating a CSV file of phenotype data for selected fields

I wanted to make a single CSV including phenotype data on all individuals from a selection of ~400 fields of interest. I want to be able to update and re-run this from time to time as I add new fields of interest.

?

There were too many fields to use the data exporter (which seems to cap out after you select 100 fields), so I used the Apache Spark cluster, building queries using PySpark.

?

The only way I could figure out how to achieve this was:

?

  1. Use spark.sql("Describe TBL") on each of the participant_* tables, to find what table each field is stored in
  2. Create spark.sql("SELECT ---") queries to select these correct fields from each table, and use df.join() to put the queries together into a single query data frame
  3. Recode all of the columns with array types to strings using col().cast()
  4. Write this query as a set of csvs to the hadoop file system using df.write.csv()
  5. Copy these csvs from the hadoop file system to the local file system using a system call to hdfs
  6. Read the CSVs back in again from the local file system, merge them together and write them back out again
  7. Upload the combined CSV to the project file system using a system call to dx upload

?

Task 2: Extracting a subset of SNPs for all samples from the exome data

I have a site VCF including ~1M variants of interest, and I'd like to get a BCF including genotypes for all individuals at these sites from the 200k exome release.

?

The only way I could figure out how to do this was:

  1. Write a Docker file to make an image that included bcftools, build it on cloud_workstation, and save it using dx upload
  2. Write a WBL script, and compile with dxCompile, to extract out the SNPs from a set of VCF files with bcftools isec (streaming the VCFs/indexes etc) and merge them into a BCF
  3. Write a bash script to generate and run ~100 job submission commands to run this script (each run being given 10 of the exome region chunks each).
  4. Run another job to download these output BCFs, merge then together and upload them again

?

Can I be doing these things more simply?

So, this all feels like I am overcomplicating things somehow. Previously they would have been just one or two short commands using our old UKB files: for the first, it would be reading a few big CSVs into R with fread and them merging and subsetting using tidyverse, for the second it would be setting off small job array script with a few bcftools commands using SLURM. But on RAP everything seems like a multi-stage process involving different servers and filesystems and programs.

?

Are the ways I am approaching these tasks on the RAP sensible, or have I gone wrong somewhere? Do people have tricks/workarounds/existing workflows/images etc for doing these sort of common tasks quickly? Or do you just get used to working in these systems eventually and adjust to the different way of doing things?

Comments

2 comments

Date Votes

Please sign in to leave a comment.