We have to annotate all UKB WES and WGS variants with VEP and various other features such as pathogenicity scores from dbnsfp and organize the annotations and carrier information in a database that can be queried quickly within our custom code (python and R) that will be run on the RAP. Similarly, we want to process all the UKB phenotypic data (e.g. correcting for covariates) and store this in a database format that can be queried quickly within our custom code. We've been using sqlite files so far to store variant annotations and carrier info, but this will likely not be suitable for the WGS data. What is the best way to store large data like that on the RAP so that it can be queried quickly?
UKB phenotypic data is stored in the form of Dataset. This can be then quickly queried using Spark / Spark SQL / dxdata Python package. I would start with the following links:
Comments
1 comment
UKB phenotypic data is stored in the form of Dataset. This can be then quickly queried using Spark / Spark SQL / dxdata Python package. I would start with the following links:
https://dnanexus.gitbook.io/uk-biobank-rap/working-on-the-research-analysis-platform/using-spark-to-analyze-tabular-data
https://dnanexus.gitbook.io/uk-biobank-rap/working-on-the-research-analysis-platform/using-spark-to-analyze-tabular-data#notebook-starting-code
https://dnanexus.gitbook.io/uk-biobank-rap/working-on-the-research-analysis-platform/using-spark-to-analyze-tabular-data#accessing-the-database-directly-using-sql
https://dnanexus.gitbook.io/uk-biobank-rap/frequently-asked-questions#databases-and-datasets
Please sign in to leave a comment.